A distributed document parsing method, device, equipment and medium

Abstract

The application discloses a distributed document analysis method, device, equipment and medium, the method comprises the following steps: obtaining a target to be analyzed document; extracting the content feature data of the target to be analyzed document; the content feature data is used to reflect the content complexity of the target to be analyzed document; according to the content feature data, the estimated time consumption of analyzing the target to be analyzed document is determined; based on load balancing, according to the estimated time consumption, the target analysis task of the target to be analyzed document is distributed to the target computing power node; so that the target computing power node executes the target analysis task to obtain the analysis result. Therefore, the content feature data of the document is used to estimate the analysis time, rather than relying on the file size to determine the estimated time consumption, which is adaptive to the time consumption fluctuation characteristics of different documents, improves the estimated time consumption accuracy, and based on load balancing, the estimated time consumption is used for computing power stage distribution, which improves the resource utilization and the load balancing effect of the system.

Description

A distributed document parsing method, apparatus, device, and medium Technical Field

[0001] This application relates to the field of computer data processing technology, and in particular to a distributed document parsing method, apparatus, device, and medium. Background Technology

[0002] In the field of distributed document processing systems, parsing PDF documents is a common and computationally intensive business scenario. In practical applications, the PDF documents to be parsed vary significantly in size, but the parsing time is not simply linearly proportional to the file size. For example, a 10MB plain text PDF may only take one minute to parse, while a 5MB PDF containing numerous scanned images, complex charts, or specially formatted content may take up to an hour to parse. This "non-linear parsing time" characteristic poses a significant challenge to the load balancing and scheduling of distributed clusters.

[0003] Currently, common distributed task scheduling solutions have significant shortcomings in this scenario. First, most systems adopt a task allocation model of "proactive push by the scheduling center," where the scheduling center only distributes tasks based on real-time hardware indicators of server nodes (such as CPU utilization and memory usage) or the static size of PDF files, and cannot effectively predict the actual time consumption of each PDF parsing task.

[0004] This task allocation strategy based on simple metrics is prone to two types of inefficiency: First, it may concentrate highly complex PDF tasks on certain nodes, causing them to be overloaded for a long time; second, some nodes may be idle after completing simple tasks early on, as they cannot obtain new tasks in time, thus resulting in an unbalanced overall load on the cluster and reduced resource utilization.

[0005] Therefore, how to adapt to the fluctuations in PDF processing time, improve resource utilization, and enhance the load balancing effect of distributed clusters are problems that urgently need to be solved by those skilled in the art. Summary of the Invention

[0006] In view of this, one aspect of this application provides a method for parsing distributed documents, the method comprising:

[0007] Obtain the target document to be parsed;

[0008] Extract content feature data from the target document to be parsed; the content feature data is used to reflect the complexity of the content of the target document to be parsed.

[0009] Based on the content feature data, determine the estimated time required to parse the target document to be parsed;

[0010] Based on load balancing, according to the estimated time, the target parsing task of the target document to be parsed is assigned to the target computing power node; so that the target computing power node can execute the target parsing task and obtain the parsing result.

[0011] Optionally, the content feature data includes at least the total number of pages in the document and special content parameters; the special content parameters include a first parameter for indicating whether scanned documents are included, a second parameter for indicating whether charts are included, a third parameter for indicating whether the document is multi-column formatted, and a fourth parameter for indicating whether annotations are included.

[0012] Optionally, determining the estimated time for parsing the target document based on the content feature data includes:

[0013] Obtain the highest resource parameter representing computing power resources in each computing power node and the pre-constructed mapping relationship; the mapping relationship is the correspondence between the influence factor of content complexity and the compensation coefficient; the influence factor includes the number of scanned documents, the area ratio of the total area of ​​charts to the total area of ​​the document, the number of layouts, and the number of annotations;

[0014] Based on the highest resource parameter, the basic single-page time for parsing a single page is determined within a preset time range; the highest resource parameter is negatively correlated with the basic single-page time; the preset time range is determined based on the historical basic single-page time within a specified historical time range.

[0015] Based on the mapping relationship and the special content parameters, the compensation time for the special content is determined; the compensation time includes the compensation time for scanned documents, the compensation time for charts, the compensation time for layout, and the compensation time for annotations.

[0016] The estimated time is determined based on the total number of pages in the document, the basic single-page time, and the compensation time.

[0017] Optionally, based on the mapping relationship and the special content parameters, the compensation time for the special content is determined, including:

[0018] If the first parameter indicates that scanned documents are included, the current number of scanned documents is determined; based on the mapping relationship, a first compensation coefficient corresponding to the current number of scanned documents is determined; the compensation time for the scanned documents is determined according to the first compensation coefficient, the total number of pages in the document, and the basic single-page time; the number of scanned documents is positively correlated with the first compensation coefficient.

[0019] If the second parameter represents a chart, determine the current ratio of the current chart area to the total area of ​​the target document to be parsed; based on the mapping relationship, determine the second compensation coefficient corresponding to the current area ratio; and determine the chart compensation time based on the second compensation coefficient, the total number of pages in the document, and the basic single-page time; the area ratio is positively correlated with the second compensation coefficient.

[0020] If the third parameter represents a multi-column layout, determine the current layout quantity; based on the mapping relationship, determine the third compensation coefficient corresponding to the current layout quantity; and determine the layout compensation time based on the third compensation coefficient, the total number of pages in the document, and the basic single-page time; the layout quantity is positively correlated with the third compensation coefficient.

[0021] If the fourth parameter characterization includes annotations, determine the current number of annotations; based on the mapping relationship, determine the fourth compensation coefficient corresponding to the current number of annotations; and determine the annotation compensation time based on the fourth compensation coefficient, the total number of pages in the document, and the basic single-page time; the number of annotations is positively correlated with the fourth compensation coefficient.

[0022] The sum of the compensation time for the scanned document, the compensation time for the chart, the compensation time for the layout, and the compensation time for the annotation is taken as the compensation time.

[0023] Optionally, obtaining the target document to be parsed includes:

[0024] Upon receiving a parsing task request, the system filters out initial documents in the database whose task status is pending execution.

[0025] Obtain the ranking influence factor of the initial document and the weight coefficient corresponding to the ranking influence factor; the ranking influence factor includes at least one of the following: file size, level parameter representing task priority, complexity parameter representing content complexity, and urgency parameter representing task urgency.

[0026] Based on the weight coefficients, the ranking factors are weighted and summed to obtain a comprehensive score;

[0027] The overall scores of the tasks are sorted in descending order, and the document with the highest overall score in the sorting results is selected as the target document to be parsed.

[0028] Optionally, the target computing node executes the target parsing task to obtain the parsing result, including:

[0029] Based on the content feature data, the corresponding execution tool is invoked to parse each page of the target document to be parsed in sequence; and it is determined whether the current page has been parsed successfully.

[0030] If the parsing is successful, save the parsing result of the current page and proceed to the parsing task of the next page until the last page is parsed, and mark the parsing result as parsing successful.

[0031] If parsing fails, perform the following steps:

[0032] Determine whether the ratio of the total number of pages that failed to be parsed to the total number of pages in the document reaches a ratio threshold;

[0033] If the ratio threshold is reached, the target parsing task ends, and the parsing result is marked as parsing failure.

[0034] If the ratio threshold is not reached, proceed to the parsing task on the next page.

[0035] Optionally, after the target computing node executes the target parsing task and obtains the parsing result, the process includes:

[0036] Obtain the actual parsing time and parsing result of the target computing node;

[0037] Determine whether the actual parsing time is not greater than the timeout threshold, and whether the parsing result is successful; the timeout threshold is a preset multiple of the estimated time consumption, and the preset multiple is greater than 1;

[0038] If so, update the task status of the target document to be parsed to "completed";

[0039] If not, determine whether the number of times the target document to be parsed has been retried for parsing has reached a threshold; if the threshold has been reached, update the task status to permanent failure; if the threshold has not been reached, update the task status to pending execution.

[0040] Another aspect of this application provides a distributed document parsing apparatus, the apparatus comprising:

[0041] The target document acquisition module is used to acquire the target document to be parsed.

[0042] The feature data extraction module is used to extract content feature data of the target document to be parsed; the content feature data is used to reflect the complexity of the content of the target document to be parsed.

[0043] The estimated time determination module is used to determine the estimated time for parsing the target document to be parsed based on the content feature data.

[0044] The task allocation module is used to allocate the target parsing task of the target document to the target computing power node based on load balancing and the estimated time consumption; so that the target computing power node can execute the target parsing task and obtain the parsing result.

[0045] Another aspect of this application provides an electronic device including a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor executes the computer program to implement the steps of the distributed document parsing method.

[0046] Another aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the distributed document parsing method.

[0047] The distributed document parsing method, apparatus, device, and medium provided in this application have the following beneficial effects: the parsing time is estimated based on the content feature data of the document, rather than solely relying on the file size to determine the estimated time. This adapts to the time fluctuation characteristics of different documents, improving the accuracy of the estimated time. At the same time, based on load balancing, computing power is allocated in stages according to the estimated time, improving resource utilization and the load balancing effect of the system. Attached Figure Description

[0048] Figure 1 is a flowchart illustrating a distributed document parsing method provided in an embodiment of this application;

[0049] Figure 2 is a schematic diagram of the principle architecture of a distributed document parsing method provided in an embodiment of this application;

[0050] Figure 3 is a flowchart of a distributed document parsing method provided in an embodiment of this application;

[0051] Figure 4 is a schematic diagram illustrating the principle of a distributed document parsing method provided in an embodiment of this application;

[0052] Figure 5 is a schematic diagram of the structure of a distributed document parsing device provided in an embodiment of this application;

[0053] Figure 6 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0054] The attached diagram is labeled as follows: 50 is the target document acquisition module, 51 is the feature data extraction module, 52 is the estimated time determination module, 53 is the task allocation module, 60 is the memory, 61 is the processor, 62 is the display screen, 63 is the input / output interface, 64 is the communication interface, 65 is the power supply, 66 is the communication bus, 601 is the computer program, 602 is the operating system, and 603 is the data. Detailed Implementation

[0055] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0056] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0057] Figure 1 is a flowchart illustrating a distributed document parsing method provided in an embodiment of this application. As shown in Figure 1, the method includes:

[0058] S10: Obtain the target document to be parsed;

[0059] S11: Extract content feature data from the target document to be parsed; the content feature data is used to reflect the complexity of the content of the target document to be parsed.

[0060] Figure 2 is a schematic diagram of the principle architecture of a distributed document parsing method provided in an embodiment of this application. As shown in Figure 2, in an optional embodiment, the distributed document parsing method provided in this application is applied to a task scheduling system to realize document parsing scheduling in a distributed cluster. In a specific embodiment, multiple computing nodes are communicatively connected to the task scheduling system. In the distributed cluster, different computing nodes are allocated according to the specific situation of the target document to be parsed, ensuring resource utilization and load balancing.

[0061] Specifically, in one optional embodiment, a target document to be parsed is obtained. This target document to be parsed may be a PDF document, which is not limited in this application; the following description uses a PDF document as an example. After obtaining the target PDF document, in order to accurately estimate the time required to parse the target PDF document, in one optional embodiment, it is necessary to extract the content feature data of the target PDF document. This content feature data can reflect the complexity of the content of the target PDF document.

[0062] It is understandable that the more pages a target PDF document has, the longer the parsing time will be. Furthermore, if the PDF document contains a large number of charts or scanned documents, it will require even more parsing time and consume more computing resources. Therefore, the content feature data can include, but is not limited to, parameters representing complex information such as the total number of pages in the document, whether it contains charts, and whether it contains scanned documents. This application does not impose any limitations on these parameters.

[0063] It should be noted that the target document to be parsed can be the document corresponding to the parsing request currently initiated by the user in real time, or it can be one of the target parsing documents stored in the database in a large number of parsing scenarios. This application does not limit this.

[0064] Correspondingly, the extraction of content feature data of the target document to be parsed can be performed in real time based on a real-time request, or it can be performed in advance when storing the target document to be parsed. This application does not make any specific restrictions on this.

[0065] S12: Based on the content feature data, determine the estimated time required to parse the target document;

[0066] In one optional embodiment, the task scheduling system deploys a task retrieval service and an algorithm service. In a specific embodiment, after the algorithm service starts, it sends a task retrieval request to the task retrieval service. Upon receiving the request, the task retrieval service obtains the target document to be parsed and acquires the content feature data from the target document.

[0067] Furthermore, in one optional embodiment, the estimated time for parsing the target document is determined based on content feature data using a pre-built time estimation model. In another optional embodiment, a specified large model can be invoked to process and evaluate the content feature data to determine the estimated time. The specified large model can be such as Qianwen, Deepseek, etc., and this application does not limit this.

[0068] It is understandable that content feature data is used to reflect the complexity of the content of the target document to be parsed. Therefore, in specific embodiments, the higher the content complexity reflected by the content feature data, the longer the estimated time.

[0069] It is worth noting that the algorithm service in this application supports horizontal scaling. When adding a new algorithm service instance, there is no need to modify the filtering logic of the task retrieval service. After the instance starts, it can directly obtain tasks by calling the task retrieval service, thereby flexibly improving the system's processing capacity. As a result, the system's processing capacity can increase linearly with the number of nodes.

[0070] S13: Based on load balancing, the target parsing task of the target document to be parsed is assigned to the target computing power node according to the estimated time; so that the target computing power node can execute the target parsing task and obtain the parsing result.

[0071] Furthermore, after accurately determining the estimated time, as shown in Figure 2, a target computing node capable of completing the target parsing task is identified among multiple computing nodes. That is, based on the estimated time, the target parsing task is allocated to a target computing node with appropriate computing resources. This avoids assigning high-complexity target parsing tasks to nodes with low computing resources, and also prevents nodes with high computing resources from completing simple tasks prematurely and becoming idle due to a lack of timely access to new tasks, thus improving load balancing.

[0072] Furthermore, in an optional embodiment, the target computing node performs a target parsing task to obtain the parsing results. Specifically, for plain text pages, the text can be extracted directly. If the target document to be parsed contains scanned copies, it needs to be recognized by Optical Character Recognition (OCR). For charts, image and text separation and extraction are required.

[0073] Therefore, the distributed document parsing method provided in this application estimates the parsing time based on the content feature data of the document, rather than solely relying on the file size to determine the estimated time. It adapts to the time fluctuation characteristics of different documents, improves the accuracy of the estimated time, and, based on load balancing, allocates computing power in stages according to the estimated time, thereby improving resource utilization and the load balancing effect of the system.

[0074] In one optional embodiment, the content feature data includes at least the total number of pages in the document and special content parameters; the special content parameters include a first parameter for indicating whether scanned documents are included, a second parameter for indicating whether charts are included, a third parameter for indicating whether the document is multi-column formatted, and a fourth parameter for indicating whether annotations are included.

[0075] Understandably, in one alternative embodiment, when the target document to be extracted is a target PDF document, parsing will take longer than plain text parsing if the document has a multi-column layout, contains scanned copies, contains complex charts, or contains annotations. Furthermore, the more pages a PDF document has, the longer the parsing time will be.

[0076] Based on this, in one optional embodiment, an image recognition model is used to determine whether the target document to be parsed contains scanned image content, and a first parameter is generated to characterize whether the scanned document is contained. In one optional embodiment, if the scanned document is contained, the first parameter is 1; if the scanned document is not contained, the first parameter is 0.

[0077] In addition, in an optional embodiment, a local Python preprocessor is launched, which can extract the total number of pages in the target PDF document using a lightweight PDF parsing tool such as pdfplumber or PyPDF2, and determine whether the document is multi-column formatted to obtain a third parameter to characterize whether it is multi-column formatted. If it is multi-column formatted, the third parameter is 1; if it is not multi-column formatted, the third parameter is 0.

[0078] Simultaneously, it identifies whether the target PDF document contains charts and annotations. Similarly, if charts are present, the second parameter is 1; if charts are not present, the second parameter is 0. If annotations are present, the fourth parameter is 1; if annotations are not present, the fourth parameter is 0. Annotations can include handwritten annotations and automatically generated annotations.

[0079] It should be noted that, in addition to the first to fourth parameters, special content parameters may also include a fifth parameter indicating that the page contains a corrupted page, a sixth parameter indicating whether it contains special fonts, etc. This application does not limit the special content parameters; any parameter that causes the page to consume additional parsing time is acceptable.

[0080] In one optional embodiment, the extraction time is calculated from the moment the content feature data of the target document to be parsed is extracted. To avoid consuming too many system resources, the extraction is interrupted when the extraction time reaches a preset time (e.g., 10 seconds), and only the currently extracted feature data is obtained.

[0081] Figure 3 is a flowchart of a distributed document parsing method provided in an embodiment of this application. In an optional embodiment, as shown in Figure 3, the estimated time for parsing the target document is determined based on content feature data, including:

[0082] S30: Obtain the highest resource parameter representing computing power resources in each computing power node and the pre-built mapping relationship; the mapping relationship is the correspondence between the influence factor of content complexity and the compensation coefficient; the influence factor includes the number of scanned documents, the area ratio of the total area of ​​charts to the total area of ​​the document, the number of layouts, and the number of annotations;

[0083] S31: Determine the basic single-page time for parsing a single page within a preset time range based on the highest resource parameter; the highest resource parameter is negatively correlated with the basic single-page time; the preset time range is determined based on the historical basic single-page time within a specified historical time range.

[0084] S32: Determine the compensation time for special content based on the mapping relationship and special content parameters; the compensation time includes the compensation time for scanned documents, charts, layout, and annotations.

[0085] S33: Determine the estimated time based on the total number of pages in the document, the basic time per page, and the compensation time.

[0086] In one specific embodiment for determining the estimated parsing time, it is necessary to first determine the base single-page parsing time, which adapts to the fluctuation of the parsing time of the current target document. The base single-page parsing time refers to the parsing time of a page where each page is text. In fact, the base single-page parsing time is related to the algorithm performance of parsing the target document; the higher the algorithm performance, the shorter the base single-page parsing time. That is, the base single-page parsing time is related to the computing power resources of the computing node.

[0087] In one optional embodiment, as shown in Figure 2, resource parameters representing computing power resources of all computing power nodes are obtained, and the highest resource parameter with the richest computing power resources is extracted from them. The higher the highest resource parameter, the richer the computing power resources of the corresponding computing power node. Further, based on the highest resource parameter, the basic single-page latency is determined. There is a negative correlation between the highest resource parameter and the basic single-page latency, and the duration of the basic single-page latency is within a preset duration range. The preset duration range is determined based on the historical basic single-page latency within a specified historical period.

[0088] It should be noted that the time taken for a single historical page within a specified historical period can be for parsing plain text or for parsing complex documents; this application does not impose any limitations on this. Of course, in a preferred embodiment, the preset time range is determined based on the time taken for parsing a single historical page containing plain text within the specified historical period. Furthermore, the average time taken for a single historical page within the specified historical period can be used as a benchmark to determine the preset time range, or the mode of the time taken for a single historical page within the specified historical period can be used as a benchmark to determine the preset time range; this application also does not impose any limitations on this.

[0089] For example, in one alternative embodiment, an initial preset duration range (e.g., 0.4 seconds to 1 second) can be set based on empirical values. During the parsing of the target document, the basic single-page time is set to 0.4 seconds based on the highest resource parameter. If the historical average basic single-page time over a week is 0.5 seconds, the preset duration is adjusted to 0.5 seconds to 1 second.

[0090] In other words, determining the basic single-page time depends not only on the computing resources of the computing nodes but also on historical basic single-page times. This allows for a high-precision determination of the basic single-page time, laying a solid foundation for estimating the total time required to parse the entire target document.

[0091] Based on the above description, it is understandable that when the target document to be parsed contains scanned documents, complex charts, or handwritten annotations, it takes more time than plain text parsing. The time consumed also varies depending on the number of scanned documents, charts, and annotations.

[0092] Therefore, to more accurately adapt to the dynamic time consumption of different target PDF documents, a mapping relationship is pre-constructed. This mapping relationship is the correspondence between the influence factors of document content complexity and the compensation coefficient. The influence factors include the number of scanned documents, the ratio of the total area of ​​charts to the total document area, the number of layouts, and the number of annotations. Table 1 is a schematic table of a mapping relationship provided in an embodiment of this application, and the following explanation will be based on Table 1.

[0093] Table 1 is a schematic diagram of one type of mapping relationship.

[0094]

[0095] As shown in Table 1, different impact factors correspond to different compensation coefficients. In specific embodiments, based on the specific content parameters in the above embodiments, it can be determined whether to determine the compensation time for descriptions, charts, layouts, and annotations. Furthermore, based on the mapping relationship in Table 1, the compensation coefficients corresponding to different impact factors can be determined, thereby determining the compensation time for different types of factors.

[0096] In one optional embodiment, the same first compensation coefficient can be configured for influence factors with a first parameter of 1 (i.e., representing the inclusion of scanned documents). Similarly, the same second compensation coefficient is configured for all area ratio influence factors, a third compensation coefficient is configured for all multi-column layouts, and the same fourth compensation coefficient is configured for all annotations (i.e., the fourth parameter is 1). The first compensation coefficient is greater than the fourth compensation coefficient, which is greater than the second compensation coefficient, which is greater than the third compensation coefficient. For example, the first compensation coefficient is set to 2, the fourth compensation coefficient is set to 1.8, the second compensation coefficient is set to 1.5, and the third compensation coefficient is set to 1.2.

[0097] Based on this, we can calculate the compensation time for different types of documents under different impact factors. Specifically, the compensation time for scanned documents = first parameter × total number of pages in the document × first compensation coefficient × basic single-page time; the compensation time for charts and graphs = second parameter × total number of pages in the document × second compensation coefficient × basic single-page time; the compensation time for typesetting = third parameter × total number of pages in the document × third compensation coefficient × basic single-page time; and the compensation time for annotations = fourth parameter × total number of pages in the document × fourth compensation coefficient × basic single-page time.

[0098] As can be seen, the estimated time for parsing the target document can be determined based on the total number of pages, the basic time per page, and the compensation time. Specifically, the estimated time = total number of pages × basic time per page + compensation time.

[0099] For example, in one optional embodiment, based on a single-page time set to 0.6 seconds, and the target document to be parsed being "doc_business.pdf", content feature data is extracted from "doc_business.pdf". The extraction results include: the document has a total of 200 pages; pages 50-80 are scanned copies, so the first parameter is marked as 1; pages 100-120 contain charts, so the second parameter is marked as 1; "doc_business.pdf" has a three-column layout, so the third parameter is marked as 1; excluding annotations, the fourth parameter is marked as 0. The process of extracting content feature data takes 4 seconds, which does not exceed the preset 10-second extraction time limit. The extraction results are written to the "Content Feature Data" field of the task record. Based on the content feature data of "doc_business.pdf" and the pre-built mapping relationship, the estimated time for parsing the current target document can be determined to be 240 seconds.

[0100] Furthermore, for the target computing node "doc_business.pdf", a parsing task is performed. Specifically, plain text pages are directly extracted from the text, scanned document pages are processed with OCR recognition, and complex chart pages are separated from the text for extraction. The actual parsing time is obtained, and the parsing results are saved.

[0101] Based on the above embodiments, as an optional embodiment, the compensation time for special content is determined according to the mapping relationship and special content parameters, including:

[0102] If the first parameter indicates that scanned documents are included, determine the current number of scanned documents; based on the mapping relationship, determine the first compensation coefficient corresponding to the current number of scanned documents; determine the scanned document compensation time based on the first compensation coefficient, the total number of pages in the document, and the basic single-page time; the number of scanned documents is positively correlated with the first compensation coefficient.

[0103] If the second parameter represents a chart, determine the current area ratio of the current chart area to the total area of ​​the target document to be parsed; based on the mapping relationship, determine the second compensation coefficient corresponding to the current area ratio; and determine the chart compensation time based on the second compensation coefficient, the total number of pages in the document, and the basic single-page time; the area ratio is positively correlated with the second compensation coefficient.

[0104] If the third parameter represents multi-column layout, determine the current layout quantity; based on the mapping relationship, determine the third compensation coefficient corresponding to the current layout quantity; and determine the layout compensation time based on the third compensation coefficient, the total number of pages in the document, and the basic single-page time; the layout quantity is positively correlated with the third compensation coefficient.

[0105] If the fourth parameter indicates that annotations are included, determine the current number of annotations; based on the mapping relationship, determine the fourth compensation coefficient corresponding to the current number of annotations; and determine the annotation compensation time based on the fourth compensation coefficient, the total number of pages in the document, and the basic single-page time; the number of annotations is positively correlated with the fourth compensation coefficient.

[0106] The total time spent compensating for scanned documents, charts, layout, and annotations is taken as the total compensation time.

[0107] It is understandable that, in specific embodiments, when the first parameter is 1 indicating the inclusion of scanned documents, different numbers of scanned documents have a significant impact on the estimated time. Specifically, the more scanned documents, the longer the estimated time. Similarly, the larger the ratio of the total area of ​​charts to the total area of ​​the document, the more charts are loaded in the target document to be parsed, and the longer the estimated time. The larger the number of layouts, the longer the estimated time; the larger the number of annotations, the longer the estimated time.

[0108] Based on the above analysis, in order to further improve the accuracy of the estimated duration, in an optional embodiment, the compensation coefficient is divided into finer granularity, that is, different compensation coefficients are set for different influencing factors.

[0109] Specifically, if the first parameter is 1, indicating that scanned items are included, a first compensation coefficient corresponding to the current number of scanned items must also be determined. The larger the number of scanned items, the larger the corresponding first compensation coefficient. Thus, the scanned item compensation time can be determined. In an optional embodiment, different values ​​can be set for the number of scanned items, with different values ​​corresponding to different first compensation coefficients; this application does not limit this.

[0110] If the second parameter is 1, indicating the inclusion of a chart, it's also necessary to determine the current ratio of the chart's area to the total area of ​​the target document to be parsed. Different area ratios in the mapping relationship correspond to different second compensation coefficients. Based on the mapping relationship, the second compensation coefficient corresponding to the current area ratio can be obtained, thus accurately determining the chart compensation time. Specifically, the larger the area ratio, the larger the corresponding second compensation coefficient.

[0111] If the third parameter is 1, it indicates that the current layout is multi-column. In the mapping relationship, different numbers of layouts correspond to different third compensation coefficients. Based on the mapping relationship, the compensation coefficient corresponding to the current number of layouts can be determined, thereby determining the layout compensation time. The larger the current number of layouts, the larger the corresponding third compensation coefficient.

[0112] If the fourth parameter is 1, it indicates that the annotation is included. In the mapping relationship, different numbers of annotations correspond to different fourth compensation coefficients. Based on this mapping relationship, the compensation coefficient corresponding to the current number of annotations can be determined, and thus the annotation compensation coefficient can be determined.

[0113] Furthermore, the sum of the compensation time for scanned documents, charts, typesetting, and annotations is taken as the compensation time.

[0114] In one optional embodiment, obtaining the target document to be parsed includes:

[0115] Upon receiving a parsing task request, the system filters out initial documents in the database whose task status is pending execution.

[0116] Obtain the initial document's ranking impact factor and the corresponding weight coefficient; the ranking impact factor includes at least one of the following: file size, level parameter representing task priority, complexity parameter representing content complexity, and urgency parameter representing task urgency.

[0117] Based on the weighting coefficients, the ranking factors are weighted and summed to obtain a comprehensive score.

[0118] The overall scores of the tasks are sorted in descending order, and the document with the highest overall score in the sorting results is selected as the target document to be parsed.

[0119] In one optional embodiment, a MySQL database is set up as a task storage carrier to store task records of PDF documents to be parsed, and each task record includes at least the task ID, PDF storage path, level parameter representing business priority, file size, content feature data, task status, number of retries, and creation time.

[0120] The task ID is used to uniquely identify the parsing task, allowing for differentiation between different parsing tasks. The PDF storage path is the storage location of the target document to be parsed in the system. The level parameter represents the business priority. In one optional embodiment, the priority can be divided into five levels from 1 to 5, with level 1 being the highest. This application does not limit the criteria for this division.

[0121] The file size is the size of the target document in bytes. Content feature data includes the total number of pages and special content parameters. Task status can include pending, executing, completed, failed, and permanently failed. The retries count records the number of times the target document was retried for parsing, and the creation time is the time the task was entered into the database.

[0122] In a specific embodiment, upon receiving a parsing task request, initial documents with a task status of "pending execution" are filtered from the MySQL database. If there is only one initial document, it is used as the target document to be parsed. If there are multiple initial documents, the target document to be parsed needs to be extracted from these multiple initial documents.

[0123] Understandably, different PDF documents have different levels of urgency for parsing. In order to avoid allocating multiple highly complex tasks to the same computing node in a short period of time, when a parsing task request is received, multiple initial documents need to be comprehensively scored, and the document with the highest comprehensive score is selected as the target document to be parsed.

[0124] Specifically, the ranking impact factors and their corresponding weight coefficients are obtained for the initial document. The ranking impact factors include at least one of the following: file size, a rank parameter representing task priority, a complexity parameter representing content complexity, and an urgency parameter representing task urgency. Furthermore, the ranking impact factors are weighted and summed, resulting in the overall score: Overall Score = File Size × First Weight + Rank Parameter × Second Weight + Complexity Parameter × Third Weight + Urgency Parameter × Fourth Weight.

[0125] In one optional embodiment, the first weight corresponding to the level parameter is greater than the fourth weight corresponding to the urgency parameter, the fourth weight is greater than the third weight corresponding to the complexity parameter, and the third weight is greater than the first weight corresponding to the file size.

[0126] After applying a comprehensive score to all initial documents, the scores are sorted in descending order to determine the document with the highest score, which is then designated as the target document to be parsed. At this point, a database row lock is used to lock the target parsing task corresponding to the target document to prevent it from being repeatedly fetched by other algorithm service instances, and the task status of the target document to be parsed is updated to "in progress".

[0127] It should be noted that the complexity parameter, which represents the complexity of the content, will result in a lower overall score if the content complexity is higher. This can help avoid assigning multiple highly complex tasks to the same node in a short period of time.

[0128] Therefore, by adopting the model of "algorithm service proactive retrieval + task retrieval service comprehensive scoring and screening", the shortcomings of the traditional "scheduling center proactive push" in predicting the time consumption are not accurate, ensuring that each algorithm service node runs continuously at full load and improving resource utilization.

[0129] In one optional embodiment, the target computing node performs a target parsing task to obtain the parsing result, including:

[0130] Based on content feature data, the corresponding execution tool is invoked to parse each page of the target document in sequence; and it is determined whether the current page has been parsed successfully.

[0131] If the parsing is successful, save the parsing result of the current page and proceed to the parsing task of the next page until the last page is parsed, and mark the parsing result as successful.

[0132] If parsing fails, perform the following steps:

[0133] Determine whether the ratio of the total number of pages with parsing failures to the total number of pages in the document reaches the ratio threshold;

[0134] If the ratio threshold is reached, end the target parsing task and mark the parsing result as a parsing failure;

[0135] If the ratio threshold is not reached, proceed to the steps of the parsing task for the next page.

[0136] It can be understood that for different content feature data, different execution tools need to be called for processing. For example, for pure text pages, the text can be directly extracted, for scanned documents, OCR needs to be started for recognition, and for complex charts, image and text separation needs to be performed. Therefore, in a specific embodiment, according to the content feature data, the corresponding execution tool needs to be called to parse each page of the target document to be parsed one by one, and determine whether the current page is successfully parsed in turn.

[0137] FIG. 4 is a schematic diagram of the principle of a distributed document parsing method provided by an embodiment of the present application. As shown in FIG. 4, in a specific embodiment, the target computing power node loads the target document according to the PDF storage path in the task, calls the PDF parsing algorithm to perform parsing, and starts a timer during the parsing process to monitor the execution time in real time. When parsing each page, determine whether the current page is successfully parsed. If it is successfully parsed, save the parsing result of the current page. At the same time, determine whether the current page is the last page. If not, continue to parse the next page until the last page is reached. If the current page is the last page, directly end the current parsing task and save the parsing results of all pages.

[0138] In another optional embodiment, if the current page parsing fails, first record the reason for the parsing failure. Further, determine whether the ratio of the current page with parsing failures to the total number of pages in the document reaches the ratio threshold. If it reaches, end the current parsing task, save the partial parsing results that have been parsed, and mark the final parsing result as a parsing failure. Of course, if the ratio threshold is not reached, it means that parsing can continue. At this time, proceed to the step of determining whether the current page is the last page, and loop like this to complete the target parsing task. For ease of understanding, an example will be given below.

[0139] For example, in an optional embodiment, the target document to be parsed is "doc_archive.pdf". This document is a new task, with a file size of 5MB, the level parameter representing the task priority is level 1, the total number of pages in the document is forty pages, and it contains 2 page damages and the special font "Fangzheng Xiaobiao Song".

[0140] After the algorithm service AS-007 obtains the "doc_archive.pdf" task (task ID T20240601004), it initializes the environment and loads the document, specifically from OSS to the local cache at / cache / AS-007 / doc_archive.pdf. The target computing node's resources include one CPU core and 512MB of memory. Executing the target parsing task requires starting the PDF parsing algorithm and a second-level timer.

[0141] In fact, the failure parsing rule provided in this application is that if the total number of pages that failed to parse reaches a threshold percentage of the total number of pages in the document (e.g., 10%), parsing will no longer be performed, and the parsing execution is considered a failure. Therefore, for a 40-page document in this embodiment, if the number of pages that failed to parse exceeds 4, the parsing execution is considered a failure.

[0142] During the parsing process for each page, the first 10 pages were parsed normally, and the extracted text showed no abnormalities. However, parsing failed on page 11. If the number of pages considered the point at which the entire document should fail to parse had not yet been reached, the parsing failure status and reason were recorded in the document parsing results. For example, if the parsing failure was due to missing fonts, the content recorded in the document parsing results would be: [Parsing Failed][Missing Fonts]. Parsing the remaining pages continued. Ultimately, 3 pages of this document failed to parse, and 37 pages were parsed successfully, which did not exceed 4 pages. Therefore, the target parsing task was considered successful, and the parsing results were stored by "Task ID". The task status of the target document to be parsed was updated to "Completed", and the completion time was recorded.

[0143] In another optional embodiment, if more than four pages fail to be parsed during the parsing process—specifically, when the fifth page fails—the parsing algorithm stops parsing the remaining pages of the document to conserve system resources. At this time, the parsing results of the previously parsed pages are saved to a temporary directory, and an alarm signal is sent to the client. The alarm signal includes the task ID, the page number that failed to be parsed, and the reason for each page's parsing failure. The alarm signal is sent via SMS, email, or other means; this application does not specifically limit the content or method of sending the alarm signal.

[0144] In an optional embodiment, in order to ensure the success rate of document parsing, if the parsing of the target document fails at the moment, it can continue to retry parsing. At this time, as shown in Figure 4, the number of retry updates for the target document to be parsed is updated first, so as to wait for the next update.

[0145] Based on the above embodiments, as an optional embodiment, after executing the target parsing task on the target computing power node and obtaining the parsing result, the following steps are included:

[0146] Obtain the actual parsing time and parsing results of the target computing power node;

[0147] Determine whether the actual parsing time is not greater than the timeout threshold, and whether the parsing result is successful; the timeout threshold is a preset multiple of the estimated time, and the preset multiple is greater than 1;

[0148] If so, update the task status of the target document to be parsed to "completed";

[0149] If not, determine whether the number of retries to parse the target document has reached the threshold; if it has, update the task status to permanent failure; if it has not reached the threshold, update the task status to pending execution.

[0150] Currently, existing solutions for timeout control of parsing tasks mostly use static fixed thresholds or simple segmentation settings based solely on file size. This approach fails to fully consider the varying complexity of PDF content, resulting in a lack of adaptability in timeout settings: if the timeout threshold is too short, the system may misjudge high-time-consuming tasks that should be executed normally as failures, triggering unnecessary retries or error handling; if the timeout threshold is set too long, nodes will continue to wait for extended periods even when tasks have actually stalled, causing resource congestion and significantly reducing the overall throughput and processing efficiency of the system.

[0151] To address the aforementioned technical issues, in one optional embodiment, to handle sudden fluctuations in parsing time, a corresponding timeout threshold is set based on the estimated timeout for different target documents to be parsed. Specifically, the timeout threshold is set as a preset multiple of the estimated timeout, and this preset multiple is greater than 1; for example, it can be set to 1.2 times. This allows for a 20% buffer space. The preset multiple can be set according to actual business needs. Therefore, the timeout threshold configuration method provided in this application, in specific embodiments, improves the accuracy of timeout threshold configuration and significantly reduces task misjudgment failures and node blocking problems caused by unreasonable timeout settings.

[0152] Based on the above embodiments, after the target computing node executes the target parsing task, the parsing result needs to be verified. Specifically, a timer is started during the parsing process to obtain the actual parsing duration. At the same time, the verification also needs to obtain the parsing result, further determining whether the actual parsing duration exceeds the timeout threshold, and whether the parsing result is successful.

[0153] If the actual parsing time does not exceed the timeout threshold, and the parsing result of the entire target document is successful, the parsing result data is written to the MySQL database, and the task status of the target document is updated to completed.

[0154] If the actual parsing time exceeds the timeout threshold or parsing fails due to other anomalies, the reason for the parsing failure is saved, and the successfully parsed data is stored. Simultaneously, it is determined whether the number of retries to parse the target document has reached a threshold. If the threshold is reached (e.g., 3 times), it indicates that the target document cannot be parsed any further. In this case, the task status of the target document is updated to permanent failure, thereby avoiding repeated retries that consume resources, and an alarm signal is sent to the client terminal.

[0155] If the threshold for the number of attempts has not been reached, it indicates that the target document to be parsed can continue to be parsed. The number of retries is updated by one, and the task status of the target document to be parsed is updated to "pending execution" so that it can be retrieved by the algorithm service later. Thus, through the retry mechanism and permanent failure markers, task omissions and repeated retries are avoided. Meanwhile, database row locks ensure task uniqueness, improving system stability and data consistency.

[0156] To help those skilled in the art better understand the technical solutions provided in this application, the following examples will provide a detailed description of the distributed document parsing method provided in this application.

[0157] For example, in one optional embodiment, the target document to be parsed is "doc_business.pdf", which is 15MB in size, has a priority level parameter of 4, and contains scanned copies and complex charts. After the user uploads "doc_business.pdf" to the system, the system automatically generates a task record in a preset MySQL task pool. The core fields include the task ID (T20240601001), the PDF storage path ( / data / pdf / doc_business.pdf), the priority parameter (level 4), the task status (pending execution), and the content feature data (initially empty).

[0158] Referring to the description of the above embodiment, after extracting content feature data from the document, the extraction results include: the document has a total of 200 pages; pages 50-80 are scanned copies, so the first parameter is marked as 1; pages 100-120 contain charts and graphs, so the second parameter is 1; and "doc_business.pdf" has a three-column layout, so the third parameter is 1; excluding annotations, the fourth parameter is 0. The process of extracting content feature data takes 4 seconds, which does not exceed the preset 10-second extraction time limit. The extraction results are written to the "Content Feature Data" field of the task record. Furthermore, the algorithm service sends a pull request to the task pull service. The task pull service selects the "doc_business.pdf" task with the highest comprehensive score from all tasks with a pending status according to the rule "Comprehensive score = file size × first weight + level parameter × second weight + complexity parameter × third weight + urgency parameter × fourth weight". It then locks "doc_business.pdf" using a locking statement (e.g., "SELECT * FROM task_pool WHERE task_id= 'T20240601001' FOR UPDATE") and updates the task status of "doc_business.pdf" to "Executing".

[0159] Based on the content feature data of "doc_business.pdf" and the pre-built mapping relationship, the estimated time for parsing the current target document can be determined to be 240 seconds, which does not exceed the timeout threshold of 288 seconds (the timeout threshold is 1.2 times the estimated time). After the parsing results pass the integrity and accuracy verification, they are sent back to the result storage service. The system updates the task status to "completed" and stores the parsing results according to the naming rule of "task ID + page number".

[0160] For example, in another alternative embodiment, the target document to be parsed is "doc_tech.pdf", which is a brand new task, has a file size of 6MB, a level parameter of 2, no special content parameters, and a task urgency of "high".

[0161] Two algorithm services (instance IDs AS-004 and AS-005 respectively) simultaneously send task fetch requests to the task fetch service. AS-004's request information includes: instance ID AS-004, CPU utilization 25%, memory utilization 20%, and request time 2024-06-01 10:00:00. AS-005's request information includes: instance ID AS-005, CPU utilization 30%, memory utilization 22%, and request time 2024-06-01 10:00:02.

[0162] The task retrieval service establishes a queuing mechanism based on the "request time order," prioritizing the processing of AS-004 requests and delaying the processing of AS-005 requests by 2 seconds. After verifying the legitimacy of both services (both are within the preset instance list), a total of 4 "pending execution" tasks are selected, including "doc_tech.pdf" (task IDT20240601002).

[0163] Based on the comprehensive scoring calculation method described in the above embodiment, the document "doc_tech.pdf" was determined to have the highest score and was selected as the target document to be parsed. Specifically, the first weight was 0.15, the second weight was 0.6, and the third weight was 0.15. In the mapping relationship, the file size was configured with 0.15 points for 0-5MB and 0.18 points for 5-10MB. For the urgency parameter, the advanced configuration was set to 1. Therefore, the comprehensive score was 2×0.6+0.18+1×0.15=1.53 points, ranking first among the four tasks to be executed.

[0164] The task retrieval service performs a row lock operation on "doc_tech.pdf" and updates the task status to "in progress" to prevent AS-005 from fetching it repeatedly. After the execution is complete and the parsing result is obtained, the task information of "doc_tech.pdf" (storage path / data / pdf / doc_tech.pdf, estimated time 35 seconds, timeout 42 seconds) is returned to AS-004 in JSON format, while AS-005 is sent a message indicating "no high-priority task available".

[0165] For example, in another optional embodiment, the target document to be parsed is "doc_report.pdf". This document is a brand new task, with a file size of 20MB, a level parameter of 5, a three-column layout, handwritten annotations, and a total of 250 pages. After "doc_report.pdf" is uploaded, the system triggers the feature extraction module, loads tool libraries including but not limited to pdfplumber, OpenCV, and PyTesseract, and enters the feature extraction preparation state.

[0166] In a specific embodiment of content feature data extraction, the total number of pages in the document was 250. OpenCV was used to detect handwritten annotations on pages 120-150 (the fourth parameter was marked as 1). Analysis of the X-coordinate distribution of the text lines revealed three concentrated intervals (left column 40-200, middle column 250-410, right column 460-620), indicating that the document was multi-column formatted; therefore, the third parameter was marked as 1. The extraction took 5 seconds, exceeding the preset time of 10 seconds. The extraction results were written to the task record (task IDT20240601003).

[0167] In a specific embodiment, parsing performance data from the previous 7 days is retrieved, and the average parsing time for plain text pages is found to be 0.45 seconds, which is lower than the preset adjustable range of 0.5-1 seconds. Therefore, the basic single-page parsing time is automatically reduced from the initial 0.5 seconds to 0.4 seconds to ensure that the parameters match the actual performance.

[0168] Based on the above basic single-page time, the estimated time is determined. Layout compensation time = 1 × 250 × 0.8 × 0.4 = 80 seconds. Based on the mapping relationship, the third compensation coefficient for multi-column layout is determined to be 0.8. Annotation compensation time = 1 × 250 × 1.8 × 0.4 = 180 seconds. Based on the mapping relationship, the fourth compensation coefficient for annotations is determined to be 1.8. Therefore, the estimated document time is: 250 × 0.4 + 80 + 180 = 360 seconds. The timeout threshold is set to 360 × 1.2 = 432 seconds. The estimated time, timeout threshold, and parameter adjustment records are synchronized to the task record.

[0169] In the above embodiments, the method for parsing distributed documents has been described in detail. This application also provides an embodiment of a device for parsing distributed documents.

[0170] Figure 5 is a schematic diagram of a distributed document parsing device provided in an embodiment of this application. As shown in Figure 5, the device includes:

[0171] Target document acquisition module 50 is used to acquire the target document to be parsed;

[0172] Feature data extraction module 51 is used to extract content feature data of the target document to be parsed; the content feature data is used to reflect the complexity of the content of the target document to be parsed.

[0173] The estimated time determination module 52 is used to determine the estimated time for parsing the target document based on content feature data.

[0174] The task allocation module 53 is used to allocate the target parsing task of the target document to the target computing power node based on load balancing and the estimated time consumption, so that the target computing power node can execute the target parsing task and obtain the parsing result.

[0175] Furthermore, the distributed document parsing apparatus provided in this application embodiment also includes:

[0176] The target acquisition module is used to acquire the highest resource parameters representing computing power resources in each computing power node and the pre-built mapping relationship. The mapping relationship is the correspondence between the influence factors of content complexity and the compensation coefficient. The influence factors include the number of scanned documents, the ratio of the total area of ​​charts to the total area of ​​the document, the number of layouts, and the number of annotations. The content feature data includes at least the total number of pages in the document and special content parameters. The special content parameters include a first parameter for indicating whether scanned documents are included, a second parameter for indicating whether charts are included, a third parameter for indicating whether it is a multi-column layout, and a fourth parameter for indicating whether annotations are included.

[0177] The basic single-page time determination module is used to determine the basic single-page time for parsing a single page within a preset time range based on the highest resource parameter; the highest resource parameter is negatively correlated with the basic single-page time; the preset time range is determined based on the historical basic single-page time within a specified historical time range;

[0178] The compensation time determination module is used to determine the compensation time for special content based on the mapping relationship and special content parameters; the compensation time includes the compensation time for scanned documents, charts, layout, and annotations.

[0179] The estimated time determination submodule is used to determine the estimated time based on the total number of document pages, the basic single-page time, and the compensation time.

[0180] The scanned document compensation time determination module uses a first parameter to characterize the number of scanned documents when they are included, and determines the current number of scanned documents; based on the mapping relationship, it determines the first compensation coefficient corresponding to the current number of scanned documents; and determines the scanned document compensation time based on the first compensation coefficient, the total number of pages in the document, and the basic single-page time; the number of scanned documents is positively correlated with the first compensation coefficient.

[0181] The chart compensation time determination module is used to determine the current area ratio of the current chart area to the total area of ​​the target document to be parsed when the second parameter represents the inclusion of a chart; based on the mapping relationship, it determines the second compensation coefficient corresponding to the current area ratio; and determines the chart compensation time based on the second compensation coefficient, the total number of pages in the document, and the basic single-page time; the area ratio is positively correlated with the second compensation coefficient.

[0182] The layout compensation time determination module is used to determine the current number of layouts when the third parameter represents a multi-column layout; based on the mapping relationship, it determines the third compensation coefficient corresponding to the current number of layouts; and determines the layout compensation time based on the third compensation coefficient, the total number of pages in the document, and the basic single-page time; the number of layouts is positively correlated with the third compensation coefficient.

[0183] The annotation compensation time determination module is used to determine the current number of annotations when the fourth parameter represents the inclusion of annotations; determine the fourth compensation coefficient corresponding to the current number of annotations based on the mapping relationship; and determine the annotation compensation time based on the fourth compensation coefficient, the total number of pages in the document, and the basic single-page time; the number of annotations is positively correlated with the fourth compensation coefficient.

[0184] The compensation time determination submodule is used to take the sum of the compensation time for scanned documents, charts, layout, and annotations as the compensation time.

[0185] The document filtering module is used to filter out initial documents with a task status of "to be executed" from the database after receiving a parsing task request;

[0186] The scoring module is used to obtain the ranking impact factor of the initial document and the weight coefficient of the ranking impact factor; and based on the weight coefficient, the ranking impact factor is weighted and summed to obtain a comprehensive score; the ranking impact factor includes at least one of the following: file size, level parameter representing task priority, complexity parameter representing content complexity, and urgency parameter representing task urgency.

[0187] The target document to be parsed module is used to sort the task's overall score in descending order and select the document with the highest overall score in the sorting results as the target document to be parsed.

[0188] The first processing module is used to call the corresponding execution tool to parse each page of the target document to be parsed in sequence based on the content feature data; and to determine whether the current page is parsed successfully; if the parsing is successful, the parsing result of the current page is saved and the parsing task of the next page is started, until the last page is parsed, and the parsing result is marked as parsing successful; if the parsing fails, the second processing module is called.

[0189] The second processing module is used to determine whether the ratio of the total number of pages that failed to be parsed to the total number of pages in the document reaches a ratio threshold. If the ratio threshold is reached, the target parsing task is terminated and the parsing result is marked as parsing failure. If the ratio threshold is not reached, the next page parsing task is started.

[0190] The parsing information acquisition module is used to obtain the actual parsing time and parsing results of the target computing power node;

[0191] The third processing module is used to determine whether the actual parsing time is not greater than the timeout threshold and whether the parsing result is successful. The timeout threshold is a preset multiple of the estimated time, and the preset multiple is greater than 1. If so, the task status of the target document to be parsed is updated to completed. If not, it is determined whether the number of times the target document to be parsed has been retried has reached the number of times thresholded. If the number of times thresholded has been reached, the task status is updated to permanent failure. If the number of times thresholded has not been reached, the task status is updated to pending execution.

[0192] Figure 6 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. As shown in Figure 6, the electronic device includes: a memory 60 for storing computer programs;

[0193] The processor 61 is configured to implement the steps of the distributed document parsing method as described in the above embodiments when executing a computer program.

[0194] The electronic devices provided in this embodiment may include, but are not limited to, laptops or desktop computers.

[0195] The processor 61 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 61 may be implemented using at least one of the following hardware forms: Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 61 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 61 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 61 may also include an Artificial Intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0196] The memory 60 may include one or more computer-readable storage media, which may be non-transitory. The memory 60 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 60 is used to store at least the following computer program 601, which, after being loaded and executed by the processor 61, is capable of implementing the relevant steps of the distributed document parsing method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 60 may also include an operating system 602 and data 603, and the storage method may be temporary or permanent storage. The operating system 602 may include Windows, Unix, Linux, etc. The data 603 may include, but is not limited to, the relevant data involved in the distributed document parsing method.

[0197] In some embodiments, the electronic device may further include a display screen 62, an input / output interface 63, a communication interface 64, a power supply 65, and a communication bus 66.

[0198] Those skilled in the art will understand that the structure shown in FIG6 does not constitute a limitation on the electronic device and may include more or fewer components than shown.

[0199] The electronic device provided in this application includes a memory and a processor. When the processor executes a program stored in the memory, it can implement the distributed document parsing method described in the above embodiments.

[0200] It should be noted that although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

Claims

1. A distributed document parsing method, characterized in that, The method includes: acquiring a target document to be parsed; extracting content feature data from the target document to be parsed; the content feature data reflecting the complexity of the content of the target document to be parsed; the content feature data including at least the total number of pages and special content parameters; the special content parameters including a first parameter indicating whether scanned documents are included, a second parameter indicating whether charts are included, a third parameter indicating whether the document is multi-column formatted, and a fourth parameter indicating whether annotations are included; determining the estimated time for parsing the target document to be parsed based on the content feature data; and, based on load balancing, allocating the target parsing task of the target document to a target computing node according to the estimated time, so that the target computing node can execute the target parsing task and obtain the parsing result; the method further includes: determining the estimated time for parsing the target document to be parsed based on the content feature data. The time estimation includes: obtaining the highest resource parameter representing computing power resources in each computing node and a pre-constructed mapping relationship; the mapping relationship is the correspondence between the influence factor of content complexity and the compensation coefficient; the influence factor includes the number of scanned documents, the area ratio of the total area of ​​charts to the total area of ​​the document, the number of layouts, and the number of annotations; determining the basic single-page time for parsing a single page within a preset time range based on the highest resource parameter; the highest resource parameter is negatively correlated with the basic single-page time; the preset time range is determined based on the historical basic single-page time within a specified historical time range; determining the compensation time for special content based on the mapping relationship and the special content parameter; the compensation time includes the compensation time for scanned documents, charts, layouts, and annotations; and determining the estimated time based on the total number of document pages, the basic single-page time, and the compensation time.

2. The distributed document parsing method as described in claim 1, characterized in that, Based on the mapping relationship and the special content parameters, the compensation time for special content is determined, including: if the first parameter indicates the inclusion of scanned documents, determining the current number of scanned documents; based on the mapping relationship, determining a first compensation coefficient corresponding to the current number of scanned documents; determining the compensation time for the scanned documents based on the first compensation coefficient, the total number of pages in the document, and the basic single-page time; the number of scanned documents is positively correlated with the first compensation coefficient; if the second parameter indicates the inclusion of charts, determining the current area ratio of the current chart area to the total area of ​​the target document to be parsed; based on the mapping relationship, determining a second compensation coefficient corresponding to the current area ratio; and determining the compensation time for the charts based on the second compensation coefficient, the total number of pages in the document, and the basic single-page time; the area ratio is positively correlated with the second compensation coefficient. If the third parameter indicates a multi-column layout, determine the current layout quantity; based on the mapping relationship, determine the third compensation coefficient corresponding to the current layout quantity; and determine the layout compensation time based on the third compensation coefficient, the total number of pages in the document, and the basic single-page time; the layout quantity is positively correlated with the third compensation coefficient; if the fourth parameter indicates the inclusion of annotations, determine the current annotation quantity; based on the mapping relationship, determine the fourth compensation coefficient corresponding to the current annotation quantity; and determine the annotation compensation time based on the fourth compensation coefficient, the total number of pages in the document, and the basic single-page time; the annotation quantity is positively correlated with the fourth compensation coefficient; the sum of the scanned document compensation time, the chart compensation time, the layout compensation time, and the annotation compensation time is taken as the compensation time.

3. The distributed document parsing method as described in claim 1, characterized in that, The process of obtaining the target document to be parsed includes: after receiving a parsing task request, filtering out initial documents with a task status of pending execution from the database; obtaining the ranking influence factor and the weight coefficient corresponding to the ranking influence factor of the initial document; the ranking influence factor includes at least one of file size, a level parameter representing task priority, a complexity parameter representing content complexity, and an urgency parameter representing task urgency; weighting and summing the ranking influence factors based on the weight coefficient to obtain a comprehensive score; sorting the comprehensive scores of the task in descending order, and selecting the document with the highest comprehensive score in the sorting results as the target document to be parsed.

4. The distributed document parsing method as described in claim 1, characterized in that, The target computing node executes the target parsing task to obtain the parsing result, including: based on the content feature data, calling the corresponding execution tool to sequentially parse each page of the target document to be parsed; and determining whether the current page is parsed successfully; if the parsing is successful, saving the parsing result of the current page and proceeding to the parsing task of the next page, until the last page is parsed, and marking the parsing result as parsing successful; if the parsing fails, performing the following steps: determining whether the ratio of the total number of pages that failed to be parsed to the total number of pages in the document reaches a ratio threshold; if the ratio threshold is reached, ending the target parsing task and marking the parsing result as parsing failed; if the ratio threshold is not reached, proceeding to the parsing task of the next page.

5. The distributed document parsing method as described in claim 1, characterized in that, After the target parsing task is executed on the target computing power node and the parsing result is obtained, the process includes: obtaining the actual parsing time and parsing result of the target computing power node; determining whether the actual parsing time is not greater than a timeout threshold and whether the parsing result is successful; the timeout threshold is a preset multiple of the estimated time consumption, and the preset multiple is greater than 1; if yes, updating the task status of the target document to be parsed to completed; if no, determining whether the number of times the target document to be parsed has been retried for parsing has reached a threshold; if the threshold has been reached, updating the task status to permanently failed; if the threshold has not been reached, updating the task status to pending execution.

6. A distributed document parsing device, characterized in that, The device includes: a target document acquisition module for acquiring a target document to be parsed; a feature data extraction module for extracting content feature data of the target document to be parsed; the content feature data reflects the complexity of the content of the target document to be parsed; the content feature data includes at least the total number of pages in the document and special content parameters; the special content parameters include a first parameter indicating whether it contains scanned documents, a second parameter indicating whether it contains charts, a third parameter indicating whether it is a multi-column layout, and a fourth parameter indicating whether it contains annotations; an estimated time determination module for determining the estimated time for parsing the target document to be parsed based on the content feature data; a task allocation module for allocating the target parsing task of the target document to be parsed to target computing nodes based on load balancing and the estimated time, so that the target computing nodes can execute the target parsing task and obtain the parsing result; and a target acquisition module for acquiring various... The system includes a highest resource parameter representing computing power resources in the computing power node and a pre-built mapping relationship; the mapping relationship is the correspondence between the influence factor of content complexity and the compensation coefficient; the influence factor includes the number of scanned documents, the area ratio of the total area of ​​charts to the total area of ​​the document, the number of layouts, and the number of annotations; a basic single-page time determination module is used to determine the basic single-page time for parsing a single page within a preset time range based on the highest resource parameter; the highest resource parameter is negatively correlated with the basic single-page time; the preset time range is determined based on the historical basic single-page time within a specified historical time range; a compensation time determination module is used to determine the compensation time for special content based on the mapping relationship and the special content parameter; the compensation time includes the compensation time for scanned documents, charts, layouts, and annotations; and an estimated time determination submodule is used to determine the estimated time based on the total number of document pages, the basic single-page time, and the compensation time.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the distributed document parsing method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the distributed document parsing method according to any one of claims 1 to 5.

Images