Document processing method and device, equipment and storage medium

Abstract

The invention discloses a document processing method and device, equipment and a storage medium, and belongs to the technical field of computers. The method comprises the steps that multiple target content blocks of a to-be-processed document and related information of the multiple target content blocks are obtained, and the related information comprises corner position information of a target frame; performing column cluster clustering on the plurality of target content blocks based on the angular point position information of the target frames of the plurality of target content blocks, and dividing the plurality of target content blocks into a plurality of column cluster lists; sorting the target content blocks in the plurality of column cluster lists on the basis of a reading sequence of the to-be-processed document; and based on the plurality of sorted target content blocks, generating knowledge of the to-be-processed document. The application can improve the reliability of document processing.

Description

Technical Field

[0001] The present application belongs to the field of computer technology, and in particular, relates to a document processing method, device, equipment and storage medium. Background Art

[0002] With the emergence of big models and the continuous improvement of model understanding capabilities, various applications based on big models are constantly innovating. Establishing a private domain expert big model based on a local knowledge base is one type of application. This type of application can effectively utilize domain data and ensure data security.

[0003] The acquisition of document knowledge is an important part of the local knowledge base. Currently, document knowledge is usually acquired through an open source framework, and then the acquired document knowledge is stored in the local knowledge base. This solution is based on the document parsing module built into the framework, extracts the text content in the document, and then splits the text content based on the set delimiter and the length of the knowledge fragment to obtain paragraph blocks, and then combines the paragraph blocks from top to bottom to obtain document knowledge. However, document typesetting is diverse and complex. The above solution is prone to chaotic arrangement of paragraph blocks, which does not conform to the reading order of the document, resulting in low reliability of document processing. Summary of the invention

[0004] Embodiments of the present application provide a document processing method, apparatus, device, and storage medium, which can solve the problem of low reliability of document processing.

[0005] In a first aspect, an embodiment of the present application provides a document processing method, the method comprising:

[0006] Acquire multiple target content blocks of the document to be processed, and related information of the multiple target content blocks, wherein the related information includes corner point position information of the target box;

[0007] Based on the corner point position information of the target boxes of the multiple target content blocks, clustering the multiple target content blocks into column clusters, and dividing the multiple target content blocks into multiple column cluster lists;

[0008] sorting the target content blocks in the plurality of column cluster lists based on the reading order of the to-be-processed document;

[0009] Based on the sorted plurality of target content blocks, knowledge of the document to be processed is generated.

[0010] In a second aspect, an embodiment of the present application provides a document processing device, the device comprising:

[0011] A first acquisition module, used to acquire multiple target content blocks of a document to be processed and related information of the multiple target content blocks, wherein the related information includes corner point position information of a target box;

[0012] A clustering module, configured to cluster the multiple target content blocks into column clusters based on the corner point position information of the target boxes of the multiple target content blocks, and divide the multiple target content blocks into multiple column cluster lists;

[0013] A sorting module, configured to sort the target content blocks in the plurality of column cluster lists based on the reading order of the to-be-processed document;

[0014] A generating module is used to generate the knowledge of the document to be processed based on the sorted multiple target content blocks.

[0015] In a third aspect, an embodiment of the present application provides a document processing device, the device comprising:

[0016] A processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the document processing method as described in the first aspect is implemented.

[0017] In a fourth aspect, an embodiment of the present application provides a computer storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the document processing method described in the first aspect is implemented.

[0018] In a fifth aspect, an embodiment of the present application provides a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the document processing method as described in the first aspect.

[0019] In the embodiment of the present application, first, in addition to obtaining multiple target content blocks of the document to be processed, the corner point position information of the target boxes of the multiple target content blocks can be further obtained. Then, based on the corner point position information of the target boxes of the multiple target content blocks, the multiple target content blocks can be clustered into column clusters, and the multiple target content blocks can be divided into multiple column cluster lists; then, based on the reading order of the document to be processed, the target content blocks in the multiple column cluster lists are sorted; based on the sorted multiple target content blocks, the knowledge of the document to be processed is generated. In this way, by firstly dividing the target content blocks into corresponding column cluster lists, and then sorting the target content blocks in the column cluster lists based on the reading order of the document to be processed to generate document knowledge, the mixed arrangement of content blocks between document columns can be avoided, so that the sorting of content blocks in the document knowledge conforms to the reading order of the document, thereby improving the reliability of document processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0021] Figure 1 It is one of the flowcharts of the document processing method provided in the embodiment of the present application;

[0022] Figure 2a is a schematic diagram of a document to be processed provided by an embodiment of the present application;

[0023] Figure 2b This is the second flowchart of the document processing method provided in the embodiment of the present application;

[0024] Figure 3 is a structural diagram of a document processing device provided in an embodiment of the present application;

[0025] Figure 4 It is a structural diagram of a document processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The features and exemplary embodiments of each aspect of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0027] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0028] The document processing method provided by the embodiment of the present application is described in detail below through some embodiments and their application scenarios in combination with the accompanying drawings.

[0029] See also Figure 1 , Figure 1 This is one of the flowcharts of the document processing method provided in the embodiment of the present application. Figure 1 As shown, the document processing method may include the following steps:

[0030] Step 101: Acquire multiple target content blocks of a document to be processed and related information of the multiple target content blocks, wherein the related information includes corner point position information of a target box.

[0031] The document to be processed may also be referred to as the document to be parsed, and may include at least one of the following contents: title, text, picture, table, etc. Furthermore, the document to be processed may be a structured document with a complete row and column structure or a frame structure, or may be a non-structured document such as Word or PDF, that is, the embodiment of the present application may be applicable to the processing of both structured and non-structured documents, thus enriching the document processing types.

[0032] The target content block of the document to be processed may be a content block initially obtained from the document to be processed (hereinafter referred to as the initial content block), or may be a content block obtained after screening the initial content block of the document to be processed. That is, the target content block of the document to be processed may include part or all of the content blocks obtained from the document to be processed.

[0033] The embodiments of the present application do not limit the method for obtaining the content block of the document to be processed. In some embodiments, a target detection model for document content block recognition can be trained, and then the content block of the document to be processed and related information thereof can be obtained through the target detection model. In other embodiments, the document to be processed can be scanned first, and then the scanned document to be processed can be split to obtain the content block of the document to be processed, and then the split content block can be further analyzed to obtain related information of the content block.

[0034] A content block may include one or more contents in a document to be processed. Further, when a content block includes multiple contents, the attributes of these contents may be the same or different, such as: in one example, a content block may include text and pictures. However, it should be noted that in an embodiment of the present application, a content block corresponds to a content attribute classification, and a content attribute classification may also be referred to as a category, that is, a content block refers to a category of content. It is worth noting that the content included in each category of content blocks can be pre-set based on actual needs, such as: in one example, content blocks can be divided into the following categories: title content block, list content block, picture content block, further, picture content blocks may include "pure picture" type content blocks and "text type picture" content blocks, and "text type picture" content blocks may include text and pictures.

[0035] The relevant information of the content block may include, but is not limited to, the corner point position information of the target box of the content block. The target box of the content block is a rectangular box that can mark the position and range of the content block. The corner point position information of the target box of the content block may indicate the position information of the content block in the document to be processed. The corner point position information of the target box of the content block may include the horizontal and vertical coordinate information of the corner points of the target box of the content block relative to the origin. The origin may be set based on actual needs. In some embodiments, the origin may be the upper left corner of the page of the document to be processed where the content block is located. In other embodiments, the origin may be the upper right corner of the page of the document to be processed where the content block is located. In an example, the corner point position information of the target frame of the content block can be expressed as (x1, y1, x2, y2), where (x1, y1) is the position information of the upper left corner point of the target frame, x1 is the horizontal coordinate of the upper left corner point of the target frame, and y1 is the vertical coordinate of the upper left corner point of the target frame; (x2, y2) is the position information of the lower right corner point of the target frame, x2 is the horizontal coordinate of the lower right corner point of the target frame, and y2 is the vertical coordinate of the lower right corner point of the target frame.

[0036] Step 102: cluster the target content blocks into column clusters based on the corner point position information of the target frames of the target content blocks, and divide the target content blocks into a plurality of column cluster lists.

[0037] In an embodiment of the present application, after obtaining multiple target content blocks of a document to be processed and before utilizing the knowledge of combining multiple target content blocks, in order to avoid mixed arrangement of content blocks between document columns, the target content blocks can be clustered into column clusters based on the corner point position information of the target boxes of the target content blocks, thereby achieving column splitting of the target content blocks and dividing the target content blocks into corresponding column cluster lists.

[0038] Column cluster clustering can be understood as clustering in the column direction or vertical direction. Each column cluster list can include one or more target content blocks.

[0039] In specific implementation, the horizontal coordinate of a corner point of the target frame of the target content block, such as the x1 coordinate of the upper left corner point, can be extracted to obtain a horizontal coordinate array, and then the horizontal coordinate array is clustered to obtain the number of column clusters k and the column cluster classification information of each target content block. Then, according to the column cluster classification information of the target content block, the target content block is divided into corresponding column cluster lists of k column cluster lists, where k is a positive integer.

[0040] The embodiments of the present application do not limit the clustering method. In some embodiments, DBSCAN can be used for clustering, but it is not limited to this.

[0041] Step 103: Sort the target content blocks in the plurality of column cluster lists based on the reading order of the to-be-processed document.

[0042] In an embodiment of the present application, after completing the column splitting of the target content block and obtaining multiple column cluster lists, the target content blocks in the column cluster list can be sorted based on the reading order of the document to be processed, so that the sorting of the target content blocks conforms to the reading order of the document to be processed, thereby improving the reliability of document knowledge acquisition.

[0043] When the reading order of the document to be processed is from top to bottom and from left to right, the target content blocks may be sorted from top to bottom within the column cluster list and from left to right between the column cluster lists.

[0044] When the reading order of the document to be processed is from top to bottom and from right to left, the target content blocks may be sorted from top to bottom within the column cluster list and from right to left between the column cluster lists.

[0045] Step 104: Generate knowledge of the document to be processed based on the sorted plurality of target content blocks.

[0046] After obtaining the target content blocks whose order matches the reading order of the document to be processed, the target content blocks may be combined to generate knowledge of the document to be processed.

[0047] In some embodiments, the sorted target content blocks may be split and combined based on the upper limit of the knowledge length to generate knowledge of the document to be processed.

[0048] In other embodiments, the relevant information of the content block may further include content attribute classification information of the content block, and the document processing device may store a first knowledge combination rule, and the first knowledge combination rule may indicate content attribute classification information that supports combination. In these embodiments, the sorted target content blocks may be split and combined in combination with the knowledge length upper limit and the knowledge combination rule to generate knowledge of the document to be processed. For specific implementation, reference may be made to the following related description.

[0049] The document processing method of the embodiment of the present application, first, in addition to obtaining multiple target content blocks of the document to be processed, can further obtain the corner point position information of the target boxes of the multiple target content blocks. Then, based on the corner point position information of the target boxes of the multiple target content blocks, the multiple target content blocks can be clustered into column clusters, and the multiple target content blocks can be divided into multiple column cluster lists; then, based on the reading order of the document to be processed, the target content blocks in the multiple column cluster lists are sorted; based on the sorted multiple target content blocks, the knowledge of the document to be processed is generated. In this way, by firstly dividing the target content blocks into corresponding column cluster lists, and then sorting the target content blocks in the column cluster lists based on the reading order of the document to be processed to generate document knowledge, the mixed arrangement of content blocks between document columns can be avoided, so that the sorting of content blocks in the document knowledge conforms to the reading order of the document, thereby improving the reliability of document processing.

[0050] The following is a detailed description of how to obtain a column cluster list.

[0051] In some embodiments, after obtaining multiple target content blocks of the document to be processed, the multiple target content blocks may be directly classified into column clusters to obtain multiple column cluster lists.

[0052] In some other embodiments, the relevant information of the target content block further includes a width coefficient of the target frame;

[0053] The clustering of the multiple target content blocks into column clusters based on the corner point position information of the target frames of the multiple target content blocks, and dividing the multiple target content blocks into multiple column cluster lists, includes:

[0054] The target content blocks whose target frame width coefficients are less than or equal to a first threshold value among the multiple target content blocks are classified into a first content block list, and the target content blocks whose target frame width coefficients are greater than the first threshold value are classified into a second content block list;

[0055] Based on the corner point position information of the target frame of the target content block in the first content block list, clustering the target content blocks in the first content block list into column clusters, and dividing the target content blocks in the first content block list into n column cluster lists, where n is a positive integer;

[0056] The multiple column cluster lists include the n column cluster lists and the second content block list.

[0057] In this embodiment, a width coefficient of the target frame of the content block is introduced, and the width coefficient can be used to split the document page vertically. The width coefficient of the target frame of the target content block can be the ratio of the width of the target frame to the page width of the document to be processed, and the value range is (0, 1].

[0058] It should be noted that the method for obtaining the width coefficient of the target frame and the method for obtaining the corner point position information of the target frame may be the same or different. In some embodiments, the width coefficient of the target frame and the corner point position information may be output by the target detection model. In other embodiments, the width coefficient of the target frame may be determined based on the corner point position information of the target frame. Specifically, the width of the target frame may be determined based on the corner point position information of the target frame, and then the ratio of the width of the target frame to the page width of the document to be processed may be calculated to obtain the width coefficient of the target frame.

[0059] In this embodiment, the target content block can be first split into two content block lists based on the width coefficient of the target frame of the target content block. In specific implementation, a first threshold can be pre-set, and for each target content block, the width coefficient of its target frame can be compared with the first threshold respectively, and then the target content blocks whose width coefficient of the target frame is less than or equal to the first threshold are divided into the first content block list, and the target content blocks whose width coefficient of the target frame is greater than the first threshold are divided into the second content block list. The first threshold can be pre-set based on actual needs, and further, can be greater than or equal to 0.5. For example, in one example, the first threshold can be set to 0.5.

[0060] It can be understood that the width of the target content blocks in the first content block list is relatively small. Therefore, the target content blocks in the first content block list may be located in different columns in the document to be processed. Based on this, the target content blocks in the first content block list can be clustered into column clusters to achieve horizontal column splitting of the target content blocks in the first content block list, and divide the target content blocks in the first content block list into n column cluster lists.

[0061] In a specific implementation, the horizontal coordinate of a corner point of the target frame of each target content block in the first content block list, such as the coordinate of the upper left corner point x1, can be extracted to obtain a horizontal coordinate array, and then the horizontal coordinate array is clustered to obtain the number of column clusters n and the column cluster classification information of each target content block. Then, according to the column cluster classification information of the target content block, the target content block is divided into the corresponding column cluster lists of the n column cluster lists.

[0062] As for the second content block list, the target content blocks included therein are of greater width and can be considered to be located in the same column. Therefore, there is no need to further perform horizontal column splitting on it and it can be directly regarded as a column cluster list.

[0063] Therefore, in this embodiment, the plurality of column cluster lists may include the second content block list, and n column cluster lists determined based on the first content block list.

[0064] For ease of understanding, combined Figure 2a For example, Figure 2aThe document to be processed shown includes the following content blocks: text_1 to text_6, and title. Based on the width coefficient of the target box of the content block, the content blocks text_1 to text_6 will be divided into the first content block list, and the title will be divided into the second content block list. Further, by clustering the content blocks in the first content block list into column clusters, the following two column cluster lists can be clustered: [text_1, text_4, text_5] and [text_2, text_3, text_6].

[0065] In this embodiment, the target content block is first split into two content block lists based on the width coefficient of the target frame of the target content block, and then the first content block list is clustered into columns to achieve column splitting of the content block. In this way, the influence of the content blocks that do not need column splitting in the second content block list on the column cluster clustering can be avoided, thereby improving the accuracy of the column cluster clustering results.

[0066] The following is a detailed description of the order of content blocks.

[0067] In some embodiments, the sizes of the horizontal and vertical coordinates of a corner point of the target box of the target content block in multiple column cluster lists can be directly compared, and then sorted in an arrangement order consistent with the reading order of the document to be processed, so that the sorting of the content blocks can be simplified.

[0068] Exemplarily, it is assumed that the reading order of the document to be processed is from top to bottom and from left to right. At the same time, it is assumed that the multiple column cluster lists include column cluster list 1 and column cluster list 2, column cluster list 1 includes target content block a and target content block b, and column cluster list 2 includes target content block c and target content block d, wherein the horizontal coordinate of the upper left corner of the target frame of target content block b is less than the horizontal coordinate of the upper left corner of the target frame of target content block a, less than the horizontal coordinate of the upper left corner of the target frame of target content block c, and less than the horizontal coordinate of the upper left corner of the target frame of target content block d; the vertical coordinate of the upper left corner of the target frame of target content block a is equal to the vertical coordinate of the upper left corner of the target frame of target content block c, less than the vertical coordinate of the upper left corner of the target frame of target content block b, and less than the vertical coordinate of the upper left corner of the target frame of target content block d.

[0069] Then, based on the reading order of the document to be processed, the arrangement order of the above four target content blocks from first to last is: target content block a, target content block b, target content block c and target content block d.

[0070] In some other embodiments, the plurality of column cluster lists include a first content block list and a second content block list, wherein a width coefficient of a target box of a target content block in the first content block list is less than a first threshold, and a width coefficient of a target box of a target content block in the second content block list is greater than the first threshold; sorting the target content blocks in the plurality of column cluster lists based on the reading order of the document to be processed includes:

[0071] Based on the first content block list, determining a horizontal boundary list, wherein the horizontal boundary values ​​in the horizontal boundary list are horizontal segmentation points of the document to be processed;

[0072] Based on the second content block list, determining a vertical boundary list, wherein the vertical boundary values ​​included in the vertical boundary list are vertical segmentation points of the document to be processed;

[0073] For each target content block in the plurality of column cluster lists, determining a row index and a column index of the target content block based on comparison results of the corner point position information of the target frame of the target content block with the horizontal boundary value and the vertical boundary value respectively;

[0074] For each target content block in the plurality of column cluster lists, adding the target content block to a third content block list in the three-dimensional list corresponding to the row index and the column index of the target content block;

[0075] Determine, based on the reading order of the to-be-processed document, a first arrangement order of the row indexes and the column indexes in the three-dimensional list, and a second arrangement order of the target content blocks in the third content block list;

[0076] The target content blocks in the three-dimensional list are sorted based on the first arrangement order and the second arrangement order.

[0077] In these embodiments, on the one hand, a horizontal boundary list can be determined based on the first content block list, wherein the horizontal boundary values ​​in the horizontal boundary list are the horizontal dividing points of the document to be processed, and are used to determine the column index of the content block. On the other hand, a vertical boundary list can be determined based on the second content block list, wherein the vertical boundary values ​​included in the vertical boundary list are the vertical dividing points of the document to be processed, and are used to determine the row index of the content block.

[0078] In addition, a three-dimensional list M is created, and the three dimensions of the three-dimensional list are: row index, column index and content block list (ie, the third content block list). One row index and one column index in the three-dimensional list correspond to one content block list.

[0079] After obtaining the horizontal boundary list and the vertical boundary list, the row index of the target content block can be determined by comparing the vertical coordinate of a corner point of the target frame of the target content block with the vertical boundary value in the vertical boundary list; the column index of the target content block can be determined by comparing the horizontal coordinate of a corner point of the target frame of the target content block with the horizontal boundary value in the horizontal boundary list; and then, the target content block is added to the content block list corresponding to its row index and column index in the three-dimensional list.

[0080] In a specific implementation, when the horizontal boundary values ​​in the horizontal boundary list are arranged in ascending order, if the horizontal coordinate of the above-mentioned corner point of the target frame of the target content block is greater than or equal to the w-1th horizontal boundary value in the horizontal boundary list, and is less than the wth horizontal boundary value in the horizontal boundary list, then the column index of the target content block can be determined to be w; if the horizontal coordinate of the above-mentioned corner point of the target frame of the target content block is greater than or equal to the wth horizontal boundary value in the horizontal boundary list, and is less than the w+1th horizontal boundary value in the horizontal boundary list, then the column index of the target content block can be determined to be w+1, where w is a positive integer. If w is 1, the w-1th horizontal boundary value can be replaced by 0.

[0081] In the case where the vertical boundary values ​​in the vertical boundary list are arranged in ascending order, if the vertical coordinate of the above-mentioned corner point of the target frame of the target content block is greater than or equal to the u-1th vertical boundary value in the vertical boundary list, and is less than the u-th vertical boundary value in the vertical boundary list, then the row index of the target content block can be determined to be u; if the vertical coordinate of the above-mentioned corner point of the target frame of the target content block is greater than or equal to the u-th vertical boundary value in the vertical boundary list, and is less than the u+1th vertical boundary value in the vertical boundary list, then the row index of the target content block can be determined to be u+1, where u is a positive integer, and if u is 1, the u-1th vertical boundary value can be replaced by 0.

[0082] After the target content blocks are added to the three-dimensional list, a first arrangement order of row indexes and column indexes in the three-dimensional list and a second arrangement order of the target content blocks in the third content block list in the three-dimensional list may be determined based on the reading order of the document to be processed. Then, the target content blocks in the three-dimensional list are sorted based on the first arrangement order and the second arrangement order.

[0083] The following example uses the reading order of the document to be processed from top to bottom and from left to right:

[0084] In the first arrangement order, the row index has the first priority and the column index has the second priority; in the second arrangement order, the vertical coordinate of the corner point of the target frame of the target content block has the first priority and the horizontal coordinate of the corner point of the target frame of the target content block has the second priority.

[0085] In a specific implementation, the third content block lists in the three-dimensional list are first arranged according to the first arrangement order. Specifically, the third content block lists corresponding to different row indexes are arranged in order of row index from small to large; the third content block lists corresponding to the same row index are arranged in order of column index from small to large.

[0086] Then, for each third content block list, the target content blocks therein are arranged according to the second arrangement order. Specifically, for target content blocks with different vertical coordinates of the corner points of the target boxes, they are arranged in order from small to large according to the vertical coordinates of the corner points of the target boxes; for target content blocks with the same vertical coordinates of the corner points of the target boxes, they are arranged in order from small to large according to the horizontal coordinates of the corner points of the target boxes.

[0087] In this way, the order of the content blocks can be matched with the reading order of the document to be processed.

[0088] For ease of understanding, continue to combine Figure 2a For example, Figure 2a According to the comparison result of the horizontal coordinate of the upper left corner point of the target frame of each content block with the horizontal boundary value, and the comparison result of the vertical coordinate with the vertical boundary value, it can be determined that: the row index and column index of text_1 are both 1; the row index of text_2 and text_3 is 1, and the column index is 2; the row index of title, text_4 and text_5 is 2, and the column index is 1; the row index of text_6 is 2, and the column index is 2. After that, the target content block is added to the content block list corresponding to its row index and column index in the three-dimensional list. Then, according to the first arrangement order of the row index and column index in the three-dimensional list, and the second arrangement order of the target content block in the third content block list in the three-dimensional list, the target content block is sorted, and the sorting results obtained are: text_1, text_2, text_3, title, text_4, text_5 and text_6. It can be seen that by using this embodiment to sort the content blocks, the sorting method of the content blocks can match the reading order of the content blocks, thereby improving the accuracy of the sorting of the document content blocks.

[0089] It can be understood that for documents to be processed in other reading orders, the target content blocks in the three-dimensional list can be sorted in the same manner, and to avoid repetition, it will not be described here.

[0090] In these embodiments, the horizontal boundary list and the vertical boundary list are first determined; then, based on the corner position information of the target frame of the target content block, the row index and the column index of the target content block are determined; and based on the row index and the column index of the target content block, the target content block is added to the corresponding content block list in the three-dimensional list. Then, based on the reading order of the document to be processed, the first arrangement order of the row index and the column index in the three-dimensional list and the second arrangement order of the target content block in the content block list are determined respectively, and then the target content blocks in the three-dimensional list are sorted based on the first arrangement order and the second arrangement order. In this way, when determining the content block sorting, the comparison of the position information between the target content blocks can be reduced, thereby improving the accuracy of the content block sorting.

[0091] The following is a detailed description of how to obtain the horizontal boundary list and the vertical boundary list.

[0092] Regarding the acquisition of the horizontal boundary list, in some embodiments, the first content block list includes at least one column cluster list;

[0093] The determining of a horizontal boundary list based on the first content block list includes:

[0094] Determine a horizontal critical value element corresponding to each column cluster list in the at least one column cluster list, wherein the horizontal critical value element corresponding to each column cluster list includes a left horizontal critical value and a right horizontal critical value of the column cluster list, the left horizontal critical value of the column cluster list is an average value of the horizontal coordinates of the left corner points of the target box of each target content block in the column cluster list, and the right horizontal critical value of the column cluster list is an average value of the horizontal coordinates of the right corner points of the target box of each target content block in the column cluster list;

[0095] According to the order of the horizontal critical values ​​in the horizontal critical value elements corresponding to each column cluster list from small to large, the horizontal critical value elements corresponding to each column cluster list are sorted to obtain a column cluster critical value list;

[0096] Based on the column cluster critical value list, a horizontal boundary list is determined, wherein the value of the i-th element in the horizontal boundary list is the average of the right horizontal critical value of the i-th element in the column cluster critical value list and the left horizontal critical value of the i+1-th element, and i is a positive integer.

[0097] In these embodiments, the first content block list includes at least one column cluster list. It should be noted that the embodiments of the present application do not limit the method for determining the column cluster list in the first content block list, and any method for dividing the first content block list into at least one column cluster list can fall within the protection scope of the embodiments of the present application.

[0098] In some embodiments, each column cluster list in the first content block list can be obtained by clustering the target content blocks in the first content block list into column clusters based on the corner point position information of the target box of the target content block in the first content block list. In these embodiments, the column cluster list included in the first content block list is the aforementioned n column cluster lists.

[0099] In other embodiments, the document to be processed can be divided into r areas in the vertical coordinate direction, and r column cluster lists corresponding to the r areas are set, where r is an integer greater than 1. In specific implementation, after determining the first content block list, for each target content block in the first content block list, the target area with the highest overlap rate among the r areas can be determined, and then the target content block can be divided into the column cluster list corresponding to the target area. It can be understood that in actual applications, there may be a column cluster list corresponding to a region where the number of target content blocks is 0, and these column cluster lists can be regarded as invalid column cluster lists. Correspondingly, the column cluster list corresponding to the region where the target content block exists can be regarded as a valid column cluster list. In these embodiments, the valid column cluster list in the r column cluster lists can be determined as the column cluster list included in the first content block. For ease of understanding, continue with Figure 2a Let's take an example:

[0100] Assuming that the document to be processed is equally divided into two regions in the vertical axis direction, based on the above method, the first content block list includes two column cluster lists: [text_1, text_4, text_5] and [text_2, text_3, text_6].

[0101] In these embodiments, the horizontal boundary list can be determined based on each column cluster list in the first content block list. In specific implementation, for each column cluster list in the first content block list, the horizontal coordinates of the left corner points of the target boxes of each target content block in the column cluster list can be obtained, and then the average value is calculated to obtain the left horizontal critical value l of the column cluster list; the horizontal coordinates of the right corner points of the target boxes of each target content block in the column cluster list can be obtained, and then the average value is calculated to obtain the right horizontal critical value r of the column cluster list; then, the left horizontal critical value l and the right horizontal critical value r of the column cluster list can be combined to obtain the horizontal critical value element (l, r) corresponding to the column cluster list.

[0102] That is, l can be calculated by formula (1), and r can be calculated by formula (2):

[0103]

[0104] Where c represents the number of target content blocks in the column cluster list, b i [1] represents the horizontal coordinate of the left corner point of the target box of the i-th target content block in the column cluster list, bi [2] represents the horizontal coordinate of the right corner point of the target box of the i-th target content block in the column cluster list.

[0105] Since the left horizontal critical value of each column cluster list is the average value of the horizontal coordinates of the left corner points of the target boxes of each target content block in the column cluster list, and the right horizontal critical value is the average value of the horizontal coordinates of the right corner points of the target boxes of each target content block in the column cluster list, the overlap of adjacent horizontal critical values ​​of different column cluster lists can be reduced, thereby improving the accuracy of the division of the target content blocks and further improving the accuracy of document knowledge acquisition.

[0106] After obtaining the horizontal critical value elements corresponding to each column cluster list, the horizontal critical value elements corresponding to the column cluster list can be sorted in the order of the horizontal critical value in the horizontal critical value elements from small to large, and the column cluster critical value list N1 is obtained. Taking n column cluster lists as an example, N1 can be expressed as formula (3):

[0107] N1=[(l1,r1),(l2,r2),...,(l n-1 ,r n-1 ),(l n ,r n )] (3)

[0108] Afterwards, for two adjacent column cluster lists, the average of the right horizontal critical value r of the previous column cluster list and the left horizontal critical value of the next column cluster list can be calculated to obtain the horizontal boundary values ​​of the two column cluster lists, and then the horizontal boundary values ​​of the adjacent column cluster lists are combined to obtain the horizontal boundary list N2. It can be understood that the element calculation logic of N2 is shown in formula (4):

[0109]

[0110] By determining the horizontal boundary list in the above manner, the overlap of adjacent horizontal critical values ​​of different column cluster lists can be reduced, thereby improving the accuracy of dividing the target content block and further improving the accuracy of document knowledge acquisition.

[0111] Of course, it can be understood that in other embodiments, the horizontal boundary values ​​of each adjacent column cluster list in the horizontal boundary list can be obtained by calculating the average value of the maximum right horizontal coordinate value of the previous column cluster list and the maximum left horizontal coordinate value of the next column cluster list.

[0112] For obtaining the vertical boundary list, the vertical boundary list N3 may be determined based on the vertical coordinate of a corner point of the target frame of each target content block in the second content block list.

[0113] In some implementations, the ordinate of a corner point of the target frame of each target content block in the second content block list may be directly used as the vertical critical value, and then these vertical critical values ​​may be combined in ascending order to obtain a vertical boundary list.

[0114] In other implementations, the vertical coordinates of a corner point of the target frame of each content block in the second content block list may be sorted in ascending order to obtain a content block critical value list; then, for two adjacent target content blocks, the vertical coordinates of the corner point of the target frame of the previous target content block and the average value of the vertical coordinate of the corner point of the target frame of the next target content block may be calculated to obtain the vertical boundary values ​​of the two target content blocks, and then the vertical boundary values ​​of the adjacent target content blocks may be combined to obtain a vertical boundary list.

[0115] The acquisition of knowledge is described in detail below.

[0116] In some embodiments, the relevant information also includes content attribute classification information;

[0117] The step of generating the knowledge of the document to be processed based on the sorted plurality of target content blocks includes:

[0118] Extracting text information from the target content block;

[0119] Based on the first knowledge combination rule, the knowledge length upper limit value and the content attribute classification information of each target content block, the sorted multiple target content blocks are split to obtain multiple sub-lists, wherein the first knowledge combination rule indicates the content attribute classification information supporting the combination;

[0120] For each sub-list, combining text information of each target content block in the sub-list to obtain knowledge corresponding to the sub-list;

[0121] According to the arrangement order of the multiple sub-lists, the knowledge corresponding to the multiple sub-lists are combined in sequence to obtain the knowledge of the document to be processed.

[0122] In this embodiment, the relevant information of the content block also includes content attribute classification information of the content block. In specific implementation, the content attribute classification information of the content block can be, but is not limited to, any of the following: page title, text, page header, page footer, picture, table, section title, chart title, item list, etc. The document processing device stores a first knowledge combination rule and a knowledge length upper limit value. The first knowledge combination rule can be, but is not limited to, indicating content attribute classification information that supports combination. The first knowledge combination rule can be pre-set based on the actual needs of content combination, and the knowledge length upper limit value can be pre-set based on the actual needs of the large model.

[0123] In specific implementation, the sorted multiple content blocks can be split into multiple sub-lists in combination with the content attribute classification information of the supported combination indicated by the first knowledge combination rule, the upper limit of the knowledge length, and the content attribute classification information of each target content block. It can be understood that each sub-list may include one or more target content blocks; the total length of the target content blocks included in each sub-list is less than or equal to the upper limit of the knowledge length; the content attribute classification information of the target content blocks included in each sub-list is the content attribute classification information of the supported combination in the first knowledge combination rule.

[0124] In addition, for each target content block, the position of the target content block in the document to be processed can be located based on the corner position information of the target frame of the target content block, and then the text information in the target content block can be extracted by using, but not limited to, optical character recognition (OCR) technology.

[0125] Thus, for each sublist, the knowledge corresponding to the sublist can be obtained by combining the text information of each target content block in the sublist. Then, according to the arrangement order of the multiple sublists, the knowledge corresponding to the multiple sublists is combined in sequence to obtain the knowledge of the document to be processed.

[0126] The knowledge acquired in the above manner can obtain the text information of any target content block of content attribute classification information and combine it into knowledge, thereby enriching the source of knowledge. In addition, by combining the target content blocks based on the first knowledge combination rule and the content attribute classification information, the text content of the combined knowledge can have a high correlation, thereby improving the reliability of document knowledge acquisition. In addition, the acquired knowledge can be made less than or equal to the upper limit of the knowledge length, thereby improving the operational reliability of the large model.

[0127] The acquisition of the target content block is described in detail below.

[0128] In some embodiments, the obtaining of multiple content blocks of the document to be processed and related information of each of the multiple content blocks includes:

[0129] Get the target image corresponding to the document to be processed;

[0130] Inputting the target image into a target detection model to obtain a plurality of initial content blocks of the document to be processed and related information of the plurality of initial content blocks, wherein the related information further includes content attribute classification information;

[0131] Based on the content attribute classification information of the multiple initial content blocks and the knowledge combination rule, the multiple initial content blocks are filtered to obtain multiple intermediate content blocks of the document to be processed; wherein the knowledge combination rule indicates the content attribute classification information that is not used as knowledge composition;

[0132] Based on the corner point position information of the target frames of the multiple intermediate content blocks, the multiple intermediate content blocks are merged to obtain multiple target content blocks of the document to be processed.

[0133] In this embodiment, the initial content block of the module to be processed can be obtained through the trained target detection model, and then the initial content block is filtered and merged to obtain the target content block of the module to be processed.

[0134] In specific implementation, the target detection model can be trained first for document detection, so the target detection model can also be called a document page target detection model. It is worth noting that the embodiment of the present application does not limit the model structure of the target detection model, which can be flexibly selected based on actual needs. For example, in one example, the model structure of the target detection model can be DETR or DeformableDETR.

[0135] The input of the target detection model is the image corresponding to the document page in the document to be processed. Before training the target detection model, different content blocks in the document page image can be marked as different categories based on the typesetting style. The classification setting is mainly based on the main page typesetting content classification of the document to be parsed in the domain, such as setting the classification category to "page title, text, page header, page footer, picture, table, section title, chart title, sub-item list", and using the target detection method to mark the position of the target box of each content block. The position marking information of each content block can be the horizontal and vertical coordinate information of the four corner points of the content block target box relative to the upper left corner point of the page. The target detection model is obtained through training with a certain amount of image annotation data; the position prediction and content attribute classification prediction of the document page are performed based on the target detection model.

[0136] For the document to be processed, the document to be processed can be first converted into pictures by page, and then the pictures are sequentially input into the target detection model to obtain the content blocks of each page of the document to be processed and related information thereof. In this embodiment, the content blocks output by the target detection model are referred to as initial content blocks. In some embodiments, the initial content blocks of the document to be processed can be put into the fourth content block list C1, and the fourth content block list C1 can also be referred to as the initial content block list C1.

[0137] In this embodiment, the document processing device stores a second knowledge combination rule, which may include but is not limited to indicating content attribute classification information that is not a knowledge component. Further, the second knowledge combination rule and the aforementioned first knowledge combination rule may be the same knowledge combination rule or different knowledge combination rules.

[0138] After obtaining the initial content blocks, for each initial content block, it can be detected whether its content attribute classification information is included in the second knowledge combination rule. If so, the initial content block can be discarded, and if not, the initial content block can be retained until all initial content blocks are traversed and the retained initial content blocks are determined as intermediate content blocks. In some embodiments, the intermediate content blocks of the document to be processed can be placed in the fifth content block list C2.

[0139] After obtaining the middle content block, the overlapping content blocks in the middle content block can be further determined based on the corner point position information of the target frame of the middle content block, and then merged to obtain the target content block of the document to be processed. In some embodiments, the target content block of the document to be processed can be put into the sixth content block list C3.

[0140] Through the above method, the target detection model is used to obtain the initial content block of the document to be processed. Compared with splitting the text content by delimiters and setting knowledge length, the content integrity and content relevance of the obtained initial content block can be improved; in addition, the target content block of the document to be processed can be obtained by filtering and merging the initial content blocks in turn, so that the reliability of document knowledge acquisition can be improved.

[0141] The present application does not limit the method of determining overlapping content blocks and the specific form of merging processing.

[0142] In some embodiments, the merging of the multiple intermediate content blocks based on the corner point position information of the target boxes of the multiple intermediate content blocks to obtain the multiple target content blocks of the document to be processed may include:

[0143] For any two intermediate content blocks among the plurality of intermediate content blocks, determining an intersection area of ​​target frames of the two intermediate content blocks;

[0144] Determine the smaller value of the area of ​​the target frames of the two middle content blocks as the target area;

[0145] Determine the ratio of the intersection area to the target area as the overlap coefficient of the two middle content blocks;

[0146] When the overlap coefficient is greater than the second threshold, the larger one of the target frames in the two middle content blocks is retained.

[0147] In this embodiment, for any two middle content blocks, the overlap coefficient between the two can be calculated by formula (5): r :

[0148]

[0149] Wherein, b1 represents the target frame of one of the middle content blocks, b2 represents the target frame of the other middle content block; intersection is a function for calculating the area of ​​the intersection of two target frames; and area is a function for calculating the area of ​​the target frame.

[0150] After calculating the overlap coefficient O of the two middle content blocks r Afterwards, it can be compared with the second threshold. r If the target area is greater than the second threshold, it can be determined that the two are overlapping content blocks, and the middle content block with the larger target box area is retained, and the middle content block with the smaller target box area is discarded. The second threshold can be set based on actual needs, such as 0.85 in one example.

[0151] It can be seen that in this embodiment, the determination of the overlapping content blocks is based on the comparison result of the overlapping coefficient of the content blocks and the second threshold. The merging process of the overlapping content blocks is specifically manifested in retaining the middle content blocks with larger areas in the overlapping content blocks. In this way, the determination accuracy of the overlapping content blocks can be improved, and then the accuracy of obtaining the target content blocks can be improved.

[0152] Of course, in other embodiments, as long as there is overlap in content blocks, they can be determined as overlapping content blocks; the merging process of overlapping content blocks can be expressed as taking the union of two overlapping content blocks, which can be determined according to actual conditions and is not limited in the embodiments of the present application.

[0153] It should be noted that the various optional implementation modes introduced in the embodiments of the present application can be implemented in combination with each other or separately if they do not conflict with each other, and the embodiments of the present application are not limited to this.

[0154] The embodiments of the present application may include the following:

[0155] 1. Based on typesetting vision, the target detection method is used to split and analyze the content blocks to obtain the location information and content attribute classification of different content blocks in the document page.

[0156] 2. Use clustering method to aggregate the detected content block position points, obtain the page column splitting cluster segmentation point information (i.e. the aforementioned horizontal boundary value), arrange each content block in a different column cluster list, and sort the content blocks from top to bottom within the column cluster list and from left to right between column cluster lists.

[0157] 3. Parse the text in each content block based on OCR to obtain the text content in each content block.

[0158] 4. Based on the content attribute classification of content blocks, the content blocks are combined into knowledge according to the knowledge combination rules.

[0159] For the first point above, different content blocks in the document page image can be marked as different categories based on the typesetting style. The classification setting is mainly based on the main page typesetting content classification of the document to be parsed in the domain, such as setting the classification category to "page title, text, page header, footer, picture, table, section title, chart title, sub-item list", and using target detection to mark the position of the target box of each content block. The position marking information of each content block is the horizontal and vertical coordinate information of the four corner points of the content block target box relative to the upper left corner point of the page. The target detection model is obtained through training with a certain amount of image annotation data; based on the target detection model, the position prediction and content attribute classification prediction of the content block of the document page are performed.

[0160] Regarding the second point above, for the location point information of each content block obtained by the target detection model, the cluster label prediction of the horizontal coordinate of the upper left corner point of the target box is performed, and a corresponding number of column cluster lists are set based on the number of clusters. The horizontal boundary points of the column cluster (i.e., the aforementioned horizontal boundary values) are obtained based on the horizontal coordinates of the content block target boxes within the column cluster. The target boxes of each content block are arranged in a column cluster list based on the column cluster boundary points, and the content blocks are sorted from top to bottom within the column cluster list and from left to right between column cluster lists.

[0161] For the above point 3, text detection and parsing extraction are performed on each content block through OCR.

[0162] For the above-mentioned point 4, based on the content blocks sorted in point 2, the adjacent content blocks are combined into knowledge based on the knowledge combination rules, such as combining "section title + text" into one piece of knowledge.

[0163] For ease of understanding, a specific scenario example is described as follows:

[0164] like Figure 2b As shown, the document processing method may include the following steps:

[0165] Step 201: training a document page object detection model.

[0166] In specific implementation, the process of training the document page object detection model may include:

[0167] 1) Define the classification of document content blocks, that is, define the classification of content attributes of content blocks. You can classify them according to the main page layout content of the document to be parsed in the domain. For example, you can set the classification categories to "page title, text, header, footer, picture, table, section title, chart title, and item list";

[0168] 2) Prepare a data set to train the target detection model, including paging and converting a certain amount of typical documents for building the knowledge base into images, and annotating them with annotation tools to prepare a data set for training the target detection model.

[0169] Step 202, document processing, converting the document into a picture list page by page.

[0170] In specific implementation, the document can be converted into pictures by page and saved.

[0171] Step 203, perform target detection prediction on each document page image, obtain a list of initial content blocks in the page predicted by the model, each content block contains target frame position and content attribute classification information, and post-process the content block list to obtain a post-processed content block list.

[0172] In the specific implementation, the trained document page target detection model is used to predict the target box of the document page image to be parsed, and the position information and content block attribute classification information of each content block in the page are obtained.

[0173] Post-processing of the content block target box can include two steps:

[0174] Process 1) filtering the initial content block list C1 based on the content block category information. This process mainly filters out the content blocks that are not knowledge components in the original document page, such as "header, footer", and obtains the filtered content block list C2, that is, the fifth content block list mentioned above;

[0175] Process 2) mainly combines overlapping content blocks. This process detects the overlap coefficient O between the target frame positions of the content blocks. r When the overlap coefficient between the two content block target boxes is O r When the value is greater than the set threshold Thr1 (i.e., the aforementioned second threshold), the two content blocks are considered to be overlapping content blocks, and only the content block with the larger area is saved to obtain the filtered content block list C3, i.e., the aforementioned sixth content block list. r It can be calculated by the above formula (5).

[0176] Step 204 , clustering the content blocks based on the position of the target frame in the page, and sorting each content block in a document logical order.

[0177] The specific implementation may include the following steps:

[0178] 1) Based on the target box width coefficient W r The content blocks in the content block list C3 are divided into two sub-lists, where the sub-list C4 (i.e., the first content block list mentioned above) is the width coefficient W of the content block target frame. r The sublist C5 (i.e., the second content block list) is the width coefficient W of the content block target frame. r The sublist is greater than the set threshold Thr2. When the width coefficient W of the content block target box r When it is greater than the set threshold Thr2, the current content block is considered to be a single-line content block and is not listed as a column layout content block. In this embodiment of the solution, Thr2 is set to 0.5.

[0179] 2) Extract the x1 coordinate of the upper left corner point of each content block target frame in the content block list C4, cluster the x1 coordinate array, and use DBSCAN for clustering in this solution example to obtain the column cluster classification information and the number of column clusters n for each content block target frame in C4. Split the content blocks in C4 into n content block column cluster lists according to the column cluster classification information, and calculate the (l, r) position information of each column cluster list, where l is the average value of the x1 coordinate point of the target frame of the content block in the column cluster list, and r is the average value of the x2 coordinate point of the target frame of the content block in the column cluster list. Extract the (l, r) values ​​of the n column cluster lists and sort them by the l value to obtain the column cluster critical value list N1. From N1, obtain the horizontal boundary list N2, where N2 is the average value of the adjacent critical values ​​in N1, and the number of elements in list N2 is n-1.

[0180] 3) Extract the y1 coordinate of the upper left corner point of each content block target frame in the content block list C5, sort the y1 coordinate data, and obtain the vertical boundary list N3, the number of elements in the list N3 is m.

[0181] 4) Construct a three-dimensional list M with m+1 rows and n columns. Based on the coordinates x1 and y1 of each content block target frame in the content block list C3, use binary search to find the index of x1 in the horizontal boundary list N2 and the index of y1 in the vertical boundary list N3. Add the content block data to the three-dimensional empty list M based on the index value.

[0182] 5) Arrange the content blocks in the three-dimensional list M in order of row and column indexes, and sort the innermost content block list with the target frame y1 coordinate first and the x1 coordinate second.

[0183] Step 205: The sorted content block list is combined into knowledge based on content attribute classification and combination rules.

[0184] The specific implementation may include the following steps:

[0185] 1) Document image cropping and content block OCR text extraction based on the position of each content block target box;

[0186] 2) Set the rules for combining and merging text block contents into knowledge (i.e., the first knowledge combination rule mentioned above), such as splitting the sorted content block list into sublists according to the index position of the "section title" and the upper limit of the knowledge length, and combining each sublist into a single piece of knowledge. The knowledge text is the text content extracted by OCR in the content block sublist, and the knowledge source data is the content block sublist data.

[0187] Based on the document processing method provided in the above embodiment, the present application also provides a specific implementation of a document processing device. Please refer to the following embodiment.

[0188] See also Figure 3 , the document processing device provided in the embodiment of the present application may include:

[0189] A first acquisition module 301 is used to acquire multiple target content blocks of a document to be processed and related information of the multiple target content blocks, wherein the related information includes corner point position information of a target box;

[0190] A clustering module 302 is configured to cluster the target content blocks into column clusters based on the corner point position information of the target boxes of the target content blocks, and divide the target content blocks into a plurality of column cluster lists;

[0191] A sorting module 303, configured to sort the target content blocks in the plurality of column cluster lists based on the reading order of the document to be processed;

[0192] The generating module 304 is configured to generate the knowledge of the document to be processed based on the sorted plurality of target content blocks.

[0193] The document processing device provided in the embodiment of the present application can implement each process implemented by the document processing device in the method embodiment, and will not be described again here to avoid repetition.

[0194] Figure 4 A schematic diagram of the hardware structure of document processing provided in an embodiment of the present application is shown.

[0195] The document processing device may include a processor 401 and a memory 402 storing computer program instructions.

[0196] Specifically, the processor 401 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0197] Memory 402 may include a large capacity memory for data or instructions. By way of example and not limitation, memory 402 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, memory 402 may include a removable or non-removable (or fixed) medium. In appropriate cases, memory 402 may be inside or outside of an integrated gateway disaster recovery device. In a specific embodiment, memory 402 is a non-volatile solid-state memory.

[0198] The memory may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Therefore, generally, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of the present disclosure.

[0199] The processor 401 implements any one of the document processing methods in the above embodiments by reading and executing computer program instructions stored in the memory 402 .

[0200] In one example, the document processing device may further include a communication interface 404 and a bus 410. Figure 4 As shown, the processor 401, the memory 402, and the communication interface 404 are connected via a bus 410 and communicate with each other.

[0201] The communication interface 404 is mainly used to implement communication between each module, device, unit and / or equipment in the embodiment of the present application.

[0202] Bus 410 includes hardware, software or both, and the parts of document processing equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 410 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.

[0203] In addition, in combination with the document processing method in the above embodiment, the embodiment of the present application can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the document processing methods in the above embodiment is implemented.

[0204] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiment, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make each kind of change, modification and addition, or change the order between the steps after understanding the spirit of the present application.

[0205] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0206] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0207] Each aspect of the present disclosure is described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0208] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of each equivalent modification or replacement within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application.

Claims

1. A document processing method, characterized in that: include: Acquire multiple target content blocks of the document to be processed, and related information of the multiple target content blocks, wherein the related information includes corner point position information of the target box; Based on the corner point position information of the target boxes of the multiple target content blocks, clustering the multiple target content blocks into column clusters, and dividing the multiple target content blocks into multiple column cluster lists; sorting the target content blocks in the plurality of column cluster lists based on the reading order of the to-be-processed document; Based on the sorted plurality of target content blocks, knowledge of the document to be processed is generated.

2. The method according to claim 1, characterized in that The relevant information also includes a width coefficient of the target frame; The clustering of the multiple target content blocks into column clusters based on the corner point position information of the target frames of the multiple target content blocks, and dividing the multiple target content blocks into multiple column cluster lists, includes: The target content blocks whose target frame width coefficients are less than or equal to a first threshold value among the multiple target content blocks are classified into a first content block list, and the target content blocks whose target frame width coefficients are greater than the first threshold value are classified into a second content block list; Based on the corner point position information of the target frame of the target content block in the first content block list, clustering the target content blocks in the first content block list into column clusters, and dividing the target content blocks in the first content block list into n column cluster lists, where n is a positive integer; The multiple column cluster lists include the n column cluster lists and the second content block list.

3. The method according to claim 1, characterized in that The multiple column cluster lists include a first content block list and a second content block list, wherein a width coefficient of a target frame of a target content block in the first content block list is less than a first threshold, and a width coefficient of a target frame of a target content block in the second content block list is greater than the first threshold; The step of sorting the target content blocks in the plurality of column cluster lists based on the reading order of the to-be-processed document comprises: Based on the first content block list, determining a horizontal boundary list, wherein the horizontal boundary values ​​in the horizontal boundary list are horizontal segmentation points of the document to be processed; Based on the second content block list, determining a vertical boundary list, wherein the vertical boundary values ​​included in the vertical boundary list are vertical segmentation points of the document to be processed; For each target content block in the plurality of column cluster lists, determining a row index and a column index of the target content block based on comparison results of the corner point position information of the target frame of the target content block with the horizontal boundary value and the vertical boundary value respectively; For each target content block in the plurality of column cluster lists, adding the target content block to a third content block list in the three-dimensional list corresponding to the row index and the column index of the target content block; Determine, based on the reading order of the to-be-processed document, a first arrangement order of the row indexes and the column indexes in the three-dimensional list, and a second arrangement order of the target content blocks in the third content block list; The target content blocks in the three-dimensional list are sorted based on the first arrangement order and the second arrangement order.

4. The method according to claim 3, characterized in that The first content block list includes at least one column cluster list; The determining of a horizontal boundary list based on the first content block list includes: Determine a horizontal critical value element corresponding to each column cluster list in the at least one column cluster list, wherein the horizontal critical value element corresponding to each column cluster list includes a left horizontal critical value and a right horizontal critical value of the column cluster list, the left horizontal critical value of the column cluster list is an average value of the horizontal coordinates of the left corner points of the target box of each target content block in the column cluster list, and the right horizontal critical value of the column cluster list is an average value of the horizontal coordinates of the right corner points of the target box of each target content block in the column cluster list; According to the order of the horizontal critical values ​​in the horizontal critical value elements corresponding to each column cluster list from small to large, the horizontal critical value elements corresponding to each column cluster list are sorted to obtain a column cluster critical value list; Based on the column cluster critical value list, a horizontal boundary list is determined, wherein the value of the i-th element in the horizontal boundary list is the average of the right horizontal critical value of the i-th element in the column cluster critical value list and the left horizontal critical value of the i+1-th element, and i is a positive integer.

5. The method according to claim 1, characterized in that The relevant information also includes content attribute classification information; The step of generating the knowledge of the document to be processed based on the sorted plurality of target content blocks includes: Extracting text information from the target content block; Based on the first knowledge combination rule, the knowledge length upper limit value and the content attribute classification information of each target content block, the sorted multiple target content blocks are split to obtain multiple sub-lists, wherein the first knowledge combination rule indicates the content attribute classification information supporting the combination; For each sub-list, combining text information of each target content block in the sub-list to obtain knowledge corresponding to the sub-list; According to the arrangement order of the multiple sub-lists, the knowledge corresponding to the multiple sub-lists are combined in sequence to obtain the knowledge of the document to be processed.

6. The method according to claim 1, characterized in that The step of obtaining multiple content blocks of the document to be processed and related information of each content block in the multiple content blocks includes: Get the target image corresponding to the document to be processed; Inputting the target image into a target detection model to obtain a plurality of initial content blocks of the document to be processed and related information of the plurality of initial content blocks, wherein the related information further includes content attribute classification information; Based on the content attribute classification information of the multiple initial content blocks and the second knowledge combination rule, the multiple initial content blocks are filtered to obtain multiple intermediate content blocks of the document to be processed; wherein the second knowledge combination rule indicates the content attribute classification information that is not used as knowledge composition; Based on the corner point position information of the target frames of the multiple intermediate content blocks, the multiple intermediate content blocks are merged to obtain multiple target content blocks of the document to be processed.

7. The method according to claim 6, characterized in that The merging of the multiple intermediate content blocks based on the corner point position information of the target boxes of the multiple intermediate content blocks to obtain the multiple target content blocks of the document to be processed includes: For any two intermediate content blocks among the plurality of intermediate content blocks, determining an intersection area of ​​target frames of the two intermediate content blocks; Determine the smaller value of the area of ​​the target frames of the two middle content blocks as the target area; Determine the ratio of the intersection area to the target area as the overlap coefficient of the two middle content blocks; When the overlap coefficient is greater than the second threshold, the larger one of the target frames in the two middle content blocks is retained.

8. A document processing device, characterized in that: include: A first acquisition module, used to acquire multiple target content blocks of a document to be processed and related information of the multiple target content blocks, wherein the related information includes corner point position information of a target box; A clustering module, configured to cluster the multiple target content blocks into column clusters based on the corner point position information of the target boxes of the multiple target content blocks, and divide the multiple target content blocks into multiple column cluster lists; A sorting module, configured to sort the target content blocks in the plurality of column cluster lists based on the reading order of the to-be-processed document; A generating module is used to generate the knowledge of the document to be processed based on the sorted multiple target content blocks.

9. A document processing device, characterized in that: The device comprises: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the document processing method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the document processing method according to any one of claims 1 to 7 is implemented.

Images