Non-vector document preprocessing method, device and equipment and readable storage medium

Abstract

The invention discloses a non-vector document preprocessing method, device and equipment and a readable storage medium, and the method comprises the steps: obtaining a to-be-recognized non-vector document which comprises information used for large language model recognition; performing format conversion on the to-be-recognized non-vector document to generate a to-be-recognized picture; the to-be-recognized picture is input to a trained layout analysis model, an identifier and a position coordinate corresponding to a local area of the to-be-recognized picture are obtained, and the layout analysis model is a Layout neural network detection model; extracting type data covered by the local area of the to-be-recognized picture through the identifier and the position coordinate corresponding to the local area; and integrating all types of data to generate target input data so as to realize identification of the large language model on the to-be-identified non-vector document. According to the method, by means of the layout analysis Layout technology, the structured information of the original non-vector document can be restored and stored, so that the recognition accuracy of the large language model is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a non-vector document preprocessing method, device, equipment and readable storage medium. Background Art

[0002] With the gradual promotion of ChatGPT technology, LLM (Large Language Model) has been widely used in various fields in my country. In order to quickly complete the pre-training or fine-tuning work on the base large model, it is necessary to collect domain data with complete knowledge in advance and perform necessary data preprocessing on the data. The training data of the large language model is generally text or text question and answer pairs, and the collected original domain data is often document-type, such as PDF, Docx, and there are many non-vector or pseudo-vector PDFs; this requires parsing the document before model training, and parsing the PDF or Docx type documents into text data before it can be used, otherwise it cannot be directly input into the large language model training.

[0003] At the same time, existing domain data analysis and preprocessing solutions are generally divided into the following two types:

[0004] Solution 1: Based on GPT-3.5 or GPT-4, the document is understood and output. For example, the large-scale models Mingyi and DoctorGLM in the medical field, as well as the large-scale model Cornucopia in the financial field, all use GPT-3.5 and GPT-4, and with the help of certain prompts, generate a number of questions and answers related to the knowledge in the field;

[0005] Option 2 is based on self-developed processing tools, such as the educational model Educhat developed by East China Normal University in the education field, which uses the self-developed data processing tool Clean-Tool;

[0006] However, the prior art has the following defects:

[0007] The defects of Solution 1 are: First, when the document contains a large amount of text content and the content is relatively fragmented, such as various types of specification data in the construction industry, including construction specifications, design specifications, economic specifications, etc., if no paragraph processing or text processing is done directly and it is directly thrown to GPT-3.5 or GPT-4, the generated effect will be very unsatisfactory and the processing results required by the user will not be obtained; second, when there are a large number of pictures and tables in the data, taking the various types of specification data in the construction industry as an example, it involves a large number of table lookups and picture examples. This type of data is directly given to GPT-3.5 or GPT-4, and the effect is also unsatisfactory, and often a very satisfactory result is not obtained.

[0008] The disadvantages of Solution 2 are mainly reflected in its own limitations. The processing tools for this type of approach are often relatively niche, with poor generalization performance and cannot be well used for data in other fields because they are completely dependent on the characteristics of the data in their own fields for customized development.

[0009] Regarding the technical problem that the improper processing of non-vector documents in the prior art leads to poor recognition effects of large language models, there is currently no effective solution. Summary of the Invention

[0010] The purpose of the present invention is to provide a preprocessing method, device, equipment and readable storage medium for non-vector documents, which can solve the technical problem that the improper processing of non-vector documents in the prior art leads to poor recognition effects of large language models.

[0011] One aspect of the present invention provides a preprocessing method for non-vector documents, the method includes: obtaining a non-vector document to be recognized, where the non-vector document to be recognized contains information for large language model recognition; performing format conversion on the non-vector document to be recognized to generate a picture to be recognized; inputting the picture to be recognized into a trained layout analysis model to obtain the identifiers and position coordinates corresponding to the local areas of the picture to be recognized, where the layout analysis model is a Layout neural network detection model; extracting the type data covered by the local areas of the picture to be recognized through the identifiers and position coordinates corresponding to the local areas; integrating all type data to generate target input data to realize the recognition of the non-vector document to be recognized by the large language model.

[0012] Optionally, extracting the type data covered by the local areas of the picture to be recognized through the identifiers and position coordinates corresponding to the local areas includes: determining a target extraction area from the picture to be recognized through the position coordinates of the local areas; obtaining the data identifiers and the extraction methods corresponding to the data identifiers stored in advance in the database; where the data identifiers include figure type, table type and text type; matching the identifier of the target extraction area with the data identifiers, and recognizing the target extraction area according to the extraction method corresponding to the successfully matched data identifier to obtain the type data covered by the target extraction area.

[0013] Optionally, if the identifier of the target extraction area matches the data identifier of the figure type, the target extraction area is identified according to the extraction method corresponding to the successfully matched data identifier, and the type data covered by the target extraction area is obtained, including: determining the picture border line in the target extraction area, intercepting the target extraction area through the picture border line to obtain the first picture data, where the first picture data is the picture content in the target extraction area; identifying the target extraction area through the OCR recognition algorithm to obtain the second picture data, where the second picture data is the picture name, page number, and note information in the target extraction area; calculating the distance between the first picture data and / or the second picture data in the target extraction area, and associating the first picture data and / or the second picture data corresponding to whether the distance is less than or equal to the first preset threshold; integrating the associated first picture data and second picture data to obtain the figure type data covered by the target extraction area.

[0014] Optionally, if the identifier of the target extraction area matches the data identifier of the table type, the target extraction area is identified according to the extraction method corresponding to the successfully matched data identifier, and the type data covered by the target extraction area is obtained, including: determining the table border line in the target extraction area, intercepting the target extraction area through the table border line to obtain the first table data, where the first table data is the table content in the form of a picture in the target extraction area; identifying the target extraction area through the OCR recognition algorithm to obtain the second table data, where the second table data is the table name, page number, and note information in the target extraction area; calculating the distance between the first picture data and / or the second picture data in the target extraction area, and associating the first table data and / or the second table data corresponding to whether the distance is less than or equal to the second preset threshold; identifying the first table data through the preset table recognition algorithm to obtain the third table data, and storing the third table data in the markdown format; integrating the associated first table data, second table data, and third table data to obtain the table type data covered by the target extraction area.

[0015] Optionally, if the identifier of the target extraction area matches the data identifier of the text type, the target extraction area is recognized according to the extraction method corresponding to the successfully matched data identifier to obtain the type data covered by the target extraction area, including: recognizing the target extraction area through the OCR recognition algorithm to obtain the first text data; receiving the division instruction of the first text data sent by the client, where the division instruction includes the division type, and the division type includes chapter division and paragraph division; if the division type is chapter division, obtain the title identifier and the page number corresponding to the title identifier from the identifier of the target extraction area, divide the first text data with the same page number by the first text data corresponding to the title identifier, and establish the belonging relationship between the first text data corresponding to the non-title identifier and the first text data corresponding to the title identifier according to the relative position between the first text data, so as to obtain the text type data covered by the target extraction area; if the division type is paragraph division, obtain the paragraph identifier and the page number corresponding to the paragraph identifier from the identifier of the target extraction area, divide the first text data by the first text data corresponding to the paragraph identifier, and associate the divided paragraphs by the distance between the first text data corresponding to different paragraph identifiers, so as to obtain the text type data covered by the target extraction area.

[0016] Optionally, the layout analysis model is trained in the following way: obtain a batch of non-vector documents, and convert the non-vector documents into pictures; add labels to the local areas of the pictures according to the layout of the pictures to generate a training set; where the labels include picture type, table type, and text type; input the training set into the initial neural network model for training, and stop training the model until the loss value of the model tends to converge, and obtain the layout analysis model.

[0017] Another aspect of the present invention provides a preprocessing device for non-vector documents, the device includes: an acquisition module, configured to acquire a non-vector document to be recognized, where the non-vector document to be recognized contains information for large language model recognition; a format conversion module, configured to perform format conversion on the non-vector document to be recognized to generate a picture to be recognized; an identification module, configured to input the picture to be recognized into the trained layout analysis model to obtain the identifier and position coordinates corresponding to the local area of the picture to be recognized, where the layout analysis model is a Layout neural network detection model; an extraction module, configured to extract the type data covered by the local area of the picture to be recognized through the identifier and position coordinates corresponding to the local area; an integration module, configured to integrate all type data to generate target input data to realize the recognition of the non-vector document to be recognized by the large language model.

[0018] Optionally, the device further includes a training module, specifically for: obtaining a batch of non-vector documents, converting the non-vector documents into pictures; adding labels to local areas of the pictures according to the layout of the pictures to generate a training set; wherein the labels include picture type, table type, and text type; inputting the training set into an initial neural network model for training, and stopping training the model until the loss value of the model tends to converge, to obtain a layout analysis model.

[0019] Another aspect of the present invention provides a computer device, which includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the computer program, it implements the preprocessing method of non-vector documents in any of the above embodiments.

[0020] Another aspect of the present invention provides a computer storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the preprocessing method of non-vector documents in any of the above embodiments. Further, the computer-readable storage medium mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function, etc.; the data storage area can store data created according to the use of blockchain nodes, etc.

[0021] The present invention converts non-vector documents into picture format, realizes the automatic parsing of non-vector PDF documents; realizes the automatic and accurate extraction of data such as text, pictures, and tables in non-vector PDF documents; at the same time, by means of the layout analysis Layout technology, it can restore and save the structured information of the original non-vector documents, ensure the integrity of non-vector document information, and improve the recognition accuracy of large language models. Description of the Drawings

[0022] By reading the detailed description of the preferred embodiments below, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present invention. Moreover, throughout the drawings, the same reference numerals are used to represent the same components. In the drawings:

[0023] Figure 1 Shows an optional flowchart of the preprocessing method of non-vector documents provided in Embodiment 1 of the present invention;

[0024] Figure 2 Shows a structural block diagram of the preprocessing device of non-vector documents provided in Embodiment 2 of the present invention; and

[0025] Figure 3 Shows a block diagram of a computer device suitable for implementing the preprocessing method of non-vector documents provided in Embodiment 3 of the present invention. Detailed implementation manners

[0026] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0027] It should be noted that in this text, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, the element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article or device including the element.

[0028] Embodiment 1

[0029] This embodiment provides a preprocessing method for non-vector documents. Figure 1 The flowchart of the preprocessing method for the non-vector document is shown. As Figure 1 shown, the preprocessing method for the non-vector document may include steps S101 to S104, where:

[0030] Step S101: Obtain the non-vector document to be recognized, where the non-vector document to be recognized contains information for recognition by a large language model;

[0031] The non-vector document to be recognized may be a non-vector PDF or Docx document in the construction field. In the non-vector document to be recognized, there may be data such as construction specifications, design specifications or economic specifications. If no parsing operation is performed on the non-vector and the non-vector document to be recognized is directly input into the large language model for recognition, the final result is often not satisfactory and the accuracy rate is low.

[0032] Step S102: Perform format conversion on the non-vector document to be recognized to generate a picture to be recognized;

[0033] The format conversion can be performed by a conventional data conversion tool, which is not limited herein. Converting the non-vector document into a picture form can make the display result of the document content more intuitive, facilitate the extraction of the non-vector document content by the subsequent layout analysis model, and improve the recognition accuracy of the large language model.

[0034] Step S103: Input the image to be recognized into the trained layout analysis model to obtain the identifiers and position coordinates corresponding to the local regions of the image to be recognized. Here, the layout analysis model is a Layout neural network detection model.

[0035] The identifiers corresponding to the local regions may include document name, document level, main title, secondary subtitle, left - hand secondary title, left - hand tertiary title, table of contents, table name, table, table note, figure, figure name, figure note, complete paragraph, cross - page paragraph, useless text, footer. Specifically, figure, figure name, and figure note are figure - type identifiers, table name, table, and table note are table - type identifiers, and the other identifiers are text - type identifiers.

[0036] The position coordinates of the local region are bbox(xmin, ymin, xmax, ymax), where xmin and ymin are the x - coordinate and y - coordinate of the upper - left corner of the position box respectively, and xmax and ymax are the x - coordinate and y - coordinate of the lower - right corner of the position box respectively.

[0037] Since the non - vector documents in the construction field contain a variety of data forms, using the Layout neural network detection model can automatically parse document data in multiple different layouts / formats. The parsing granularity can reach the level of paragraphs, sections, and specific page numbers, with high accuracy in document data parsing. By using layout analysis to eliminate non - text interference, it can also improve the accuracy of large - language model recognition.

[0038] Step S104: Extract the type data covered by the local regions of the image to be recognized through the identifiers and position coordinates corresponding to the local regions.

[0039] By using the identifiers and position coordinates corresponding to the local regions to extract the type data of non - vector documents, it is possible to restore and save the structured information of the original non - vector documents, which is conducive to the effective recognition of subsequent large - language models and improves the accuracy of large - language model recognition.

[0040] Step S105: Integrate all type data to generate target input data to enable the large - language model to recognize the non - vector document to be recognized.

[0041] Integrating all type data in the non - vector document avoids the omission of individual data and ensures the smooth progress of the recognition process of the large - language model.

[0042] Preferably, step S104 may include steps S1041 to S1043, where:

[0043] Step S1041: Determine the target extraction region from the image to be recognized through the position coordinates of the local region.

[0044] The target extraction area can be a rectangular area enclosed by the position coordinates of any local area; the picture to be recognized is preset with a two-dimensional coordinate system, and the position coordinates of a local area are sequentially extracted, and position matching is performed in the picture to be recognized based on the position coordinates of the local area, and the target extraction area is circled. The circling method can be marked with a special color, or other marked colors, which are not limited here. Since the position coordinates of the local areas are different, the target extraction areas will also be different.

[0045] Step S1042, obtain the data identifiers pre-stored in the database and the extraction methods corresponding to the data identifiers; among them, the data identifiers include figure types, table types, and text types;

[0046] The data identifier and the extraction method are uniquely corresponding to achieve targeted extraction of different types of data, ensuring the accuracy of obtaining non-vector document content.

[0047] Step S1043, match the identifier of the target extraction area with the data identifier, and identify the target extraction area according to the extraction method corresponding to the successfully matched data identifier, so as to obtain the type of data covered by the target extraction area.

[0048] Since figure types, table types, and text types belong to different dimensions of data, it is difficult to completely identify the above types of data based on a fixed / single recognition method, resulting in the large language model being unable to accurately restore non-vector document information. By specifically identifying the target extraction area according to the extraction method corresponding to the successfully matched data identifier, accurate extraction of all-dimensional data is realized, the original content and structural layout of the non-vector document information are restored, and the recognition accuracy of the large language model is improved.

[0049] Preferably, if the identifier of the target extraction area matches the data identifier of the figure type, the target extraction area is identified according to the extraction method corresponding to the successfully matched data identifier, and the type of data covered by the target extraction area is obtained, including steps A1 to A4, where:

[0050] Step A1, determine the picture border line in the target extraction area, and intercept the target extraction area through the picture border line to obtain the first picture data, where the first picture data is the picture content in the target extraction area;

[0051] Specifically, the picture border line is the contour of the image in the target extraction area, and the interception method can be to cut the target extraction area along the picture border line to obtain the first picture data. The first picture data refers to the picture content of the target extraction area, that is, the image itself covered by the target extraction area.

[0052] Step A2: Identify the target extraction area through the OCR recognition algorithm to obtain second picture data, where the second picture data is the picture name, page number, and note information in the target extraction area;

[0053] Specifically, the OCR recognition algorithm can also be replaced by a conventional text recognition algorithm, which is not limited here.

[0054] Step A3: Calculate the distance between the first picture data and / or the second picture data in the target extraction area, and associate the first picture data and / or the second picture data for which the distance is less than or equal to the first preset threshold;

[0055] There may be multiple images in the target extraction area, and the meanings represented by different images may be related and are generally concentrated. For this situation, it is necessary to calculate the distance between different first picture data and / or second picture data. When the distance is less than or equal to the first preset threshold, it indicates that there is an association relationship between the corresponding images. Therefore, the corresponding first picture data and / or second picture data are associated. The first preset threshold can be set according to actual needs and is not limited here.

[0056] It should be noted that when calculating the distance between different first picture data and / or second picture data, as long as the distance in any case meets the preset situation, the calculation process stops, and then the corresponding picture data can be associated. For example, when the distance between different first picture data is less than the first preset threshold, at this time, it is not necessary to calculate the distance between different second picture data anymore to shorten the picture processing time and thus improve the efficiency of non-vector document preprocessing. The principles of other situations are similar to this example and will not be elaborated here.[[ID=##]]

[0057] Step A4: Integrate the associated first picture data and second picture data to obtain the graph type data covered by the target extraction area.

[0058] The integration method can be to summarize according to the arrangement layout of the first picture data and the second picture data in the target extraction area.

[0059] Preferably, if the identifier of the target extraction area matches the data identifier of the table type, identify the target extraction area according to the extraction method corresponding to the successfully matched data identifier to obtain the type data covered by the target extraction area, including steps B1 to B5, where:

[0060] Step B1: Determine the table border line in the target extraction area, and intercept the target extraction area through the table border line to obtain first table data, where the first table data is the table content in the form of a picture in the target extraction area;

[0061] Step B2: Identify the target extraction area through the OCR recognition algorithm to obtain the second table data, where the second table data is the table name, page number, and note information in the target extraction area;

[0062] Step B3: Calculate the distance between the first picture data and / or the second picture data in the target extraction area, and associate the corresponding first table data and / or second table data whose distance is less than or equal to the second preset threshold;

[0063] The principle of extracting table type data in Steps B1 to B3 is similar to that of extracting picture type data (Steps A1 to A3), which will not be elaborated here.

[0064] Step B4: Identify the first table data through a preset table recognition algorithm to obtain the third table data, and store the third table data in markdown format;

[0065] The preset table recognition algorithm can be a conventional table recognition technology on the market, which is not limited here.

[0066] Storing the third table data in json in markdown format can completely retain the original structure of the table, which is beneficial to the effective recognition of the large language model.

[0067] Step B5: Integrate the associated first table data, second table data, and third table data to obtain the table type data covered by the target extraction area.

[0068] This integration method is similar to the principle of Step A4, which will not be elaborated here.

[0069] Preferably, if the identifier of the target extraction area matches the data identifier of the text type, identify the target extraction area according to the extraction method corresponding to the successfully matched data identifier to obtain the type data covered by the target extraction area, including Steps C1 to C4, where:

[0070] Step C1: Identify the target extraction area through the OCR recognition algorithm to obtain the first text data;

[0071] Step C2: Receive the division instruction of the first text data sent by the client, where the division instruction includes the division type, and the division type includes chapter division and paragraph division;

[0072] Step C3. If the division type is chapter division, obtain the title identifier and the page number corresponding to the title identifier from the identifiers in the target extraction area, divide the first text data with the same page number by the first text data corresponding to the title identifier, and establish the belonging relationship between the first text data corresponding to the non-title identifier and the first text data corresponding to the title identifier according to the relative positions between the first text data, so as to obtain the text type data covered by the target extraction area;

[0073] When establishing the belonging relationship between the first text data corresponding to the non-title identifier and the title identifier, the specific implementation method is as follows:

[0074] Determine the title identifier adjacent to the first text data of the non-title identifier, and judge whether the first text data corresponding to the adjacent title identifier is above the first text data of the non-title identifier. If so, establish the belonging relationship between the first text data corresponding to the non-title identifier and the first text data of the upper title identifier.

[0075] Step C4. If the division type is paragraph division, obtain the paragraph identifier and the page number corresponding to the paragraph identifier from the identifiers in the target extraction area, divide the first text data by the first text data corresponding to the paragraph identifier, and associate the divided paragraphs according to the distances between the first text data corresponding to different paragraph identifiers, so as to obtain the text type data covered by the target extraction area.

[0076] Specifically, the first text data corresponding to each paragraph identifier belongs to a separate paragraph, and based on this rule, the first text data can be divided in terms of content.

[0077] When associating the divided paragraphs according to the distances between the first text data corresponding to different paragraph identifiers, if the first distance is less than or equal to the third preset threshold, it indicates that the corresponding paragraphs have similar representational meanings and need to be associated. The third preset threshold can be set according to actual needs and is not limited here.

[0078] Preferably, the layout analysis model is trained through the following steps D1 to D3:

[0079] Step D1. Obtain a batch of non-vector documents and convert the non-vector documents into pictures;

[0080] Step D2. Add labels to local areas according to the layout of the pictures to generate a training set; where the labels include figure type, table type, and text type;

[0081] The labels include: document name, document level, main title, secondary subtitle, left secondary title, left tertiary title, table of contents, table name, table, table note, figure, figure name, complete paragraph, multi-page paragraph, useless text, footer. Among them, figure, figure name, and figure note are figure type identifiers, table name, table, and table note are table type identifiers, and the other identifiers are text type identifiers.

[0082] Step D3: Input the training set into the initial neural network model for training. Stop training the model until the loss value of the model tends to converge, and obtain the layout analysis model.

[0083] The initial neural network model is trained using the Yolo5-m framework. Among them, the backbone network is darknet53, and the loss function consists of three parts, namely: localization loss L bbox , object confidence loss L conf and classification loss L cls , and the formula is as follows: L loss = L bbox + L conf + L cls ,

[0084] Among them:

[0085] The sum of the third and fourth terms is the penalty term. b is the Bounding box predicted by the model, b gt is the ground truth box, IoU refers to the intersection over union of the above two boxes, ρ 2 refers to the square of the distance between the centers of the above two boxes, c is the length of the diagonal of the smallest box, α is the weight coefficient, and v is used to measure the similarity of the aspect ratio, and is defined as follows:

[0086]

[0087] w and h are the width and height of b, ω gt and h gt are the width and height of b gt ;

[0088] L conf and L cls are the same,

[0089] Here, N = 16, two categories of objects and background, binary cross-entropy loss;

[0090] Use the existing detection framework to train the model, tune it until the model converges, and export the optimal model.

[0091] This embodiment converts non-vector documents into image format, realizes the automatic parsing of non-vector PDF documents, and realizes the automatic and accurate extraction of text, images, tables and other data in non-vector PDF documents. At the same time, by using layout analysis technology, it can restore and preserve the structured information of the original non-vector document, ensure the integrity of the non-vector document information, and thus improve the recognition accuracy of the large language model.

[0092] Example 2

[0093] The second embodiment of the present invention further provides a non-vector document preprocessing device. This non-vector document preprocessing device corresponds to the non-vector document preprocessing method provided in the first embodiment above. The corresponding technical features and technical effects are not described in detail in this embodiment. For relevant details, please refer to the first embodiment above. Specifically, Figure 2 FIG. 4 shows a structural block diagram of the preprocessing device for the non-vector document. Figure 2 As shown, the non-vector document preprocessing device 200 includes an acquisition module 501, a format conversion module 202, a recognition module 203, an extraction module 204 and an integration module 205, wherein:

[0094] An acquisition module 201 is configured to acquire a non-vector document to be identified, wherein the non-vector document to be identified contains information for large language model identification;

[0095] The format conversion module 202 is connected to the acquisition module 201 and is used to convert the format of the non-vector document to be identified and generate an image to be identified;

[0096] The recognition module 203 is connected to the format conversion module 202 and is used to input the image to be recognized into the trained layout analysis model to obtain the identification and position coordinates corresponding to the local area of the image to be recognized, wherein the layout analysis model is a layout neural network detection model;

[0097] The extraction module 204 is connected to the recognition module 203 and is used to extract the type data covered by the local area of the image to be recognized through the identifier and position coordinates corresponding to the local area;

[0098] The integration module 205 is connected to the extraction module 204 and is used to integrate all types of data to generate target input data to enable the large language model to recognize the non-vector document to be recognized.

[0099] Optionally, the extraction module includes: a determination unit configured to determine a target extraction area from the picture to be recognized based on the position coordinates of the local area; an acquisition unit configured to acquire the data identifier pre-stored in the database and the corresponding extraction method of the data identifier; wherein the data identifier includes picture type, table type, and text type; a matching unit configured to match the identifier of the target extraction area with the data identifier, and recognize the target extraction area according to the extraction method corresponding to the successfully matched data identifier to obtain the type data covered by the target extraction area.

[0100] Optionally, the matching unit is specifically configured to: determine the picture border line in the target extraction area, intercept the target extraction area through the picture border line to obtain first picture data, where the first picture data is the picture content in the target extraction area; recognize the target extraction area through the OCR recognition algorithm to obtain second picture data, where the second picture data is the picture name, page number, and note information in the target extraction area; calculate the distance between the first picture data and / or the second picture data in the target extraction area, and associate the corresponding first picture data and / or second picture data whose distance is less than or equal to a first preset threshold; integrate the associated first picture data and second picture data to obtain the picture type data covered by the target extraction area.

[0101] Optionally, the matching unit is further configured to: determine the table border line in the target extraction area, intercept the target extraction area through the table border line to obtain first table data, where the first table data is the table content in the form of a picture in the target extraction area; recognize the target extraction area through the OCR recognition algorithm to obtain second table data, where the second table data is the table name, page number, and note information in the target extraction area; calculate the distance between the first picture data and / or the second picture data in the target extraction area, and associate the corresponding first table data and / or second table data whose distance is less than or equal to a second preset threshold; recognize the first table data through a preset table recognition algorithm to obtain third table data, and store the third table data in markdown form; integrate the associated first table data, second table data, and third table data to obtain the table type data covered by the target extraction area.

[0102] Optionally, the matching unit is further configured to: identify the target extraction area through an OCR recognition algorithm to obtain first text data; receive a division instruction for the first text data sent by the client, where the division instruction includes a division type, and the division type includes chapter division and paragraph division; if the division type is chapter division, obtain a title identifier and the page number corresponding to the title identifier from the identifier of the target extraction area, perform content division on the first text data with the same page number through the first text data corresponding to the title identifier, and establish a belonging relationship between the first text data corresponding to the non-title identifier and the first text data corresponding to the title identifier according to the relative position between the first text data, so as to obtain the text type data covered by the target extraction area; if the division type is paragraph division, obtain a paragraph identifier and the page number corresponding to the paragraph identifier from the identifier of the target extraction area, perform content division on the first text data through the first text data corresponding to the paragraph identifier, and associate the divided paragraphs through the distance between the first text data corresponding to different paragraph identifiers, so as to obtain the text type data covered by the target extraction area.

[0103] Optionally, the device further includes a training module, configured to: obtain a batch of non-vector documents, and convert the non-vector documents into pictures; add labels to local areas of the pictures according to the layout of the pictures to generate a training set; where the labels include picture type, table type, and text type; input the training set into an initial neural network model for training, and stop training the model until the loss value of the model tends to converge, so as to obtain a layout analysis model.

[0104] Embodiment III

[0105] Figure 3 FIG. shows a block diagram of a computer device suitable for implementing the preprocessing method of non-vector documents provided in Embodiment III of the present invention. In this embodiment, the computer device 300 may be a smart phone, a tablet computer, a notebook computer, a desktop computer, a rack server, a blade server, a tower server, or a cabinet server (including an independent server, or a server cluster composed of multiple servers) that executes a program. As Figure 3 shown, the computer device 300 of this embodiment at least includes, but is not limited to: a memory 301, a processor 302, and a network interface 303 that can communicate with each other through a system bus. It should be noted that Figure 3 only the computer device 300 with components 301-303 is shown, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented.

[0106] In this embodiment, the memory 303 includes at least one type of computer-readable storage medium. The readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disc, etc. In some embodiments, the memory 301 may be an internal storage unit of the computer device 300, such as the hard disk or memory of the computer device 300. In other embodiments, the memory 301 may also be an external storage device of the computer device 300, such as a plug-in hard disk equipped on the computer device 300, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Of course, the memory 301 may also include both the internal storage unit and the external storage device of the computer device 300. In this embodiment, the memory 301 is generally used to store the operating system installed on the computer device 300 and various application software, such as the program code of the preprocessing method for non-vector documents.

[0107] In some embodiments, the processor 302 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 302 is generally used to control the overall operation of the computer device 300. For example, it executes control and processing related to data interaction or communication with the computer device 300. In this embodiment, the processor 302 is used to run the program code of the steps of the preprocessing method for non-vector documents stored in the memory 301.

[0108] In this embodiment, the preprocessing method for non-vector documents stored in the memory 301 may also be divided into one or more program modules and executed by one or more processors (in this embodiment, the processor 302) to complete the present invention.

[0109] The network interface 303 may include a wireless network interface or a wired network interface, which is generally used to establish a communication link between the computer device 300 and other computer devices. For example, the network interface 303 is used to connect the computer device 300 to an external terminal through a network, and to establish a data transmission channel and a communication link between the computer device 300 and the external terminal. The network may be a wireless or wired network such as an enterprise intranet (Intranet), the Internet, Global System of Mobile communication (GSM for short), Wideband Code Division Multiple Access (WCDMA for short), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0110] Embodiment 4

[0111] This embodiment also provides a computer-readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disc, server, App application mall, etc., on which a computer program is stored, and when the computer program is executed by a processor, the steps of the preprocessing method for non-vector documents are implemented.

[0112] Obviously, those skilled in the art should understand that the above-mentioned modules or steps of the embodiments of the present invention can be implemented by a general-purpose computing device. They can be concentrated on a single computing device or distributed on a network composed of multiple computing devices. Optionally, they can be implemented by program codes executable by the computing device, so that they can be stored in a storage device and executed by the computing device. And in some cases, the steps shown or described can be executed in a different order than here, or they can be separately made into individual integrated circuit modules, or multiple modules or steps among them can be made into a single integrated circuit module to implement. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0113] It should be noted that the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0114] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general-purpose hardware platform. Of course, they can also be implemented by hardware, but in many cases, the former is a better implementation method.

[0115] The above are only the preferred embodiments of the present invention, and do not limit the patent scope of the present invention accordingly. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, shall be similarly included in the patent protection scope of the present invention.

Claims

1. A preprocessing method for non-vector documents, characterized in that The method includes: Obtain a non-vector document to be recognized, where the non-vector document to be recognized contains information for large language model recognition; Convert the format of the non-vector document to be recognized to generate a picture to be recognized; Input the picture to be recognized into a trained layout analysis model to obtain the identifiers and position coordinates corresponding to the local areas of the picture to be recognized, where the layout analysis model is a Layout neural network detection model; Extract the type data covered by the local area of the picture to be recognized through the identifiers and position coordinates corresponding to the local area; Integrate all type data to generate target input data to enable the large language model to recognize the non-vector document to be recognized.

2. The method according to claim 1, characterized in that The extracting the type data covered by the local area of the picture to be recognized through the identifiers and position coordinates corresponding to the local area includes: Determine a target extraction area from the picture to be recognized through the position coordinates of the local area; Obtain the data identifiers pre-stored in the database and the extraction methods corresponding to the data identifiers; where the data identifiers include figure type, table type, and text type; Match the identifier of the target extraction area with the data identifiers, and identify the target extraction area according to the extraction method corresponding to the successfully matched data identifier to obtain the type data covered by the target extraction area.

3. The method according to claim 2, wherein If the identifier of the target extraction area matches the data identifier of the figure type, the identifying the target extraction area according to the extraction method corresponding to the successfully matched data identifier to obtain the type data covered by the target extraction area includes: Determine the picture border line in the target extraction area, and intercept the target extraction area through the picture border line to obtain first picture data, where the first picture data is the picture content in the target extraction area; Recognize the target extraction area through an OCR recognition algorithm to obtain second picture data, where the second picture data is the picture name, page number, and note information in the target extraction area; Calculate the distance between the first picture data and / or the second picture data in the target extraction area, and associate the first picture data and / or the second picture data for which the distance is less than or equal to a first preset threshold; Integrate the associated first picture data and second picture data to obtain the figure type data covered by the target extraction area.

4. The method according to claim 2, wherein If the identifier of the target extraction area matches the data identifier of the table type, the identifying the target extraction area according to the extraction method corresponding to the successfully matched data identifier to obtain the type data covered by the target extraction area includes: Determine the table border line in the target extraction area, and intercept the target extraction area through the table border line to obtain first table data, where the first table data is the table content in the form of a picture in the target extraction area; The target extraction area is recognized by an OCR recognition algorithm to obtain second table data, where the second table data is the table name, page number, and note information in the target extraction area; Calculate the distance between the first picture data and / or the second picture data in the target extraction area, and associate the corresponding first table data and / or second table data where the distance is less than or equal to a second preset threshold; The first table data is recognized by a preset table recognition algorithm to obtain third table data, and the third table data is stored in markdown format; Integrate the associated first table data, second table data, and the third table data to obtain the table type data covered by the target extraction area.

5. The method according to claim 2, characterized in that If the identifier of the target extraction area matches the data identifier of the text type, the target extraction area is recognized according to the extraction method corresponding to the successfully matched data identifier to obtain the type data covered by the target extraction area, including: The target extraction area is recognized by an OCR recognition algorithm to obtain first text data; Receive a division instruction for the first text data sent by the client, where the division instruction includes a division type, and the division type includes chapter division and paragraph division; If the division type is chapter division, obtain a title identifier and the page number corresponding to the title identifier from the identifier of the target extraction area, divide the first text data with the same page number by the first text data corresponding to the title identifier, and establish an ownership relationship between the first text data corresponding to the non-title identifier and the first text data corresponding to the title identifier according to the relative position between the first text data, so as to obtain the text type data covered by the target extraction area; If the division type is paragraph division, obtain a paragraph identifier and the page number corresponding to the paragraph identifier from the identifier of the target extraction area, divide the first text data by the first text data corresponding to the paragraph identifier, and associate the divided paragraphs by the distance between the first text data corresponding to different paragraph identifiers, so as to obtain the text type data covered by the target extraction area.

6. The method according to any one of claims 1-5, characterized in that, The layout analysis model is trained in the following way: Obtain a batch of non-vector documents and convert the non-vector documents into pictures; Add labels to the local areas of the pictures according to the layout of the pictures to generate a training set; where the labels include picture type, table type, and text type; Input the training set into an initial neural network model for training, and stop training the model until the loss value of the model tends to converge to obtain a layout analysis model.

7. A preprocessing device for a non-vector document, characterized in that, The device includes: An acquisition module, configured to acquire a non-vector document to be recognized, where the non-vector document to be recognized contains information for large language model recognition; A format conversion module, configured to perform format conversion on the non-vector document to be recognized to generate a picture to be recognized; An identification module, configured to input the picture to be identified into a trained layout analysis model, and obtain the identification and position coordinates corresponding to the local area of the picture to be identified, where the layout analysis model is a Layout neural network detection model; An extraction module, configured to extract the type data covered by the local area of the picture to be identified through the identification and position coordinates corresponding to the local area; An integration module, configured to integrate all type data to generate target input data, so as to implement the identification of the non-vector document to be identified by the large language model.

8. The method according to claim 7, wherein The device includes a training module, specifically used for: Obtaining a batch of non-vector documents and converting the non-vector documents into pictures; Adding labels to the local areas of the pictures according to the layout of the pictures to generate a training set; where the labels include picture type, table type, and text type; Inputting the training set into an initial neural network model for training, and stopping training the model until the loss value of the model tends to converge, so as to obtain a layout analysis model.

9. A computer device, the computer device comprising: A memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 6 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program, when executed by the processor, implements the method according to any one of claims 1 to 6.

Images