Document identification method and system based on large model, terminal and storage medium

Abstract

The invention provides a document recognition method and system based on a large model, a terminal and a storage medium, and the method comprises the steps: training a target large model according to model training data after data preprocessing until the target large model converges; performing data cleaning on a to-be-recognized document, and inputting the to-be-recognized document after data cleaning into the converged target large model for recognition to obtain a word tag prediction result; and performing label analysis on the word label prediction result to obtain a label analysis result, and performing format processing on the label analysis result to obtain a document identification result. According to the embodiment of the invention, the target large model is trained based on the model training data subjected to data preprocessing, so that the trained target large model can effectively identify the to-be-identified document to obtain the word tag prediction result, and the title and the text in the to-be-identified document can be effectively determined through tag analysis, so that the word tag prediction result is obtained. And the title and the text do not need to be marked manually, so that the document identification efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a document recognition method, system, terminal, and storage medium based on a large model. Background Technology

[0002] In a knowledge management system, knowledge management begins with the import of large-scale documents. This typically requires identifying the distribution of headings and body text within document paragraphs. The identification of heading and body text distribution is crucial for knowledge retrieval and large-scale question answering models.

[0003] In existing document recognition processes, manual annotation is generally used to mark the titles and body text of the documents to be recognized, which makes the operation cumbersome for users and reduces the efficiency of document recognition. Summary of the Invention

[0004] The purpose of this invention is to provide a document recognition method, system, terminal, and storage medium based on a large model, aiming to solve the problem of low efficiency in existing document recognition methods.

[0005] This invention is implemented as follows: a document recognition method based on a large model, the method comprising:

[0006] Obtain the document recognition requirements, and determine the target large model and model training data based on the document recognition requirements;

[0007] The model training data is preprocessed, and the target large model is trained based on the preprocessed model training data until the target large model converges.

[0008] The document to be identified is cleaned, and the cleaned document is input into the converged target large model for identification to obtain word tag prediction results.

[0009] The word tag prediction results are parsed to obtain tag parsing results, and the tag parsing results are formatted to obtain document recognition results. The tag parsing results include segment titles and body text.

[0010] Preferably, the document to be identified undergoes data cleaning, including:

[0011] The document to be identified is segmented into words to obtain a segmented document, and the segmented document is subjected to character detection. The segmented document stores at least one segmented word.

[0012] Based on the character detection results, the segmented document is cleaned, and then word embedding is performed on the cleaned segmented document.

[0013] Preferably, the word tag prediction results are parsed to obtain tag parsing results, including:

[0014] The word tags in the word tag prediction results are matched with tags, and the word segments corresponding to the same word tags are merged according to the tag matching results to obtain the tag parsing results.

[0015] Preferably, the tag parsing results are formatted to obtain document recognition results, including:

[0016] The title features of each segment title are obtained separately, and the title level is determined based on the title features, which include character size, word entity type and character position;

[0017] The headings of each segment are sorted according to the heading level to obtain the heading recognition result, and the body text is reorganized according to the heading recognition result to obtain the body text recognition result;

[0018] The document recognition results include the title recognition results and the body text recognition results.

[0019] Preferably, the model training data undergoes data preprocessing, including:

[0020] Missing values ​​are processed in the model training data, and outlier processing is performed on the model training data after missing value processing.

[0021] The model training data after outlier processing is subjected to duplicate detection, and the model training data is deleted according to the duplicate detection results to obtain the model training data after data preprocessing.

[0022] Preferably, the target large model and model training data are determined according to the document recognition requirements, including:

[0023] The task attributes are determined based on the task description in the document recognition requirements, and the candidate model and the training data for the model are determined based on the task attributes.

[0024] The candidate models are filtered based on the amount of model data required for document recognition to obtain the target large model.

[0025] Another objective of this invention is to provide a document recognition system based on a large model, the system comprising:

[0026] The model determination module is used to obtain document recognition requirements and determine the target large model and model training data based on the document recognition requirements.

[0027] The training module is used to preprocess the model training data and train the target large model based on the preprocessed model training data until the target large model converges.

[0028] The tag prediction module is used to clean the data of the document to be identified, and input the cleaned document to be identified into the converged target large model for identification to obtain the word tag prediction result.

[0029] The format processing module is used to perform tag parsing on the word tag prediction results to obtain tag parsing results, and to perform format processing on the tag parsing results to obtain document recognition results.

[0030] Another objective of this invention is to provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0031] Another objective of this invention is to provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0032] In this embodiment of the invention, by acquiring document recognition requirements, the target large model and model training data can be effectively determined based on the document recognition requirements. By preprocessing the model training data, the accuracy of the model training data is effectively improved. The target large model is trained based on the preprocessed model training data, so that the trained target large model can effectively recognize the document to be recognized and obtain word tag prediction results. By parsing the word tag prediction results, the title and body text in the document to be recognized can be effectively determined without the need for manual labeling of the title and body text, thus improving the document recognition efficiency. By formatting the label parsing results, the recognition results of the title and body text can be effectively converted into document recognition results. Attached Figure Description

[0033] Figure 1 This is a flowchart of the document recognition method based on a large model provided in the first embodiment of the present invention;

[0034] Figure 2 This is a flowchart of the document recognition method based on a large model provided in the second embodiment of the present invention;

[0035] Figure 3 This is a schematic diagram of the structure of the document recognition system based on a large model provided in the third embodiment of the present invention;

[0036] Figure 4This is a schematic diagram of the structure of the terminal device provided in the fourth embodiment of the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0038] To illustrate the technical solution described in this invention, specific embodiments are described below.

[0039] Example 1

[0040] Please see Figure 1 This is a flowchart of a document recognition method based on a large model provided in the first embodiment of the present invention. This method can be applied to any terminal device or system, and includes the following steps:

[0041] Step S10: Obtain document recognition requirements, and determine the target large model and model training data based on the document recognition requirements;

[0042] The document recognition requirements include a task description and a model data volume, both of which can be set according to user needs. The task description represents the knowledge domain of the document to be recognized, such as the medical field, news field, or product field. The model data volume represents the required size of the target model.

[0043] Optionally, the target large model and model training data are determined based on the document recognition requirements, including:

[0044] The task attributes are determined based on the task description in the document recognition requirements, and the candidate model and the training data for the model are determined based on the task attributes.

[0045] The candidate models are filtered based on the amount of model data in the document recognition requirements to obtain the target large model;

[0046] The process involves matching the task description with a model lookup table to obtain the candidate model and its training data. The model lookup table stores the correspondence between different task descriptions and their corresponding candidate models and training data.

[0047] Furthermore, in this step, the task description can also be determined based on the document to be identified, including: extracting the document to be identified to obtain the extracted document, performing entity recognition on the extracted document to obtain the document entity type, and determining the task description based on the document entity type.

[0048] Step S20: Perform data preprocessing on the model training data, and train the target large model based on the preprocessed model training data until the target large model converges;

[0049] In this step, the accuracy of the model training data is effectively improved by preprocessing the model training data, so that the target large model can better understand and learn. In this step, the target large model is fine-tuned using the labeled model training data, and the parameters of each input sample are optimized by a supervised learning algorithm to reduce the gap between the model prediction value and the true label. When the training of the target large model reaches the convergence state, the training of the target large model is completed.

[0050] Step S30: Clean the data of the document to be identified, and input the cleaned document to be identified into the converged target large model for identification to obtain word tag prediction results;

[0051] Among them, by cleaning the data of the document to be identified, special characters and irrelevant information in the document to be identified can be effectively removed, which can effectively facilitate the identification of the document by the target large model. The word label prediction result includes the label prediction of each word segment in the document to be identified.

[0052] Optionally, data cleaning may be performed on the document to be identified, including:

[0053] The document to be identified is segmented into words to obtain a segmented document, and character detection is performed on the segmented document; wherein, the segmented document stores at least one segmented word, and segmenting the document to be identified into words divides the document to be identified into word units, which is beneficial for data input into the target large model;

[0054] Based on the character detection results, the segmented document is cleaned, and then word embedding is performed on the cleaned segmented document. Regular expressions and other methods can be used to clean the segmented document to remove special characters and irrelevant information, such as whitespace and punctuation marks. In this step, by performing word embedding on the cleaned segmented document, the cleaned segmented document can be effectively converted into a numerical vector form that the target large model can directly process.

[0055] Step S40: Perform tag parsing on the word tag prediction results to obtain tag parsing results, and perform format processing on the tag parsing results to obtain document recognition results;

[0056] In particular, by parsing the word tag prediction results, the title and body text in the document to be identified can be effectively determined without the need for manual labeling of the title and body text, thus improving the document recognition efficiency.

[0057] Optionally, the word tag prediction results are parsed to obtain tag parsing results, including:

[0058] The word tags in the word tag prediction results are matched with tags, and the word segments corresponding to the same word tags are merged according to the tag matching results to obtain the tag parsing results. The tag parsing results include the segment titles and the body text.

[0059] Furthermore, the tag parsing results are formatted to obtain document recognition results, including:

[0060] The title features of each segment title are obtained separately, and the title level is determined based on the title features, wherein the title features include character size, word entity type and character position;

[0061] The headings of each segment are sorted according to the heading level to obtain the heading recognition result, and the body text is reorganized according to the heading recognition result to obtain the body text recognition result;

[0062] The document recognition result includes the title recognition result and the body text recognition result. In this step, the title is re-identified and classified according to the level as needed. The identified segmented titles and body text are reordered and reorganized to obtain the document recognition result. The processed document recognition result is output in a specified format for subsequent application.

[0063] In this embodiment, the document title and body text can be effectively identified. By using a target large model and fine-tuning it, the title and body text in the document can be effectively identified, thus solving the problem that the existing technology has low recognition accuracy and cannot efficiently distinguish between titles and body text when processing large-scale document content.

[0064] The fully automated processing workflow, including document preprocessing, model training, recognition, and result post-processing, significantly improves processing efficiency and reduces manual intervention. Furthermore, standardized output allows the processing results to be directly used for subsequent tasks, resolving issues such as the need for extensive manual processing and data type conversion in existing technologies.

[0065] In this embodiment, by acquiring document recognition requirements, the target large model and model training data can be effectively determined based on these requirements. By preprocessing the model training data, the accuracy of the model training data is effectively improved. The target large model is trained based on the preprocessed model training data, enabling the trained target large model to effectively recognize the document to be recognized and obtain word tag prediction results. By parsing the word tag prediction results, the title and body text in the document to be recognized can be effectively determined without the need for manual labeling of the title and body text, thus improving document recognition efficiency. By formatting the label parsing results, the recognition results of the title and body text can be effectively converted into document recognition results.

[0066] Example 2

[0067] Please see Figure 2 This is a flowchart of a document recognition method based on a large model provided in the second embodiment of the present invention. This embodiment further refines step S20 in the first embodiment, including the following steps:

[0068] Step S21: Perform missing value processing on the model training data, and perform outlier processing on the model training data after missing value processing.

[0069] Among these steps, by handling missing values ​​in the model training data, missing data can be effectively filled in. In this step, methods such as Lagrange interpolation or Newton interpolation can be used to fill in missing data in the model training data. By handling outliers in the model training data after handling missing values, outliers can be effectively removed from the model training data, thereby improving the accuracy of the model training data.

[0070] Step S22: Perform duplicate detection on the model training data after outlier processing, and delete data from the model training data according to the duplicate detection results to obtain the model training data after data preprocessing.

[0071] Specifically, by performing duplicate detection on the model training data after outlier processing, the accuracy of the model training data is effectively improved by removing duplicate data.

[0072] In this embodiment, by performing missing value processing, outlier processing, and duplicate detection on the model training data, the accuracy of the model training data is effectively improved, further enhancing the training effect of the target large model.

[0073] Example 3

[0074] Please see Figure 3This is a schematic diagram of the structure of a document recognition system 100 based on a large model provided in the third embodiment of the present invention, including:

[0075] The model determination module 10 is used to acquire document recognition requirements and determine the target large model and model training data based on the document recognition requirements. The document recognition requirements include a task description and a model data volume, both of which can be set according to user needs. The task description represents the knowledge domain of the document to be recognized, such as the medical field, news field, or product field. The model data volume represents the required size of the target large model.

[0076] Optionally, the model determination module 10 is further configured to: determine task attributes based on the task description in the document recognition requirements, and determine candidate models and model training data based on the task attributes;

[0077] The candidate models are filtered based on the amount of model data required for document recognition to obtain the target large model.

[0078] Training module 11 is used to preprocess the model training data and train the target large model based on the preprocessed model training data until the target large model converges. Preprocessing the model training data effectively improves the accuracy of the model training data, allowing the target large model to better understand and learn. The labeled model training data is used to fine-tune the target large model, and a supervised learning algorithm is used to optimize the parameters of each input sample to reduce the gap between the model's predicted values ​​and the true labels. Training of the target large model is complete when it converges.

[0079] Optionally, the training module 11 is further configured to: process the missing values ​​in the model training data, and process the outliers in the model training data after processing the missing values.

[0080] The model training data after outlier processing is subjected to duplicate detection, and the model training data is deleted according to the duplicate detection results to obtain the model training data after data preprocessing.

[0081] The label prediction module 12 is used to clean the data of the document to be identified, and then input the cleaned document into the converged target large model for identification to obtain the word label prediction result. By cleaning the data of the document to be identified, special characters and irrelevant information can be effectively removed, which effectively facilitates the identification of the document by the target large model. The word label prediction result includes the label prediction for each segmented word in the document to be identified.

[0082] Optionally, the tag prediction module 12 is further configured to: segment the document to be identified into words to obtain a segmented document, and perform character detection on the segmented document, wherein the segmented document stores at least one segmented word;

[0083] Based on the character detection results, the segmented document is cleaned, and then word embedding is performed on the cleaned segmented document.

[0084] The format processing module 13 is used to perform tag parsing on the word tag prediction results to obtain tag parsing results, and to perform format processing on the tag parsing results to obtain document recognition results. The tag parsing results include segment titles and body text. Specifically, by performing tag parsing on the word tag prediction results, the titles and body text in the document to be recognized can be effectively determined without the need for manual tagging of titles and body text, thus improving document recognition efficiency.

[0085] Optionally, the format processing module 13 is further configured to: perform tag matching on the word tags in the word tag prediction results, and merge the word segments corresponding to the same word tags according to the tag matching results to obtain the tag parsing results.

[0086] Furthermore, the format processing module 13 is also used to: obtain the title features of each segment title respectively, and determine the title level according to the title features, wherein the title features include character size, word entity type and character position;

[0087] The headings of each segment are sorted according to the heading level to obtain the heading recognition result, and the body text is reorganized according to the heading recognition result to obtain the body text recognition result;

[0088] The document recognition results include the title recognition results and the body text recognition results.

[0089] In this embodiment, by acquiring document recognition requirements, the target large model and model training data can be effectively determined based on these requirements. By preprocessing the model training data, the accuracy of the model training data is effectively improved. The target large model is trained based on the preprocessed model training data, enabling the trained target large model to effectively recognize the document to be recognized and obtain word tag prediction results. By parsing the word tag prediction results, the title and body text in the document to be recognized can be effectively determined without the need for manual labeling of the title and body text, thus improving document recognition efficiency. By formatting the label parsing results, the recognition results of the title and body text can be effectively converted into document recognition results.

[0090] Example 4

[0091] Figure 4 This is a structural block diagram of a terminal device 2 provided in the fourth embodiment of this application. For example... Figure 4 As shown, the terminal device 2 in this embodiment includes a processor 20, a memory 21, and a computer program 22 stored in the memory 21 and executable on the processor 20, such as a program for a document recognition method based on a large model. When the processor 20 executes the computer program 22, it implements the steps in the various embodiments of the document recognition method based on the large model described above.

[0092] For example, the computer program 22 may be divided into one or more modules, which are stored in the memory 21 and executed by the processor 20 to complete this application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 22 in the terminal device 2. The terminal device may include, but is not limited to, the processor 20 and the memory 21.

[0093] The processor 20 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0094] The memory 21 can be an internal storage unit of the terminal device 2, such as a hard drive or memory of the terminal device 2. The memory 21 can also be an external storage device of the terminal device 2, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the terminal device 2. Furthermore, the memory 21 can include both internal and external storage units of the terminal device 2. The memory 21 is used to store the computer program and other programs and data required by the terminal device. The memory 21 can also be used to temporarily store data that has been output or will be output.

[0095] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0096] If an integrated module is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. This computer-readable storage medium can be non-volatile or volatile. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the contents of a computer-readable storage medium may be appropriately added to or subtracted from the contents as required by the legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, a computer-readable storage medium may not include electrical carrier signals and telecommunication signals.

[0097] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A document recognition method based on a large model, characterized in that, The method includes: Obtain the document recognition requirements, and determine the target large model and model training data based on the document recognition requirements; The model training data is preprocessed, and the target large model is trained based on the preprocessed model training data until the target large model converges. The document to be identified is cleaned, and the cleaned document is input into the converged target large model for identification to obtain word tag prediction results. The word tag prediction results are parsed to obtain tag parsing results, and the tag parsing results are formatted to obtain document recognition results. The tag parsing results include segment titles and body text.

2. The document recognition method based on a large model as described in claim 1, characterized in that, Data cleaning of the documents to be identified includes: The document to be identified is segmented into words to obtain a segmented document, and the segmented document is subjected to character detection. The segmented document stores at least one segmented word. Based on the character detection results, the segmented document is cleaned, and then word embedding is performed on the cleaned segmented document.

3. The document recognition method based on a large model as described in claim 2, characterized in that, The word tag prediction results are parsed to obtain tag parsing results, including: The word tags in the word tag prediction results are matched with tags, and the word segments corresponding to the same word tags are merged according to the tag matching results to obtain the tag parsing results.

4. The document recognition method based on a large model as described in claim 3, characterized in that, The tag parsing results are formatted to obtain document recognition results, including: The title features of each segment title are obtained separately, and the title level is determined based on the title features, which include character size, word entity type and character position; The headings of each segment are sorted according to the heading level to obtain the heading recognition result, and the body text is reorganized according to the heading recognition result to obtain the body text recognition result; The document recognition results include the title recognition results and the body text recognition results.

5. The document recognition method based on a large model as described in claim 1, characterized in that, Data preprocessing of the model training data includes: Missing values ​​are processed in the model training data, and outlier processing is performed on the model training data after missing value processing. The model training data after outlier processing is subjected to duplicate detection, and the model training data is deleted according to the duplicate detection results to obtain the model training data after data preprocessing.

6. The document recognition method based on a large model as described in any one of claims 1 to 5, characterized in that, Based on the document recognition requirements, the target large model and model training data are determined, including: The task attributes are determined based on the task description in the document recognition requirements, and the candidate model and the training data for the model are determined based on the task attributes. The candidate models are filtered based on the amount of model data required for document recognition to obtain the target large model.

7. A document recognition system based on a large model, characterized in that, The system includes: The model determination module is used to obtain document recognition requirements and determine the target large model and model training data based on the document recognition requirements. The training module is used to preprocess the model training data and train the target large model based on the preprocessed model training data until the target large model converges. The tag prediction module is used to clean the data of the document to be identified, and input the cleaned document to be identified into the converged target large model for identification to obtain the word tag prediction result. The format processing module is used to perform tag parsing on the word tag prediction results to obtain tag parsing results, and to perform format processing on the tag parsing results to obtain document recognition results. The tag parsing results include segment titles and body text.

8. The document recognition system based on a large model as described in claim 7, characterized in that, The label prediction module is also used for: The document to be identified is segmented into words to obtain a segmented document, and the segmented document is subjected to character detection. The segmented document stores at least one segmented word. Based on the character detection results, the segmented document is cleaned, and then word embedding is performed on the cleaned segmented document.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 6.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

Images