Text recognition method and device of target file, storage medium and electronic device

Abstract

The invention discloses a character recognition method and device of a target file, a storage medium and an electronic device, and relates to the field of computers.The character recognition method of the target file comprises the steps that character recognition is conducted on the target file, and character recognition content is obtained; the character recognition content is restored through a large language model, a character restoration result is obtained, and the character restoration result comprises at least one sub-restoration result; processing the character recognition content, a file picture corresponding to the target file and the character restoration result through a recognition content correction model to obtain a final recognition result of the target file; by adopting the scheme, the problems of low image contrast, character inclination and blurring of the file image obtained by scanning the paper file in the related technology are solved.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a method and device for identifying text in a target file, a storage medium, and an electronic device. Background Art

[0002] In many industries represented by energy and electricity, post-project evaluation is an important link. In addition to massive structured production and operational financial data, the data used for post-project evaluation also includes a large amount of data in file types such as feasibility study reports and approval documents. Most of these data are stored in unstructured forms such as Word and PDF, including a large number of paper documents. These historical documents often contain a lot of valuable information and are an important basis for evaluating projects. Therefore, electronic processing of paper documents, rapid discovery of knowledge from massive unstructured texts, and representation of this textual knowledge in a form that computers can "understand" and structured management of unstructured documents are important ways to achieve automated post-project evaluation.

[0003] However, the document images obtained by scanning paper documents often face problems such as low image contrast, tilted text, blur, smudges, and the amount of ink received by printing. These problems will cause frequent errors in the recognized text content.

[0004] Regarding the related technologies, the document images obtained by scanning paper documents often face the problems of low image contrast, tilted text, blur, etc., which lead to frequent errors in the recognized text content. Currently, no effective solution has been proposed.

[0005] Therefore, it is necessary to improve the related technology to overcome the above-mentioned defects in the related technology. Summary of the invention

[0006] The embodiments of the present application provide a method and device for text recognition of a target file, a storage medium and an electronic device, so as to at least solve the problems in the related art that the document images obtained by scanning paper documents often face the problems of low image contrast, tilted text and blur.

[0007] According to one aspect of an embodiment of the present application, a method for text recognition of a target file is provided, comprising: performing text recognition on a target file to obtain text recognition content; repairing the text recognition content through a large language model to obtain a text repair result, wherein the text repair result includes at least one sub-repair result; processing the text recognition content, a file image corresponding to the target file, and the text repair result through a recognition content correction model to obtain a final recognition result of the target file.

[0008] In an exemplary embodiment, before repairing the text recognition content through the large language model to obtain the text repair result, the method also includes: obtaining a first prompt template in the large language model, wherein the first prompt template is used to instruct the large language model to classify the text recognition content; filling the first prompt template according to the text recognition content to obtain a filled first prompt template; processing the filled first prompt template through the large language model to obtain a classification result of the text recognition content, wherein the classification result is used to indicate whether the text recognition content needs to be repaired; when the classification result indicates that the text recognition content needs to be repaired, repairing the text recognition content through the large language model to obtain the text repair result.

[0009] In an exemplary embodiment, repairing the text recognition content by using the large language model to obtain the text repair result includes: obtaining a second prompt template in the large language model, wherein the second prompt template is used to instruct the large language model to repair the text recognition content; filling the second prompt template according to the text recognition content to obtain a filled second prompt template; and processing the filled second prompt template by using the large language model to obtain the text repair result.

[0010] In an exemplary embodiment, the text recognition content, the file image corresponding to the target file, and the text repair result are processed by a recognition content correction model to obtain a final recognition result of the target file, including: respectively encoding the text recognition content, the file image, and the text repair result to obtain a first code, a second code, and a third code, wherein the first code corresponds to the text recognition content, the second code corresponds to the file image, and the third code corresponds to the text repair result; correcting the first code, the second code, and the third code by a co-attention module to obtain a corrected code; decoding the corrected code to obtain the final recognition result.

[0011] In an exemplary embodiment, the text recognition content, the file image and the text repair result are respectively encoded to obtain a first code, a second code and a third code, including: encoding the text recognition content and the text repair result through a text content encoder to obtain a fourth code and a fifth code, and encoding the file image through an image encoder to obtain a sixth code, wherein the fourth code corresponds to the text recognition content, the fifth code corresponds to the text repair result, and the fifth code includes at least one sub-code, and the at least one sub-code corresponds one-to-one with the at least one sub-repair result; fusing the at least one sub-code through a fusion module to obtain a seventh code; and dimensionally converting the fourth code, the sixth code and the seventh code through a linear module to obtain the first code, the second code and the third code, wherein the first code, the second code and the third code have the same dimension.

[0012] In an exemplary embodiment, the first code, the second code and the third code are corrected by the attention module to obtain a corrected code, including: M +WF G +(WF M )C M +(WF G )C G ) performs correction processing on the first code, the second code and the third code to obtain the corrected code, wherein H is the corrected code, ReLU is the activation function, W is the module parameter of the co-attention module, and F M For the third code, F G For the first encoding, F P is the second encoding.

[0013] According to another aspect of an embodiment of the present application, a target file text recognition device is also provided, including: a text recognition module, used to perform text recognition on the target file to obtain text recognition content; a repair module, used to repair the text recognition content through a large language model to obtain a text repair result, wherein the text repair result includes at least one sub-repair result; a correction module, used to process the text recognition content, the file image corresponding to the target file and the text repair result through a recognition content correction model to obtain a final recognition result of the target file.

[0014] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the text recognition method for the target file when running.

[0015] According to another aspect of the embodiment of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the text recognition method of the target file through the computer program.

[0016] According to another aspect of the embodiments of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps of the method described in each embodiment of the present application are implemented.

[0017] Through this application, the target file is firstly recognized for text to obtain preliminary text recognition content; the text recognition content is then repaired through a large language model to repair errors in the text recognition content; finally, the text recognition content, the file image corresponding to the target file and the text repair result are processed through a recognition content correction model to further correct the text repair result and obtain the final recognition result of the target file; adopting the above scheme, this application proposes a recognition content correction model based on deep learning and the combination of language and image, which utilizes the excellent language understanding ability of a large language model and the information of the image itself to achieve better file scan image recognition content repair and obtain more accurate text recognition results; thereby solving the problem in the related art that the file images obtained by scanning paper documents often face problems such as low image contrast, tilted text, blur, etc., resulting in frequent errors in the recognized text content. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] Figure 1 It is a hardware structure block diagram of a computer terminal of a method for character recognition of a target file in an embodiment of the present application;

[0021] Figure 2 is a flow chart of a method for text recognition of a target file according to an embodiment of the present application;

[0022] Figure 3 is a flowchart of an optional text recognition method according to an embodiment of the present application;

[0023] Figure 4 is a schematic diagram of an architecture of an identification content correction model according to an embodiment of the present application;

[0024] Figure 5 It is a structural block diagram of a text recognition device for a target file according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] The method embodiments provided in the embodiments of the present application can be executed in a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 1 1 is a hardware structure block diagram of a computer terminal for a method for recognizing text in a target file according to an embodiment of the present application. Figure 1 As shown, the computer terminal may include one or more ( Figure 1Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor (Central Processing Unit, MCU) or a programmable logic device (Field Programmable Gate Array, FPGA)) and a memory 104 for storing data, wherein the above-mentioned computer terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0028] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the text recognition method of the target file in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0029] A wireless network provided by a communication provider of a computer terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0030] In this embodiment, a method for character recognition of a target file is provided. Figure 2 is a flow chart of a method for text recognition of a target file according to an embodiment of the present application. Figure 2 As shown, the process includes the following steps S202-S206:

[0031] Step S202: performing text recognition on the target file to obtain text recognition content;

[0032] It should be noted that the text recognition of the target file can be completed using OCR character recognition technology or other text recognition technologies, and this application does not impose any restrictions on this.

[0033] Step S204: repairing the text recognition content by using a large language model to obtain a text repair result, wherein the text repair result includes at least one sub-repair result;

[0034] Step S206: Processing the text recognition content, the file image corresponding to the target file and the text repair result through the recognition content correction model to obtain a final recognition result of the target file.

[0035] Through the above steps, the target file is firstly recognized for text to obtain preliminary text recognition content; the text recognition content is then repaired through the large language model to repair errors in the text recognition content; finally, the text recognition content, the file image corresponding to the target file and the text repair result are processed through the recognition content correction model to further correct the text repair result and obtain the final recognition result of the target file; adopting the above scheme, the present application proposes a recognition content correction model based on deep learning and the combination of language and image, which utilizes the excellent language understanding ability of the large language model and the information of the image itself to achieve better file scan image recognition content repair and obtain more accurate text recognition results; thereby solving the problem in the related art that the file images obtained by scanning paper documents often face problems such as low image contrast, tilted text, blur, etc., resulting in frequent errors in the recognized text content.

[0036] In an exemplary embodiment, before repairing the text recognition content through the large language model to obtain the text repair result, the method also includes: obtaining a first prompt template in the large language model, wherein the first prompt template is used to instruct the large language model to classify the text recognition content; filling the first prompt template according to the text recognition content to obtain a filled first prompt template; processing the filled first prompt template through the large language model to obtain a classification result of the text recognition content, wherein the classification result is used to indicate whether the text recognition content needs to be repaired; when the classification result indicates that the text recognition content needs to be repaired, repairing the text recognition content through the large language model to obtain the text repair result.

[0037] The analysis and judgment of content is realized by means of a large language model, which usually utilizes the excellent language understanding and question-answering capabilities of the large language model. There is a mainstream method called "prompt learning", which guides the large language model to output the desired results by constructing a prompt template. In this embodiment, in order to determine whether the text recognition content needs to be repaired, a set of prompt templates (i.e., the first prompt template mentioned above) is designed to identify whether the text recognition content is fluent. The template structure is shown in Table 1, where [CLS] is the sentence start marker, the content recognized by OCR (i.e., the text recognition content mentioned above) is filled in at <Recognition content>, and [SEP] is the sentence segmentation / segmentation marker. The recognition content is directly filled into the template and input into the large language model. If the large language model outputs "False" or contains "False", it means that the recognition content needs to be modified; if the large language model outputs "True" or contains "True", it means that the recognition content does not need to be modified and can be directly output for use. If the output of the large language model does not meet any of the above conditions, it means that the large language model does not match the language or its language understanding ability is insufficient to support the task, and another large language model needs to be replaced.

[0038] Table 1

[0039]

[0040] Based on the above steps, the text recognition content is repaired by the large language model to obtain the text repair result, including: obtaining a second prompt template in the large language model, wherein the second prompt template is used to instruct the large language model to repair the text recognition content; filling the second prompt template according to the text recognition content to obtain a filled second prompt template; and processing the filled second prompt template by the large language model to obtain the text repair result.

[0041] Repairing the text recognition content also uses the prompt learning method to guide the large language model to generate the content we need. In this embodiment, a set of prompt templates (i.e., the second prompt template mentioned above) is designed as shown in Table 2, where <separator> is used to separate the output content to facilitate subsequent automatic reading and processing. <separator> can be any symbol that is convenient for separation and not easily confused with the text content, such as "@", "&", etc. Fill the template with the recognition content and the separator, and the large language model will output the modified content (i.e., the text repair result mentioned above).

[0042] Table 2

[0043]

[0044] Optionally, the text recognition content, the file image corresponding to the target file, and the text repair result are processed by a recognition content correction model to obtain a final recognition result of the target file, including: respectively encoding the text recognition content, the file image, and the text repair result to obtain a first code, a second code, and a third code, wherein the first code corresponds to the text recognition content, the second code corresponds to the file image, and the third code corresponds to the text repair result; correcting the first code, the second code, and the third code by a co-attention module to obtain a corrected code; decoding the corrected code to obtain the final recognition result.

[0045] Although the large language model has proposed several content modification suggestions (i.e., at least one sub-repair result), since the large language model only infers and repairs from the possibility of language and logic, it may cause inconsistency with the meaning of the original sentence. For example, the original sentence is "I disagree with the proposal", and due to unclearness, stains, etc., OCR recognizes it as "I agree to the proposal". Then, after being repaired by the large language model, it may be repaired to "I agree to the proposal" for logical and semantic rationality, resulting in completely opposite content. In order to solve this problem, an embodiment of the present application proposes a recognition content correction model based on deep learning that combines language and image, and inputs the repaired content given by the large language model (i.e., the above-mentioned text repair result), the original recognition content (i.e., the above-mentioned text recognition content), and the original scanned file image (i.e., the above-mentioned file picture) together, so that the model understands the text semantics and image semantics to achieve better repair content output.

[0046] The overall processing flow of the recognition content correction model is as follows: first, the text recognition content, file image and text repair results are encoded separately to obtain the corresponding encoding results, where the encoding results are in matrix form; then the co-attention module enables the model to learn image-guided corrections by respectively representing the relationship between the text recognition content and the file image, and the relationship between the text repair result and the file image, and then combines this correction with the modified content output by the large language model and the text information of the original recognition content (that is, the above-mentioned text recognition content) to obtain the correction code H. Finally, with H as input, the decoding part of the Transformer model is used as a decoder, and the final output is the repaired recognition content, that is, the above-mentioned final recognition result.

[0047] Optionally, the text recognition content, the file image and the text repair result are encoded respectively to obtain a first code, a second code and a third code, including: encoding the text recognition content and the text repair result through a text content encoder to obtain a fourth code and a fifth code, and encoding the file image through an image encoder to obtain a sixth code, wherein the fourth code corresponds to the text recognition content, the fifth code corresponds to the text repair result, and the fifth code includes at least one sub-code, and the at least one sub-code corresponds one-to-one to the at least one sub-repair result; fusing the at least one sub-code through a fusion module to obtain a seventh code; and dimensionally converting the fourth code, the sixth code and the seventh code through a linear module to obtain the first code, the second code and the third code, wherein the first code, the second code and the third code have the same dimension.

[0048] Specifically, the process of encoding the text recognition content, the file image and the text repair result by the recognition content correction model includes: on the one hand, using the text content encoder to encode each sub-repair result in the text recognition content and the text repair result respectively to obtain the fourth code and the fifth code; and encoding the file image through the image encoder to obtain the sixth code. The n sub-codes corresponding to the n sub-repair results are fused through the fusion module, and the fusion formula is M′=W1Bert(m1)+W2Bert(m2)+...+W n Bert(m n )+b, where W1-W n and b are both trainable parameters. Afterwards, the dimensions of the encoded matrices of each input (i.e., the fourth encoding, the sixth encoding, and the seventh encoding) are unified through three linear modules to obtain F M (i.e. the third code), F G (i.e. the first code), F P (ie the second encoding).

[0049] Further, the first code, the second code and the third code are corrected by the co-attention module to obtain a corrected code, including: by the formula H=ReLU(WF M +WF G +(WF M )C M +(WF G )C G ) performs correction processing on the first code, the second code and the third code to obtain the corrected code, wherein H is the corrected code, ReLU is the activation function, W is the module parameter of the co-attention module, and F MFor the third code, F G For the first encoding, F P is the second encoding.

[0050] The co-attention module first represents the relationship between the file image and the text repair results separately The relationship between file images and text recognition content Then, by the formula H = ReLU (WF M +WF G +(WF M )C M +(WF G )C G ) enables the model to learn image-guided corrections, and combines these corrections with the text restoration results and text recognition content to obtain correction codes.

[0051] In an optional embodiment, the present application embodiment provides an optional text recognition method, the process is as follows: Figure 3 As shown, the following steps are included:

[0052] Step 1: Scan the paper document to obtain a scanned document image (i.e., the target document mentioned above);

[0053] Step 2: Perform OCR recognition (text recognition) on the scanned document image to obtain text recognition content;

[0054] Step 3: Use the large language model to determine whether the text recognition content needs to be repaired; if so, proceed to step 4; if not, proceed to step 6;

[0055] Step 4: Repair the text recognition content through the large language model;

[0056] Step 5: Process the repaired result through a recognition content correction model based on deep learning combining language and image to obtain the repaired recognition content (i.e., the above-mentioned final recognition result);

[0057] In this embodiment, the architecture of the recognition content correction model is as follows: Figure 4 As shown, the following combination Figure 4 The process of correcting text repair results by the recognition content correction model is described, which specifically includes the following steps:

[0058] Step 5.1, using the Bert model as a text content encoder, respectively encoding each modification result (i.e., the above-mentioned text repair result) and the original recognition content (i.e., the above-mentioned text recognition content); using the ResNet model as an image encoder, encoding the document scanned image (i.e., the above-mentioned document image);

[0059] Step 5.2: Since the text repair result M contains n modification results, and n is not a fixed value, a fusion module is designed to fuse the variable number of n modification results. The formula is M′=W1Bert(m1)+W2Best(m2)+...+W n Bert(m n )+b, where W1-W n and b are both trainable parameters;

[0060] Step 5.3: Use three linear modules to unify the dimensions of the encoded matrices of each input to obtain F M (i.e. the third code), F G (i.e. the first code), F P (i.e. the second code);

[0061] Step 5.4: In order to make the model learn the correlation between the modified content output by the large language model, the original recognition content and the scanned image of the file and make corrections, a co-attention module is designed. The co-attention module enables the model to learn the correction guided by the image by representing the relationship between the modified content output by the large language model and the scanned image and the original recognition content, respectively. Then, this correction is combined with the text information of the modified content output by the large language model and the meta-recognition content. The specific formula is as follows, where W is a trainable parameter:

[0062]

[0063] H = ReLU(WF M +WF G +(WF M )C M +(WF G )C G );

[0064] Step 5.5. Finally, take H (i.e. the above-mentioned corrected code) as input, use the decoding part of the Transformer model as the decoder, and the final output is the repaired recognition content (i.e. the above-mentioned final recognition result).

[0065] Step 6: Output the obtained recognition content.

[0066] Through this embodiment, a method for repairing the content of document scanned image recognition assisted by a large language model is proposed, in which the recognized text content is judged and modification suggestions are given with the help of a large language model; at the same time, a recognition content correction model based on deep learning language and image recognition is designed, and the modified content generated by the large language model and the original scanned document image content are used to achieve further recognition content correction.

[0067] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0068] In the present embodiment, a text recognition device for a target file is also provided, and the device is used to implement the above-mentioned embodiment and preferred implementation mode, and the descriptions thereof are omitted. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the equipment described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and contemplated.

[0069] Figure 5 : is a structural block diagram of a text recognition device for a target file according to an embodiment of the present application, the device comprising:

[0070] The text recognition module 52 is used to perform text recognition on the target file to obtain text recognition content;

[0071] A repair module 54, configured to repair the text recognition content by using a large language model to obtain a text repair result, wherein the text repair result includes at least one sub-repair result;

[0072] The correction module 56 is used to process the text recognition content, the file image corresponding to the target file and the text repair result through the recognition content correction model to obtain the final recognition result of the target file.

[0073] Through the above-mentioned device, the target file is firstly recognized for text to obtain preliminary text recognition content; then the text recognition content is repaired through the large language model, and the errors in the text recognition content are repaired; finally, the text recognition content, the file image corresponding to the target file and the text repair result are processed through the recognition content correction model, and the text repair result is further corrected to obtain the final recognition result of the target file; adopting the above-mentioned scheme, the present application proposes a recognition content correction model based on deep learning and the combination of language and image, which utilizes the excellent language understanding ability of the large language model and the information of the image itself to achieve better file scan image recognition content repair and obtain more accurate text recognition results; thereby solving the problem in the related art that the file image obtained by scanning paper documents often faces the problems of low image contrast, tilted text, blur, etc., resulting in frequent errors in the recognized text content.

[0074] Optionally, the above-mentioned repair module 54 is also used to obtain a first prompt template in the large language model, wherein the first prompt template is used to instruct the large language model to classify the text recognition content; fill the first prompt template according to the text recognition content to obtain the filled first prompt template; process the filled first prompt template through the large language model to obtain a classification result of the text recognition content, wherein the classification result is used to indicate whether the text recognition content needs to be repaired; when the classification result indicates that the text recognition content needs to be repaired, the text recognition content is repaired through the large language model to obtain the text repair result.

[0075] The analysis and judgment of content is realized by means of a large language model, which usually utilizes the excellent language understanding and question-answering capabilities of the large language model. There is a mainstream method called "prompt learning", which guides the large language model to output the desired results by constructing a prompt template. In this embodiment, in order to determine whether the text recognition content needs to be repaired, a set of prompt templates (i.e., the first prompt template mentioned above) is designed to identify whether the text recognition content is fluent. The template structure is shown in Table 1, where [CLS] is the sentence start marker, the content recognized by OCR (i.e., the text recognition content mentioned above) is filled in at <Recognition content>, and [SEP] is the sentence segmentation / segmentation marker. The recognition content is directly filled into the template and input into the large language model. If the large language model outputs "False" or contains "False", it means that the recognition content needs to be modified; if the large language model outputs "True" or contains "True", it means that the recognition content does not need to be modified and can be directly output for use. If the output of the large language model does not meet any of the above conditions, it means that the large language model does not match the language or its language understanding ability is insufficient to support the task, and another large language model needs to be replaced.

[0076] Optionally, the above-mentioned repair module 54 is also used to obtain a second prompt template in the large language model, wherein the second prompt template is used to instruct the large language model to repair the text recognition content; fill the second prompt template according to the text recognition content to obtain a filled second prompt template; and process the filled second prompt template through the large language model to obtain the text repair result.

[0077] Repairing the text recognition content also uses the prompt learning method to guide the large language model to generate the content we need. In this embodiment, a set of prompt templates (i.e., the second prompt template mentioned above) is designed as shown in Table 2, where <separator> is used to separate the output content to facilitate subsequent automatic reading and processing. <separator> can be any symbol that is convenient for separation and not easily confused with the text content, such as "@", "&", etc. Fill the template with the recognition content and the separator, and the large language model will output the modified content (i.e., the text repair result mentioned above).

[0078] Optionally, the correction module 56 is further used to encode the text recognition content, the file image and the text repair result respectively to obtain a first code, a second code and a third code, wherein the first code corresponds to the text recognition content, the second code corresponds to the file image, and the third code corresponds to the text repair result; the first code, the second code and the third code are corrected by the co-attention module to obtain a correction code; the correction code is decoded to obtain the final recognition result.

[0079] Although the large language model has proposed several content modification suggestions (i.e., at least one sub-repair result), since the large language model only infers and repairs from the possibility of language and logic, it may cause inconsistency with the meaning of the original sentence. For example, the original sentence is "I disagree with the proposal", and due to unclearness, stains, etc., OCR recognizes it as "I agree to the proposal". Then, after being repaired by the large language model, it may be repaired to "I agree to the proposal" for logical and semantic rationality, resulting in completely opposite content. In order to solve this problem, an embodiment of the present application proposes a recognition content correction model based on deep learning that combines language and image, and inputs the repaired content given by the large language model (i.e., the above-mentioned text repair result), the original recognition content (i.e., the above-mentioned text recognition content), and the original scanned file image (i.e., the above-mentioned file picture) together, so that the model understands the text semantics and image semantics to achieve better repair content output.

[0080] The overall processing flow of the recognition content correction model is as follows: first, the text recognition content, file image and text repair results are encoded separately to obtain the corresponding encoding results, where the encoding results are in matrix form; then the co-attention module enables the model to learn image-guided corrections by respectively representing the relationship between the text recognition content and the file image, and the relationship between the text repair result and the file image, and then combines this correction with the modified content output by the large language model and the text information of the original recognition content (that is, the above-mentioned text recognition content) to obtain the correction code H. Finally, with H as input, the decoding part of the Transformer model is used as a decoder, and the final output is the repaired recognition content, that is, the above-mentioned final recognition result.

[0081] Optionally, the correction module 56 is further used to encode the text recognition content and the text repair result through a text content encoder to obtain a fourth code and a fifth code, and to encode the file image through an image encoder to obtain a sixth code, wherein the fourth code corresponds to the text recognition content, the fifth code corresponds to the text repair result, and the fifth code includes at least one sub-code, and the at least one sub-code corresponds one-to-one with the at least one sub-repair result; the at least one sub-code is fused through a fusion module to obtain a seventh code; the fourth code, the sixth code and the seventh code are respectively dimensionally converted through a linear module to obtain the first code, the second code and the third code, wherein the first code, the second code and the third code have the same dimension.

[0082] Specifically, the process of encoding the text recognition content, the file image and the text repair result by the recognition content correction model includes: on the one hand, using the text content encoder to encode each sub-repair result in the text recognition content and the text repair result respectively to obtain the fourth code and the fifth code; and encoding the file image through the image encoder to obtain the sixth code. The n sub-codes corresponding to the n sub-repair results are fused through the fusion module, and the fusion formula is M′=W1Bert(m1)+W2Bert(m2)+...+W n Bert(m n )+b, where W1-W n and b are both trainable parameters. Afterwards, the dimensions of the encoded matrices of each input (i.e., the fourth encoding, the sixth encoding, and the seventh encoding) are unified through three linear modules to obtain F M (i.e. the third code), F G (i.e. the first code), F P (ie the second encoding).

[0083] Optionally, the correction module 56 is further configured to pass the formula H=ReLU(WF M +WF G +(WF M )C M +(WF G )C G ) performs correction processing on the first code, the second code and the third code to obtain the corrected code, wherein H is the corrected code, ReLU is the activation function, W is the module parameter of the co-attention module, and F M For the third code, F G For the first encoding, F P is the second encoding.

[0084] The co-attention module first represents the relationship between the file image and the text repair results separately The relationship between file images and text recognition content Then, by the formula H = ReLU (WF M +WF G +(WF M )C M +(WF G )C G ) enables the model to learn image-guided corrections, and combines these corrections with the text restoration results and text recognition content to obtain correction codes.

[0085] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when run.

[0086] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0087] S1, perform text recognition on the target file to obtain text recognition content;

[0088] S2, repairing the text recognition content by using a large language model to obtain a text repair result, wherein the text repair result includes at least one sub-repair result;

[0089] S3, processing the text recognition content, the file image corresponding to the target file and the text repair result through a recognition content correction model to obtain a final recognition result of the target file.

[0090] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0091] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0092] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0093] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0094] S1, perform text recognition on the target file to obtain text recognition content;

[0095] S2, repairing the text recognition content by using a large language model to obtain a text repair result, wherein the text repair result includes at least one sub-repair result;

[0096] S3, processing the text recognition content, the file image corresponding to the target file and the text repair result through a recognition content correction model to obtain a final recognition result of the target file.

[0097] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0098] An embodiment of the present application further provides a computer program product, comprising a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program product, and when the computer program is executed by a processor, the steps of the method described in each embodiment of the present application are implemented.

[0099] Optionally, in this embodiment, the above computer program may be configured to implement the following steps when executed by a processor:

[0100] S1, perform text recognition on the target file to obtain text recognition content;

[0101] S2, repairing the text recognition content by using a large language model to obtain a text repair result, wherein the text repair result includes at least one sub-repair result;

[0102] S3, processing the text recognition content, the file image corresponding to the target file and the text repair result through a recognition content correction model to obtain a final recognition result of the target file.

[0103] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0104] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0105] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for character recognition of a target file, characterized in that: include: Perform text recognition on the target file to obtain text recognition content; Repairing the text recognition content by using a large language model to obtain a text repair result, wherein the text repair result includes at least one sub-repair result; The text recognition content, the file image corresponding to the target file and the text repair result are processed by a recognition content correction model to obtain a final recognition result of the target file.

2. The method according to claim 1, characterized in that Before repairing the text recognition content by using a large language model to obtain a text repair result, the method further includes: Acquire a first prompt template in the large language model, wherein the first prompt template is used to instruct the large language model to classify the text recognition content; Filling the first prompt template according to the text recognition content to obtain a filled first prompt template; Processing the filled first prompt template through the large language model to obtain a classification result of the text recognition content, wherein the classification result is used to indicate whether the text recognition content needs to be repaired; When the classification result indicates that the text recognition content needs to be repaired, the text recognition content is repaired by using the large language model to obtain the text repair result.

3. The method according to claim 2, characterized in that Repairing the text recognition content by using the large language model to obtain the text repair result includes: Acquire a second prompt template in the large language model, wherein the second prompt template is used to instruct the large language model to repair the text recognition content; Filling the second prompt template according to the text recognition content to obtain a filled second prompt template; The filled second prompt template is processed by the large language model to obtain the text repair result.

4. The method according to claim 1, characterized in that: The text recognition content, the file image corresponding to the target file and the text repair result are processed by the recognition content correction model to obtain the final recognition result of the target file, including: Encoding the text recognition content, the file image, and the text repair result respectively to obtain a first code, a second code, and a third code, wherein the first code corresponds to the text recognition content, the second code corresponds to the file image, and the third code corresponds to the text repair result; Correcting the first code, the second code and the third code by using a co-attention module to obtain a corrected code; The correction code is decoded to obtain the final recognition result.

5. The method according to claim 4, characterized in that Encoding the text recognition content, the file image, and the text repair result respectively to obtain a first code, a second code, and a third code, including: Encoding the text recognition content and the text repair result by a text content encoder to obtain a fourth code and a fifth code, and encoding the file image by an image encoder to obtain a sixth code, wherein the fourth code corresponds to the text recognition content, the fifth code corresponds to the text repair result, and the fifth code includes at least one sub-code, and the at least one sub-code corresponds to the at least one sub-repair result in a one-to-one manner; Performing fusion processing on the at least one sub-code by a fusion module to obtain a seventh code; The fourth code, the sixth code and the seventh code are respectively converted in dimension by a linear module to obtain the first code, the second code and the third code, wherein the first code, the second code and the third code have the same dimension.

6. The method according to claim 4, characterized in that Correcting the first code, the second code, and the third code by a co-attention module to obtain a corrected code includes: By formula H = ReLU (WF M +WF G +(WF M )C M +(WF G )C G ) performs correction processing on the first code, the second code and the third code to obtain the corrected code, wherein H is the corrected code, ReLU is the activation function, W is the module parameter of the co-attention module, and F M For the third code, F G For the first encoding, F P is the second encoding.

7. A text recognition device for a target file, characterized in that: include: A text recognition module is used to perform text recognition on a target file to obtain text recognition content; A repair module, used to repair the text recognition content by using a large language model to obtain a text repair result, wherein the text repair result includes at least one sub-repair result; The correction module is used to process the text recognition content, the file image corresponding to the target file and the text repair result through the recognition content correction model to obtain the final recognition result of the target file.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method according to any one of claims 1 to 6 when executed.

9. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 6 through the computer program.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images