Text extraction method and device, electronic equipment and storage medium

Abstract

The invention discloses a text extraction method and device, electronic equipment and a storage medium, belongs to a text recognition technology, and is used for improving the text processing efficiency and accuracy. The method comprises the following steps: identifying a first text image to obtain a first text in the first text image; the first text is input into a first model, a first target text of a target type in the first text and the position of the first target text in the first text are output through the first model, and the first model is trained through an attention mechanism of a Transform architecture.

Description

Technical Field

[0001] The present application belongs to the technical field of text recognition, and specifically relates to a text extraction method, device, electronic device and storage medium. Background Art

[0002] Optical Character Recognition (OCR) technology is a technology that converts written or printed text content into machine-editable text format through image processing technology. The core steps of this technology mainly include image preprocessing, character segmentation, feature extraction and character recognition. The development of OCR technology has made it possible to recognize text in various scenarios, from books and documents to road signs and advertisements, which has greatly promoted the process of automation and informatization. It is widely used in banking, file management, autonomous driving, and smart assistants.

[0003] However, currently text extraction is mostly performed through post-processing. A major drawback of this method is that even in the same field, the text content extracted by OCR is very similar, and it is difficult to extract corresponding texts of different types or complex text information. Summary of the invention

[0004] The embodiments of the present application provide a text extraction method, device, electronic device and storage medium, which can solve the problem that different types of text or complex text information are difficult to extract accordingly.

[0005] In a first aspect, an embodiment of the present application provides a text extraction method, the method comprising: recognizing a first text image to obtain a first text in the first text image; inputting the first text into a first model, and outputting a first target text of a target type in the first text and a position of the first target text in the first text through the first model, wherein the first model is trained through the attention mechanism of the Transformer architecture.

[0006] In a second aspect, an embodiment of the present application provides a text extraction device, which includes: a processing module, used to identify a first text image to obtain a first text in the first text image; an extraction module, used to input the first text into a first model, and output a first target text of a target type in the first text and the position of the first target text in the first text through the first model, wherein the first model is trained through the attention mechanism of the Transformer architecture.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

[0008] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0009] In an embodiment of the present application, a first text image is recognized to obtain a first text in the first text image; the first text is input into a first model, and the first model outputs a first target text of a target type in the first text and a position of the first target text in the first text, wherein the first model is trained through the attention mechanism of the Transformer architecture, and can extract the first target text of the target type in the first text, thereby improving text processing efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a flowchart of a text extraction method provided in an embodiment of the present application;

[0011] Figure 2 It is a flowchart of a model training provided in an embodiment of the present application;

[0012] Figure 3 is a structural schematic diagram of a text extraction device provided in an embodiment of the present application;

[0013] Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0014] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0015] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0016] In the following, in conjunction with the accompanying drawings, a text extraction method, device, electronic device and storage medium provided in the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

[0017] Figure 1 A text extraction method provided by an embodiment of the present application is shown, and the method can be executed by an electronic device, and the electronic device may include: a server and / or a terminal device. In other words, the method can be executed by software or hardware installed in the electronic device, and the method includes the following steps:

[0018] Step 102: Recognize the first text image to obtain the first text in the first text image.

[0019] Specifically, OCR recognition can be performed on the first text image to be recognized, and the first text in the first text image can be obtained from the first text image. In the embodiment of the present application, the first text can be saved in a text list.

[0020] Step 104: input the first text into a first model, and output a first target text of a target type in the first text and a position of the first target text in the first text through the first model.

[0021] Wherein, the first model is trained by the attention mechanism of the Transformer architecture. Specifically, the first text in the text list can be input into the first model, and the first model can be a BERT model, which is a natural language processing (Natural Language Processing, NLP) pre-training model based on the Transformers structure. In an embodiment of the present application, the first model can be pre-trained by the attention mechanism of the Transformer architecture, and the attention mechanism allows the first model to focus on key parts when processing texts and ignore irrelevant information, thereby improving processing efficiency and accuracy. In this way, after the first text is input into the first model, the attention mechanism can be used to output the first target text of the target type in the first text and the position of the first target text in the first text through the first model. In an embodiment of the present application, the target type can be set according to actual needs when training the first model.

[0022] A text extraction method provided in an embodiment of the present application obtains a first text in the first text image by recognizing a first text image; inputs the first text into a first model, and outputs a first target text of a target type in the first text and a position of the first target text in the first text through the first model, wherein the first model is trained through the attention mechanism of the Transformer architecture, and can solve the problem of difficulty in corresponding extraction for different types of text or complex text information, and can extract the first target text of the target type in the first text, thereby improving text processing efficiency and accuracy, reducing OCR post-processing work, and reducing the later manual work of extracting text content.

[0023] In one implementation, before the first text image is recognized to obtain the first text in the first text image, the method further includes: acquiring a second text image; recognizing the second text image to obtain the second text in the second text image, wherein the second text carries a first label for indicating a type and a second label for indicating a position in the second text; and inputting the second text into the first model to train the first model.

[0024] Specifically, before recognizing the first text image and obtaining the first text in the first text image, the first model needs to be trained. In an embodiment of the present application, a data set for training the first model can be obtained, and the data set includes second text images, and the number of second text images is not limited here.

[0025] Perform OCR recognition on the second text image, extract the second text in the second text image, and generate a text list. Perform text data word embedding vector on the second text, and the text data word embedding vector content includes text word embedding, classification label (its own type), and position label (its own position in the text list), so that the second text will carry a first label for indicating the type and a second label indicating the position in the second text. As an example, for example, the second text [A, B, C, D] in the text list, each first label of the second text is ["question", "answer", "other", "other"], at this time, the type of text A is question, the type of text B is answer, and the type of text C and D is other. Then the second text can be input into the first model and the first model can be trained. In this way, by setting the first label and the second label for the second text, the first model trained by the second text can recognize the type of the input text and the position in the text.

[0026] In one implementation, inputting the second text into the first model to train the first model includes:

[0027] Input the second text into the first model; process the second text through the first model to obtain an output vector; add the first label and the second label to the output vector; process the output vector through the attention mechanism, and output a second target text of the target type in the second text and the position of the second target text in the second text through the first model.

[0028] Specifically, the second text is input into the first model, and the first model uses the BERT model as a pre-trained model. Figure 2As shown, the input layer of the first model is each second text in the text list, such as sentence A, sentence B, sentence C, and sentence D. The second text is processed by the first model, that is, the input second text (sentence) is embedded once by the bert model to obtain an output vector. Then the first label cls and the second label cls are added to the output vector. The output vector is processed by the attention mechanism, and the output vector is processed by the encode_only module based on the attention mechanism, and finally the second target text of the target type in the second text and the position of the second target text in the second text are output by the first model. In this way, the first model trained by the encode_only module of the attention mechanism can solve the problem that it is difficult to extract correspondingly for different types of text or complex text information, and can extract the first target text of the target type in the first text, thereby improving the efficiency and accuracy of text processing, reducing the OCR post-processing work, and reducing the work of manual extraction of text content in the later stage.

[0029] In one implementation, before the first text image is recognized to obtain the first text in the first text image, the method further includes: in the event that the first text image is angularly offset, inputting the first text image into a second model so that the second model outputs the first text image after rotation correction according to the offset angle of the first text image.

[0030] Specifically, before performing OCR recognition on the first text image, the first text image can be input into the second model, and the second model is used to determine whether the first text image has an angle shift. The second model can be a text image regressor. In the case where the first text image has an angle shift, the first text image can be rotationally corrected according to the offset angle of the first text image by the second model, and then the first text image after rotation correction is output. If the first text image has no angle shift, the first text image can be directly OCR recognized. In an embodiment of the present application, no matter what angle the first text image has shifted, the first text image can be rotationally corrected according to the angle of the shift, not just a few simple directions (such as 90 degrees, 180 degrees, 270 degrees, these more typical directional deflections), so that the image presents a normal direction for OCR recognition, which improves the accuracy of the OCR recognition result and improves the efficiency and accuracy of text processing.

[0031] In one implementation, after acquiring the second text image, it also includes: based on the second text image, using the mean square error as the loss function and the offset angle of the second text image as the regression target, training the second model with a regression algorithm; in the event that the second text image has an angle offset, inputting the second text image into the second model so that the second model outputs the second text image after rotation correction according to the offset angle of the second text image.

[0032] In an embodiment of the present application, the second text image can be provided with a label of the degree of rotation, such as the degree of clockwise rotation, and the data set can be expanded by data enhancement technology. According to the second model whose backbone network is resnet, a regression algorithm is performed, and the second model can be a CNN model, using mean square error (MSE) as a loss function (loss), using Adam optimizer for training to make the model converge faster, and the regression target is the offset angle of the second text image, such as the clockwise offset angle. In the case where the second text image has an angle offset, the second text image is input into the second model, and for the offset angle, opencv is used to rotate the image, and the filling method is used to make it consistent with the original image size, so that the second model outputs the second text image after rotation correction according to the offset angle of the second text image. In this way, no matter what angle the text image is offset, the input text image can be rotated and corrected according to the offset angle by the trained second model, not just a few simple directions, so that the image presents a normal direction for OCR recognition, which improves the accuracy of the OCR recognition result and improves the efficiency and accuracy of text processing.

[0033] In one implementation, the target type includes a question and an answer corresponding to the question, the first target text includes a first sub-target text and a second sub-target text, and outputting the first target text of the target type in the first text and the position of the first target text in the first text through the first model includes: outputting the first sub-target text and the position of the first sub-target text in the first text whose target type is a question through the first model; and outputting the second sub-target text and the position of the second sub-target text in the first text whose target type is an answer through the first model.

[0034] In one implementation, the first model can be used to extract question-answer pairs in the first text, and text pairs with a target type of question and an answer corresponding to the question in the first text can be extracted. That is, the first sub-target text with a target type of question and the second sub-target text with a target type of answer are extracted. For example, "Question: Account. Answer: "******". Where "Account" and "******" are both the content of the first text, and the first sub-target text with a target type of question and the position of the first sub-target text in the first text can be output through the first model. The second sub-target text with a target type of answer in the first text and the position of the second sub-target text are output through the first model. Users can search for corresponding question keywords in the output results according to the questions they want to extract and find their corresponding answers. (For example: account opening contract, search for account number, find questions related to account number, and find the content of the answer according to the position of the answer to the question). In the embodiment of the present application, the first text is judged to be a question or an answer, and the position of the corresponding answer and question is given, which can reduce the OCR post-processing work, reduce the work of manual extraction of text content in the later stage, and improve the efficiency and accuracy of text processing.

[0035] It should be noted that the text extraction method provided in the embodiment of the present application can be executed by a text extraction device or a control module in the text extraction device for executing the text extraction method. In the embodiment of the present application, the text extraction device provided in the embodiment of the present application is described by taking the text extraction method executed by the text extraction device as an example.

[0036] Figure 3 Schematic diagram of the structure of the text extraction device according to the embodiment of the present application. Figure 3 As shown, the text extraction device 300 includes: a processing module 310 and an extraction module 320 .

[0037] The processing module 310 is used to recognize the first text image and obtain the first text in the first text image; the extraction module 320 is used to input the first text into a first model, and output a first target text of a target type in the first text and the position of the first target text in the first text through the first model, wherein the first model is trained through the attention mechanism of the Transformer architecture.

[0038] In one implementation, the processing module 310 is further used to obtain a second text image; recognize the second text image to obtain a second text in the second text image, wherein the second text carries a first label for indicating a type and a second label for indicating a position in the second text; and input the second text into the first model to train the first model.

[0039] In one implementation, the processing module 310 is also used to input the second text into the first model; process the second text through the first model to obtain an output vector; add the first label and the second label to the output vector; process the output vector through the attention mechanism, and output a second target text of the target type in the second text and the position of the second target text in the second text through the first model.

[0040] In one implementation, the processing module 310 is further used to input the first text image into the second model when the first text image is angularly offset, so that the second model outputs the first text image after rotation correction according to the offset angle of the first text image.

[0041] In one implementation, the processing module 310 is also used to train the second model through a regression algorithm based on the second text image, using the mean square error as a loss function and the offset angle of the second text image as a regression target; when the second text image has an angle offset, the second text image is input into the second model so that the second model outputs the second text image after rotation correction according to the offset angle of the second text image.

[0042] In one implementation, the target type includes a question and an answer corresponding to the question, the first target text includes a first sub-target text and a second sub-target text, and the extraction module 320 is used to output the first sub-target text and the position of the first sub-target text in the first text whose target type is a question through the first model; and output the second sub-target text and the position of the second sub-target text in the first text whose target type is an answer through the first model.

[0043] The text extraction device in the embodiment of the present application can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a palmtop computer, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc., and the non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0044] The text extraction device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0045] The text extraction device provided in the embodiment of the present application can achieve Figure 1 to Figure 2 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0046] like Figure 4 As shown, an embodiment of the present application further provides an electronic device 400, including a processor 401 and a memory 402, wherein the memory 402 stores a program or instruction that can be run on the processor 401, and when the program or instruction is executed by the processor 401, it implements: recognizing a first text image to obtain a first text in the first text image; inputting the first text into a first model, and outputting a first target text of a target type in the first text and the position of the first target text in the first text through the first model, wherein the first model is trained by the attention mechanism of the Transformer architecture.

[0047] In one implementation, before the first text image is recognized to obtain the first text in the first text image, a second text image is acquired; the second text image is recognized to obtain the second text in the second text image, wherein the second text carries a first label for indicating a type and a second label for indicating a position in the second text; and the second text is input into the first model to train the first model.

[0048] In one implementation, the second text is input into the first model; the second text is processed by the first model to obtain an output vector; the first label and the second label are added to the output vector; the output vector is processed by the attention mechanism, and a second target text of the target type in the second text and the position of the second target text in the second text are output by the first model.

[0049] In one implementation, before the first text image is recognized to obtain the first text in the first text image, in the case where the first text image is angularly offset, the first text image is input into a second model so that the second model outputs the first text image after rotation correction according to the offset angle of the first text image.

[0050] In one implementation, after acquiring the second text image, based on the second text image, the second model is trained with a regression algorithm using a mean square error as a loss function and an offset angle of the second text image as a regression target; in the event that the second text image has an angle offset, the second text image is input into the second model so that the second model outputs the second text image after rotation correction according to the offset angle of the second text image.

[0051] In one implementation, the target type includes a question and an answer corresponding to the question, the first target text includes a first sub-target text and a second sub-target text, and the first model is used to output the first sub-target text and the position of the first sub-target text whose target type is a question in the first text; and the first model is used to output the second sub-target text and the position of the second sub-target text whose target type is an answer in the first text.

[0052] The specific execution steps can refer to the various steps of the above-mentioned text extraction method embodiment, and can achieve the same technical effect. To avoid repetition, they will not be repeated here.

[0053] It should be noted that the electronic devices in the embodiments of the present application include: servers, terminals, or other devices except terminals.

[0054] The above electronic device structure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, the input unit may include a graphics processing unit (GPU) and a microphone, and the display unit may be configured with a display panel in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit includes a touch panel and at least one of other input devices. The touch panel is also called a touch screen. Other input devices may include, but are not limited to, a physical keyboard, function keys (such as volume control buttons, switch buttons, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0055] The memory can be used to store software programs and various data. The memory may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory may include a volatile memory or a non-volatile memory, or the memory may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DRRAM).

[0056] The processor may include one or more processing units; optionally, the processor integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to the operating system, user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor.

[0057] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned text extraction method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0058] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as ROM, RAM, magnetic disk or optical disk.

[0059] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0060] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0061] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A text extraction method, characterized in that: include: Recognize the first text image to obtain the first text in the first text image; The first text is input into a first model, and a first target text of a target type in the first text and a position of the first target text in the first text are output through the first model, wherein the first model is trained through an attention mechanism of a Transformer architecture.

2. The method according to claim 1, characterized in that Before the first text image is recognized to obtain the first text in the first text image, the method further includes: Acquire a second text image; Recognize the second text image to obtain a second text in the second text image, wherein the second text carries a first tag for indicating a type and a second tag indicating a position in the second text; The second text is input into the first model to train the first model.

3. The method according to claim 2, characterized in that The step of inputting the second text into the first model to train the first model includes: inputting the second text into the first model; Processing the second text by using the first model to obtain an output vector; adding the first label and the second label to the output vector; The output vector is processed by the attention mechanism, and a second target text of the target type in the second text and a position of the second target text in the second text are output by the first model.

4. The method according to claim 2, characterized in that: Before the first text image is recognized to obtain the first text in the first text image, the method further includes: In the case where the first text image is angularly offset, the first text image is input into the second model so that the second model outputs the first text image after rotation correction according to the offset angle of the first text image.

5. The method according to claim 4, characterized in that After acquiring the second text image, the method further includes: According to the second text image, using mean square error as a loss function and the offset angle of the second text image as a regression target, performing regression algorithm training on the second model; In the case where the second text image is angularly offset, the second text image is input into the second model so that the second model outputs the second text image after rotation correction according to the offset angle of the second text image.

6. The method according to claim 1, characterized in that The target type includes a question and an answer corresponding to the question, the first target text includes a first sub-target text and a second sub-target text, and outputting the first target text of the target type in the first text and the position of the first target text in the first text through the first model includes: Outputting, by the first model, the first sub-target text whose target type is a question and the position of the first sub-target text in the first text; The second sub-target text whose target type is an answer and the position of the second sub-target text in the first text are outputted through the first model.

7. A text extraction device, characterized in that: include: A processing module, configured to recognize the first text image and obtain the first text in the first text image; An extraction module is used to input the first text into a first model, and output a first target text of a target type in the first text and a position of the first target text in the first text through the first model, wherein the first model is trained through an attention mechanism of a Transformer architecture.

8. The device according to claim 7, characterized in that The processing module is further used to obtain a second text image; Recognize the second text image to obtain a second text in the second text image, wherein the second text carries a first tag for indicating a type and a second tag indicating a position in the second text; The second text is input into the first model to train the first model.

9. An electronic device, characterized in that: The method comprises a processor, a memory and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the text extraction method according to any one of claims 1 to 6.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the text extraction method according to any one of claims 1 to 6 are implemented.

Images