Text recognition method, device, electronic device, storage medium and program product
By using the first feature extraction module, the second feature extraction module and the feature decoding module in the text recognition model, the feature mapping and text line position representation of text images are solved, and the recognition speed is improved and accuracy.
Patent Information
- Application Number
- CN202210374079.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-11
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-04-11
AI Technical Summary
The prior art is difficult to effectively identify images containing multiple lines of text, and problems such as errors in splitting and low recognition accuracy.
Using a text recognition model including the first feature extraction module, the second feature extraction module and the feature decoding module, the first feature map of the text image and the characterization of the position of the Chinese text line is extracted to obtain text recognition output, and multiple lines of text are supported. Identification.
Accurate recognition of multi-line text images is achieved, the recognition speed and accuracy are improved, and the errors caused by manual splitting are avoided.
Smart Images

Figure CN114842463B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing, and in particular to a text recognition method, device, electronic device, storage medium and program product. Background Art
[0002] Text recognition is the process of identifying a sequence of characters in a picture with text (for Chinese, a character is a Chinese character, and for English, a character is a letter). Text recognition technology in the prior art is mostly used to recognize single-line text. Single-line text recognition is subject to many restrictions in actual application. For example, if you encounter a picture containing multiple lines of text, you need to split the picture into multiple picture blocks containing single-line text. If you use manual splitting, it will greatly affect the speed of image recognition. If you use an algorithm to split, splitting errors may occur, and ultimately the final recognition accuracy is affected due to the splitting. How to recognize images containing multiple lines of text is one of the technical problems in this field. Summary of the invention
[0003] According to one aspect of the present disclosure, a method for text recognition is provided, comprising:
[0004] Obtain the text image to be recognized;
[0005] Inputting the text image into a text recognition model; wherein the text recognition model comprises: a first feature extraction module, a second feature extraction module and a feature decoding module;
[0006] Extracting a first feature map of the text image by a first feature extraction module; wherein a height of the first feature map is greater than or equal to 1;
[0007] Extracting the representation of the text line position in the first feature map by a second feature extraction module to obtain a second feature map;
[0008] The second feature map is processed by the feature decoding module to obtain the text recognition output.
[0009] According to another aspect of the present disclosure, a text recognition device includes:
[0010] An acquisition module, used for acquiring a text image to be recognized;
[0011] An input module, used to input a text image into a text recognition model; wherein the text recognition model includes: a first feature extraction module, a second feature extraction module and a feature decoding module;
[0012] A first feature extraction module, used for the first feature extraction module, used to extract a first feature map of the text image through the first feature extraction module; wherein the height of the first feature map is greater than or equal to 1;
[0013] A second feature extraction module, configured to extract a representation of a text line position in the first feature map through the second feature extraction module to obtain a second feature map;
[0014] The feature decoding module is used to process the second feature map through the feature decoding module to obtain a text recognition output.
[0015] According to another aspect of the present disclosure, there is provided an electronic device, comprising:
[0016] Processor; and
[0017] Memory for storing programs,
[0018] The program includes instructions, and when the instructions are executed by a processor, the processor executes any one of the above methods.
[0019] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute any of the above methods.
[0020] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the computer program implements any of the above methods.
[0021] One or more technical solutions provided in the embodiments of the present disclosure can achieve the technical effect of recognizing text images containing multiple lines of text. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:
[0023] Figure 1 A flowchart of a method for text recognition according to an exemplary embodiment of the present disclosure is shown;
[0024] Figure 2 A flowchart of another text recognition method according to an exemplary embodiment of the present disclosure is shown;
[0025] Figure 3 A schematic block diagram of a text recognition device according to an exemplary embodiment of the present disclosure is shown;
[0026] Figure 4 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION
[0027] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0028] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0029] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0030] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0031] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0032] The following describes the solution of the present disclosure with reference to the accompanying drawings:
[0033] Figure 1 A flowchart of a method for text recognition according to an exemplary embodiment of the present disclosure is shown; the method comprises:
[0034] Step S101, obtaining a text image to be recognized;
[0035] Step S102, inputting the text image into a text recognition model; wherein the text recognition model includes: a first feature extraction module, a second feature extraction module and a feature decoding module;
[0036] Step S103, extracting a first feature map of the text image by a first feature extraction module; wherein the height of the first feature map is greater than or equal to 1;
[0037] Step S104, extracting the representation of the text line position in the first feature map by a second feature extraction module to obtain a second feature map;
[0038] Step S105, processing the second feature map through a feature decoding module to obtain a text recognition output.
[0039] The execution subject of the method disclosed herein may be a desktop computer, a tablet computer or other terminal with strong computing power. The application scenario of the method disclosed herein is natural scene text recognition, corresponding to CNN (Convolutional Neural Networks). The text image to be recognized may be paper or electronic. If it is paper, the data of the text image can be obtained by scanning or taking a photo. The data of the text image may be the RGB (Red, Green, Blue) data of each pixel on the text image. The present disclosure may have certain restrictions on the text image to be recognized according to the text recognition model used to recognize text. For example, the pixels of the text image cannot exceed the maximum number of pixels that the text recognition model can recognize, the number of text lines in the text image cannot exceed the maximum number of lines that the text recognition model can recognize at a time, and the number of words included in a single text line in the text image cannot exceed the maximum number of words supported by the text recognition model.
[0040] Since the height of the first feature map extracted by the present disclosure can be greater than 1, that is, the first feature map corresponds to a feature map of multiple rows of pixels. The text model in the present disclosure can extract multi-line text feature maps, and thus can be used to recognize multi-line text. The first feature map may include the following contents: height, number of channels, quantity, and width. The size of the first feature map is height × width, and the ratio to the text image to be recognized can be greater than 1. The number of channels indicates the number of types of feature values used for text recognition, and the quantity indicates the number of recognized texts. Each pixel in the feature map corresponding to the first feature map corresponds to an area of a specific size of the recognized text image. For example, the text image is an area of 32×160 (rows × columns), the feature map of the first feature map is 4x40 (rows × columns) pixels, and each pixel in the first feature map corresponds to an area of 8×4 (rows × columns) pixels in the text image.
[0041] As some embodiments, the first feature extraction module includes one or more residual blocks. The first feature extraction module can be a Resnet18 network. Resnet18 is composed of 4 blocks (residual blocks), each of which is composed of several convolution operations. The output of each block is the input of the next block. The ratio of the input size of the first feature extraction module (i.e., the area of the pixel points of the text image to be recognized) to the output size (the area of the feature map corresponding to the first feature vector) is set to R1. For example, the input size is 32×160 (rows×columns), the ratio is 4, and the output size is 8×40 (height×width). Each pixel point of the first feature map can be used to recognize a character, and the entire text recognition model can be trained to recognize text images including 8 lines of text or less. The ratio of the input size and the output size of the image extraction module disclosed in the present invention is adjusted by adjusting the step size of the convolution kernel movement.
[0042] The feature decoding module can obtain the probability matrix of the corresponding characters of each pixel point in the feature map corresponding to the first feature map according to the first feature map, and for a pixel, determine the probability character corresponding to the pixel point as the character of the pixel point to complete the text recognition of the text image.
[0043] The text recognition output can be the recognized text information or the text probability matrix corresponding to each pixel on the feature map. According to the text probability matrix, the text with the highest probability corresponding to each pixel on the feature map can be determined.
[0044] The text recognition model of the embodiment of the present disclosure provides a model framework for extracting graphic features of feature mapping, and the text recognition model including the link of extracting graphic features can more accurately recognize text images. In addition, the second feature map with enhanced graphic features also helps to determine the position information such as the text box where the text line is located, and distinguishes valid information from invalid information with the help of the position information, so that the text image can be recognized more accurately.
[0045] Specifically, the size of the convolution kernel used by the second feature extraction module is the same as the size of the feature map of the first feature map.
[0046] As some embodiments, the second feature extraction module is a plurality of equal-width convolutional layers connected in series.
[0047] For example, the second feature extraction module is three layers of equal-width convolutional layers connected in series.
[0048] As some embodiments, the text recognition model is trained in the following manner: obtaining multiple annotated first training samples; wherein at least some of the first training samples include text images carrying multiple text lines; the annotated first annotation information includes a binary image of the positions of all text lines in each first training sample and a character sequence corresponding to all text lines in each first training sample; obtaining a first position result corresponding to each first training sample through a second feature map of the text recognition model; compensating the second feature map according to an L2 norm loss function, a first position result corresponding to each first training sample, and a binary image of the positions of all text lines in each first training sample; obtaining a first recognition result corresponding to each first training sample through recognition by the text recognition model; and compensating the text recognition model according to each first recognition result, a character sequence corresponding to all text lines in each first training sample, and a CTC (Connectionist temporal classification) loss function.
[0049] Specifically, the character sequence is the characters actually contained in the text image to be recognized. The embodiment of the present disclosure provides a method for training a text recognition model, so that the trained text recognition model can recognize text more accurately.
[0050] As some embodiments, the text recognition model also includes a third feature extraction module, which, after extracting the first feature map of the text image through the first feature extraction module, further includes: enhancing the representation of at least part of the context information of the characters in the text line in the second feature map through the third feature extraction module to obtain a third feature map; wherein, processing the second feature map through the feature decoding module to obtain the text recognition output includes: processing the third feature map through the feature decoding module to obtain the text recognition output.
[0051] The third feature extraction module of the disclosed embodiment can enhance the context modeling feature, that is, combine long-range information and short-range information.
[0052] As some embodiments, the third feature extraction module includes at least two self-attention layers.
[0053] In practical applications, the third feature extraction module can process the vectors corresponding to all pixels in the feature map corresponding to the second feature map. This full calculation method has a large amount of calculation. Alternatively, the vectors of some pixels can be selected for processing.
[0054] As some embodiments, a third feature map is obtained by enhancing the representation of at least a portion of the context information of characters within a text line in a second feature map through a third feature extraction module, including: extracting context modeling characteristics of a multi-channel feature map of one or more rows of pixels in a feature map corresponding to the second feature map to obtain a multi-channel feature map of one or more pixel rows; and determining the multi-channel feature maps of all one or more pixel rows as a third feature map.
[0055] For example, the third feature extraction module may randomly extract one row out of every four rows of pixels in the feature map corresponding to the second feature map, and extract the context modeling characteristics of the multi-channel feature map of this row of pixels.
[0056] As an embodiment, a text recognition model including the second feature extraction module, the third feature extraction module, the first feature extraction module and the feature decoding module in the present disclosure is trained, and the loss function used may also be: L2 norm loss function and CTC loss function.
[0057] The present disclosure trains the model, and the sign of the end of training is the convergence of the loss function.
[0058] As some embodiments, before obtaining the text image to be recognized, the process further includes generating a dictionary based on all annotated character sequences in one or more annotated text images.
[0059] The disclosed embodiment can generate a dictionary based on the annotated character sequence in the annotated text image used for training. Therefore, the dictionary is modifiable, and the more training is performed, the more character types are included in the dictionary, and the more types of characters and texts can be recognized, providing better scalability. The dictionary in the disclosed embodiment can be fixed, or it can be obtained and expanded through training as described above.
[0060] As some embodiments, the convolution kernel size of the convolution layer of the feature decoding module is the same as the size of the feature map of the first feature map.
[0061] As some embodiments, obtaining the text recognition output corresponding to the second feature map through the feature decoding module includes obtaining the text recognition output corresponding to the second feature map according to the dictionary and the second feature map through the feature decoding module.
[0062] As an embodiment, the feature decoding module can determine the character probability matrix through the feature mapping (the second feature mapping or the third feature mapping) in the above text, that is, each pixel in the feature map corresponding to the feature mapping corresponds to the word with the highest probability in the dictionary, and this word is determined as the recognition result of the pixel point.
[0063] Figure 2A flowchart of another text recognition method according to an exemplary embodiment of the present disclosure is shown. In some embodiments,
[0064] Step S201, obtaining a text image to be recognized;
[0065] Step S202, inputting the text image into a text recognition model;
[0066] Step S203, extracting a first feature map of the text image by a first feature extraction module;
[0067] Step S204, extracting the representation of the text line position in the first feature map by a second feature extraction module to obtain a second feature map;
[0068] Step S205, extracting the context modeling characteristics of the second feature map by a third feature extraction module to obtain a third feature map;
[0069] Step S206: Obtain text recognition output corresponding to the second feature map and the third feature map according to the second feature map and the third feature map through the feature decoding module.
[0070] As some embodiments, the text recognition model also includes a position prediction module, which extracts the representation of the text line position in the first feature map through the second feature extraction module. After obtaining the second feature map, it also includes obtaining the center point score map of all text lines in the second feature map, the left endpoint score map of all text lines, and the right endpoint score map of all text lines through the position prediction module.
[0071] Specifically, the center point, left endpoint, and right endpoint in the text are all points in the feature map. The center point score map contains the center point of each text line, the left endpoint score map contains the left endpoint of each text line, and the right endpoint score map contains the right endpoint of each text line.
[0072] As some embodiments, the position prediction module includes multiple layers of convolutional layers connected in series and multiple layers of deconvolutional layers connected in series.
[0073] As some embodiments, the third feature map is processed by a feature decoding module to obtain text recognition output, including: through the feature decoding module, the text recognition output is obtained based on the center point score map of all text lines, the left endpoint score map of all text lines, the right endpoint score map of all text lines, and the third feature map.
[0074] As some embodiments, a feature decoding module is used to obtain text recognition output based on a center point score map of all text lines, a left endpoint score map of all text lines, a right endpoint score map of all text lines, and a third feature map, including: obtaining a first pixel point through which an oblique line passes through an oblique text line based on a center point score map of each text line and a left endpoint score map of each text line, or obtaining a second pixel point on the oblique line corresponding to the oblique text line based on a center point score map of each text line and a right endpoint score map of each text line through a feature decoding module; and obtaining a text recognition output based on the second pixel point and a third feature map.
[0075] As an embodiment, to determine the oblique line where the oblique text is located, the uppermost and lowermost of all left endpoints of the oblique text, and the uppermost and lowermost of all right endpoints can be found. The slope is determined based on the upper left endpoint and the lower right endpoint, or based on the lower left endpoint and the upper right endpoint. The specific position of the oblique line where the text line is located can be determined based on the position and slope of the upper left endpoint, the lower left endpoint, the upper right endpoint or the lower right endpoint in the feature map, and the location area where the recognized oblique text is located can be determined based on the specific position of the oblique line. The specific characters of the oblique text are then determined using data including context modeling information.
[0076] As some embodiments, the text recognition model is trained and obtained by: obtaining multiple annotated second training samples; wherein at least part of the second training samples include text images carrying multiple text lines; the annotated second annotation information includes a binary image of the positions of all text lines in each second training sample, the center point position of all text lines in each second training sample, the left boundary point position of all text lines in each second training sample, the right boundary point position of all text lines in each second training sample, and the character sequence corresponding to all text lines in each second training sample; obtaining the second position result corresponding to each second training sample through the second feature map of the text recognition model; compensating the second feature map according to the L2 norm loss function, each second position result and the binary image of the positions of all text lines in each second training sample; obtaining the center point score map of all text lines corresponding to each second training sample, the left endpoint score map of all text lines corresponding to each second training sample, and the right endpoint score map of all text lines corresponding to each second training sample through the position prediction module of the text recognition model; according to the center loss Focal Loss function, each of the center point positions, each of the left boundary point positions, each of the right boundary point positions, each of the center point score maps, each of the left endpoint score maps, and each of the right endpoint score maps are used to compensate the position prediction module; the second recognition result corresponding to each second training sample is obtained through the text recognition model recognition; the text recognition model is compensated according to each of the second recognition results, the character sequences corresponding to all text lines in each of the second training samples, and the CTC loss function.
[0077] Figure 3 A schematic block diagram of a text recognition device according to an exemplary embodiment of the present disclosure is shown, and the device 300 includes:
[0078] An acquisition module 301 is used to acquire a text image to be recognized;
[0079] An input module 302 is used to input a text image into a text recognition model; wherein the text recognition model includes: a first feature extraction module, a second feature extraction module and a feature decoding module;
[0080] A first feature extraction module 303, used for the first feature extraction module, used to extract a first feature map of the text image through the first feature extraction module; wherein the height of the first feature map is greater than or equal to 1;
[0081] A second feature extraction module 304, configured to extract the representation of the text line position in the first feature map through the second feature extraction module to obtain a second feature map;
[0082] The feature decoding module 305 is used to process the second feature map through the feature decoding module to obtain a text recognition output.
[0083] The exemplary embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to cause the electronic device to perform the method according to the embodiment of the present disclosure when executed by the at least one processor.
[0084] Exemplary embodiments of the present disclosure also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute the method according to an embodiment of the present disclosure.
[0085] The exemplary embodiments of the present disclosure also provide a computer program product, including a computer program, wherein the computer program is used to cause the computer to perform the method according to the embodiments of the present disclosure when executed by a processor of the computer.
[0086] Figure 4 The block diagram of an exemplary electronic device that can be used to implement an embodiment of the present disclosure is shown. The block diagram of an electronic device 400 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0087] like Figure 4 As shown, the electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the device 400 can also be stored. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.
[0088] A plurality of components in the electronic device 400 are connected to the I / O interface 405, including: an input unit 406, an output unit 407, a storage unit 408, and a communication unit 409. The input unit 406 may be any type of device capable of inputting information to the electronic device 400, and the input unit 406 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 407 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 408 may include, but is not limited to, a disk, an optical disk. The communication unit 409 allows the electronic device 400 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.
[0089] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above. For example, in some embodiments, all of the above methods may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 400 via ROM 402 and / or communication unit 409. In some embodiments, the computing unit 401 may be configured to perform all of the above methods in any other appropriate manner (e.g., by means of firmware).
[0090] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0091] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0092] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.
[0093] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0094] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0095] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.
Claims
1. A method for text recognition, characterized in that: include: Obtain the text image to be recognized; Inputting the text image into a text recognition model; wherein the text recognition model comprises: a first feature extraction module, a second feature extraction module and a feature decoding module; Extracting a first feature map of the text image by the first feature extraction module; wherein the height of the first feature map is greater than 1; Extracting the representation of the text line position in the first feature map by the second feature extraction module to obtain a second feature map; Processing the second feature map by the feature decoding module to obtain a text recognition output; The text recognition model further includes a third feature extraction module, and after the first feature extraction module extracts the first feature map of the text image, further includes: Extracting representations of at least part of the context information of characters in the text line in the second feature map by the third feature extraction module to obtain a third feature map; the third feature extraction module supports selecting vectors of a part of pixel points in the second feature map for processing; Wherein, processing the second feature map by the feature decoding module to obtain the text recognition output includes: processing the third feature map by the feature decoding module to obtain the text recognition output; The text recognition model further includes a position prediction module, and after extracting the representation of the text line position in the first feature map through the second feature extraction module and obtaining the second feature map, it also includes: Obtaining, by the position prediction module, a center point score map of all text lines in the second feature map, a left endpoint score map of all text lines, and a right endpoint score map of all text lines; The step of processing the third feature map by the feature decoding module to obtain the text recognition output includes: The text recognition output is obtained by the feature decoding module according to the center point score map of all text lines, the left endpoint score map of all text lines, the right endpoint score map of all text lines, and the third feature map, including: Through the feature decoding module, according to the center point score map of each text line and the left endpoint score map of each text line, or, according to the center point score map of each text line and the right endpoint score map of each text line, the first pixel point through which the oblique line where the oblique text line is located passes is obtained, and then the second pixel point corresponding to the oblique text line on the oblique line is obtained; wherein, the position area where the oblique text line is located is determined based on the specific position of the oblique line where the oblique text line is located, and the specific position of the oblique line where the oblique text line is located is determined based on the position and slope of the upper left endpoint, lower left endpoint, upper right endpoint or lower right endpoint of the oblique text line in the feature map; The text recognition output is obtained according to the second pixel point and the third feature map through the feature decoding module.
2. The method according to claim 1, characterized in that The step of enhancing the representation of at least part of the context information of characters in the text line in the second feature map by the third feature extraction module to obtain the third feature map includes: Extracting context modeling characteristics of a multi-channel feature map of one or more rows of pixels in a feature map corresponding to the second feature map to obtain a multi-channel feature map of one or more pixel rows; The multi-channel feature maps of all the one or more pixel rows are determined as the third feature map.
3. The method according to claim 1, characterized in that The first feature extraction module includes one or more residual blocks.
4. The method according to claim 1, characterized in that The second feature extraction module includes multiple layers of equal-width convolutional layers connected in series.
5. The method according to claim 1, characterized in that The third feature extraction module includes at least two self-attention layers.
6. The method according to claim 1, characterized in that The position prediction module includes multiple convolutional layers connected in series and multiple deconvolutional layers connected in series.
7. The method according to claim 1, characterized in that The convolution kernel size of the convolution layer of the feature decoding module is the same as the size of the feature map of the first feature map.
8. The method according to claim 1, characterized in that The text recognition model is trained in the following way: Acquire a plurality of annotated first training samples; wherein at least some of the first training samples include text images carrying a plurality of text lines; the annotated first annotation information includes a binary image of the positions of all text lines in each of the first training samples and a character sequence corresponding to all text lines in each of the first training samples; Obtaining a first position result corresponding to each first training sample through a second feature map of the text recognition model; Compensating the second feature map according to an L2 norm loss function, a first position result corresponding to each of the first training samples, and a binary image of positions of all text lines in each of the first training samples; Obtaining a first recognition result corresponding to each first training sample through the text recognition model; The text recognition model is compensated according to each of the first recognition results, character sequences corresponding to all text lines in each of the first training samples, and a connection time classification (CTC) loss function.
9. The method according to claim 1, characterized in that The text recognition model is trained in the following way: Acquire a plurality of annotated second training samples; wherein at least part of the second training samples include text images carrying a plurality of text lines; the annotated second annotation information includes a binary image of the positions of all text lines in each of the second training samples, the positions of the center points of all text lines in each of the second training samples, the positions of the left boundary points of all text lines in each of the second training samples, the positions of the right boundary points of all text lines in each of the second training samples, and a character sequence corresponding to all text lines in each of the second training samples; Obtaining a second position result corresponding to each second training sample through the second feature map of the text recognition model; Compensating the second feature map according to an L2 norm loss function, each of the second position results, and a binary image of the positions of all text lines in each of the second training samples; Obtaining, by means of the position prediction module of the text recognition model, a center point score map of all text lines corresponding to each second training sample, a left endpoint score map of all text lines corresponding to each second training sample, and a right endpoint score map of all text lines corresponding to each second training sample; Compensating the position prediction module according to the Focal Loss loss function, each of the center point positions, each of the left boundary point positions, each of the right boundary point positions, each of the center point score maps, each of the left endpoint score maps, and each of the right endpoint score maps; Obtaining a second recognition result corresponding to each second training sample through the text recognition model; The text recognition model is compensated according to each of the second recognition results, character sequences corresponding to all text lines in each of the second training samples, and a CTC loss function.
10. A device for text recognition, characterized in that: include: An acquisition module, used for acquiring a text image to be recognized; An input module, used for inputting the text image into a text recognition model; wherein the text recognition model comprises: a first feature extraction module, a second feature extraction module and a feature decoding module; the text recognition model also comprises a third feature extraction module and a position prediction module; A first feature extraction module, used for extracting a first feature map of the text image through the first feature extraction module; wherein the height of the first feature map is greater than 1; A second feature extraction module, configured to extract a representation of a text line position in the first feature map through the second feature extraction module to obtain a second feature map; A third feature extraction module, used to extract the representation of at least part of the context information of the characters in the text line in the second feature map through the third feature extraction module to obtain a third feature map; the third feature extraction module supports selecting a vector of a part of the pixel points in the second feature map for processing; A position prediction module, used to obtain a center point score map of all text lines in the second feature map, a left endpoint score map of all text lines, and a right endpoint score map of all text lines through the position prediction module; A feature decoding module, used for processing the second feature map by the feature decoding module to obtain a text recognition output, including: processing the third feature map by the feature decoding module to obtain the text recognition output; the processing of the third feature map by the feature decoding module to obtain the text recognition output includes: obtaining the text recognition output according to the center point score map of all text lines, the left endpoint score map of all text lines, the right endpoint score map of all text lines, and the third feature map by the feature decoding module, including: Through the feature decoding module, according to the center point score map of each text line and the left endpoint score map of each text line, or, according to the center point score map of each text line and the right endpoint score map of each text line, the first pixel point through which the oblique line where the oblique text line is located passes is obtained, and then the second pixel point corresponding to the oblique text line on the oblique line is obtained; wherein, the position area where the oblique text line is located is determined based on the specific position of the oblique line where the oblique text line is located, and the specific position of the oblique line where the oblique text line is located is determined based on the position and slope of the upper left endpoint, lower left endpoint, upper right endpoint or lower right endpoint of the oblique text line in the feature map; The text recognition output is obtained according to the second pixel point and the third feature map through the feature decoding module.
11. An electronic device, characterized in that: include: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 9.
12. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-9.
13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.
