Text recognition method, device, electronic device and storage medium
By cropping and stitching the text images of natural scenes, and using feature extraction and coding modules to identify short text sub-images, the problem of difficulty and accuracy of long text recognition is solved, and a more efficient recognition process is achieved.
Patent Information
- Application Number
- CN202210375788.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-11
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-04-11
AI Technical Summary
In natural scene text recognition, as the text length increases, the recognition difficulty increases, the model training time is longer and the accuracy decreases, making it difficult to improve the accuracy of long text recognition and reduce the model training time.
By cropping the text images to be recognized, segmenting them into short text sub-images. Each sub-image is input into a text recognition model for recognition. After obtaining the sub-text lines, it is spliced to form the final text lines, and using the feature extraction module, encoder and feature decoding module Perform processing.
It reduces the training time of the text recognition model and improves the recognition accuracy, achieving faster and more accurate long text recognition, taking into account the accuracy of long text.
Smart Images

Figure CN114627464B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a text recognition method, device, electronic device and storage medium. Background Art
[0002] Natural scene text recognition is the process of identifying character sequences in images with text (for Chinese, a character is a Chinese character, for English, a character is a letter). It is a very challenging task. In addition to factors such as complex image background and lighting changes, the complexity of the recognition output space is also a major difficulty.
[0003] The difficulty of natural scene text recognition is closely related to the length of the text in the text image being recognized. As the length of the text increases, the difficulty of text recognition will increase rapidly, the training time of the text recognition model will increase greatly, and the accuracy of text recognition will also decrease significantly. How to improve the accuracy of natural scene long text recognition and reduce the training time of the long text recognition model is one of the technical problems in this field. Summary of the invention
[0004] According to one aspect of the present disclosure, a method for text recognition is provided, comprising:
[0005] Obtain the text image to be recognized;
[0006] The text image is cropped to obtain at least two sub-images; the number of characters contained in each sub-image is less than the number of characters that can be recognized by the text recognition model;
[0007] Input each sub-image into a text recognition model; wherein the text recognition model includes: a feature extraction module, an encoder, and a feature decoding module;
[0008] Extracting a first feature map of each sub-image by a feature extraction module;
[0009] Encoding each first feature map by an encoder to obtain a second feature map for each sub-image;
[0010] Processing each second feature map through a feature decoding module to obtain a sub-text line corresponding to each sub-image;
[0011] The sub-text lines corresponding to each sub-image are concatenated to obtain the text line corresponding to the text image.
[0012] According to another aspect of the present disclosure, there is provided a device for text recognition, comprising:
[0013] An acquisition module, used for acquiring a text image to be recognized;
[0014] A cropping module, used for cropping the text image to obtain at least two sub-images; the number of characters contained in each sub-image is less than the number of characters that can be recognized by the text recognition model;
[0015] An input module, used to input each sub-image into a text recognition model; wherein the text recognition model includes: a feature extraction module, an encoder, and a feature decoding module;
[0016] A feature extraction module, used to extract a first feature map of a text recognition model for each sub-image;
[0017] An encoder, configured to encode each first feature map to obtain a second feature map for each sub-image;
[0018] A feature decoding module, used for processing each second feature map to obtain a sub-text line corresponding to each sub-image;
[0019] The splicing module is used to splice the sub-text lines corresponding to each sub-image to obtain the text lines corresponding to the text image.
[0020] According to another aspect of the present disclosure, there is provided an electronic device, comprising:
[0021] Processor; and
[0022] Memory for storing programs,
[0023] The program includes instructions, and when the instructions are executed by a processor, the processor executes any one of the above methods.
[0024] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute any of the above methods.
[0025] One or more technical solutions provided in the embodiments of the present disclosure can crop a long text, recognize the cropped short texts, obtain recognition results of the short texts, and then splice the recognition results into a long text. Therefore, the text recognition model used to obtain the recognition results is a short text recognition model. The training time of the text recognition model is shorter, the recognition accuracy is higher, and the recognition results of the short texts can be spliced into a long text, taking into account the accuracy of long text recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:
[0027] Figure 1 A flowchart of a method for text recognition according to an exemplary embodiment of the present disclosure is shown;
[0028] Figure 2 The structure block diagram of the basic module 1 in the encoder of the Transformer model in the prior art is shown;
[0029] Figure 3 A schematic diagram showing a principle of determining a maximum pooling window according to an exemplary embodiment of the present disclosure is shown;
[0030] Figure 4 A principle block diagram of a text recognition device 400 according to an exemplary embodiment of the present disclosure is shown;
[0031] Figure 5 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION
[0032] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0033] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0034] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0035] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0036] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0037] Figure 1 A flowchart of a method for text recognition according to an exemplary embodiment of the present disclosure is shown, comprising:
[0038] Step S101, obtaining a text image to be recognized;
[0039] Step S102, cropping the text image to obtain at least two sub-images; the number of characters contained in each sub-image is less than the number of characters that can be recognized by the text recognition model;
[0040] Step S103, inputting each sub-image into a text recognition model; wherein the text recognition model includes: a feature extraction module, an encoder, and a feature decoding module;
[0041] Step S104, extracting a first feature map of each sub-image by a feature extraction module;
[0042] Step S105, encoding each first feature map by an encoder to obtain a second feature map for each sub-image;
[0043] Step S106, processing each second feature map by a feature decoding module to obtain a sub-text line corresponding to each sub-image;
[0044] Step S107, concatenating the sub-text lines corresponding to each sub-image to obtain the text lines corresponding to the text image.
[0045] The execution subject of the method disclosed in the present invention may be a desktop computer, a tablet computer or other terminal with strong computing power. The application scenario of the method disclosed in the present invention is natural scene text recognition, corresponding to CNN (Convolutional Neural Networks). The text image to be recognized may be paper or electronic. If it is paper, the text image can be obtained by scanning or taking a photo. The present invention is to crop the text image to be recognized for the purpose of improving the recognition speed and accuracy of the text recognition model, and crop the text image of the long text into a sub-image of the short text. The number of characters contained in the sub-image is less than the maximum recognition capability of the text recognition model. For example, the pixels of the sub-image may not exceed the maximum number of pixels that the text recognition model can recognize, the number of text lines in the sub-image may be one line, and the number of characters included in a single text line in the sub-image may not exceed the maximum number of characters in a single line supported by the text recognition model (for example, 10 characters). The text line may be straight or curved. If the number of characters on the initial text image exceeds 10, it can be cropped.
[0046] Since the embodiments of the present disclosure can crop a text image containing long text into a text image containing shorter text, it can reduce the recognition difficulty of the text recognition model, speed up the recognition speed and increase the recognition accuracy, and can splice the short texts recognized by the text recognition model, thereby achieving faster and more accurate recognition of long texts.
[0047] The embodiments of the present disclosure are mainly divided into the following parts:
[0048] (I) Text image cropping and splicing recognition results
[0049] In the disclosed embodiment, the cropping method and the splicing method of the text image correspond to each other.
[0050] As some embodiments, a text image is cropped to obtain at least two sub-images, including: cropping the text image into at least two sub-images without overlapping parts; splicing the sub-text lines corresponding to each sub-image to obtain the text lines corresponding to the text image, including: splicing the sub-text lines corresponding to each sub-image in the order in which each sub-image is input into a text recognition model to obtain the text lines corresponding to the text image.
[0051] In practical applications, after using the text model to recognize each sub-image in turn, the recognition results (sub-text lines) can be arranged in order according to the recognition order, and the recognition results can be spliced in the arrangement order.
[0052] As some embodiments, a text image is cropped to obtain at least two sub-images, including: cropping the text image into at least two sub-images without overlapping parts, and adding the same label to each sub-image; splicing the sub-text lines corresponding to each sub-image to obtain the text lines corresponding to the text image, including: splicing the sub-text lines corresponding to each sub-image containing the same label to obtain the text lines corresponding to the text image.
[0053] As some embodiments, a text image is cropped to obtain at least two sub-images, including: cropping the text image into at least two sub-images with overlapping portions; splicing the sub-text lines corresponding to each sub-image to obtain a text line corresponding to the text image, including: for any two of all the sub-text lines, if a predetermined number of characters at the left end of one sub-text line is the same as a predetermined number of characters of another sub-text line, then splicing the two sub-text lines to obtain a text line; or splicing the sub-text lines corresponding to each sub-image in the order in which each sub-image is input into a text recognition model to obtain a text line corresponding to the text image.
[0054] In practical applications, when cutting, two adjacent parts leave an overlapping part of a predetermined area, and a certain number of identical characters can be identified. After completing text recognition and obtaining text lines, these characters are used to determine whether two or more text lines should be spliced together. Therefore, the splicing method can be to determine the predetermined number of characters (for example, 2 characters) at the left and right ends of the text line. If it is found that the predetermined number of characters at the right end (end end) of a text line is the same as the predetermined number of characters at the left end of another text line, the two text lines should be spliced together. The splicing method can be that the repeated characters are located at the right end in front, the repeated characters are located at the left end in the back, and a duplicate part in the middle is deleted. For example, "The weather is good today" is cut into "Today's weather" and "The weather is good". When splicing, one of the repeated "weather" is deleted to obtain the result "The weather is good today".
[0055] (II) Recognition through text recognition model
[0056] Specifically, the relationship between multiple modules of the text recognition model can be serial.
[0057] (1) Feature extraction module
[0058] As some embodiments, the first feature extraction module includes one or more residual blocks; the feature decoding module includes a fully connected layer. The first feature extraction module can use a Resnet18 network, Resnet18 consists of 4 Block blocks (residual blocks), each block consists of several convolution operations, the output of each block is the input of the next block, and the first feature map is obtained after passing through the first feature extraction module. The size of the first feature map is 1 / 4 of the original input text image.
[0059] Specifically, the first feature map may include the following contents: height, number of channels, quantity, and width. The size of the first feature map is height × width, and the height can be 1 (corresponding to the recognition of straight text) or greater than 1 (corresponding to the recognition of curved text). The number of channels represents the number of types of feature values used for text recognition, and the quantity represents the number of recognized sub-images. In the disclosed embodiment, the length of the feature map of the first feature map can be 10, and each pixel on the feature map can be used to decode a word, corresponding to a predictive capability of recognizing up to 10 words in a row, and the height of the sub-image used for recognition can generally be set to 32, then the height is 8 after passing through the feature extraction module, which can be used to recognize a line of curved text.
[0060] (2) Encoder
[0061] The encoder in step S104 may be based on an encoder in a Transformer model. The Transformer model generally uses six basic modules, but in the embodiment of the present disclosure, four basic modules may be used to prevent overfitting.
[0062] As some embodiments, basic module one also includes: multi-head attention layers, point-by-point operations and normalization operations, and equal-width convolutions connected in series.
[0063] Part of the structure of the basic module one in the encoder of the embodiment of the present disclosure may be different from the basic module one commonly used in the Transformer model. Figure 2 The structure block diagram of the basic module 1 in the encoder of the Transformer model in the prior art is shown. The fully connected layer in the basic module 1 in the embodiment of the present disclosure can be changed to a 3*3 equal-width convolution layer.
[0064] The 3x3 equal-width convolution layer is a smaller window. The reason for changing the fully connected layer to a 3x3 equal-width convolution layer is that we need to obtain the contextual information of the local area: the relationship between long texts is not tight, and obtaining the contextual information of long texts is not as useful as obtaining the contextual information of the local area.
[0065] In addition, the positional encoding part of the encoder can be changed to adaptive two-dimensional positional encoding.
[0066] As some embodiments, the encoder includes one or more basic modules connected in series and a position encoding operation; the position encoding operation includes: obtaining a fourth feature map according to the first feature map; maximizing pooling the fourth feature map according to an adaptive window; obtaining the fourth feature map according to the first feature map, including: obtaining a two-dimensional encoding value of each pixel point of the feature map corresponding to the first feature map according to the first feature map; adding the two-dimensional position value of each pixel point to the first feature map one by one to obtain the fourth feature map; wherein obtaining the two-dimensional encoding value of any pixel point includes: obtaining the first sine-cosine code of the pixel point in units of columns, and obtaining the second sine-cosine code of the pixel point in units of rows; the first sine-cosine code and the second sine-cosine code are the two-dimensional encoding values of the pixel point;
[0067] The fourth feature map is subjected to maximum pooling according to an adaptive window, including: determining the window corresponding to the pixel; determining that for any pixel in the feature map, the range of the row corresponding to the window corresponding to the pixel is from the bottom row of the feature map to the row where the pixel is located, and the range of the column is from the rightmost column of the feature map to the column where the pixel is located; and using the window to perform a maximum pooling operation on the fourth feature map. The effect of maximum pooling is that because the feature map (fourth feature map) extracted each time is different, for each text image, after the maximum pooling operation, the obtained value is different, which can adjust the encoding weights of the horizontal and vertical positions to better represent the character position information.
[0068] Specifically, the sine and cosine functions for obtaining the two-dimensional code value in the embodiment of the present invention may be:
[0069]
[0070]
[0071] Among them, pos represents the position of the pixel in the feature map corresponding to the first feature code, and i represents the dimension.
[0072] The fourth feature map is a two-dimensional feature encoding with two-dimensional feature values added to the first feature map. The advantage of using an adaptive window to perform maximum pooling on the fourth feature map is that each pixel corresponds to a different pooling window, which can adjust the encoding weights of the horizontal and vertical positions and better represent the character position information.
[0073] Specifically, the maximum pooling method is to perform the maximum pooling operation from bottom to top for the columns and from right to left for the rows (taking the maximum value. Here, the maximum pooling window size is different for different positions. Figure 3 A schematic diagram of the principle of determining the maximum pooling window according to an exemplary embodiment of the present disclosure is shown. In terms of columns, for the pixel at the (0,0) position on the 8x8 feature map of the fourth feature map (the same as the feature map of the first feature map), the pooling window is 8x8, and for (1,1), the pooling window is 7x7 in size, which is to perform maximum pooling using the feature data after each pixel of the feature map.
[0074] (3) Feature Enhancement
[0075] As some embodiments, the text recognition model also includes a list feature acquisition module, which, after encoding each first feature map through an encoder and before processing each second feature map through a feature decoding module, also includes: acquiring the list feature vector of each second feature map through the list feature acquisition module, and then obtaining a third feature map of each sub-image; wherein, each second feature map is processed through the feature decoding module to obtain a sub-text row corresponding to each sub-image, including: processing each second feature map through the feature decoding module to obtain a sub-text row corresponding to each sub-image.
[0076] The function of the list feature acquisition module is to select and enhance features to improve the accuracy of text recognition.
[0077] As some embodiments, the column feature acquisition module includes at least one layer of many-to-one bidirectional long short-term memory network LSTM.
[0078] LSTM (Long short-term memory) can input multiple vectors contained in each column in sequence and finally obtain a feature vector for this column.
[0079] (4) Feature decoding module
[0080] As described above, the feature decoding module includes a fully connected layer. The fully connected layer of this embodiment can decode and obtain the recognition result of the cropped short text image (sub-image).
[0081] (III) Text recognition model training
[0082] As some embodiments, the text recognition model is trained in the following manner: obtaining training data including multiple labeled training samples; wherein the multiple training samples are obtained by cropping one or more training text images; the cropping is performed in a manner of cropping each training text image into at least two training samples with overlapping or non-overlapping parts; the number of characters contained in each training sample is less than the maximum number of characters that the text recognition model can process; the labeled content includes the characters of each training sample; and the text recognition model is trained for the first time using the training data and the connection time classification CTC loss function.
[0083] Specifically, the CTC loss function can be used to compare the recognition result of the cropped subtext with the actual characters of the annotated subtext to obtain a loss value. According to the loss value, the parameters in the text recognition model are adjusted to improve the recognition ability of the cropped short text.
[0084] As some embodiments, the text recognition model is first trained using training data and a connection time classification CTC loss function, including: obtaining a first recognition result corresponding to each training sample through the text recognition model; and compensating the text recognition model according to each first recognition result, the labeled characters of each training sample and the connection time classification CTC loss function.
[0085] The above embodiments provide specific methods for using training results, which can enable more effective training.
[0086] As some embodiments, the annotated content also includes labels; all training samples cropped from the same training text image have the same label; after the text recognition model is trained for the first time using the training data and the connection time classification CTC loss function, it also includes: splicing all first recognition results with the same label to obtain one or more second recognition results; and training the text recognition model for a second time using each second recognition result, the characters corresponding to each training text image, and the CTC loss function.
[0087] There are two aspects of the text recognition model that need to be trained: one is the ability to recognize short texts, and the other is the ability to accurately splice long texts. This embodiment provides a specific training method for accurately splicing long texts. In practical applications, the recognition ability training can be performed once, and the splicing ability training can be performed again. The two training methods can be performed alternately.
[0088] Figure 4 A principle block diagram of a text recognition device 400 according to an exemplary embodiment of the present disclosure is shown;
[0089] An acquisition module 401 is used to acquire a text image to be recognized;
[0090] A cropping module 402 is used to crop the text image to obtain at least two sub-images; the number of characters contained in each sub-image is less than the number of characters that can be recognized by the text recognition model;
[0091] An input module 403 is used to input each sub-image into a text recognition model; wherein the text recognition model includes: a feature extraction module, an encoder, and a feature decoding module;
[0092] A feature extraction module 404 is used to extract a first feature map of a text recognition model for each sub-image;
[0093] An encoder 405 is used to encode each first feature map to obtain a second feature map for each sub-image;
[0094] A feature decoding module 406 is used to process each second feature map to obtain a sub-text line corresponding to each sub-image;
[0095] The splicing module 407 is used to splice the sub-text lines corresponding to each sub-image to obtain the text lines corresponding to the text image.
[0096] The exemplary embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to cause the electronic device to perform the method according to the embodiment of the present disclosure when executed by the at least one processor.
[0097] The exemplary embodiments of the present disclosure also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform the method according to the embodiments of the present disclosure.
[0098] The exemplary embodiments of the present disclosure further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform the method according to the embodiments of the present disclosure.
[0099] refer to Figure 5 , a block diagram of an electronic device 500 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0100] Figure 5 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. Figure 5 As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.
[0101] A plurality of components in the electronic device 500 are connected to the I / O interface 505, including: an input unit 506, an output unit 507, a storage unit 508, and a communication unit 509. The input unit 506 may be any type of device capable of inputting information to the electronic device 500, and the input unit 506 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 507 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 504 may include, but is not limited to, a disk, an optical disk. The communication unit 509 allows the electronic device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.
[0102] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above. For example, in some embodiments, all of the above methods may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 500 via ROM 502 and / or communication unit 509. In some embodiments, the computing unit 501 may be configured to perform all of the above methods in any other appropriate manner (e.g., by means of firmware).
[0103] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0104] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0105] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.
[0106] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0107] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0108] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.
Claims
1. A method for text recognition, characterized in that: include: Obtain the text image to be recognized; Cropping the text image to obtain at least two sub-images; The number of characters contained in each of the sub-images is less than the number of characters that can be recognized by the text recognition model; Input each of the sub-images into the text recognition model; wherein the text recognition model comprises: a feature extraction module, an encoder, and a feature decoding module; Extracting a first feature map of each of the sub-images by the feature extraction module; Encoding each of the first feature maps by the encoder to obtain a second feature map for each of the sub-images; Processing each of the second feature maps by the feature decoding module to obtain a sub-text line corresponding to each of the sub-images; splicing the sub-text lines corresponding to each of the sub-images to obtain the text lines corresponding to the text image; The encoder includes one or more basic modules connected in series and a position encoding operation; the position encoding operation includes: obtaining a fourth feature map according to the first feature map; performing maximum pooling on the fourth feature map according to an adaptive window; The obtaining of the fourth feature map according to the first feature map includes: obtaining the two-dimensional code value of each pixel point of the feature map corresponding to the first feature map according to the first feature map; adding the two-dimensional code value of each pixel point to the first feature map one by one to obtain the fourth feature map; wherein obtaining the two-dimensional code value of any one of the pixel points includes: obtaining the first sine-cosine code of the pixel point in units of columns, and obtaining the second sine-cosine code of the pixel point in units of rows; the first sine-cosine code and the second sine-cosine code are the two-dimensional code values of the pixel point; The maximizing pooling of the fourth feature map according to the adaptive window includes: determining a window corresponding to a pixel point, wherein, for any pixel point in the fourth feature map, a range of rows corresponding to the window corresponding to the pixel point is from the bottom row of the fourth feature map to the row where the pixel point is located, and a range of columns is from the rightmost column of the fourth feature map to the column where the pixel point is located; and using the window to perform a maximum pooling operation on the fourth feature map.
2. The method according to claim 1, characterized in that The text recognition model further includes a list feature acquisition module, which, after encoding each of the first feature maps by the encoder and before processing each of the second feature maps by the feature decoding module, further includes: Acquire the list feature vector of each second feature map through the list feature acquisition module, and then obtain the third feature map of each sub-image; The step of processing each of the second feature maps by the feature decoding module to obtain a sub-text line corresponding to each of the sub-images includes: Each of the third feature maps is processed by the feature decoding module to obtain a sub-text line corresponding to each of the sub-images.
3. The method according to claim 1, characterized in that The step of cropping the text image to obtain at least two sub-images includes: Cutting the text image into at least two sub-images without overlapping parts; The step of splicing the sub-text lines corresponding to each of the sub-images to obtain the text line corresponding to the text image includes: The sub-text lines corresponding to each of the sub-images are concatenated in the order in which each of the sub-images is input into the text recognition model to obtain the text lines corresponding to the text image.
4. The method according to claim 1, characterized in that The step of cropping the text image to obtain at least two sub-images includes: Cropping the text image into at least two sub-images without overlapping parts, and adding the same label to each of the sub-images; The step of splicing the sub-text lines corresponding to each of the sub-images to obtain the text line corresponding to the text image includes: The sub-text lines corresponding to each of the sub-images containing the same label are concatenated to obtain the text lines corresponding to the text image.
5. The method according to claim 1, characterized in that The step of cropping the text image to obtain at least two sub-images includes: Cropping the text image into at least two sub-images having overlapping portions; The step of splicing the sub-text lines corresponding to each of the sub-images to obtain the text line corresponding to the text image includes: For any two of all the sub-text lines, if a predetermined number of characters at the left end of one of the sub-text lines is the same as a predetermined number of characters of another sub-text line, the two sub-text lines are concatenated to obtain the text line; or The sub-text lines corresponding to each of the sub-images are concatenated in the order in which each of the sub-images is input into the text recognition model to obtain the text lines corresponding to the text image.
6. The method according to claim 1, characterized in that The basic module 1 also includes: The multi-head attention sublayer, point-by-point operation sublayer, normalization operation sublayer and equal-width convolution sublayer are connected in series.
7. The method according to claim 1, characterized in that The feature extraction module includes one or more residual blocks; the feature decoding module includes a fully connected layer.
8. The method according to claim 2, characterized in that The list feature acquisition module includes at least one layer of many-to-one bidirectional long short-term memory network LSTM.
9. The method according to claim 1, characterized in that The text recognition model is trained in the following way: Acquire training data including a plurality of annotated training samples; wherein the plurality of training samples are obtained by cropping one or more training text images; the cropping is performed by cropping each of the training text images into at least two training samples with overlapping or non-overlapping parts; the number of characters contained in each of the training samples is less than the maximum number of characters that the text recognition model can process; and the annotated content includes the characters of each of the training samples; The text recognition model is trained for the first time using the training data and a connection time classification (CTC) loss function.
10. The method according to claim 9, characterized in that The first training of the text recognition model using the training data and the connection time classification CTC loss function includes: Obtaining a first recognition result corresponding to each of the training samples through the text recognition model; The text recognition model is compensated according to each of the first recognition results, the labeled characters of each training sample and the connection time classification CTC loss function.
11. The method according to claim 10, characterized in that The annotated content also includes a label; all the training samples cropped from the same training text image have the same label; After the text recognition model is trained for the first time using the training data and the connection time classification CTC loss function, the method further includes: concatenating all the first recognition results having the same label to obtain one or more second recognition results; The text recognition model is trained for a second time using each of the second recognition results, the characters corresponding to each of the training text images, and the CTC loss function.
12. An electronic device, characterized in that: include: processor; as well as Memory for storing programs, The program comprises instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 11.
13. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: in, The computer instructions are used to make a computer execute the method according to any one of claims 1-11.
Citation Information
Patent Citations
Text recognition method, device and equipment
CN111695385A
Text recognition method and device, equipment and medium
CN113283427A
