A text recognition method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a text recognition method and device, electronic equipment and storage medium, and relate to the technical field of image processing. The method comprises: obtaining a to-be-recognized image; text content is displayed in the to-be-recognized image; inputting the to-be-recognized image into a pre-trained text recognition model to obtain a probability matrix corresponding to each image region in the to-be-recognized image output by the text recognition model; each column vector in the probability matrix corresponds to an image region; the elements in the vector include a first element representing the probability that the corresponding image region does not exist characters, and a second element representing the probability that the characters in the corresponding image region are preset characters when the corresponding image region exists characters; and obtaining the text content in the to-be-recognized image based on the probability matrix corresponding to each image region in the to-be-recognized image, which can avoid the situation that the user operation is complicated and improve the efficiency of the electronic equipment in obtaining the text content in the to-be-recognized image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a text recognition method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of image processing technology, in scenarios such as document digitization, form processing, and note conversion, paper with written text can be scanned to obtain an image displaying the text content. Then, when the device needs to process the text content in this image, such as copying text content, it first needs to acquire the text content from the image. At this point, the user needs to read the text content from the image and manually input it into the device.

[0003] However, having users manually input the text content from the image into the device is cumbersome for users, and the device is not very efficient at acquiring the text content from the image. Summary of the Invention

[0004] The purpose of this invention is to provide a text recognition method, apparatus, electronic device, and storage medium to avoid cumbersome user operations and improve the efficiency of electronic devices in acquiring text content from images to be recognized. The specific technical solution is as follows:

[0005] In a first aspect of the present invention, a text recognition method is provided, the method comprising: acquiring an image to be recognized; wherein the image to be recognized displays text content; inputting the image to be recognized into a pre-trained text recognition model to obtain a probability matrix corresponding to each image region in the image to be recognized output by the text recognition model; wherein each column vector in the probability matrix corresponds to an image region; the elements in the vector include: a first element representing the probability that the corresponding image region does not contain a character, and a second element representing the probability that the character in the corresponding image region is a preset character when the character exists in the corresponding image region; and obtaining the text content in the image to be recognized based on the probability matrix corresponding to each image region in the image to be recognized.

[0006] Optionally, the text recognition model includes a feature extraction network and a probability prediction network; the step of inputting the image to be recognized into the pre-trained text recognition model to obtain the probability matrix corresponding to each image region in the image to be recognized output by the text recognition model includes: inputting the image to be recognized into the feature extraction network to obtain the target image features of each image region in the image to be recognized output by the feature extraction network; wherein, the target image feature of an image region is: a target feature vector used to represent the content displayed in the image region; inputting the target image features of each image region in the image to be recognized into the probability prediction network to obtain the probability matrix corresponding to each image region in the image to be recognized output by the probability prediction network.

[0007] Optionally, the feature extraction network includes a convolutional module and a self-attention module; the step of inputting the image to be identified into the feature extraction network to obtain the target image features of each image region in the image to be identified output by the feature extraction network includes: inputting the image to be identified into the convolutional module, performing feature extraction on the image to be identified through the convolutional module to obtain the first image features of each image region in the image to be identified; inputting the first image features of each image region in the image to be identified into the self-attention module, and processing the first image features of each image region in the image to be identified according to the self-attention mechanism through the self-attention module to obtain the target image features of each image region in the image to be identified.

[0008] Optionally, the convolution module includes a deformable convolution submodule; the step of extracting features from the image to be identified through the convolution module to obtain the first image features of each image region in the image to be identified includes: for each image region in the image to be identified, performing convolution processing on the image region through the deformable convolution submodule to obtain the offset corresponding to the image region; determining the offset region corresponding to the image region according to the offset corresponding to the image region, and extracting features from the offset region through the deformable convolution submodule to obtain the first image features of the image region.

[0009] Optionally, the self-attention module includes: a masking submodule, a position determination submodule, and a self-attention submodule; the step of inputting the first image features of each image region in the image to be identified into the self-attention module, and processing the first image features of each image region in the image to be identified according to the self-attention mechanism to obtain the target image features of each image region in the image to be identified, includes: inputting the first image features of each image region in the image to be identified into the masking submodule, and replacing the first image features of a specified image region in each image region with a preset feature vector through the masking submodule to obtain the second image features of each image region output by the masking submodule. The second image features of each image region output by the masking submodule are input to the position determination submodule. Based on the positional relationship of each image region, the position determination submodule obtains the third image features of each image region output by the position determination submodule. The third image features represent the arrangement order of the corresponding image region with other image regions. The third image features of each image region output by the position determination submodule are input to the self-attention submodule. The self-attention submodule determines the weight of each element in the third image features of each image region. Based on the weight of each element in the third image features of each image region, the target image features of each image region in the image to be identified are determined.

[0010] Optionally, the self-attention submodule includes: a self-attention unit, a position feedforward unit, and a residual connection unit; the position feedforward unit includes multiple linear subunits; the step of inputting the third image features of each image region output by the position determination submodule into the self-attention submodule, determining the weight of each element in the third image features of each image region through the self-attention submodule, and determining the target image features of each image region in the image to be identified based on the weight of each element in the third image features of each image region, includes: inputting the third image features of each image region output by the position determination submodule into the self-attention unit, determining the weight of each element in the third image features of each image region through the self-attention unit, and calculating the product of the weight of each element in the third image features of each image region and the third image features of that image region to obtain the self-attention. The force unit outputs the fourth image features of each image region in the image to be identified; the self-attention unit outputs the fourth image features of each image region in the image to be identified and inputs them into the position feedforward unit. The position feedforward unit performs a linear transformation on the fourth image features of each image region in the image to be identified through multiple linear sub-units to obtain the fifth image features of each image region in the image to be identified output by the position feedforward unit; the position feedforward unit outputs the fifth image features of each image region in the image to be identified and inputs them into the residual connection unit. For each image region in the image to be identified, the residual connection unit calculates the sum of the fourth image features and the fifth image features of the image region and normalizes the calculation results to obtain the target image features of each image region in the image to be identified.

[0011] Optionally, obtaining the text content in the image to be identified based on the probability matrix corresponding to each image region in the image to be identified includes: for each image region in the image to be identified, if the first element of the column vector corresponding to the image region in the probability matrix is ​​greater than a preset threshold, it is determined that the image region does not contain characters; if the first element of the column vector corresponding to the image region in the probability matrix is ​​not greater than the preset threshold, it is determined that the image region contains characters; for each image region in the image to be identified containing characters, a preset character corresponding to the largest second element in the column vector corresponding to the image region in the probability matrix is ​​determined, and a preset character in the image region is obtained; based on the preset characters in each image region containing characters in the image to be identified, the text content in the image to be identified is obtained.

[0012] Optionally, obtaining the image to be identified includes: obtaining an original image and extracting the smallest image region occupied by each text line in the original image to obtain the image to be identified.

[0013] In a second aspect of the present invention, a text recognition device is provided, the device comprising:

[0014] An image acquisition module is used to acquire an image to be recognized; wherein the image to be recognized displays text content;

[0015] The probability matrix acquisition module is used to input the image to be recognized into a pre-trained text recognition model to obtain the probability matrix corresponding to each image region in the image to be recognized output by the text recognition model; wherein, each column vector in the probability matrix corresponds to an image region; the elements in the vector include: a first element representing the probability that the corresponding image region does not contain a character, and a second element representing the probability that the character in the corresponding image region is a preset character when the character exists in the corresponding image region;

[0016] The text content acquisition module is used to obtain the text content in the image to be identified based on the probability matrix corresponding to each image region in the image to be identified.

[0017] Optionally, the text recognition model includes a feature extraction network and a probability prediction network; the probability matrix acquisition module is specifically used for: inputting the image to be recognized into the feature extraction network to obtain the target image features of each image region in the image to be recognized output by the feature extraction network; wherein, the target image feature of an image region is: a target feature vector used to represent the content displayed in the image region; inputting the target image features of each image region in the image to be recognized into the probability prediction network to obtain the probability matrix corresponding to each image region in the image to be recognized output by the probability prediction network.

[0018] Optionally, the feature extraction network includes a convolutional module and a self-attention module; the probability matrix acquisition module is specifically used for: inputting the image to be identified into the convolutional module, extracting features from the image to be identified through the convolutional module to obtain first image features of each image region in the image to be identified; inputting the first image features of each image region in the image to be identified into the self-attention module, and processing the first image features of each image region in the image to be identified according to the self-attention mechanism through the self-attention module to obtain target image features of each image region in the image to be identified.

[0019] Optionally, the convolution module includes a deformable convolution submodule; the probability matrix acquisition module is specifically used for: performing convolution processing on each image region in the image to be identified through the deformable convolution submodule to obtain the offset corresponding to the image region; determining the offset region corresponding to the image region according to the offset corresponding to the image region, and extracting features from the offset region through the deformable convolution submodule to obtain the first image feature of the image region.

[0020] Optionally, the self-attention module includes: a masking submodule, a position determination submodule, and a self-attention submodule; the probability matrix acquisition module is specifically used for: inputting the first image features of each image region in the image to be identified into the masking submodule, and through the masking submodule, replacing the first image features of a specified image region in each image region with a preset feature vector to obtain the second image features of each image region output by the masking submodule; inputting the second image features of each image region output by the masking submodule into the position determination submodule, and through the position determination submodule, obtaining the third image features of each image region output by the position determination submodule based on the positional relationship of each image region; wherein, the third image feature represents: the arrangement order of the corresponding image region with other image regions; inputting the third image features of each image region output by the position determination submodule into the self-attention submodule, and through the self-attention submodule, determining the weight of each element in the third image features of each image region, and determining the target image features of each image region in the image to be identified based on the weight of each element in the third image features of each image region.

[0021] Optionally, the self-attention submodule includes: a self-attention unit, a position feedforward unit, and a residual connection unit; the position feedforward unit includes multiple linear subunits; the probability matrix acquisition module is specifically used for: inputting the third image features of each image region output by the position determination submodule into the self-attention unit, determining the weight of each element in the third image features of each image region through the self-attention unit, and calculating the product of the weight of each element in the third image features of each image region and the third image features of that image region to obtain the fourth image features of each image region in the image to be recognized output by the self-attention unit; and inputting the third image features of each image region in the image to be recognized output by the self-attention unit into the image region... The fourth image feature is input to the position feedforward unit. The position feedforward unit performs a linear transformation on the fourth image feature of each image region in the image to be identified through multiple linear subunits, thereby obtaining the fifth image feature of each image region in the image to be identified output by the position feedforward unit. The fifth image feature of each image region in the image to be identified output by the position feedforward unit is input to the residual connection unit. For each image region in the image to be identified, the residual connection unit calculates the sum of the fourth image feature and the fifth image feature of the image region, and normalizes the calculation result to obtain the target image feature of each image region in the image to be identified.

[0022] Optionally, the text content acquisition module is specifically used for: for each image region in the image to be recognized, if the first element of the column vector corresponding to the image region in the probability matrix is ​​greater than a preset threshold, determining that the image region does not contain characters; if the first element of the column vector corresponding to the image region in the probability matrix is ​​not greater than the preset threshold, determining that the image region contains characters; for each image region in the image to be recognized containing characters, determining the preset character corresponding to the largest second element in the column vector corresponding to the image region in the probability matrix, and obtaining the preset character in the image region; and obtaining the text content in the image to be recognized based on the preset characters in each image region containing characters in the image to be recognized.

[0023] Optionally, the image acquisition module is specifically used to: acquire an original image and extract the smallest image region occupied by each text line in the original image to obtain the image to be recognized.

[0024] In a third aspect of the present invention, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is used to store computer programs; and the processor is used to implement any of the steps of the method described in the first aspect when executing the program stored in the memory.

[0025] In a fourth aspect of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements the steps of any of the methods described in the first aspect above.

[0026] This invention also provides a computer program product containing instructions that, when run on a computer, causes the computer to execute any of the text recognition methods described above.

[0027] This invention provides a text recognition method, comprising: acquiring an image to be recognized; wherein the image to be recognized displays text content; inputting the image to be recognized into a pre-trained text recognition model to obtain a probability matrix corresponding to each image region in the image to be recognized output by the text recognition model; wherein each column vector in the probability matrix corresponds to an image region; the elements in the vector include: a first element representing the probability that the corresponding image region does not contain a character, and a second element representing the probability that the character in the corresponding image region is a preset character when the character exists in the corresponding image region; and obtaining the text content in the image to be recognized based on the probability matrix corresponding to each image region in the image to be recognized.

[0028] Based on the above processing, electronic devices can obtain the probability matrix corresponding to each image region in the image to be recognized through a pre-trained text recognition model. Then, based on the probability matrix corresponding to each image region in the image to be recognized, the text content in the image to be recognized can be obtained. In other words, electronic devices can automatically recognize the text content in the image to be recognized, without requiring the user to read the text in the image and manually input the text into the device. This avoids cumbersome user operations and improves the efficiency of electronic devices in acquiring the text content in the image to be recognized.

[0029] Of course, implementing any product or method of the present invention does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0031] Figure 1 This is a first flowchart of a text recognition method provided in an embodiment of the present invention;

[0032] Figure 2 This is a second flowchart of the text recognition method provided in an embodiment of the present invention;

[0033] Figure 3 This is a third flowchart of the text recognition method provided in the embodiments of the present invention;

[0034] Figure 4 This is a fourth flowchart of the text recognition method provided in the embodiments of the present invention;

[0035] Figure 5 A fifth flowchart of the text recognition method provided in the embodiments of the present invention;

[0036] Figure 6 A schematic diagram of a deformable convolutional submodule provided in an embodiment of the present invention;

[0037] Figure 7 A schematic diagram illustrating the working principle of HCFResNet (Handwriting Character FeaturesResNet) provided in an embodiment of the present invention;

[0038] Figure 8 The sixth flowchart of the text recognition method provided in the embodiments of the present invention;

[0039] Figure 9 This is a first schematic diagram of the text recognition method provided in an embodiment of the present invention;

[0040] Figure 10 A schematic diagram of a Transformer encoder provided for an embodiment of the present invention;

[0041] Figure 11 This is a second schematic diagram of the text recognition method provided in an embodiment of the present invention;

[0042] Figure 12 A seventh flowchart of the text recognition method provided in the embodiments of the present invention;

[0043] Figure 13(a) is a first comparison diagram of the text recognition effect provided by the embodiment of the present invention;

[0044] Figure 13(b) is a second comparison diagram of the text recognition effect provided by the embodiment of the present invention;

[0045] Figure 13(c) is a third comparison diagram of the text recognition effect provided by the embodiment of the present invention;

[0046] Figure 14 A structural diagram of the text recognition model provided in an embodiment of the present invention;

[0047] Figure 15 A structural diagram of a text recognition device provided in an embodiment of the present invention;

[0048] Figure 16 This is a structural diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art based on the present invention are within the scope of protection of the present invention.

[0050] In the existing technology, when the device needs to obtain the text content in the image, the user has to manually input the text content into the device. This operation is cumbersome for the user and the device is not efficient in obtaining the text content in the image.

[0051] To address the aforementioned problems, this invention provides a text recognition method applied to an electronic device, which can be a server. The electronic device acquires an image to be recognized, wherein the image displays text content. The image is input into a pre-trained text recognition model to obtain a probability matrix corresponding to each image region in the image output by the text recognition model. Each column vector in the probability matrix corresponds to an image region. The elements of this vector include a first element representing the probability that the corresponding image region does not contain a character, and a second element representing the probability that the character in the corresponding image region is a preset character when a character is present. Based on the probability matrix corresponding to each image region in the image to be recognized, the text content in the image is obtained. The electronic device can automatically recognize the text content in the image to be recognized, avoiding cumbersome user operations and improving the efficiency of acquiring the text content in the image.

[0052] In one application scenario, after an electronic device recognizes the text content in an image to be recognized, it writes the recognized text content into an editable text document. Subsequently, the user can directly edit the text document, thus editing the text content in the image to be recognized. Alternatively, when the text written on paper is unclear, such as due to poor paper quality causing ink to bleed and become illegible, the electronic device scans the paper to obtain an image displaying the text content. After recognizing the text content in this image, the electronic device directly displays the recognized text content to the user for easy reading.

[0053] See Figure 1 , Figure 1 This is a first flowchart of a text recognition method provided in an embodiment of the present invention. The method may include the following steps:

[0054] S101: Obtain the image to be recognized. The image to be recognized contains text content.

[0055] S102: Input the image to be recognized into a pre-trained text recognition model to obtain the probability matrix corresponding to each image region in the image to be recognized output by the text recognition model. Each column vector in the probability matrix corresponds to an image region; the elements in this vector include: a first element representing the probability that the corresponding image region does not contain a character, and a second element representing the probability that the character in the corresponding image region is a preset character when a character exists in the corresponding image region.

[0056] S103: Based on the probability matrix corresponding to each image region in the image to be identified, obtain the text content in the image to be identified.

[0057] Based on the text recognition method provided in this invention, an electronic device can obtain the probability matrix corresponding to each image region in the image to be recognized through a pre-trained text recognition model. Then, based on the probability matrix corresponding to each image region in the image to be recognized, the text content in the image to be recognized is obtained. In other words, the electronic device can automatically recognize the text content in the image to be recognized, without requiring the user to read the text in the image and manually input the text into the device. This avoids cumbersome user operations and improves the efficiency of the electronic device in obtaining the text content in the image to be recognized.

[0058] In step S101, the image to be recognized is the image from which the text content to be identified and displayed needs to be identified. The text content in the image to be recognized can be handwritten text. Because different people have different writing styles and habits, even the same character may have different shapes in different handwritten texts.

[0059] In some embodiments, the text content in the image to be recognized is a single line of text. However, the image acquired by the electronic device (i.e., the original image) may display multiple lines of text. Therefore, in order to recognize the text content in the original image, the electronic device needs to preprocess the original image to obtain an image to be recognized that displays only one line of text.

[0060] exist Figure 1 Based on this, see Figure 2 Step S101 may include the following steps:

[0061] S1011: Obtain the original image and extract the smallest image region occupied by each text line in the original image to obtain the image to be recognized.

[0062] After an electronic device acquires an original image displaying multiple lines of text, it can use an image segmentation algorithm to extract the smallest image region occupied by each line of text within the original image. The extracted smallest image region will then display only one line of text, and thus, this extracted image region can be used as the image to be recognized. Image segmentation algorithms can include threshold-based segmentation methods, edge detection-based segmentation methods, feature encoding-based segmentation methods, etc.

[0063] Based on the above processing, the electronic device can obtain the image to be recognized by extracting the smallest image region occupied by each line of text in the original image. Subsequently, the electronic device can obtain the text content in the original image by recognizing the text content in each image to be recognized. This means it can recognize line-level handwritten text images. This eliminates the need for users to read the text in the original image and manually input it into the device, avoiding cumbersome user operations and improving the efficiency of the electronic device in acquiring the text content from the original image.

[0064] Regarding step S102, in order to identify the text content in the image to be identified, the electronic device can process the image to be identified using a pre-trained text recognition model to obtain the probability matrix corresponding to each image region in the image to be identified. The training method of the text recognition model can be found in the detailed description of subsequent embodiments.

[0065] In one implementation, the text recognition model can be a CNN (Convolutional Neural Network) model, an RNN (Recurrent Neural Network) model, or an LSTM (Long Short-Term Memory) model, etc.

[0066] In another implementation, the text recognition model includes a feature extraction network and a probability prediction network. Figure 1 Based on this, see Figure 3 Step S102 may include the following steps:

[0067] S1021: Input the image to be recognized into the feature extraction network to obtain the target image features of each image region in the image to be recognized output by the feature extraction network. Among them, the target image feature of an image region is: a target feature vector used to represent the content displayed in the image region.

[0068] S1022: Input the target image features of each image region in the image to be identified into the probability prediction network to obtain the probability matrix corresponding to each image region in the image to be identified output by the probability prediction network.

[0069] After acquiring the image to be recognized, the electronic device first inputs the image into the feature extraction network of the text recognition model. The feature extraction network then performs convolution processing on the image to be recognized, obtaining the target image features of each image region in the image to be recognized.

[0070] In one implementation, the feature extraction network includes multiple convolutional layers. After the image to be recognized is input into the feature extraction network, the multiple convolutional layers of the feature extraction network are used to extract features from the image to be recognized, thereby obtaining the target image features of each image region in the image to be recognized.

[0071] In another implementation, the feature extraction network includes a convolutional module and a self-attention module. Figure 3 Based on this, see Figure 4 Step S1021 may include the following steps:

[0072] S10211: Input the image to be recognized into the convolution module, and extract features from the image to be recognized through the convolution module to obtain the first image features of each image region in the image to be recognized.

[0073] S10212: Input the first image features of each image region in the image to be identified into the self-attention module. Through the self-attention module, the first image features of each image region in the image to be identified are processed according to the self-attention mechanism to obtain the target image features of each image region in the image to be identified.

[0074] A convolution module consists of multiple convolution sub-modules. A convolution sub-module can be a convolutional layer, and a convolutional layer can perform a convolution process on the input data once.

[0075] In one implementation, the convolution module includes multiple convolutional sub-modules that can be standard convolutional sub-modules, and each standard convolutional sub-module can be a standard convolutional layer. After the image to be recognized is input into the convolution module, features are extracted from the image through the multiple standard convolutional layers included in the convolution module to obtain the first image features of each image region in the image to be recognized.

[0076] In another implementation, the convolution module includes deformable convolution submodules, and a deformable convolution submodule can be a deformable convolutional layer. Figure 4 Based on this, see Figure 5 Step S10211 may include the following steps:

[0077] S102111: For each image region in the image to be identified, the image region is convolved by the deformable convolution submodule to obtain the offset corresponding to the image region.

[0078] S102112: Determine the offset region corresponding to the image region according to the offset amount corresponding to the image region, and extract features from the offset region through the deformable convolution submodule to obtain the first image feature of the image region.

[0079] The deformable convolution submodule can perform multiple convolution processes on the input data. For each image region in the image to be recognized, the deformable convolution submodule first performs a convolution process on that region to obtain the corresponding offset. This offset can include both horizontal and vertical offsets, which are the number of pixels shifted. The horizontal offset is along the width of the image region and can be to the left or right. The vertical offset is along the length of the image region and can be upward or downward. For example, the offset for this image region could be 2 pixels to the left and 2 pixels upward.

[0080] The electronic device calculates the sum of the horizontal coordinate of the center point of the image region and the corresponding horizontal offset, and the sum of the vertical coordinate of the center point and the corresponding vertical offset, based on the offset of the image region. This yields the coordinates of the offset center point. Based on the length and width of the image region, an image region with the same length and width as the offset center point is determined as the offset region corresponding to the original image region.

[0081] Alternatively, an image region can be a rectangle. For each vertex of the rectangle, the electronic device can calculate the coordinates of that vertex after offset according to the offset amount corresponding to the image region, thus obtaining the coordinates of the offset vertex corresponding to that vertex. The electronic device can calculate the coordinates of the offset vertices corresponding to the four vertices of the rectangle. Based on the calculated coordinates of the four offset vertices, the electronic device can determine the four offset vertices and the rectangle with these four offset vertices as its vertices. The image region corresponding to this determined rectangle is the offset region corresponding to the image region.

[0082] Furthermore, the electronic device can extract features from the offset region corresponding to the image region, and use the extracted image features from the offset region as the first image feature of the image region. The effectiveness of the image features obtained by extracting features from the offset region corresponding to the image region is higher than that of the first image features obtained by directly extracting features from the image region.

[0083] For example, see Figure 6 , Figure 6 This is a schematic diagram of a deformable convolutional submodule provided in an embodiment of the present invention. Figure 6 In the input feature map, the white area represents an image region in the image to be recognized, containing N pixels. The deformable convolution submodule first performs a convolution process on the white area, resulting in 2N offset fields. Each pixel corresponds to two offset fields, including a horizontal offset field and a vertical offset field. These offset fields represent the values ​​of the offsets in the previous embodiment. For example, if the horizontal offset field is 2, the horizontal offset is 2 pixels to the left; if the horizontal offset field is -2, the horizontal offset is 2 pixels to the right.

[0084] Then, based on the 2N offset fields obtained from the first convolution process, for each small region within the white area, the coordinates of the center point of the corresponding offset region are calculated. For example, for... Figure 6 In the white area, the small region in the upper left corner, if the horizontal offset field of this small region is 2 and the vertical offset field is 2, then we can add 2 to the horizontal coordinates and 2 to the vertical coordinates of the center point of this small region. The point at these coordinates is the offset center point corresponding to the center point of this small region. Accordingly, the image region with the same length and width as the small region, centered at this offset center point, is the offset region corresponding to this small region.

[0085] Since the offset of the small area in the upper left corner of this white area can represent a leftward or upward offset, it means that this small area needs to be offset to the left and upward. Figure 6 In the white area, the image region defined by the black border at the top left corner of the small area at the top left is the offset region corresponding to that small area. Similarly, the offset regions corresponding to other small areas within the white area can be determined in a similar way. For example, the offset of the small area at the bottom right of the white area can represent a rightward or downward offset; that is, the small area needs to be offset to the right and downward. Therefore, the offset region corresponding to the small area at the bottom right of the white area is... Figure 6 In the image, the lower right corner of the small area in the lower right corner of the white area is defined by a black border.

[0086] Then, deformable convolution is performed on the offset regions corresponding to each small region within the white region, which is essentially a second convolution process, to obtain the output feature map corresponding to the white region. The output feature map corresponding to the white region is the first image feature of this image region.

[0087] Based on the above processing, for each image region in the image to be recognized, a first convolution is performed to calculate the offset of each pixel in that region. Then, based on the offsets of each pixel in that region, the corresponding offset region is determined. In other words, by adding the learned two-dimensional offset to each pixel, the sampling grid used in the standard convolution operation is changed, i.e., by adjusting the spatial configuration of the input data, a more robust and effective feature representation is obtained.

[0088] In some embodiments, for example, the convolutional module can be: HCFResNet (Handwriting Character Features ResNet) built based on a deep staggered convolutional network.

[0089] See Figure 7 , Figure 7 This is a schematic diagram illustrating the working principle of HCFResNet provided in an embodiment of the present invention. HCFResNet includes three residual sub-modules, and each residual sub-module includes four convolutional sub-modules, all of which are deformable convolutional sub-modules. Figure 7 The Input in this context refers to the image to be recognized in the aforementioned embodiments. Figure 7The first convolutional submodule is denoted as 3×3conv(convolution)192stride(stride)(2,1). The kernel size of the first convolutional submodule is 3×3, the number of channels is 192, and the stride in the horizontal direction is 2, while the stride in the vertical direction is 1. The pooling submodule is denoted as 3×3Pool stride(pooling stride)(2,1). The pooling window size is 3×3, and the pooling stride in the horizontal direction is 2, while the pooling stride in the vertical direction is 1.

[0090] Figure 7 The residual submodule 1 in the model includes a first convolutional submodule and a second convolutional submodule. The second convolutional submodule is denoted as 3×3conv 192, and its convolutional kernel size is 3×3 with 192 channels. Figure 7 The residual submodule 2 in the model includes a third convolutional submodule. The third convolutional submodule is denoted as 3×3conv 384, and its convolutional kernel size is 3×3 with 384 channels. Figure 7 The residual submodule 3 in the model includes a fourth convolutional submodule. The fourth convolutional submodule is denoted as 3×3conv 768, and its convolutional kernel size is 3×3 with 768 channels.

[0091] After the image to be recognized is input into HCFResNet, the first convolution submodule in HCFResNet first performs a convolution operation on the image to be recognized to obtain the image features output by the first convolution submodule. Then, the pooling submodule performs pooling processing on the image features output by the first convolution submodule to obtain the image features output by the pooling submodule.

[0092] Next, the image features output by the pooling submodule are input into residual submodule 1. The first and second convolution submodules in residual submodule 1 perform convolution processing on the second feature to obtain the image features output by the second convolution submodule. Then, the sum of the image features output by the second convolution submodule and the image features output by the pooling submodule is calculated, and this sum is used as the output data of residual submodule 1. The output data of residual submodule 1 is then input into residual submodule 2. The third convolution submodule in residual submodule 2 performs convolution processing on the fourth feature to obtain the image features output by the third convolution submodule. Then, the sum of the image features output by the third convolution submodule and the output data of residual submodule 1 is calculated, and this sum is used as the output data of residual submodule 2. The output data of residual submodule 2 is input into residual submodule 3. The fourth convolution submodule in residual submodule 3 performs convolution processing on the sixth feature to obtain the image feature output by the fourth convolution submodule. Then, the sum of the image feature output by the fourth convolution submodule and the output data of residual submodule 2 is calculated, and the calculated sum is used as the first image feature of each image region in the image to be identified.

[0093] Compared to the existing ResNet-18 (Residual Network-18), ResNet-18 includes four residual sub-modules, while Figure 7 The HCFResNet shown includes three residual sub-modules, which means one residual sub-module has been removed. Furthermore, for the deformable convolutional sub-modules within each residual sub-module, the stride of the deformable convolutional sub-modules has been adjusted. This allows for the extraction of relatively accurate first image features for each image region in the image to be recognized using a smaller model. In other words, it maintains the two-dimensional nature of the task while generating features with sufficient character recognition information.

[0094] In some embodiments, the self-attention module includes: a masking submodule, a position determination submodule, and a self-attention submodule. For example, the masking submodule is a network layer based on Span Masking Features; the position determination submodule is a network layer based on Positional Encoding; and the self-attention submodule is a network layer based on Transformer Encoder. Figure 4 Based on this, see Figure 8 Step S10212 includes the following steps:

[0095] S102121: Input the first image features of each image region in the image to be identified into the masking submodule. Through the masking submodule, replace the first image features of the specified image region in each image region with a preset feature vector to obtain the second image features of each image region output by the masking submodule.

[0096] S102122: The second image features of each image region output by the masking submodule are input to the position determination submodule. Based on the positional relationship of each image region, the position determination submodule obtains the third image features of each image region output by the position determination submodule. The third image feature represents the arrangement order of the corresponding image region relative to other image regions.

[0097] S102123: Input the third image features of each image region output by the location determination submodule into the self-attention submodule, determine the weight of each element in the third image features of each image region through the self-attention submodule, and determine the target image features of each image region in the image to be identified based on the weight of each element in the third image features of each image region.

[0098] The first image features of each image region in the image to be recognized output by the convolution module form an L×C feature sequence. Here, L represents the number of image regions in the image to be recognized, C represents the dimension of the first image feature of each image region, the first image feature of an image region is a feature vector, and C is the number of elements contained in each feature vector.

[0099] The specified image region is a subset of the image regions defined from the image regions to be recognized. The number of specified image regions (also known as the specified number) and their positions within the image to be recognized (also known as the specified positions) can be set according to actual needs. For example, the specified number can be 10, or it can be 20% of the total number of image regions in the image to be recognized.

[0100] The electronic device can randomly select a specified number of image regions from the image regions of the image to be recognized as designated image regions. The electronic device can replace the first image feature of the designated image region with a preset feature vector. Each element in the preset feature vector is a preset value, and the dimension of the preset feature vector is the same as the dimension of the first image feature. The preset value can be 0, or it can be 1, etc., without specific limitations.

[0101] Furthermore, the masking submodule outputs the second image features of each image region. For each image region other than the specified image region, the second image feature of that image region is also the first image feature of that image region. For each specified image region, the second image feature of that specified image region is also the preset feature vector.

[0102] The second image features of each image region output by the masking submodule are input into the position determination submodule, which determines the positional relationship of each image region in the image to be recognized. For example, based on the order of the image regions in the image to be recognized, the position determination submodule can determine which position each image region is in the sequence, and use the determined position as the positional relationship of each image region in the image to be recognized, thus obtaining the third image feature of each image region. Accordingly, the third image feature represents the arrangement order of the corresponding image region with other image regions. Furthermore, based on the third image features of each image region, the image regions can be sorted. Subsequently, the preset characters in each recognized image region can be sorted according to the order of the image regions, and then the text content in the image to be recognized can be determined based on the preset characters in each image region.

[0103] For example, see Figure 9 , Figure 9 This is a first schematic diagram of a text recognition method provided in an embodiment of the present invention. Figure 9 In the process, based on the third image features of each image region, the sorting results of the image regions are as follows: Image Region 1, Image Region 2, Image Region 3, and Image Region 4. If Image Region 1 is a blank area without displayed characters, the preset character in Image Region 2 is A, the preset character in Image Region 3 is B, and Image Region 4 is a blank area without displayed characters, then the electronic device can sort the preset characters in each image region according to the sorting results obtained from sorting the image regions, resulting in "AB", that is, the text content in the image to be recognized is "AB". Then, the third image features of each image region output by the position determination submodule are input to the self-attention submodule. The self-attention submodule can be a Transformer encoder.

[0104] In some embodiments, the self-attention submodule includes: a self-attention unit, a position feedforward unit, and a residual connection unit; the position feedforward unit includes multiple linear subunits. Step S102123 may include the following steps:

[0105] Step 1: Input the third image features of each image region output by the location determination submodule into the self-attention unit. The self-attention unit determines the weight of each element in the third image features of each image region, and calculates the product of the weight of each element in the third image features of each image region and the third image features of that image region to obtain the fourth image features of each image region in the image to be identified output by the self-attention unit.

[0106] Step 2: Input the fourth image features of each image region in the image to be recognized output by the self-attention unit to the position feedforward unit. The position feedforward unit performs a linear transformation on the fourth image features of each image region in the image to be recognized through multiple linear sub-units to obtain the fifth image features of each image region in the image to be recognized output by the position feedforward unit.

[0107] Step 3: Input the fifth image features of each image region in the image to be identified from the position feedforward unit to the residual connection unit. For each image region in the image to be identified, the residual connection unit calculates the sum of the fourth image features and the fifth image features of the image region, and normalizes the calculation results to obtain the target image features of each image region in the image to be identified.

[0108] In some embodiments, since the text content in the image to be recognized may include multiple words, and each word is composed of multiple preset characters, and each word may have multiple semantics and functions (e.g., a word can be a noun, a subject, etc.), in order to obtain the multiple semantics and functions of the words composed of preset characters in each image region of the image to be recognized, and to obtain feature vectors that can better represent the preset characters in each image region, the self-attention unit needs to perform multiple processing on the third image features of each image region for each image region. Furthermore, since the process by which the self-attention unit determines the weights of each element in the third image features of each image region is a vector embedding process, in order to fully utilize the embedding subspace of the self-attention unit, the self-attention unit can be a multi-head self-attention unit, that is, the self-attention unit performs multiple processing on the third image features of the image region in parallel.

[0109] The self-attention unit performs various processing operations on the third image features of the image region to obtain the weights of each element in the third image features of the image region. This weight better represents the correlation between the element and other elements in the third image features of the image region. The higher the weight of an element, the more correlated that element is with other elements in the third image features of the image region.

[0110] The fourth image feature of an image region is obtained by multiplying the weights of each element in the third image feature of the image region by the third image feature of the image region using a self-attention unit. This fourth image feature is essentially an image feature that integrates the weights of each element in the third image feature of the image region. Compared to the third image feature, the fourth image feature of the image region can better represent the content displayed in that image region.

[0111] In some embodiments, in order to further improve the accuracy of the fourth image features of each image region and reduce the mean and standard deviation of the fourth image features of each image region to obtain more stable fourth image features of each image region and reduce the influence of covariance shift, after the self-attention unit outputs the fourth image features of each image region, for each image region, the third image feature and the fourth image feature of the image region can be calculated by using Element-Wise addition through the Residual Connections unit connected after the self-attention unit, and the calculation result is normalized. Then, the result of the normalization is used as the fourth image feature of the image region output by the self-attention unit.

[0112] For example, the residual connection unit can calculate the sum of the third image feature and the fourth image feature of the image region based on the following formula (1):

[0113] Add = x + Sublayer(x) (1)

[0114] Where Add represents the sum of the third image feature and the fourth image feature of the image region, x represents the fourth image feature of the image region output by the self-attention unit, and Sublayer(x) represents the third image feature of the image region.

[0115] Furthermore, the calculation results output by the residual connection unit can be normalized based on the following formula (2):

[0116] Norm=LayerNorm(x+Sublayer(x)) (2)

[0117] Where Norm represents the result of normalization, LayerNorm represents the normalization process, x represents the fourth image feature of the image region output by the self-attention unit, and Sublayer(x) represents the third image feature of the image region.

[0118] Then, the fourth image features of each image region output by the self-attention unit are input into the position feedforward unit. The position feedforward unit includes multiple linear sub-units. For each image region, these multiple linear sub-units perform a linear transformation on the fourth image region of that image region according to preset weights and preset position offsets. The position feedforward unit can be an FFN (Feed Forward Neural Network), and the linear sub-units can use activation functions to perform linear transformations on the fourth image features of each image region, such as the ReLU (Rectified Linear Unit) function.

[0119] For example, the position feedforward unit includes two linear sub-units. For each image region, the fourth image region of the image region can be processed based on the following formula (3) to obtain the fifth image feature of the image region:

[0120] FFN(x)=ReLU((x×W1+b1)×W2+b2) (3)

[0121] Wherein, FFN(x) is the fifth image feature of the image region, ReLU is the activation function, x is the fourth image feature of the image region, W1 is the weight parameter of the first linear subunit, b1 is the offset parameter of the first linear subunit, W2 is the weight parameter of the second linear subunit, and b2 is the offset parameter of the second linear subunit.

[0122] In some embodiments, when the plurality of linear sub-units process the fourth image region of the image region, the corresponding weights are the same.

[0123] Then, the fifth image features of each image region output by the position feedforward unit are input to the residual connection unit connected after the position feedforward unit. The Element-Wise addition method is used to calculate the sum of the fourth image features and the fifth image features of the image region. The calculation result is normalized and then the result of the normalization is used as the target image features of each image region in the image to be identified.

[0124] Based on the above processing, for each image region, the self-attention submodule enriches the third image features of the image region according to the elements in the third image features of that image region, obtaining target image features that can represent the content displayed in that image region. When the image region is a specified image region, the second image feature of that image region is also a preset feature vector. Therefore, the third image feature of that image region is also a preset feature vector that can represent the positional relationship between that image region and other image regions. The fourth image feature of that image region, obtained by processing the third image features of that image region through the self-attention submodule, is the image feature that integrates the weights of each element in the third image features of that image region. In other words, the self-attention submodule can iterate over the elements in the preset feature vector based on the elements in the third image features of that image region.

[0125] See Figure 10 , Figure 10 This is a schematic diagram of a Transformer encoder provided in an embodiment of the present invention. Figure 10 The Inputs in this context are the inputs to the self-attention submodule, which are the third image features of each image region in the image to be identified in the aforementioned embodiment. The third image features of each image region are obtained by Positional Encoding Word Embedding based on the positional relationship of each image region. Positional Encoding Word Embedding is the position determination submodule.

[0126] Multi-Head Attention (MHA) is a self-attention unit. It determines the weights of each element in the third image feature of each image region, and calculates the product of the weights of each element in the third image feature of each image region and the third image feature of that region, resulting in the fourth image feature of each region. Then, `Add` represents calculating the sum of the third and fourth image features of each image region, and `Norm` represents normalizing the result obtained from `Add`, resulting in the fourth image feature output by the self-attention unit. `residualconnections` represent the `Add` and `Norm` operations on the third and fourth image features of the image region, which can be denoted as `Add&Norm`. A Transformer encoder may contain Nx Multi-Head Attention units.

[0127] Position-Wise Feed Forward (intelligent position feedforward) is the position feedforward unit in the aforementioned embodiment. Specifically, the position feedforward unit performs a linear transformation on the fourth image features of each image region in the image to be recognized, obtaining the fifth image features of each image region output by the position feedforward unit. Then, Add indicates that for each image region, the sum of the fourth and fifth image features of that region is calculated, yielding the result. Norm indicates that the result obtained from Add is normalized to obtain the target image features of each image region in the image to be recognized. These target image features are the features themselves.

[0128] Based on the above processing, for each image region, the self-attention submodule enriches the third image features of the image region according to each element in the third image features of the image region, thereby obtaining target image features that can represent the content displayed in the image region.

[0129] The electronic device inputs the target image features of each image region into a probabilistic prediction network. Based on these features, the network performs probability prediction for each image region, thus predicting the probability of the image to be recognized and obtaining the probability matrix corresponding to each image region. This probabilistic prediction network can be based on CTC (Connectionist Temporal Classification).

[0130] The probability matrix comprises multiple column vectors, each corresponding to an image region in the image to be identified. For each column vector in the probability matrix, the column vector includes: a first element representing the probability that the image region corresponding to the column vector does not contain a character, and a second element representing the probability that, when the image region corresponding to the column vector contains a character, the character in the image region corresponding to the column vector is a preset character. Accordingly, the column vector can represent: whether the image region corresponding to the column vector contains a character, and, when the image region corresponding to the column vector contains a character, which preset character the character in the image region corresponding to the column vector is.

[0131] After the target image features of each image region in the image to be identified are input into the probability prediction network, since the target image features of an image region are target feature vectors used to represent the content displayed in the image region, for each image region, the probability prediction network obtains the probability that there are no characters in the image region and the probability that the characters in the image region are preset characters, according to the elements in the target feature vectors representing the content displayed in the image region.

[0132] For example, see Figure 9 , Figure 9 The images to be recognized include image region 1, image region 2, image region 3, and image region 4; the preset characters include A and B; when an image region does not contain a character, the symbol "-" is used to indicate that there is no character in the image region. For each image region, the column vector corresponding to that image region in the probability matrix of the image to be recognized includes three elements. The first element represents the probability that the character present in the image region is the preset character A; the second element represents the probability that the character present in the image region is the preset character B; and the third element represents the probability that there is no character in the image region, which is the probability of the symbol "-" indicating that there is no character in the image region.

[0133] The column vector corresponding to image region 1 is denoted as {0.2, 0.2, 0.8}, meaning the probability of the character present in image region 1 being the preset character "A" is 0.2; the probability of the character present being the preset character "B" is 0.2; and the probability of the symbol "-" indicating no character in the image region is 0.8. The column vector corresponding to image region 2 is denoted as {0.7, 0.2, 0.1}, meaning the probability of the character present in image region 2 being the preset character "A" is 0.7; the probability of the character present being the preset character "B" is 0.2; and the probability of the symbol "-" indicating no character in the image region is 0.1. The column vector corresponding to image region 3 is denoted as {0.3, 0.6, 0.1}, meaning the probability of the character present in image region 3 being the preset character "A" is 0.3; the probability of the character present being the preset character "B" is 0.6; and the probability of the symbol "-" indicating no character in the image region is 0.1. The column vector corresponding to image region 4 is denoted as {0.1, 0.3, 0.6}, which means that the probability of the character present in image region 4 being the preset character "A" is 0.1; the probability of the character present being the preset character "B" is 0.3; and the probability of the symbol "-" being absent in the image region is 0.6.

[0134] In some embodiments, the text recognition model can be obtained by training based on sample images displaying sample text content and sample probability matrices corresponding to each sample image region in the sample images.

[0135] For each sample image, the electronic device can input it into a text recognition model. Based on the input sample image, the text recognition model can output a prediction probability matrix corresponding to each sample image region in the sample image. Each prediction column vector in the prediction probability matrix corresponds to a sample image region. The prediction elements in the prediction column vectors represent the probability that the corresponding sample image region does not contain a sample character, and the probability that the character in the corresponding sample image region is a preset sample character when a sample character is present. Each prediction element in the prediction probability matrix is ​​used as a prediction node, resulting in multiple prediction nodes. The number of prediction nodes corresponding to each sample image region is the same as the number of prediction elements in the prediction column vector corresponding to that sample image region. A prediction node indicates that the corresponding sample image region in the sample image has no character, or that it contains a preset sample character.

[0136] Furthermore, for each sample image region, the electronic device can select any one of the prediction nodes corresponding to that sample image region as the node to be connected. Then, according to the arrangement order of the first to last sample image region in the sample image, the electronic device sequentially connects the nodes to be connected for each sample image region to obtain a prediction path. Correspondingly, the electronic device can obtain multiple prediction paths based on the prediction nodes corresponding to each sample image region. Each prediction path includes multiple nodes to be connected according to the arrangement order of the sample image regions in the sample image.

[0137] For example, such as Figure 11 As shown, Figure 11 The sample image in the diagram includes four sample image regions. Dots represent prediction nodes. A prediction node is determined based on one sample image region in the horizontal direction and one element in the vertical direction. This prediction node represents the probability that the content displayed in the sample image region corresponding to that node is the element corresponding to that node. For example, the prediction node in the first column and first row represents the probability that the character in sample image region 1 is the preset character "A".

[0138] Furthermore, starting from sample image region 1, the electronic device can select any prediction node from the prediction nodes corresponding to sample image region 1 as the first node to be connected in a prediction path, such as selecting the prediction node in the third row of the first column as the first node to be connected in the prediction path. Then, according to the positional relationship of each sample image region in the sample image, the electronic device can select any prediction node from the prediction nodes corresponding to sample image region 2 as the second node to be connected in the prediction path, such as selecting the prediction node in the first row of the second column as the second node to be connected in the prediction path. Then, according to the positional relationship of each sample image region, the electronic device can select any prediction node from the prediction nodes corresponding to sample image region 3 as the third node to be connected in the prediction path, such as selecting the prediction node in the second row of the third column. Then, the electronic device selects any prediction node from the prediction nodes corresponding to sample image region 4 according to the positional relationship of each sample image region. For example, selecting the prediction node in the third row of the fourth column is used as the last node to be connected in the prediction path, thus obtaining a prediction path. This path includes: the prediction node in the third row of the first column, the prediction node in the first row of the second column, the prediction node in the second row of the third column, and the prediction node in the third row of the fourth column. Correspondingly, this prediction path corresponds to "-, A, B, -".

[0139] The electronic device can obtain multiple prediction paths based on the prediction nodes corresponding to each sample image region. For example, a prediction path may include: prediction nodes in the first row of the first column, prediction nodes in the first row of the second column, prediction nodes in the third row of the third column, and prediction nodes in the third row of the fourth column. This prediction path corresponds to "A, A, -, -".

[0140] Subsequently, for each predicted path, the electronic device can calculate the probability corresponding to that predicted path based on the probabilities of each predicted node included in the predicted path, and decode the string corresponding to the predicted path, which includes preset characters and symbols representing no characters, to obtain the predicted recognition result obtained through that predicted path. Subsequently, for each predicted recognition result, the electronic device can calculate the sum of the probabilities corresponding to the predicted paths that yield that predicted recognition result, to obtain the probability corresponding to that predicted recognition result.

[0141] For each predicted path, when the electronic device decodes the string corresponding to that predicted path, it can first identify the symbol "-" representing no character included in the string. It can also add the symbol "-" before the first element and after the last element in the string. Then, multiple identical sample preset characters located between two "-" symbols are merged into one sample preset character, and all "-" symbols are removed to obtain the predicted recognition result obtained through that predicted path. For example, if the string corresponding to a predicted path is "-AA-", then the predicted recognition result obtained through that predicted path can be "A".

[0142] For example, for each predicted path, the electronic device can calculate the probability corresponding to that predicted path based on the following formulas (4) and (5):

[0143]

[0144] P(a,Y|X)=P(a|X)×P(Y) (5)

[0145] Where P(a|X) represents the probability of the predicted path a obtained from the input sample image X; T represents the total number of image regions obtained from the input sample image X; P(a t |) represents the probability corresponding to the t-th node in the prediction path a; the value of t ranges from 1 to T. P(a, Y|X) represents the probability that the prediction path a can identify the recognition result Y; the recognition result Y represents the predicted recognition result obtained by decoding the string corresponding to the prediction path a, and P(Y) represents the prior probability of the output recognition result Y. For example, P(Y) can be the frequency of the recognition result Y in all sample recognition results obtained from the sample probability matrix.

[0146] Furthermore, for each predicted recognition result, the electronic device can calculate the probability corresponding to the predicted recognition result based on the following formula (6).

[0147] P(Y|X)=∑ alignments P(a, Y|X) (6)

[0148] Where P(Y|X) represents the probability corresponding to the recognition result Y; P(a, Y|X) represents the probability that the predicted path a corresponding to the recognition result Y can be identified; ∑ alignments P(a, Y|X) represents the sum of probabilities of all predicted paths that can be identified and yield the recognition result Y.

[0149] Then, the electronic device can obtain the loss function value based on the following formula (7), such as the loss function value being CTCLoss (connectionist temporal classification loss):

[0150] CTC Loss(X,Y)=-log P(Y|X) (7)

[0151] Wherein, CTC Loss(X, Y) represents the loss function value corresponding to the recognition result Y obtained from the input sample image X; P(Y|X) represents the probability corresponding to the recognition result Y.

[0152] Subsequently, the electronic device can adjust the parameters of the text recognition model based on the calculated loss function value until the preset convergence condition is met, thus obtaining the trained text recognition model.

[0153] In step S103, the electronic device can determine the text content in the image to be recognized based on the probability matrix corresponding to each image region in the image to be recognized. For each image region in the image to be recognized, the image region corresponds to a column vector in the probability matrix.

[0154] In one implementation, the electronic device can generate multiple paths from the first image region to the last image region in the image to be recognized. Each path includes multiple nodes connected according to the order of the image regions in the image to be recognized; a node indicates that the corresponding image region in the image to be recognized has no character, or contains a preset character. Then, based on the probability matrix corresponding to each image region in the image to be recognized, the probability corresponding to each path is calculated. Then, from the generated multiple paths, the path with the highest probability is determined as the target path from the first image region to the last image region in the image to be recognized. The string corresponding to the target path is then obtained and decoded to obtain the text content in the image to be recognized. The method by which the electronic device determines the path from the image to be recognized is similar to the method of determining the predicted path from the sample image in the aforementioned embodiments, and can be referred to the relevant descriptions in the aforementioned embodiments; the method by which the electronic device calculates the probability corresponding to each path is similar to the method by which it calculates the probability corresponding to each predicted path is similar to the method by which it decodes the string corresponding to the predicted path is similar to the method by which it decodes the string by which it decodes the string by which it decodes the predicted path is similar to the method by which it decodes the string by which it decodes the predicted path is similar to the method by which it decodes the string by which it decodes the target ...

[0155] In another implementation, Figure 1 Based on this, see Figure 12 Step S103 may include the following steps:

[0156] S1031: For each image region in the image to be identified, if the first element of the column vector corresponding to the image region in the probability matrix is ​​greater than a preset threshold, it is determined that there is no character in the image region; if the first element of the column vector corresponding to the image region in the probability matrix is ​​not greater than the preset threshold, it is determined that there is a character in the image region.

[0157] S1032: For each image region containing a character in the image to be recognized, determine the preset character corresponding to the second largest element in the column vector of the probability matrix corresponding to the image region, and obtain the preset character in the image region.

[0158] S1033: Based on the preset characters in each image region containing characters in the image to be recognized, obtain the text content in the image to be recognized.

[0159] For each image region in the image to be recognized, the electronic device can determine the first element representing the probability that no character exists in that image region from the column vector corresponding to that image region in the probability matrix, and then determine whether the first element is greater than a preset threshold. If the first element is greater than the preset threshold, that is, the probability that no character exists in that image region is relatively high, therefore, the electronic device can determine that no character exists in that image region, and correspondingly, the electronic device can determine that the image region corresponds to a preset symbol representing the absence of a character. The preset threshold can be set according to actual needs, such as a preset threshold of 0.5.

[0160] If the first element is not greater than a preset threshold, meaning the probability that the image region does not contain a character is low, the electronic device can determine that the image region contains a character. Furthermore, for each image region containing a character in the image to be recognized, the electronic device can directly determine the preset character corresponding to the largest second element from the column vector corresponding to that image region in the probability matrix, thus obtaining the preset character in that image region.

[0161] Based on preset characters in each image region containing characters and preset symbols in each image region where characters are not present in the image to be recognized, a string composed of preset characters and preset symbols is obtained. Since an image region may not include a complete character, multiple adjacent image regions may jointly display a single character in the image to be recognized, and multiple identical characters may be recognized from these adjacent image regions. The electronic device needs to decode the string composed of preset characters and preset symbols according to a preset decoding method and use the decoding result as the text content in the image to be recognized. The method by which the electronic device decodes the string composed of preset characters and preset symbols is similar to the method by which the string corresponding to the predicted path is decoded in the aforementioned embodiments, and can be referred to the relevant descriptions in the aforementioned embodiments.

[0162] For example, the preset threshold is 0.5, such as... Figure 9 As shown, based on the aforementioned embodiment, the column vector corresponding to image region 2 is denoted as {0.7, 0.2, 0.1}, where the third element is the first element, and the first and second elements are the second elements. For image region 2, since the first element is less than a preset threshold, a character exists in image region 2. Therefore, the preset character corresponding to the largest second element in the column vector corresponding to image region 2 in the probability matrix is ​​determined, i.e., the preset character "A" corresponding to 0.7, which is the preset character in image region 2.

[0163] Accordingly, the column vector corresponding to image region 1 is denoted as {0.2, 0.2, 0.8}; that is, image region 1 does not contain any characters, and image region 1 is determined to correspond to the preset symbol "-"; the column vector corresponding to image region 3 is denoted as {0.3, 0.6, 0.1}, that is, the preset character in image region 3 is "B"; the column vector corresponding to image region 4 is denoted as {0.1, 0.3, 0.6}, that is, image region 4 does not contain any characters, and image region 4 is determined to correspond to the preset symbol "-". The electronic device obtains the string "-AB-" composed of preset characters and preset symbols according to the arrangement order of these four image regions in the image to be recognized. Then, the string composed of preset characters and preset symbols is decoded to obtain "AB", and the text content in the text to be recognized is "AB".

[0164] Based on the above processing, electronic devices can directly determine the preset characters in each image region containing characters based on the probability matrix corresponding to each image region in the image to be recognized, and obtain the text content in the image to be recognized based on the preset characters in each image region containing characters. There is no need to calculate the probability corresponding to each possible recognition result, and then determine the text content in the image to be recognized according to the probability corresponding to each possible recognition result, which can improve the efficiency of image recognition.

[0165] For example, see Figures 13(a) to 13(c)Figure 13(a) is a first comparison diagram of the text recognition effect provided by the embodiment of the present invention. In each comparison diagram in Figure 13(a), the upper image is the image to be recognized displaying handwritten English text content. The first line of text below the image to be recognized is GT (Ground Truth), which is the text content that should be displayed in the image to be recognized as manually annotated by the technician; the second line of text is Ours (this solution), which is the text content in the image to be recognized obtained according to the technical solution provided by the embodiment of the present invention. For example, in the comparison diagram of the second row and second column in Figure 13(a), the annotation of GT is the same as the content recognized by Ours, that is, Ours can get the accurate recognition result "He gave her no chance of any private". In the comparison diagram of the second row and first column in Figure 13(a), GT is annotated as "table to the environs", where "environs" does not match the text content in the image to be recognized, that is, GT is an incorrect annotation, while Ours can still get the accurate recognition result, that is, "table to the environs".

[0166] Figure 13(b) is a second comparison diagram of the text recognition effect provided by the embodiment of the present invention. In each image in Figure 13(b), the upper image is the image to be recognized displaying handwritten German text content, the first line of text below the image to be recognized is GT; the second line of text is Ours. For example, as shown in the comparison diagram of the second row and first column in Figure 13(b), the annotation content of GT is the same as the content recognized by Ours, that is, Ours can get the accurate recognition result "Notedie vortschickhūng.ist weg".

[0167] Figure 13(c) shows a third comparison of the text recognition results provided by the embodiments of the present invention. In each image in Figure 13(c), the upper image is the image to be recognized displaying handwritten Italian text content, and the first line of text below the image to be recognized is GT; the second line of text is Ours. For example, as shown in the comparison image of the first row and first column in Figure 13(c), the annotation content of GT is the same as the content recognized by Ours, that is, Ours can obtain the accurate recognition result "intanto il lavoro si dee attribuire alla Po-".

[0168] As can be seen, based on the technical solution provided by the embodiments of the present invention, the text content in the image to be identified can be accurately identified.

[0169] See Figure 14 , Figure 14 This is a structural diagram of the text recognition model provided in an embodiment of the present invention. Figure 14 The image displaying the text "when only 20 percent of them stood to gain" is the handwritten text image that needs to be input into the text recognition model, i.e., the image to be recognized in the aforementioned embodiment.

[0170] Deformable CNN (Deformable Convolutional Neural Network) is the convolutional module in the aforementioned embodiments, specifically HCFResNet built upon a deep staggered convolutional network. The convolutional module includes a deformable convolutional submodule. For each image region in the image to be recognized, the deformable convolutional submodule performs convolution processing on that region to obtain its corresponding offset. Based on this offset, the corresponding offset region is determined, and the deformable convolutional submodule extracts features from this offset region to obtain the first image feature of that region. Figure 14 In this context, a white cuboid represents the first image feature of an image region, and the number of white cuboids is L. This means that Deformable CNN extracts L first image features from the image to be recognized.

[0171] Span Masking Features, also known as the masking submodule in the aforementioned embodiments, and Learnable Mask Token, also known as the preset feature vector in the aforementioned embodiments, are used to replace the first image features of each image region in the image to be identified with the preset feature vector. The masking submodule then replaces the first image features of the specified image regions with the preset feature vector, resulting in the second image features of each image region output by the masking submodule. Figure 14 In the text, a gray cuboid represents a Learnable Mask Token. Figure 14 Two sets of Learnable Mask Tokens were determined, with each set containing S Learnable Mask Tokens. Correspondingly, the calculation result of 2×S is the specified number in the aforementioned embodiment. Let α be the pre-set hyperparameter used to define the mask ratio. Then α is the ratio of the number of preset feature vectors in the feature sequence to the total number of first image features in the feature sequence; that is, a total of α×L specified image regions' first image features are replaced with preset feature vectors.

[0172] Positional Encoding, also known as the position determination submodule in the aforementioned embodiment, determines the positional relationship between each image region and obtains the third image feature of each image region that can represent the arrangement order between the corresponding image region and other image regions.

[0173] The Transformer Encoder (self-attention encoding) is the self-attention submodule in the aforementioned embodiments. Specifically, the self-attention submodule determines the weights of each element in the third image features of each image region, and based on these weights, determines the target image features of each image region in the image to be recognized.

[0174] exist Figure 14 In the diagram, black cuboids represent the target image features of each image region. The preset feature vector, also known as a learnable token, is iterated through its elements by a self-attention submodule; that is, a self-attention mechanism is used to enrich each token with the contextual information of the entire sentence.

[0175] CTC (Connectionist Temporal Classification Loss) is the probabilistic prediction network in the aforementioned embodiments. It involves inputting the target image features of each image region in the image to be identified into the probabilistic prediction network, obtaining the probability matrix corresponding to each image region in the image to be identified output by the network. Based on the probability matrix output by the probabilistic prediction network and the pre-acquired probability matrix of the image to be identified, the CTC Loss is calculated. The parameters of the text recognition model are then adjusted based on the CTC Loss to train the text recognition model.

[0176] Based on the above processing, the electronic device can train a text recognition model. Subsequently, the probability matrix corresponding to each image region in the image to be recognized can be obtained through the text recognition model. Then, based on the probability matrix corresponding to each image region in the image to be recognized, the text content in the image to be recognized can be obtained. In other words, the electronic device can automatically recognize the text content in the image to be recognized, without requiring the user to read the text in the image and manually input the text into the device. This avoids cumbersome user operations and improves the efficiency of the electronic device in acquiring the text content in the image to be recognized.

[0177] Furthermore, when training a text recognition model using Deformable CNN, only tens of thousands of sample images are needed. Compared to existing technologies that require millions of sample images to train a model capable of recognizing text content in images, fewer sample images are needed, resulting in a simpler structure and higher efficiency, better performance, wider applicability, and stronger robustness in recognizing text content. Even when the quality of the paper on which the writing is written is poor, causing the text to be blurred and consequently the image to be recognized, the text recognition model can still effectively recognize the text content in the image by using a self-attention mechanism to enrich each token with the contextual information of the entire sentence. Moreover, it can effectively recognize images scanned from text written by users with specific styles and writing habits.

[0178] Based on the same inventive concept as the text recognition method described above, embodiments of the present invention also provide a text recognition device. See [link to related document]. Figure 15 , Figure 15 A structural diagram of a text recognition device provided in an embodiment of the present invention is shown. The device includes:

[0179] Image acquisition module 1501 is used to acquire an image to be recognized; wherein, the image to be recognized displays text content;

[0180] The probability matrix acquisition module 1502 is used to input the image to be recognized into a pre-trained text recognition model to obtain the probability matrix corresponding to each image region in the image to be recognized output by the text recognition model; wherein, each column vector in the probability matrix corresponds to an image region; the elements in the vector include: a first element representing the probability that a character exists in the corresponding image region, and a second element representing the probability that the character in the corresponding image region is a preset character when a character exists in the corresponding image region;

[0181] The text content acquisition module 1503 is used to obtain the text content in the image to be identified based on the probability matrix corresponding to each image region in the image to be identified.

[0182] Optionally, the text recognition model includes a feature extraction network and a probability prediction network; the probability matrix acquisition module 1502 is specifically used for: inputting the image to be recognized into the feature extraction network to obtain the target image features of each image region in the image to be recognized output by the feature extraction network; wherein, the target image feature of an image region is: a target feature vector used to represent the content displayed in the image region; inputting the target image features of each image region in the image to be recognized into the probability prediction network to obtain the probability matrix corresponding to each image region in the image to be recognized output by the probability prediction network.

[0183] Optionally, the feature extraction network includes a convolutional module and a self-attention module; the probability matrix acquisition module 1502 is specifically used for: inputting the image to be identified into the convolutional module, extracting features from the image to be identified through the convolutional module to obtain the first image features of each image region in the image to be identified; inputting the first image features of each image region in the image to be identified into the self-attention module, and processing the first image features of each image region in the image to be identified according to the self-attention mechanism through the self-attention module to obtain the target image features of each image region in the image to be identified.

[0184] Optionally, the convolution module includes a deformable convolution submodule; the probability matrix acquisition module 1502 is specifically used for: performing convolution processing on each image region in the image to be identified through the deformable convolution submodule to obtain the offset corresponding to the image region; determining the offset region corresponding to the image region according to the offset corresponding to the image region, and extracting features from the offset region through the deformable convolution submodule to obtain the first image feature of the image region.

[0185] Optionally, the self-attention module includes: a masking submodule, a position determination submodule, and a self-attention submodule; the probability matrix acquisition module 1502 is specifically used for: inputting the first image features of each image region in the image to be identified into the masking submodule, and through the masking submodule, replacing the first image features of a specified image region in each image region with a preset feature vector to obtain the second image features of each image region output by the masking submodule; inputting the second image features of each image region output by the masking submodule into the position determination submodule, and through the position determination submodule, obtaining the third image features of each image region output by the position determination submodule based on the positional relationship of each image region; wherein, the third image feature represents: the arrangement order of the corresponding image region with other image regions; inputting the third image features of each image region output by the position determination submodule into the self-attention submodule, and through the self-attention submodule, determining the weight of each element in the third image features of each image region, and determining the target image features of each image region in the image to be identified based on the weight of each element in the third image features of each image region.

[0186] Optionally, the self-attention submodule includes: a self-attention unit, a position feedforward unit, and a residual connection unit; the position feedforward unit includes multiple linear subunits; the probability matrix acquisition module 1502 is specifically used for: inputting the third image features of each image region output by the position determination submodule into the self-attention unit, determining the weight of each element in the third image features of each image region through the self-attention unit, and calculating the product of the weight of each element in the third image features of each image region and the third image features of that image region to obtain the fourth image features of each image region in the image to be recognized output by the self-attention unit; and inputting the third image features of each image region in the image to be recognized output by the self-attention unit into the image region to be recognized into the position determination submodule. The fourth image feature of a region is input to the position feedforward unit. The position feedforward unit performs a linear transformation on the fourth image feature of each image region in the image to be identified through multiple linear subunits, resulting in the fifth image feature of each image region in the image to be identified output by the position feedforward unit. The fifth image feature of each image region in the image to be identified output by the position feedforward unit is input to the residual connection unit. For each image region in the image to be identified, the residual connection unit calculates the sum of the fourth image feature and the fifth image feature of that image region, and normalizes the calculation result to obtain the target image feature of each image region in the image to be identified.

[0187] Optionally, the text content acquisition module 1503 is specifically used for: for each image region in the image to be recognized, if the first element of the column vector corresponding to the image region in the probability matrix is ​​greater than a preset threshold, determining that the image region does not contain characters; if the first element of the column vector corresponding to the image region in the probability matrix is ​​not greater than the preset threshold, determining that the image region contains characters; for each image region in the image to be recognized containing characters, determining the preset character corresponding to the largest second element in the column vector corresponding to the image region in the probability matrix, and obtaining the preset character in the image region; and obtaining the text content in the image to be recognized based on the preset characters in each image region containing characters in the image to be recognized.

[0188] Optionally, the image acquisition module 1501 is specifically used to: acquire an original image and extract the smallest image region occupied by each text line in the original image to obtain the image to be recognized.

[0189] Based on the text recognition device provided in this embodiment of the invention, an electronic device can obtain the probability matrix corresponding to each image region in the image to be recognized through a pre-trained text recognition model. Then, based on the probability matrix corresponding to each image region in the image to be recognized, the text content in the image to be recognized is obtained. In other words, the electronic device can automatically recognize the text content in the image to be recognized, without requiring the user to read the text in the image and manually input the text into the device. This avoids cumbersome user operations and improves the efficiency of the electronic device in obtaining the text content in the image to be recognized.

[0190] This invention also provides an electronic device, such as... Figure 16 As shown, the system includes a processor 1601, a communication interface 1602, a memory 1603, and a communication bus 1604. The processor 1601, the communication interface 1602, and the memory 1603 communicate with each other through the communication bus 1604. The memory 1603 is used to store computer programs. When the processor 1601 executes the program stored in the memory 1603, it implements the steps of any of the text recognition methods in the above embodiments.

[0191] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus. The communication interface is used for communication between the above electronic devices and other devices.

[0192] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor. The aforementioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0193] In another embodiment of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements the steps of any of the above-described text recognition methods.

[0194] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the text recognition methods described above.

[0195] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0196] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0197] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, computer-readable storage media, and computer program products are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0198] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A text recognition method, characterized in that, The method includes: Obtain an image to be recognized; wherein the image to be recognized displays text content; The image to be recognized is input into a pre-trained text recognition model to obtain a probability matrix corresponding to each image region in the image to be recognized, output by the text recognition model; wherein, each column vector in the probability matrix corresponds to an image region; the elements in the vector include: a first element representing the probability that the corresponding image region does not contain a character, and a second element representing the probability that the character in the corresponding image region is a preset character when the character exists in the corresponding image region; Based on the probability matrix corresponding to each image region in the image to be identified, the text content in the image to be identified is obtained; The text recognition model includes: a feature extraction network and a probability prediction network; The step of inputting the image to be recognized into a pre-trained text recognition model to obtain the probability matrix corresponding to each image region in the image to be recognized output by the text recognition model includes: The image to be identified is input into the feature extraction network to obtain the target image features of each image region in the image to be identified, output by the feature extraction network; wherein, the target image feature of an image region is: a target feature vector used to represent the content displayed in the image region; The target image features of each image region in the image to be identified are input into the probability prediction network to obtain the probability matrix corresponding to each image region in the image to be identified output by the probability prediction network. The feature extraction network includes: a convolutional module and a self-attention module; The step of inputting the image to be identified into the feature extraction network to obtain the target image features of each image region in the image to be identified output by the feature extraction network includes: The image to be identified is input into the convolution module, and the convolution module performs feature extraction on the image to be identified to obtain the first image features of each image region in the image to be identified; The first image features of each image region in the image to be identified are input into the self-attention module. The self-attention module processes the first image features of each image region in the image to be identified according to the self-attention mechanism to obtain the target image features of each image region in the image to be identified. The self-attention module includes: a masking submodule, a position determination submodule, and a self-attention submodule; The step of inputting the first image features of each image region in the image to be identified into the self-attention module, and processing the first image features of each image region in the image to be identified according to the self-attention mechanism through the self-attention module to obtain the target image features of each image region in the image to be identified includes: The first image features of each image region in the image to be identified are input into the masking submodule. The masking submodule replaces the first image features of the specified image regions in each image region with a preset feature vector to obtain the second image features of each image region output by the masking submodule. The second image features of each image region output by the masking submodule are input to the position determination submodule. Based on the positional relationship of each image region, the position determination submodule obtains the third image features of each image region output by the position determination submodule. The third image feature represents the arrangement order of the corresponding image region with other image regions. The third image features of each image region output by the location determination submodule are input to the self-attention submodule. The self-attention submodule determines the weight of each element in the third image features of each image region. Based on the weight of each element in the third image features of each image region, the target image features of each image region in the image to be identified are determined.

2. The method according to claim 1, characterized in that, The convolution module includes a deformable convolution submodule; The step of extracting features from the image to be identified using the convolution module to obtain the first image features of each image region in the image to be identified includes: For each image region in the image to be identified, the deformable convolution submodule is used to perform convolution processing on the image region to obtain the offset corresponding to the image region. Based on the offset corresponding to the image region, the offset region corresponding to the image region is determined, and the first image feature of the image region is obtained by extracting features from the offset region through the deformable convolution submodule.

3. The method according to claim 1, characterized in that, The self-attention submodule includes: a self-attention unit, a position feedforward unit, and a residual connection unit; the position feedforward unit includes multiple linear subunits. The step of inputting the third image features of each image region output by the location determination submodule into the self-attention submodule, determining the weight of each element in the third image features of each image region through the self-attention submodule, and determining the target image features of each image region in the image to be identified based on the weight of each element in the third image features of each image region, includes: The third image features of each image region output by the location determination submodule are input to the self-attention unit. The self-attention unit determines the weight of each element in the third image features of each image region and calculates the product of the weight of each element in the third image features of each image region with the third image features of that image region to obtain the fourth image features of each image region in the image to be identified output by the self-attention unit. The fourth image features of each image region in the image to be identified, output by the self-attention unit, are input to the position feedforward unit. The fourth image features of each image region in the image to be identified are linearly transformed by multiple linear subunits included in the position feedforward unit to obtain the fifth image features of each image region in the image to be identified, output by the position feedforward unit. The fifth image features of each image region in the image to be identified, output by the position feedforward unit, are input to the residual connection unit. For each image region in the image to be identified, the residual connection unit calculates the sum of the fourth image feature and the fifth image feature of the image region, and normalizes the calculation result to obtain the target image features of each image region in the image to be identified.

4. The method according to claim 1, characterized in that, The process of obtaining the text content in the image to be identified based on the probability matrix corresponding to each image region in the image to be identified includes: For each image region in the image to be identified, if the first element of the column vector corresponding to the image region in the probability matrix is ​​greater than a preset threshold, it is determined that there are no characters in the image region. If the first element of the column vector corresponding to the image region in the probability matrix is ​​not greater than a preset threshold, it is determined that the image region contains a character. For each image region containing characters in the image to be identified, determine the preset character corresponding to the largest second element in the column vector of the image region in the probability matrix, and obtain the preset character in the image region; Based on preset characters in each image region containing characters in the image to be recognized, the text content in the image to be recognized is obtained.

5. The method according to claim 1, characterized in that, The acquisition of the image to be identified includes: The original image is acquired, and the smallest image region occupied by each line of text in the original image is extracted to obtain the image to be identified.

6. A text recognition device, characterized in that, The device includes: An image acquisition module is used to acquire an image to be recognized; wherein the image to be recognized displays text content; The probability matrix acquisition module is used to input the image to be recognized into a pre-trained text recognition model to obtain the probability matrix corresponding to each image region in the image to be recognized output by the text recognition model; wherein, each column vector in the probability matrix corresponds to an image region; the elements in the vector include: a first element representing the probability that the corresponding image region does not contain a character, and a second element representing the probability that the character in the corresponding image region is a preset character when the character exists in the corresponding image region; The text content acquisition module is used to obtain the text content in the image to be identified based on the probability matrix corresponding to each image region in the image to be identified; The text recognition model includes: a feature extraction network and a probability prediction network; The probability matrix acquisition module is specifically used for: The image to be identified is input into the feature extraction network to obtain the target image features of each image region in the image to be identified, output by the feature extraction network; wherein, the target image feature of an image region is: a target feature vector used to represent the content displayed in the image region; The target image features of each image region in the image to be identified are input into the probability prediction network to obtain the probability matrix corresponding to each image region in the image to be identified output by the probability prediction network. The feature extraction network includes: a convolutional module and a self-attention module; The probability matrix acquisition module is specifically used for: The image to be identified is input into the convolution module, and the convolution module performs feature extraction on the image to be identified to obtain the first image features of each image region in the image to be identified; The first image features of each image region in the image to be identified are input into the self-attention module. The self-attention module processes the first image features of each image region in the image to be identified according to the self-attention mechanism to obtain the target image features of each image region in the image to be identified. The self-attention module includes: a masking submodule, a position determination submodule, and a self-attention submodule; The probability matrix acquisition module is specifically used for: The first image features of each image region in the image to be identified are input into the masking submodule. The masking submodule replaces the first image features of the specified image regions in each image region with a preset feature vector to obtain the second image features of each image region output by the masking submodule. The second image features of each image region output by the masking submodule are input to the position determination submodule. Based on the positional relationship of each image region, the position determination submodule obtains the third image features of each image region output by the position determination submodule. The third image feature represents the arrangement order of the corresponding image region with other image regions. The third image features of each image region output by the location determination submodule are input to the self-attention submodule. The self-attention submodule determines the weight of each element in the third image features of each image region. Based on the weight of each element in the third image features of each image region, the target image features of each image region in the image to be identified are determined.

7. The apparatus according to claim 6, characterized in that, The convolution module includes a deformable convolution submodule; The probability matrix acquisition module is specifically used for: For each image region in the image to be identified, the deformable convolution submodule is used to perform convolution processing on the image region to obtain the offset corresponding to the image region. Based on the offset corresponding to the image region, the offset region corresponding to the image region is determined, and the first image feature of the image region is obtained by extracting features from the offset region through the deformable convolution submodule.

8. The apparatus according to claim 6, characterized in that, The self-attention submodule includes: a self-attention unit, a position feedforward unit, and a residual connection unit; the position feedforward unit includes multiple linear subunits. The probability matrix acquisition module is specifically used for: The third image features of each image region output by the location determination submodule are input to the self-attention unit. The self-attention unit determines the weight of each element in the third image features of each image region and calculates the product of the weight of each element in the third image features of each image region with the third image features of that image region to obtain the fourth image features of each image region in the image to be identified output by the self-attention unit. The fourth image features of each image region in the image to be identified, output by the self-attention unit, are input to the position feedforward unit. The fourth image features of each image region in the image to be identified are linearly transformed by multiple linear subunits included in the position feedforward unit to obtain the fifth image features of each image region in the image to be identified, output by the position feedforward unit. The fifth image features of each image region in the image to be identified, output by the position feedforward unit, are input to the residual connection unit. For each image region in the image to be identified, the residual connection unit calculates the sum of the fourth image feature and the fifth image feature of the image region, and normalizes the calculation result to obtain the target image features of each image region in the image to be identified.

9. The apparatus according to claim 6, characterized in that, The text content acquisition module is specifically used for: For each image region in the image to be identified, if the first element of the column vector corresponding to the image region in the probability matrix is ​​greater than a preset threshold, it is determined that there are no characters in the image region. If the first element of the column vector corresponding to the image region in the probability matrix is ​​not greater than a preset threshold, it is determined that the image region contains a character. For each image region containing characters in the image to be identified, determine the preset character corresponding to the largest second element in the column vector of the image region in the probability matrix, and obtain the preset character in the image region; Based on preset characters in each image region containing characters in the image to be recognized, the text content in the image to be recognized is obtained.

10. The apparatus according to claim 6, characterized in that, The image acquisition module is specifically used for: The original image is acquired, and the smallest image region occupied by each line of text in the original image is extracted to obtain the image to be identified.

11. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-5.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-5.

Images