Text recognition method, text recognition network training method and device

Abstract

The present invention provides a text recognition method, a text recognition network training method, and a device. The method comprises: inputting an image to be recognized into a text recognition network, obtaining character information and text region information output by the text recognition network, wherein the character information at least indicates the characters contained in the image and the positions of the characters; the text region information at least indicates the text region within the image where the text is located; determining the text region where the character is located based on the character positions and the text region information; and combining characters within the same text region according to the character positions to obtain the text content contained in the text region.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a text recognition method, a training method and device for a text recognition network, an electronic device, and a storage medium. Background Art

[0002] Text recognition is the process of using computers to process, analyze, and understand text images to identify their content. It is a practical application of deep learning algorithms. Text recognition technology is rapidly developing and is widely used in various industries.

[0003] In related technologies, text recognition techniques typically first detect the location of text in an image, then crop and align the text region, and finally recognize the cropped text region. Alternatively, a neural network model is used to capture the text region in an image, and then features of the text region are extracted and recognized to obtain the text content of the text image. However, these solutions require first detecting the text region and then recognizing it, resulting in low recognition efficiency and inability to recognize irregular text. Summary of the Invention

[0004] Embodiments of the present invention provide a text recognition method, a training method and device for a text recognition network, an electronic device, and a storage medium.

[0005] The technical solution of the embodiment of the present invention is achieved as follows:

[0006] In a first aspect, an embodiment of the present invention provides a text recognition method, comprising:

[0007] Inputting an image to be recognized into a text recognition network, obtaining character information and text region information output by the text recognition network, wherein the character information at least indicates the characters contained in the image and the positions of the characters; and the text region information at least indicates the text region in the image where the text is located;

[0008] Determining the text area where the character is located according to the character position and the text area information;

[0009] According to the character positions, the characters located in the same text area are combined to obtain the text content contained in the text area.

[0010] Optionally, inputting the image to be recognized into a text recognition network to obtain character information and text region information output by the text recognition network includes:

[0011] Inputting the image to be recognized into the text recognition network;

[0012] Performing feature extraction on the image through one or more convolutional layers in the text recognition network to obtain a feature map;

[0013] Performing character detection and classification on the feature graph through a character detection and classification network located at the output end of the convolutional layer in the text recognition network to obtain the character information;

[0014] The text segmentation network located at the output end of the convolutional layer in the text recognition network is used to perform text segmentation on the feature map to obtain the text area information.

[0015] Optionally, extracting features from the image using one or more convolutional layers in the text recognition network to obtain a feature map includes:

[0016] Performing feature extraction on the image through the convolutional layer to obtain a plurality of initial feature maps of different sizes;

[0017] The initial feature maps of different sizes are fused to obtain a feature map for obtaining the character information and the text area information.

[0018] Optionally, fusing the initial feature maps of different sizes to obtain the feature map for obtaining the character information and the text region information includes:

[0019] The initial feature maps of different sizes are fused through Feature Pyramid Networks (FPN) to obtain a feature map for obtaining the character information and the text area information.

[0020] Optionally, performing character detection and classification on the feature map by a character detection and classification network located at the output end of the convolutional layer in the text recognition network to obtain the character information includes:

[0021] Based on the first convolution kernel, performing a border regression process on the feature map containing the characters to obtain the character positions in the image;

[0022] Based on the second convolution kernel, character classification is performed on the feature map to obtain the characters contained in the image.

[0023] In a second aspect, an embodiment of the present invention provides a method for training a text recognition network, comprising:

[0024] Acquire a training sample set of an image and annotation information of the training sample set; the annotation information includes at least characters, character positions, and text area information;

[0025] Inputting the training sample set into an initial network to obtain character prediction information and text region prediction information output by the initial network; wherein the character prediction information at least indicates the characters contained in the image and the character positions of the characters predicted by the initial network; and the text region prediction information at least indicates the text region in the image where the text is predicted to be located by the initial network;

[0026] Determining a loss function value of the initial network based on the character prediction information, the text region prediction information, and the annotation information;

[0027] According to the loss function value, the parameters to be trained of the initial network are adjusted to obtain a text recognition network.

[0028] Optionally, determining the loss function value of the initial network according to the character prediction information, the text region prediction information, and the annotation information includes:

[0029] Calculating a first loss function value based on the characters included in the character prediction information and the characters in the annotation information;

[0030] Calculating a second loss function value according to the character position in the character prediction information and the character position in the annotation information;

[0031] Calculating a third loss function value based on the text region prediction information and the text region information in the annotation information;

[0032] The loss function value of the initial network is calculated according to the first loss function value, the second loss function value and the third loss function value.

[0033] Optionally, the initial network includes:

[0034] at least one convolutional layer for outputting a feature map based on the input image;

[0035] A character detection and classification network, located at the output end of the convolutional layer, is used to output character prediction information based on the feature map;

[0036] The text segmentation network is located at the output end of the convolutional layer and is used to output text region prediction information according to the feature map.

[0037] In a third aspect, an embodiment of the present invention provides a text recognition device, comprising:

[0038] A text recognition module inputs an image to be recognized into a text recognition network and obtains character information and text region information output by the text recognition network, wherein the character information at least indicates the characters contained in the image and the positions of the characters; and the text region information at least indicates the text region within the image where the text is located;

[0039] The character merging module is used to determine the text area where the character is located according to the character position and the text area information; and to combine the characters located in the same text area according to the character position to obtain the text content contained in the text area.

[0040] Optionally, the text recognition module includes:

[0041] An image preprocessing module is used to input the image to be recognized into the text recognition network; perform feature extraction on the image through one or more convolutional layers in the text recognition network to obtain a feature map;

[0042] A character detection and classification module, configured to perform character detection and classification on the feature map through a character detection and classification network located at an output end of the convolutional layer in the text recognition network to obtain the character information;

[0043] The text segmentation module is used to segment the feature map text through the text segmentation network located at the output end of the convolution layer in the text recognition network to obtain the text area information.

[0044] Optionally, the image preprocessing module includes:

[0045] A feature extraction module, configured to extract features from the image through the convolutional layer to obtain a plurality of initial feature maps of different sizes;

[0046] The feature fusion module is used to fuse the initial feature maps of different sizes to obtain a feature map for obtaining the character information and the text area information.

[0047] Optionally, the feature fusion module is further configured to:

[0048] The initial feature maps of different sizes are fused through FPN to obtain a feature map for obtaining the character information and the text area information.

[0049] Optionally, the character detection and classification module is further used to:

[0050] Based on the first convolution kernel, performing a border regression process on the feature map containing the characters to obtain the character positions in the image;

[0051] Based on the second convolution kernel, character classification is performed on the feature map to obtain the characters contained in the image.

[0052] In a fourth aspect, an embodiment of the present invention provides a training device for a text recognition network, the training device comprising:

[0053] A sample set acquisition module, configured to acquire a training sample set of an image and annotation information of the training sample set; the annotation information includes information about characters, character positions, and text regions;

[0054] Text recognition network acquisition module, used for:

[0055] Inputting the training sample set into an initial network to obtain character prediction information and text region prediction information output by the initial network; wherein the character prediction information at least indicates the characters contained in the image and the character positions of the characters predicted by the initial network; and the text region prediction information at least indicates the text region in the image where the text is predicted to be located by the initial network;

[0056] Determining a loss function value of the initial network based on the character prediction information, the text region prediction information, and the annotation information;

[0057] According to the loss function value, the parameters to be trained of the initial network are adjusted to obtain a text recognition network.

[0058] Optionally, the text recognition network acquisition module includes:

[0059] A first calculation module, configured to calculate a first loss function value based on the characters included in the character prediction information and the characters in the annotation information;

[0060] A second calculation module, configured to calculate a second loss function value according to the character position in the character prediction information and the character position in the annotation information;

[0061] a third calculation module, configured to calculate a third loss function value based on the text region prediction information and the text region information in the annotation information;

[0062] A loss function calculation module is used to calculate the loss function value of the initial text recognition network model based on the first loss function value, the second loss function value and the third loss function value.

[0063] In a fifth aspect, an embodiment of the present invention provides an electronic device, including:

[0064] a memory for storing executable instructions;

[0065] The processor is configured to implement the text recognition method provided by one or more of the aforementioned technical solutions when executing the executable instructions stored in the memory.

[0066] In a sixth aspect, an embodiment of the present invention provides an electronic device, including:

[0067] a memory for storing executable instructions;

[0068] The processor is configured to implement the text recognition network training method provided by one or more of the aforementioned technical solutions when executing the executable instructions stored in the memory.

[0069] In a seventh aspect, an embodiment of the present invention provides a computer-readable storage medium, which stores computer-executable instructions; after the computer-executable instructions are executed by a processor, the text recognition method provided by one or more of the aforementioned technical solutions can be implemented.

[0070] In an eighth aspect, an embodiment of the present invention provides a computer-readable storage medium, which stores computer-executable instructions; after the computer-executable instructions are executed by a processor, the training method of the text recognition network provided by one or more of the aforementioned technical solutions can be implemented.

[0071] The text recognition method, text recognition network training method, device, electronic device and storage medium provided by the embodiments of the present invention input the image to be recognized into the text recognition network, obtain character information output by the text recognition network for indicating the characters contained in the image and the character positions of the characters, and text area information for indicating the text area where the text in the image is located, and determine the text area where the character is located based on the character information and the text area information, and then combine the characters in the same text area according to the character positions to obtain the text content contained in the text area.

[0072] First, the present invention does not require a process of first detecting and then recognizing the text area of ​​an image. Instead, the text recognition network directly outputs the characters contained in the image and the character information of the characters. The character information in the image can be recognized through a single detection, thereby improving recognition efficiency.

[0073] Secondly, the present invention outputs text area information indicating the text area where the text in the image is located through a text recognition network, and determines the characters in each text area and the text content contained in the text area through the text area information and character information; there is no need to perform area cropping or alignment operations on the image, and irregular text can be recognized. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 11 is a flowchart of a text recognition method provided by an embodiment of the present invention;

[0075] Figure 2 yes Figure 1 Detailed flowchart of step 101 in the corresponding embodiment;

[0076] Figure 3 This is a flow chart of a text recognition network training method provided by an embodiment of the present invention;

[0077] Figure 4 yes Figure 3 Detailed flowchart of step 303 in the corresponding embodiment;

[0078] Figure 5 is a structural diagram of a text recognition device provided by an embodiment of the present invention;

[0079] Figure 6 1 is a structural diagram of a text recognition network training device provided by an embodiment of the present invention;

[0080] Figure 7 A schematic diagram of a flow chart of a single-stage end-to-end text recognition method provided by an embodiment of the present invention;

[0081] Figure 8 1. A text recognition network architecture diagram provided by an embodiment of the present invention;

[0082] Figure 9 Schematic diagram of the structure of the text segmentation network provided by an embodiment of the present invention;

[0083] Figure 10 Schematic diagram of the structure of the character detection and classification network provided by an embodiment of the present invention;

[0084] Figure 11 This is a schematic diagram of the structure of a single-stage end-to-end text recognition method provided by an embodiment of the present invention;

[0085] Figure 12 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0086] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting the present invention. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0087] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0088] In the following description, the terms "first\second\third" are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present invention described herein can be implemented in an order other than that illustrated or described herein.

[0089] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention pertains. The terms used herein are for the purpose of describing embodiments of the present invention only and are not intended to limit the present invention.

[0090] There are three main text recognition methods:

[0091] (1) The text position is obtained from the image through the text detection network model. After the text is cut out and aligned from the image, it is input into the sequence recognition network model to obtain the text content. This serial method of first detecting and then recognizing has low recognition efficiency, and the feature alignment operation cannot align irregular text, which affects subsequent recognition.

[0092] (2) The shared features of the text detection network and the sequence recognition network are merged into a single neural network. After the detection branch of the network obtains the text location, it extracts text features from the shared feature map, aligns these features, and sends them to the recognition branch for content recognition. This method forces the text detection network and the sequence recognition network, which have significant differences, to share features. This makes the network model more difficult to train, and the recognition accuracy of the sequence recognition network is not as good as that of the separate recognition model.

[0093] (3) Use the region recommendation network to obtain the approximate location of the text box, crop and align the feature map at that location, and then feed it into the segmentation network to obtain the precise location of the text frame, which is then fed into the sequence recognition network to obtain the text content. This method of first generating candidate regions through the region recommendation network and then performing sequence recognition separately is less efficient; and the region recommendation network based on anchor boxes cannot handle very long texts, resulting in the inability to recognize long texts.

[0094] The embodiment of the present invention provides a text recognition method. Figure 1 FIG. 1 is a flow chart of a text recognition method provided by an embodiment of the present invention. Figure 1 As shown, the following steps are included:

[0095] Step 101: Input an image to be recognized into a text recognition network, and obtain character information and text region information output by the text recognition network; wherein the character information at least indicates the characters contained in the image and the positions of the characters; and the text region information at least indicates the text region in the image where the text is located.

[0096] Step 102, determining the text area where the character is located based on the character position and the text area information;

[0097] Step 103 : combining characters in the same text region according to the character positions to obtain text content contained in the text region.

[0098] The text recognition method described in the embodiments of the present invention can be applied to electronic devices. Here, electronic devices include mobile terminals and fixed terminals. Mobile terminals include mobile phones, tablet computers, laptop computers, etc.; fixed terminals include personal computers. In other optional embodiments, the data processing method can also be run on network-side devices, including servers, processing centers, etc.

[0099] In practical applications, the text content recognized from the image can be further translated, edited, stored, etc. The text recognition method provided by the embodiment of the present invention can be applied to text recognition tasks in any scenario to realize the recognition of text content in images, such as natural scene text images, advertising images, ID cards, driver's licenses, business cards, license plates, etc.

[0100] In embodiments of the present invention, the image to be recognized can be acquired in real time, such as by a camera of an electronic device; can be an image sent to the recognition device by another terminal; or can be a pre-stored image. The image to be recognized can be a single image or multiple images, which is not specifically limited here.

[0101] It should be noted that the text recognition process for multiple images to be recognized is the same as the text recognition process for a single image to be recognized. Therefore, for ease of description, the embodiment of the present invention is described using a single image to be recognized.

[0102] In step 101, the image to be recognized is input into a text recognition network, which outputs character information and text region information; wherein the text recognition network is pre-trained.

[0103] It should be noted that the image to be recognized corresponds to at least one text region, and each text region contains at least one character. Here, characters include but are not limited to Chinese characters, numeric characters, English characters, and / or punctuation marks. The character position refers to the position of the character in the image, for example, the coordinates of the four vertices of the character bounding box are used as the character position. In the embodiment of the present invention, because text may have irregular characters or irregular text arrangement, the text region can be a rectangular region or a polygonal region, which is not specifically limited here.

[0104] In some embodiments, the character position can be represented by the coordinates of the four vertices of a rectangle circumscribing the character. For example, an image containing the regular character "CN" to be recognized is input into a text recognition network, and the output is the characters "C", "N", the corner coordinates of the character "C" (x3, y3), (x3, y4), (x4, y3), (x4, y4), the corner coordinates of the character "C" (x5, y5), (x5, y6), (x6, y5), (x6, y6), and the corner coordinates of the rectangular text area (x1, y1), (x1, y2), (x2, y1), (x2, y2).

[0105] In other embodiments, the character position may be represented by the coordinates of the center point of a rectangle circumscribing the character and the length and width of the rectangle.

[0106] In some embodiments, the image to be recognized can be input into a text recognition network, and a feature map can be obtained by performing feature extraction on the image to be recognized; and text segmentation can be performed on the feature map to obtain text area information of the image.

[0107] For example, text segmentation can be performed on the feature map using a DBNet network model; the feature map is input into the DBNet network model to obtain a text region probability matrix, a text region threshold matrix, and a text region center probability matrix in the image; a text region border binary map corresponding to the text region threshold matrix and a text region center binary matrix corresponding to the text region center probability map are obtained; a connected domain detection is performed on the text region center binary matrix, and the cluster centers of the text regions are determined based on the text detection results; the cluster centers corresponding to each pixel are determined based on the similarity between the pixels of the text region border in the text region border binary matrix and the cluster centers, thereby determining the pixels in the text region border corresponding to each cluster center, and the positions of each text region in the image are determined by calculating the circumscribed polygon of each text region border. Here, the text region probability matrix is ​​the probability matrix of each pixel in the image belonging to text; the text region threshold matrix is ​​the probability matrix of the text region border in the image; and the text region center probability matrix is ​​the probability matrix composed of the pixels at the center position in the image.

[0108] In actual implementation, the image to be recognized is input into a pre-trained text recognition network, and character information for indicating the characters contained in the image and the character positions of the characters, and text region information for indicating the text region within the text region where the text is located in the image are obtained from the output of the text recognition network.

[0109] In step 102, according to the character positions and text region information output by the text recognition network, the text regions corresponding to each character are determined.

[0110] In an embodiment of the present invention, the character information output by the text recognition network is used to indicate the position of the character in the image; the text region information is used to indicate the position of the text region corresponding to the text line in the image. According to the character information and the text region information, the text regions to which each character belongs are determined.

[0111] For example, the image is input into the text recognition network, and the text region information and character information of two non-overlapping text boxes contained in the image are output; among them, the coordinates of the four corner points of the text box area A are (x1, y1), (x1, y2), (x2, y1), (x2, y2) respectively, and the coordinates of the four corner points of the text box area B are (x5, x5), (x5, x6), (x6, x5), (x6, x6) respectively. And the character coordinates of the character C, that is, the coordinates of the four corner points of the character border, are (x3, y3), (x3, y4), (x4, y3), (x4, y4) respectively. Where x1 < x3 < x4 < x2, y1 < y3 < y4 < y2; then the character C is the character within the text box A.

[0112] In step 103, all the characters within the same text region are arranged and combined according to the character positions to obtain the text content contained in the text region.

[0113] In an embodiment of the present invention, according to the character positions and text region information output by the text recognition network, the text regions corresponding to each character are determined. Then, according to the character positions, the characters within the same text region are combined to obtain the text content contained in the text region.

[0114] In actual implementation, when combining characters within the same text area according to character positions, in addition to considering the character positions, a specific combination order also needs to be considered. For example, according to the character positions, the characters can be combined first from left to right and then from top to bottom, or first from top to bottom and then from left to right, etc.; multiple alternative texts are obtained after combining in different combination orders. By identifying the expression content of the alternative texts, the final text content is determined according to the expression content. Here, the combination order determines whether the final output text content is correct. For example, if the combination order is incorrect, the combined text content may be text without meaning or with chaotic meaning, resulting in an incorrect final recognition result output by the text recognition network. For example, the alternative text is input into another deep learning model, and the final determined text content is output by this deep learning model.

[0115] In some other embodiments, a combination order can also be preset in advance. When combining characters within the same text area, the characters are combined according to the character positions and the preset combination order to obtain the combined text content, which is the final output text content.

[0116] Exemplarily, for a text area containing two characters, where the character positions of character 1 are respectively (x3, y3), (x3, y4), (x4, y3), (x4, y4); the character positions of character 2 are respectively (x5, y5), (x5, y6), (x6, y5), (x,6 y6); x3 < x4 < x5 < x6, y3 = y5, y4 = y6, then it can be determined that character 1 and character 2 are two并列 characters, and character 1 is in front of character 2. Thus, according to the character positions, the characters within the same text area are combined to obtain the text content included in the text area.

[0117] In this way, the text recognition network directly performs parallel processing of text segmentation, character detection, and classification on the image to be recognized, outputs character information and text area information, and the recognition of character information in the image can be obtained through a single detection, improving the recognition efficiency; and according to the text area information and character information, the characters in each text area and the text content included in the text area are determined; there is no need to perform operations such as regional cropping or alignment on the image, and irregular texts can be recognized.

[0118] Optionally, as Figure 2 shown, Figure 2 is <I Figure 1 a detailed flowchart of step 101 in the corresponding embodiment. Step 101 includes:

[0119] In step 1011, the image to be recognized is input into the text recognition network;

[0120] In step 1012, feature extraction is performed on the image through one or more convolutional layers in the text recognition network to obtain a feature map;

[0121] In step 1013, character detection and classification are performed on the feature map by a character detection and classification network located at the output end of the convolutional layer in the text recognition network to obtain the character information;

[0122] In step 1014, the text segmentation network located at the output end of the convolutional layer in the text recognition network is used to perform text segmentation on the feature map to obtain the text region information.

[0123] In the embodiments of the present invention, the convolutional layer is the fundamental structure of a convolutional neural network. It is composed of several convolutional units, each of which has parameters optimized using a backpropagation algorithm. The purpose of the convolution operation is to extract different features of the input parameters; a convolutional neural network consists of one or more convolutional layers and a fully connected layer at the top.

[0124] In step 1012, features are extracted from the image through one or more convolutional layers to obtain a corresponding feature map.

[0125] For example, let's take the VGG16 convolutional neural network as an example. The VGG16 convolutional neural network contains 13 convolutional layers. After the image to be recognized is input into the VGG16 convolutional neural network, it first passes through two convolutional layers with the same structure to perform two convolutions for the first pooling. Then, it passes through two convolutional layers with the same structure to perform two convolutions for the second pooling. After that, it passes through two convolutional layers with the same structure to perform two convolutions for the third pooling. Finally, it passes through a fully connected layer and outputs the feature map of the image to be recognized. Here, each convolutional layer contains a different number of convolution kernels.

[0126] In an embodiment of the present invention, after the image is feature extracted through the convolutional layer, the output feature map is input into the character detection and classification network and the text segmentation network respectively. After the character detection and classification network receives the feature map, it extracts the character information from the feature map; after the text segmentation network receives the feature map, it can extract the text box area information containing the text information from the received feature map.

[0127] Specifically, the text box area information can be expressed in the form of coordinates; for example, the coordinates of the text box area are {x1, y1, x2, y2}, where x1 and x2 represent the horizontal coordinates of the four corner points of the text box area, and y1 and y2 represent the vertical coordinates of the four focal points of the text box area. By combining the horizontal coordinates and the vertical coordinates in pairs, the coordinates of the four corner points of the text box area can be obtained, which are (x1, y1), (x1, y2), (x2, y1), and (x2, y2), respectively.

[0128] In practical applications, the character detection and classification network refers to a neural network that can determine the characters and their locations in an image based on a feature map, such as a Region Proposal Network (RPN). The text segmentation network refers to a neural network that can determine the locations of text regions in an image based on a feature map, such as a Mask-RCNN network model or a DBNet network model.

[0129] Exemplarily, the RPN network can be used to detect and classify characters on the feature map; the RPN model may include a convolution layer, a pooling layer, an activation function, and a fully connected layer; first, an anchor point is generated with each pixel in the feature map as the center, and candidate boxes with different aspect ratios and area sizes are generated based on the anchor point to cover the entire feature map; then two filters are used to convolve each candidate box separately, wherein the convolution kernels of the two filters are different. The output result of one of the filters is input into the activation function (such as the softmax function) for pre-classification, and the pre-classification result is integrated with the output result of the other filter, and classification and regression are performed using a fully connected layer and an activation function (such as the Relu function) to obtain the position of the character candidate box in the image and the classification result of the corresponding category of the character in the character candidate box, thereby determining the characters and the positions of the characters in the image.

[0130] Here, the convolution kernels and convolution layers of the two filters can be determined based on the size of the candidate box. The aspect ratio of the candidate box is usually set to 0.5, 1, and 2, and the corresponding area ratio is usually set to 4, 8, and 16. In this way, through the character detection and classification network and the text segmentation network, character detection, classification, and text segmentation are performed on the feature map respectively, achieving character detection, classification, and text segmentation in one step, thereby improving recognition efficiency. Based on the text area information output by text segmentation and the character information output by character detection and classification, the text content contained in the image is determined. There is no need to perform region cropping or alignment operations on the image, and irregular text can be recognized.

[0131] Optionally, step 1012 includes:

[0132] Performing feature extraction on the image through the convolutional layer to obtain a plurality of initial feature maps of different sizes;

[0133] The initial feature maps of different sizes are fused to obtain a feature map for obtaining the character information and the text area information.

[0134] It should be noted that the initial feature map is essentially a matrix whose elements are the features extracted by the convolutional layer. The size of the initial feature map refers to the size of the matrix. For example, if the initial feature map is a u*v matrix, then the size of the initial feature map is u*v.

[0135] In the embodiment of the present invention, the convolution kernels of different sizes and different step lengths included in the convolution layer perform feature extraction on the input image through the convolution layer, and initial feature maps of different sizes can be obtained.

[0136] Here, the feature extraction network for text recognition can be a residual network ResNet, SKNet network, etc., which is not specifically limited here.

[0137] For example, the text recognition network can be a 50-layer deep ResNet network structure, which is composed of residual blocks, each of which includes three convolutional layers. By setting different stride values, the residual blocks can obtain features of different sizes at {stride4, stride8, stride16, stride32, stride64}. The more residual blocks an image passes through, the more detailed the image feature information extracted, and the better the network model's subsequent test results.

[0138] In practical applications, multiple feature maps of different sizes output by the convolutional layer in the text recognition network can be upsampled layer by layer in sequence and then fused to obtain a fused feature map. For example, the pixel values ​​contained in the largest-sized feature map are used as the pixel values ​​contained in the other feature maps after upsampling. One or more feature maps formed after upsampling are fused pixel by pixel with the largest-sized feature map to obtain the fused feature map.

[0139] For example, the VGG16 convolutional neural network, a feature extraction network for text recognition, contains 13 convolutional layers, each with a different number of convolution kernels. The multiple feature maps of different scales output by the convolutional layers are arranged in ascending order and upsampled layer by layer to obtain multiple feature maps of the same scale. These multiple feature maps of the same scale are then fused to obtain a fused feature map.

[0140] In some embodiments, fusing the initial feature maps of different sizes to obtain a feature map for obtaining the character information and the text region information includes:

[0141] The initial feature maps of different sizes are fused through FPN to obtain a feature map for obtaining the character information and the text area information.

[0142] In an embodiment of the present invention, the initial feature maps of different sizes output by the convolutional layer are fused through FPN, thereby obtaining a fused feature map that incorporates features from each level.

[0143] It can be understood that FPN can be used to fuse the initial feature map of a larger size and the initial feature map of a smaller size to obtain a fused feature map. Since the feature map of a larger size contains more position information, and the feature map of a smaller size contains more feature information, the fused feature map obtained by fusing the initial feature maps of different sizes contains both more position information and more feature information.

[0144] In this way, FPN can be used to fuse multiple initial feature maps of different sizes output by the convolutional layer to obtain a fused feature map that integrates features from each level.

[0145] Optionally, step 1013 includes:

[0146] Based on the first convolution kernel, performing a border regression process on the feature map containing the characters to obtain the character positions in the image;

[0147] Based on the second convolution kernel, character classification is performed on the feature map to obtain the characters contained in the image.

[0148] In an embodiment of the present invention, each of the multiple feature maps is processed as follows: a sliding window process is performed on the feature map using a convolution with a convolution kernel size of 1*1, that is, an anchor point is generated with each pixel in the feature map as the center, thereby generating anchor frames with different aspect ratios and different areas to cover all feature maps.

[0149] In actual implementation, the number of first convolution kernels is determined according to the number of categories of the anchor box N_anchor; the number of second convolution kernels is determined according to the number of categories of the anchor box N_anchor and the number of character categories N_class. Among them, N_anchor is determined by the number of preset aspect ratios of the anchor box; here, the aspect ratio of the anchor box can be set according to the aspect ratio of the character. For example, the aspect ratio of the character is generally between 0.5 and 2. The aspect ratio of the anchor box is set to {0.5, 1, 1.5, 2}, then N_anchor is 4. N_class is the number of characters in the preset dictionary. For example, if the dictionary contains 4000 commonly used characters, then N_class is 4000. In some embodiments, the width and height of the character can be determined according to the pixel size of the image and the size of the image. In other embodiments, the width and height of the character can be preset empirical values.

[0150] In some embodiments, the number of the first convolution kernels is N_anchor×4; the number of the second convolution kernels is N_anchor×(N_class+1).

[0151] For example, considering that the aspect ratio of a character is generally between 0.5 and 2, the size ratio of the anchor box is set to {0.5, 1, 2}, the area is set to {16*16, 32*32, 64*64, 128*128, 256*256}, the number of the first convolution kernel is 12, and the number of the second convolution kernel is 12003.

[0152] In an embodiment of the present invention, character bounding box regression processing is performed based on the feature map through the first convolution kernel, and the character position of each character contained in the image is output.

[0153] Specifically, the character position can be expressed in the form of coordinates. For example, the coordinates of the character border are {x3, y3, x4, y4}, where x3 and x4 represent the horizontal coordinates of the four corner points of the character border, and y3 and y4 represent the vertical coordinates of the four foci of the character border. By combining the horizontal coordinates and the vertical coordinates in pairs, the coordinates of the four corner points of the character border can be obtained, which are (x3, y3), (x3, y4), (x4, y3), and (x4, y4), respectively.

[0154] In this embodiment of the present invention, a convolution calculation is performed on the feature map using a second convolution kernel to extract character sequence features. Characters are then classified based on the character feature sequence, and the recognition results for each character are output. For example, assuming a dictionary contains 4,000 commonly used characters, the probability of the character output after classification based on the character feature sequence belonging to each character in the dictionary is the character with the highest probability in the dictionary. Thus, for multiple characters in an image, the recognition results for each character can be output, resulting in the characters contained in the image.

[0155] In this way, the first convolution kernel regresses the bounding box containing the characters on the feature map, improving the accuracy of the bounding box so that it is closer to the character edge and the accuracy of the character position. At the same time, the second convolution kernel classifies the characters on the feature map, improving the accuracy of character recognition and, in turn, improving the accuracy of text recognition.

[0156] Next, the embodiment of the present invention provides a training method for a text recognition network, such as Figure 3 As shown, Figure 3 1 is a flow chart of a method for training a text recognition network according to an embodiment of the present invention. The method includes:

[0157] In step 301, a training sample set of an image and annotation information of the training sample set are obtained; the annotation information at least includes characters, character positions, and text area information;

[0158] In step 302, the training sample set is input into the initial network to obtain character prediction information and text region prediction information output by the initial network;

[0159] In step 303, a loss function value of the initial network is determined based on the character prediction information, the text region prediction information, and the annotation information;

[0160] In step 304, the parameters to be trained of the initial network are adjusted according to the loss function value to obtain a text recognition network.

[0161] In an embodiment of the present invention, the character prediction information and text region prediction information are character information and text region information output by an initial network based on the training image.

[0162] Here, the character prediction information at least indicates the characters contained in the image and the character positions of the characters predicted by the initial network; the text area prediction information at least indicates the text area where the text is located in the image predicted by the initial network;

[0163] In step 301, a large number of images are acquired to form a training sample set; and the characters, character positions, and text area positions contained in each image in the training sample set are annotated to obtain the annotation information of the image.

[0164] In practical applications, the training sample set is data used to adjust the parameters to be trained in the initial network. Each training image in the training sample set contains annotation information, which includes at least information about characters, character positions, and text regions. In step 304, the parameters to be trained in the initial network are updated based on the calculated loss function value.

[0165] In actual implementation, it is necessary to pre-set the parameters to be trained of the initial network; the parameters to be trained include one or more of the following parameters: number of iterations, batch size, image size, learning rate, and learning rate decay value.

[0166] Here, the number of iterations refers to the number of times all training images in the training sample set are trained, and is generally determined by the number of training images and the initial network size. The batch size refers to the number of images loaded into the network during a single forward propagation. The image size refers to the image size that needs to be adjusted to an appropriate size before training, and the image size is changed according to the predefined length and width. The learning rate controls the learning rate of the model. If the learning rate is set too small, the trained network model will converge too slowly; if the learning rate is set too large, it will cause the loss function to oscillate. Therefore, in order to avoid unreasonable learning rate settings during training, a dynamic learning rate adjustment method is usually adopted. By setting the learning rate decay value, the learning rate will continue to decrease exponentially with the number of training rounds during training, converging to the learning step size of gradient descent.

[0167] In this embodiment of the present invention, images from a training sample set are fed into an initial network to generate character prediction information and text region prediction information. Based on this character prediction information, text region prediction information, and annotation information, a loss function value is calculated based on a preset loss function of the initial network. Backpropagation gradients are calculated based on this loss function value, and the parameters to be trained in the initial network are updated. These steps are then repeated until the loss function converges, resulting in a more optimized text recognition network.

[0168] Exemplarily, when training the initial network, after any training image is input into the initial network, the training image is subjected to feature extraction through one or more convolutional layers contained in the initial network to obtain a corresponding feature map, and the feature map is respectively input into the character detection and classification network and the text segmentation network at the output end of the convolutional layer to obtain the character prediction information output by the character detection and classification network and the text area prediction information output by the text segmentation network. The output character prediction information and text area prediction information, as well as the character information and text area information in the annotation information of the training image are input into the loss function of the initial network to calculate the loss function value of the initial network. Whether the initial network meets the requirements is judged based on the loss function value, for example, whether the loss function value is greater than a preset loss threshold. When the initial network does not meet the requirements, the initial network is optimized using a preset optimization function, the parameters of the initial network are adjusted, and the above operation is repeated until the calculated loss function value meets the requirements, and a text recognition network can be obtained.

[0169] In this way, the loss function is calculated through the character prediction information and text area prediction information output by the initial network and the annotation information of the training sample set, and the initial network is trained based on the loss function to obtain a text recognition network with better performance.

[0170] Alternatively, as Figure 4 As shown, Figure 4 for Figure 3The detailed flow chart of step 303 in the corresponding embodiment includes:

[0171] Step 3031: Calculate a first loss function value based on the characters included in the character prediction information and the characters in the annotation information;

[0172] Step 3032: Calculate a second loss function value based on the character position in the character prediction information and the character position in the annotation information;

[0173] Step 3033: Calculate a third loss function value based on the text region prediction information and the text region information in the annotation information;

[0174] Step 3034: Calculate the loss function value of the initial network based on the first loss function value, the second loss function value, and the third loss function value.

[0175] In practical applications, the loss function of the initial network used in the training process is:

[0176] Loss = αloss clc +βloss regress +γloss segment ;

[0177] Among them, Loss is the total loss function; loss cls is the first loss function value; loss regress is the second loss function value; loss segment is the value of the third loss function; α, β, γ are weight coefficients.

[0178] Here, the first loss function value is the loss of character classification; the second loss function value is the loss of character bounding box regression; and the third loss function value is the loss of text area bounding box regression.

[0179] In practical applications, Focal Loss can be used as the first loss function to avoid the problem of low accuracy caused by imbalanced sample categories. Smooth L1 loss function can be used as the second loss function, which not only makes the function converge faster, but also is insensitive to outliers and abnormal values, with relatively smaller gradient changes and good robustness. The third loss function can be determined based on the network used in the text segmentation network and is not specifically limited here.

[0180] Among them, the Focal Loss function expression is: FL(p t )=-α t (1-p t ) γ log(p t );

[0181] Here, FL(p t ) is the loss function value; p t is the classification probability of different categories; parameter α t and γ are fixed values, where parameter α t The parameter γ is a decimal between [0,1] to solve the imbalance between positive and negative samples; the parameter γ is a value greater than 0 to solve the imbalance between difficult and easy samples.

[0182] It should be noted that Focal Loss is modified based on the standard cross entropy loss. It is a loss function used to solve category imbalance and large differences in classification difficulty in classification problems.

[0183] Function expression of Smooth L1:

[0184] Among them, SmoothL1(x) is the loss function value; x=f(x i )-y i is the difference between the predicted value and the labeled true value, f(x i ) is the predicted character position of the i-th training image output by the initial network, y i is the character position in the annotation information of the i-th training image.

[0185] Optionally, the initial network includes:

[0186] at least one convolutional layer for outputting a feature map based on the input image;

[0187] A character detection and classification network, located at the output end of the convolutional layer, is used to output character prediction information based on the feature map;

[0188] The text segmentation network is located at the output end of the convolutional layer and is used to output text region prediction information according to the feature map.

[0189] In an embodiment of the present invention, a character detection and classification network and a text segmentation network are respectively set after the convolution layer in the initial network to perform character detection, classification and text segmentation on the feature map output by the convolution layer. Through this network model structure, text segmentation and character detection and classification of the image are achieved in one step, thereby improving recognition efficiency.

[0190] Next, an embodiment of the present invention provides a text recognition device 50, such as Figure 5 As shown, Figure 5 : is a schematic diagram of the structure of a text recognition device provided by an embodiment of the present invention. The device includes:

[0191] The text recognition module 51 inputs the image to be recognized into a text recognition network and obtains character information and text region information output by the text recognition network, wherein the character information at least indicates the characters contained in the image and the positions of the characters; the text region information at least indicates the text region in the image where the text is located;

[0192] The character merging module 52 is used to determine the text area where the character is located based on the character position and the text area information; and to combine the characters located in the same text area according to the character position to obtain the text content contained in the text area.

[0193] Optionally, the text recognition module 51 includes:

[0194] An image preprocessing module 511 is configured to input an image to be recognized into the text recognition network; perform feature extraction on the image through one or more convolutional layers in the text recognition network to obtain a feature map;

[0195] A character detection and classification module 512 is configured to perform character detection and classification on the feature graph using a character detection and classification network located at the output end of the convolutional layer in the text recognition network to obtain character information;

[0196] The text segmentation module 513 is used to segment the text of the feature map through the text segmentation network located at the output end of the convolution layer in the text recognition network to obtain the text area information.

[0197] Optionally, the image preprocessing module 511 includes:

[0198] A feature extraction module 5111 is configured to extract features from the image using the convolutional layer to obtain a plurality of initial feature maps of different sizes;

[0199] The feature fusion module 5112 is used to fuse the initial feature maps of different sizes to obtain a feature map for obtaining the character information and the text area information.

[0200] In an embodiment of the present invention,

[0201] Optionally, the feature fusion module is further configured to:

[0202] The initial feature maps of different sizes are fused through FPN to obtain a feature map for obtaining the character information and the text area information.

[0203] Optionally, the character detection and classification module 512 is further configured to:

[0204] Based on the first convolution kernel, performing a border regression process on the feature map containing the characters to obtain the character positions in the image;

[0205] Based on the second convolution kernel, character classification is performed on the feature map to obtain the characters contained in the image.

[0206] In this way, the text recognition device performs parallel processing of text segmentation, character detection and classification on the image to be recognized, outputs character information and text area information, and can obtain the recognition of character information in the image through a single detection, thereby improving the recognition efficiency; based on the text area information and character information, the characters in each text area and the text content contained in the text area are determined; there is no need to perform area cropping or alignment operations on the image, and irregular text can be recognized.

[0207] Next, an embodiment of the present invention provides a training device 60 for a text recognition network, such as Figure 6 As shown, Figure 6 1 is a schematic diagram of a text recognition network training device provided by an embodiment of the present invention. The device includes:

[0208] The sample set acquisition module 61 is used to acquire a training sample set of an image and annotation information of the training sample set; the annotation information includes characters, character positions and text area information;

[0209] The text recognition network acquisition module 62 is used to:

[0210] Inputting the training sample set into an initial network to obtain character prediction information and text region prediction information output by the initial network; wherein the character prediction information at least indicates the characters contained in the image and the character positions of the characters predicted by the initial network; and the text region prediction information at least indicates the text region in the image where the text is predicted to be located by the initial network;

[0211] Determining a loss function value of the initial network based on the character prediction information, the text region prediction information, and the annotation information;

[0212] According to the loss function value, the parameters to be trained of the initial network are adjusted to obtain a text recognition network.

[0213] Optionally, the text recognition network acquisition module 62 includes:

[0214] A first calculation module 621 is configured to calculate a first loss function value based on the characters included in the character prediction information and the characters in the annotation information;

[0215] A second calculation module 622 is used to calculate a second loss function value according to the character position in the character prediction information and the character position in the annotation information;

[0216] A third calculation module 623 is used to calculate a third loss function value based on the text region prediction information and the text region information in the annotation information;

[0217] The loss function calculation module 624 is used to calculate the loss function value of the initial text recognition network model according to the first loss function value, the second loss function value and the third loss function value.

[0218] In this way, the loss function is calculated by outputting character prediction information, text area prediction information, and the annotation information of the training sample set, and the initial network is trained based on the loss function to obtain a text recognition network with better performance.

[0219] In combination with the above embodiments of the present invention, an exemplary application of the embodiments of the present invention in a practical application scenario will be described below.

[0220] This example provides a single-stage end-to-end text recognition method. Figure 7 As shown, Figure 7 A flowchart of a single-stage end-to-end text recognition method provided by an embodiment of the present invention.

[0221] Step 701, annotation data preparation;

[0222] A large number of images are obtained to form a training sample set; and each image in the training sample set is annotated, the characters and character positions contained in the image are annotated, and the position of each text line is annotated.

[0223] Step 702: Initial network establishment;

[0224] like Figure 8 As shown, Figure 8 The following is a diagram of a text recognition network architecture provided by an embodiment of the present invention. Specifically:

[0225] A stacked convolutional neural network is set at the beginning of the initial network. Here, the convolutional neural network includes but is not limited to networks such as ResNet and SKNet.

[0226] The features at {stride4, stride8, stride16, stride32, stride64, stride128} are extracted through the convolutional neural network to obtain multiple initial feature maps of different sizes; the initial feature maps of different sizes are fused through FPN to obtain a fused feature map.

[0227] The text segmentation network located at the output end of the convolutional neural network in the initial network is used to perform text segmentation on the feature map to obtain text area information.

[0228] Specifically, if Figure 9 As shown, Figure 9 Schematic diagram of the text segmentation network provided by an embodiment of the present invention. The feature maps at {stride4, stride8, stride16, stride32} output by the FPN are input into the text segmentation network, where the text segmentation network can be a DBNet network or other text segmentation network.

[0229] The character detection and classification network located at the output end of the convolutional neural network in the initial network is used to perform character detection and classification on the feature map to obtain character information.

[0230] Specifically, if Figure 10 As shown, Figure 10 It is a structural diagram of the character detection and classification network provided by an embodiment of the present invention. The feature map at {stride4, stride8, stride16, stride32, stride64, stride128} output by FPN is input into the character detection and classification network. Taking into account that the aspect ratio of the character is generally between 0.5 and 2, anchor frames with areas of {16*16, 32*32, 64*64, 128*128, 256*256, 512*512} are set in sequence on the feature maps of different sizes, and N_anchor×4 1*1 first convolution kernels are connected to the feature maps of different sizes for character border regression, and N_anchor×(N_class+1) 1*1 second convolution kernels are connected to the feature maps of different sizes for character classification.

[0231] In this way, the initial network is composed of convolutional neural network, FPN, text segmentation network and character detection and classification network.

[0232] Step 703, loss function design;

[0233] The loss function of the initial network used during training is:

[0234] Loss = αloss clc +βloss regress +γloss segment ;

[0235] Among them, Loss is the total loss function; loss cls is the first loss function value; loss regress is the second loss function value; loss segmentis the value of the third loss function; α, β, γ are weight coefficients.

[0236] Here, the first loss function value is the loss for character classification, and the Focal Loss loss function can be used. The second loss function value is the loss for character bounding box regression, and the Smooth L1 loss function can be used. The third loss function value is the loss for text region bounding box regression. The loss function needs to be determined based on the text segmentation network used in the initial network. For example, if the text segmentation network used in the initial network adopts DBNet, the loss function corresponding to DBNet is used.

[0237] Step 704: network training;

[0238] The training sample set is input into the initial network to obtain character prediction information and text area prediction information output by the initial network; the loss function value of the initial network is determined based on the character prediction information, the text area prediction information and the annotation information; and the parameters to be trained of the initial network are adjusted based on the loss function value to obtain a text recognition network.

[0239] Specifically, the acquired training sample set is input into the initial network, and a forward computation is performed to obtain the network's output character prediction information and text region prediction information. Based on this character prediction information, text region prediction information, and annotation information, the initial network's loss function is calculated. Backpropagation gradients are then calculated based on this loss function value, and the initial network parameters are updated until the loss function converges. The network parameters are then saved.

[0240] Step 705, text recognition;

[0241] like Figure 11 As shown, Figure 11 This is a schematic diagram of the structure of a single-stage end-to-end text recognition method provided by an embodiment of the present invention. The image to be recognized is input into a text recognition network, and character information and text region information are output by the text recognition network. Based on the character positions and text region information, the text region in which the character is located is determined. Based on the character positions, characters within the same text region are grouped together to obtain the text content within the text region.

[0242] Specifically, a text recognition network is obtained by loading trained parameters into an initial network. The image to be recognized is input into the text recognition network, and after forward computation, character information and text region information are output, respectively. The character information at least indicates the characters contained in the image and their positions; the text region information at least indicates the text region within the image where the text is located. Based on the character positions and text region information, the text region where the characters are located is determined. Then, based on the character positions, the characters within the text region are combined in a left-to-right and top-to-bottom priority order to obtain the text content of the text region. Finally, the position coordinates and text content of the text region are output.

[0243] This achieves text detection and recognition in a single network model. By using a single-stage network, both detection and recognition are achieved in one step, resulting in higher recognition efficiency. Furthermore, without using region cropping or alignment operations, the input text's shape is preserved, allowing for recognition of irregular text.

[0244] An embodiment of the present invention further provides an electronic device, comprising:

[0245] a memory for storing executable instructions;

[0246] The processor is configured to implement a text recognition method provided by an embodiment of the present invention when executing the executable instructions stored in the memory.

[0247] An embodiment of the present invention further provides an electronic device, comprising:

[0248] a memory for storing executable instructions;

[0249] The processor is configured to implement a text recognition network training method provided by an embodiment of the present invention when executing the executable instructions stored in the memory.

[0250] The hardware structure of an electronic device of a text recognition method provided by an embodiment of the present invention is described in detail below. The electronic device includes but is not limited to a server or a terminal. Figure 12 , Figure 12 7 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The text recognition device 70 includes: at least one processor 701, a memory 70, and optionally, the text recognition device 70 may further include at least one communication interface 703. The various components in the text recognition device 70 are coupled together via a bus system 704. It can be understood that the bus system 704 is used to achieve connection and communication between these components. In addition to the data bus, the bus system x4 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 12 Various buses are labeled as bus system 704 .

[0251] It is understood that memory 702 can be volatile memory or non-volatile memory, or can include both volatile and non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disk, or compact disc read-only memory (CD-ROM); magnetic surface memory can be magnetic disk memory or tape memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus random access memory (DRRAM).The memory x2 described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0252] The memory 702 in the embodiment of the present invention is used to store various types of data to support the operation of the text recognition device 50. Examples of such data include any computer program used to operate on the text recognition device 50, such as stored sample data, prediction models, etc. The program for implementing the method of the embodiment of the present invention may be included in the memory 702.

[0253] The methods disclosed in the above embodiments of the present invention can be applied to or implemented by processor 701. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor or by software instructions. The above processor may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present invention can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium located in a memory. The processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0254] In an exemplary embodiment, the text recognition device 70 can be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the above-mentioned method.

[0255] In the several embodiments provided herein, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is merely a logical functional division. In actual implementation, other divisions may be employed, such as combining multiple units or components, integrating them into another system, or omitting or disabling certain features. Furthermore, the coupling, direct coupling, or communication connection between the components shown or discussed may be through interfaces. The indirect coupling or communication connection between devices or units may be electrical, mechanical, or other. The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of these units may be selected to achieve the objectives of the present embodiments based on actual needs. Furthermore, the functional units in the various embodiments of the present invention may all be integrated into a single processing unit, each unit may be a separate unit, or two or more units may be integrated into a single unit. These integrated units may be implemented in hardware or as hardware plus software functional units.

[0256] The embodiment of the present invention further provides a computer storage medium, wherein the computer storage medium stores a computer program. After the computer program is executed by a processor, the text recognition method provided by one or more of the above technical solutions is executed. For example, the following can be executed: Figure 1 The method shown.

[0257] The embodiment of the present invention further provides a computer storage medium, wherein the computer storage medium stores a computer program. After the computer program is executed by a processor, the training method of the text recognition network provided by one or more technical solutions described above is executed. For example, the following can be executed: Figure 3 The method shown.

[0258] The computer storage media provided in the embodiments of the present invention include various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. Optionally, the computer storage media may be non-transient storage media. Non-transient storage media herein may also be referred to as non-volatile storage media.

[0259] In some embodiments, the computer-readable storage medium may be a memory device such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface mount memory, optical disk, or CD-ROM; or various devices including any one or any combination of the above memories. The computer may be various computing devices including smart terminals and servers.

[0260] In some embodiments, executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0261] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0262] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0263] The above description is merely an embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present invention are included in the scope of protection of the present invention.

Claims

1. A text recognition method, characterized in that: include: Inputting an image to be recognized into a text recognition network, obtaining character information and text region information output by the text recognition network, wherein the character information at least indicates the characters contained in the image and the positions of the characters; the text region information at least indicates the text region in the image where the text is located; the text recognition network is a single-stage network; Determining the text area where the character is located according to the character position and the text area information; Combining characters located in the same text region according to the character positions to obtain text content contained in the text region; The step of inputting the image to be recognized into the text recognition network and obtaining character information and text area information output by the text recognition network includes: Inputting the image to be recognized into the text recognition network; Performing feature extraction on the image through one or more convolutional layers in the text recognition network to obtain a feature map; The feature map is processed respectively by a character detection and classification network and a text segmentation network provided in parallel in the text recognition network to obtain character information and text region information output by the text recognition network; Among them, the character detection and classification network is used to sequentially set multiple anchor frames with different aspect ratios and different areas on the input feature map; and based on the first convolution kernel, the feature map is respectively subjected to character border regression processing to obtain the character position of each character contained in the image; based on the second convolution kernel, the feature map is respectively subjected to character classification to obtain the character recognition result of each character contained in the image; the number of the first convolution kernel and the number of the second convolution kernel are related to the number of categories of the anchor frames in the feature map.

2. The method according to claim 1, characterized in that The text segmentation network is used to perform text segmentation on the input feature map to obtain the text area information.

3. The method according to claim 1, characterized in that The step of extracting features from the image through one or more convolutional layers in the text recognition network to obtain a feature map includes: Performing feature extraction on the image through the convolutional layer to obtain a plurality of initial feature maps of different sizes; The initial feature maps of different sizes are fused to obtain a feature map for obtaining the character information and the text area information.

4. The method according to claim 3, characterized in that The fusing of the initial feature maps of different sizes to obtain a feature map for obtaining the character information and the text region information includes: The initial feature maps of different sizes are fused through a feature pyramid network (FPN) to obtain a feature map for obtaining the character information and the text area information.

5. A training method for a text recognition network, characterized in that: The method comprises: Acquire a training sample set of an image and annotation information of the training sample set; the annotation information includes at least characters, character positions, and text area information; Inputting the training sample set into an initial network to obtain character prediction information and text region prediction information output by the initial network; wherein the character prediction information at least indicates the characters contained in the image and the character positions of the characters predicted by the initial network; and the text region prediction information at least indicates the text region in the image where the text is predicted to be located by the initial network; Determining a loss function value of the initial network based on the character prediction information, the text region prediction information, and the annotation information; According to the loss function value, the parameters to be trained of the initial network are adjusted to obtain a text recognition network; the text recognition network is a single-stage network; The initial network includes: at least one convolutional layer for outputting a feature map based on the input image; A character detection and classification network and a text segmentation network, wherein the character detection and classification network and the text segmentation network are connected in parallel at the output end of the convolution layer, and the character detection and classification network is used to sequentially set a plurality of anchor frames with different aspect ratios and different areas on the input feature map; and based on the first convolution kernel, perform character bounding box regression processing on the feature map respectively to obtain the character position of each character contained in the image; based on the second convolution kernel, perform character classification on the feature map respectively to obtain the character recognition result of each character contained in the image; wherein the number of the first convolution kernel and the number of the second convolution kernel are related to the number of categories of the anchor frames in the feature map; and the text segmentation network is used to output text area prediction information according to the feature map.

6. The training method according to claim 5, characterized in that The determining the loss function value of the initial network according to the character prediction information, the text region prediction information, and the annotation information includes: Calculating a first loss function value based on the characters included in the character prediction information and the characters in the annotation information; Calculating a second loss function value according to the character position in the character prediction information and the character position in the annotation information; Calculating a third loss function value based on the text region prediction information and the text region information in the annotation information; The loss function value of the initial network is calculated according to the first loss function value, the second loss function value and the third loss function value.

7. A text recognition device, characterized in that: The device comprises: A text recognition module is used to input the image to be recognized into a text recognition network; perform feature extraction on the image through one or more convolutional layers in the text recognition network to obtain a feature map; the text recognition network is a single-stage network; the feature map is processed respectively by a character detection and classification network and a text segmentation network set in parallel in the text recognition network to obtain character information and text area information output by the text recognition network; wherein the character information at least indicates the characters contained in the image and the character positions of the characters; the text area information at least indicates the text area where the text is located in the image; the character detection and classification network is used to sequentially set a plurality of anchor frames with different aspect ratios and different areas on the input feature map; and perform character border regression processing on the feature map based on the first convolution kernel to obtain the character position of each character contained in the image; perform character classification on the feature map based on the second convolution kernel to obtain the character recognition result of each character contained in the image; the number of the first convolution kernel and the number of the second convolution kernel are related to the number of categories of the anchor frames in the feature map; The character merging module is used to determine the text area where the character is located according to the character position and the text area information; and to combine the characters located in the same text area according to the character position to obtain the text content contained in the text area.

8. A training device for a text recognition network, characterized in that: The training device comprises: A sample set acquisition module, configured to acquire a training sample set of an image and annotation information of the training sample set; the annotation information includes information about characters, character positions, and text regions; Text recognition network acquisition module, used for: Inputting the training sample set into an initial network to obtain character prediction information and text region prediction information output by the initial network; wherein the character prediction information at least indicates the characters contained in the image and the character positions of the characters predicted by the initial network; and the text region prediction information at least indicates the text region in the image where the text is predicted to be located by the initial network; Determining a loss function value of the initial network based on the character prediction information, the text region prediction information, and the annotation information; According to the loss function value, the parameters to be trained of the initial network are adjusted to obtain a text recognition network; the text recognition network is a single-stage network; The initial network includes: at least one convolutional layer for outputting a feature map based on the input image; A character detection and classification network and a text segmentation network, wherein the character detection and classification network and the text segmentation network are connected in parallel to the output end of the convolution layer, the character detection and classification network is used to sequentially set a plurality of anchor frames with different aspect ratios and different areas on the input feature map; and based on the first convolution kernel, the feature map is respectively subjected to character border regression processing to obtain the character position of each character contained in the image; based on the second convolution kernel, the feature map is respectively subjected to character classification to obtain the character recognition result of each character contained in the image; the number of the first convolution kernel and the number of the second convolution kernel are related to the number of categories of the anchor frames in the feature map; the text segmentation network is used to output text area prediction information according to the feature map.

9. An electronic device comprising: a memory for storing executable instructions; The processor is configured to implement the text recognition method according to any one of claims 1 to 4 when executing the executable instructions stored in the memory.

10. An electronic device comprising: a memory for storing executable instructions; The processor is configured to implement the text recognition network training method according to claim 5 or 6 when executing the executable instructions stored in the memory.

11. A computer storage medium, characterized in that Executable instructions are stored, and when the executable instructions are executed by a processor, the text recognition method according to any one of claims 1 to 4 is implemented.

12. A computer storage medium, characterized in that Executable instructions are stored, and when the executable instructions are executed by the processor, the training method of the text recognition network according to claim 5 or 6 is implemented.

Images