A method for recognizing text and terminal device

Abstract

An embodiment of the present application relates to a method for recognizing text, the method comprising: scaling an image to be recognized according to a first scaling ratio; determining first coordinate information corresponding to a text line area in the image to be recognized after scaling; determining second coordinate information corresponding to the first coordinate information according to the first scaling ratio, the second coordinate information being coordinate information of the text line area in the image to be recognized; performing text recognition on the text line image corresponding to the second coordinate information through a recognition model, and determining the text line content corresponding to the text line image; the image to be recognized includes a text line image. An embodiment of the present application avoids the waste of resources caused by multiple calls to the NPU using sub-images segmented by sliding windows. Combined with the actual requirements of low response delay and low power consumption for mobile terminal applications, based on the NPU technical specifications, the advantages of the dedicated AI chip on the end side can be brought into play, multiple text lines can be processed concurrently at one time, the NPU utilization rate can be improved, and the user experience of OCR products can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a text recognition method and terminal device. Background Art

[0002] Currently, mobile terminals are developing very rapidly, and many manufacturers are promoting artificial intelligence (AI) mobile phones. Among them, vision-related fields have become the key points of mobile terminal AI technology because of their convenient collection, friendly interaction, rich information, terminal measurement and calculation, and high application efficiency. Optical character recognition (OCR) has become a difficult and key point in visual processing due to its diverse characters, complex scenes, and rich semantic information.

[0003] In some existing text recognition schemes, the original image is segmented into multiple sub-images with overlapping areas by using sliding window technology (hereinafter referred to as sliding window) on the input image, and then the multiple segmented sub-images are sent to the neural network for detection. However, since the original image is segmented into multiple sub-images using sliding windows, each sub-image needs to call the model, which will cause multiple calls to the model, resulting in a very large computational overhead in processing the image. At the same time, since multiple sub-images have overlapping areas, it will cause serious waste of resources. Since it is segmented into multiple sub-images, an additional merging operation is required at the end. This scheme as a whole will cause serious waste and consumption of resources. For text recognition of high-quality images, the problem of waste and consumption of resources will become more obvious.

[0004] In other schemes, a sliding window is also used to slide and select the text line image. The sliding step size of 4 or 8 is set according to the different Chinese and English characters. At the same time, during the sliding process, the sliding window is filled at the edge of the image to take care of the edge text, and then sent to the neural network for recognition. Since the step size is set to 4 or 8, when it is sent to the neural network for recognition calculation, it will cause 4 or 8 times of repeated calculation, resulting in waste. When the sliding window is truncated on the original layer, all the sliding windows of an image are spliced ​​and the connectionist temporal classification (CTC) based on the neural network is used to calculate the loss of the input sequence and the sample ground truth (gt) sequence, which will make the utility of CTC not really play out. And when processing an image, because each text line needs to be operated on by sliding windows, but the neural network processing unit (NPU) at each moment can only process one line of text at a time, and the parallel ability between text lines is extremely poor. Therefore, it is necessary to call the NPU for processing multiple times.

[0005] For some existing text recognition solutions, the use of sliding windows for image segmentation causes serious waste of resources and additional computing overhead. At the same time, the NPU needs to be called multiple times, resulting in a long text recognition response time, which seriously affects the experience. Summary of the invention

[0006] The embodiments of the present application provide a method and terminal device for recognizing text, which finds the coordinates of the text lines in the picture by scaling the original image, and then restores it to the original image. Find the corresponding text line image from the original image, and send the text line image to the recognition model for text recognition. Avoid the waste of resources caused by multiple calls to the NPU using sub-images segmented by sliding windows. Combined with the actual needs of low response latency and low power consumption for mobile terminal applications, based on the NPU technical specifications, it is possible to give full play to the advantages of the terminal-side AI dedicated chip, which can process multiple text lines at a time concurrently, improve the NPU utilization rate, and improve the OCR product user experience.

[0007] In a first aspect, an embodiment of the present application provides a method for recognizing text, the method comprising: scaling an image to be recognized according to a first scaling ratio; determining first coordinate information corresponding to a text line area in the image to be recognized after scaling; determining second coordinate information corresponding to the first coordinate information according to the first scaling ratio, the second coordinate information being coordinate information of the text line area in the image to be recognized; performing text recognition on the text line image corresponding to the second coordinate information through a recognition model, and determining the text line content corresponding to the text line image; the image to be recognized includes a text line image.

[0008] In one possible implementation, the image to be identified includes at least one text line area; determining the first coordinate information corresponding to the text line area in the scaled image to be identified includes: using a neural network to detect the text area of ​​the scaled image to be identified to obtain the confidence of at least one candidate area; the confidence is a probability value that the at least one candidate area contains a text line; deleting at least one candidate area whose confidence is less than a first confidence threshold, and sorting the confidences of the remaining at least one candidate area to select the candidate area with the highest confidence; combining the selected candidate area with the unselected candidate area, sorting the confidences of at least one uncombined candidate area, selecting the candidate area with the highest confidence, combining the selected candidate area with the unselected candidate area, until all the candidate areas are combined, and determining the first coordinate information of the at least one combined text line area.

[0009] In one possible implementation, the selected candidate region and the unselected candidate region are combined, including: when the ratio of the overlapping area of ​​the unselected candidate region and the selected candidate region is greater than or equal to a first area ratio threshold, the candidate region with low confidence in the two candidate regions is deleted by a non-maximum suppression algorithm; when the unselected candidate region and the selected candidate region are adjacent in the long side direction or the overlapping area in the long side direction is less than the first area ratio threshold, the unselected candidate region and the selected candidate region are merged into one region.

[0010] In a possible implementation, before the step of identifying the text line image corresponding to the second coordinate information through the recognition model, the method also includes: obtaining the text line image corresponding to the second coordinate information from the image to be identified; gray-scaling the text line image; and sending the gray-scaled text line image to the recognition model for text recognition.

[0011] In a possible implementation, before the grayscaled text line image is passed through a recognition model for text recognition, the method also includes: dividing the grayscaled text line image into three categories: horizontal, vertical, and non-horizontal and non-vertical; performing an affine transformation on the non-horizontal and non-vertical text line images so that all text line images are in the horizontal or vertical direction; traversing all vertical text line images, splitting each vertical text line image into multiple horizontal text line images and annotating them.

[0012] In a possible implementation, each vertical text line image is split into multiple horizontal text line images and annotated, including: using horizontal annotation to split each vertical text line image into multiple single-character horizontal text line images and annotate them.

[0013] In a possible implementation, performing text recognition on the text line image corresponding to the second coordinate information by using a recognition model further includes: scaling the text line image corresponding to the second coordinate information according to a second scaling ratio, and performing text recognition on the scaled text line image.

[0014] In a possible implementation manner, scaling the text line image to the second pixel ratio includes: geometric scaling, equal-width scaling, equal-length scaling, tile scaling, and zero-filling scaling.

[0015] In a possible implementation, the recognition model is also used to recognize space characters.

[0016] In one possible implementation, the operators within the recognition model do not include sliding window and recurrent neural network operator layers outside of convolution operations.

[0017] In a second aspect, an embodiment of the present application provides a method for recognizing text, including: obtaining multiple text line data; marking space characters in the multiple text line data with preset labels; updating a recognition model based on the marked text line data, and the updated recognition model is also used to recognize space characters.

[0018] In a third aspect, an embodiment of the present application provides a terminal device, comprising: a processor, a detector, and a text recognizer; the processor is used to scale the image to be recognized according to a first scaling ratio; the detector is used to determine first coordinate information corresponding to a text line area in the image to be recognized after scaling; second coordinate information corresponding to the first coordinate information is determined according to the first scaling ratio, the second coordinate information being the coordinate information of the text line area in the image to be recognized; the text recognizer is used to perform text recognition on the text line image corresponding to the second coordinate information through a recognition model, and determine the text line content corresponding to the text line image; the image to be recognized includes a text line image.

[0019] In one possible implementation, the image to be identified includes at least one text line area; the detector is also used to: use a neural network to perform text area detection on the scaled image to be identified to obtain the confidence of at least one candidate area; the confidence is a probability value that at least one candidate area contains a text line; delete at least one candidate area whose confidence is less than a first confidence threshold, and sort the confidences of the remaining at least one candidate area to select the candidate area with the highest confidence; combine the selected candidate area with the unselected candidate area, sort the confidences of the at least one uncombined candidate area, select the candidate area with the highest confidence, combine the selected candidate area with the unselected candidate area, until all the candidate areas are combined, and determine the first coordinate information of the at least one text line area after the combination.

[0020] In one possible implementation, the selected candidate region and the unselected candidate region are combined, including: when the ratio of the overlapping area of ​​the unselected candidate region and the selected candidate region is greater than or equal to a first area ratio threshold, the candidate region with low confidence in the two candidate regions is deleted by a non-maximum suppression algorithm; when the unselected candidate region and the selected candidate region are adjacent in the long side direction or the overlapping area in the long side direction is less than the first area ratio threshold, the unselected candidate region and the selected candidate region are merged into one region.

[0021] In a possible implementation, the detector is further used to: obtain a text line image corresponding to the second coordinate information from the image to be recognized; grayscale the text line image; and send the grayscaled text line image to a text recognizer for text recognition.

[0022] In a possible implementation, before the grayscaled text line image is passed through a recognition model for text recognition, the detector is also used to: divide the grayscaled text line image into three categories: horizontal, vertical, and non-horizontal and non-vertical; perform affine transformation on the non-horizontal and non-vertical text line images so that all text line images are in the horizontal or vertical direction; traverse all vertical text line images, split each vertical text line image into multiple horizontal text line images and label them.

[0023] In a possible implementation, each vertical text line image is split into multiple horizontal text line images and annotated, including: using horizontal annotation to split each vertical text line image into multiple single-character horizontal text line images and annotate them.

[0024] In a possible implementation, the detector is further configured to: scale the text line image corresponding to the second coordinate information according to a second scaling ratio, and perform text recognition on the scaled text line image.

[0025] In a possible implementation manner, scaling the text line image to the second pixel ratio includes: geometric scaling, equal-width scaling, equal-length scaling, tile scaling, and zero-filling scaling.

[0026] In a possible implementation, the recognition model is also used to recognize space characters.

[0027] In one possible implementation, the operators within the recognition model do not include sliding window and recurrent neural network operator layers outside of convolution operations.

[0028] In a fourth aspect, an embodiment of the present application provides a terminal device, including: a convolutional network model trainer; a convolutional network model trainer, used to obtain multiple text line data; mark the space characters in the multiple text line data through preset labels; update the recognition model according to the marked text line data, and the updated recognition model is also used to recognize space characters.

[0029] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, which, when executed on a computer, cause the computer to execute any method according to the first aspect.

[0030] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, which, when executed on a computer, cause the computer to execute any method according to the second aspect.

[0031] In a seventh aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any method as described in the first aspect.

[0032] In an eighth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any method as described in the second aspect.

[0033] The embodiment of the present application provides a method and terminal device for recognizing text. In text area detection, the input image is scaled to a certain pixel ratio and then sent to the text area model for detection. The present application solution only needs to call the NPU once; based on the constructed real data test set, the time consumption of an input image under the current solution is effectively reduced compared with the original solution. In text content detection, the input text line image is resized and directly sent to the recognition model for recognition operation; the sliding window mechanism with a step size of 8 and a size of 32×32 is compared to recognize a typical book text line, and the time consumption is also effectively reduced when recognizing text lines of the same specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A schematic diagram of a network structure for extracting features from a neural network;

[0035] Figure 2 A schematic diagram of a text recognition method;

[0036] Figure 3 A flow chart of a text recognition method provided in an embodiment of the present application;

[0037] Figure 4 A flowchart of another text recognition method provided in an embodiment of the present application;

[0038] Figure 5A flowchart of another text recognition method provided in an embodiment of the present application;

[0039] Figure 6 A flowchart of another text recognition method provided in an embodiment of the present application;

[0040] Figure 7 A schematic diagram of a terminal for recognizing text provided in an embodiment of the present application;

[0041] Figure 8 A schematic diagram of another terminal for recognizing text provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The technical solution in the embodiment of the present application will be described below in conjunction with the accompanying drawings in the embodiment of the present application. It should be noted that the terminal device in the embodiment of the present application can be a mobile phone, a tablet computer, a wearable device, etc.

[0043] The embodiment of the present application is applied to a terminal device, and the image to be identified can be captured by a camera on the terminal device, or can be obtained from the Internet or other means. Then, text recognition is performed on the image to be identified to obtain the text line content in the image to be identified.

[0044] Figure 1 A schematic diagram of a network structure for extracting features from a neural network.

[0045] like Figure 1 As shown in the figure, in the detection stage of the image to be recognized, a sliding window of 640*640 is used for the input image to be recognized, where 640*640 is the pixel ratio. The input image to be recognized is segmented by a sliding window with overlapping areas to obtain multiple sub-images. The multiple sub-images are then sent to the neural network for learning. On a feature map with a height of 160 and a width of 160, each pixel is regressed to the four corner points of the border of the text area, and each pixel is judged to confirm whether the current pixel is a text area, and finally the text area detection is completed for the image to be recognized.

[0046] The network structure diagram is as follows: Figure 1 As shown in Figure 1, multiple convolution operations and pooling operations are performed on the input image to be recognized to extract image features. Figure 1In the process, the pooling unit is downsampled with a step size of 2, and then in the process of downsampling in pooling unit 3, pooling unit 4 and pooling unit 5, the feature layer of this size is extracted and spliced ​​in the upsampling process of the second half of the network structure. The splicing of multi-scale feature layers enables the neural network to have a better balance between the local features of the input image to be identified and the patch features of the larger area. However, this solution still has many problems. Since the image to be identified is segmented using a sliding window, multiple sub-images need to call the NPU multiple times. By calling the NPU model multiple times or by multi-batch processing (batch), the full capacity of the NPU parallel acceleration is not brought into play, and the problem of long time consumption is caused. Moreover, the multi-batch solution is limited by the current NPU's restriction that the input must be fixed in size, and the batch must be determined when the model is preset, so it causes a very large computational overhead when processing the image. In this solution, the input image to be identified is segmented by a sliding window with overlapping areas, and the overlapping areas will cause a large waste of resources in actual operation. At the same time, since multiple sub-images are segmented, when the final text area is merged after post-processing, the multi-word image requires an additional layer of processing between the sub-images in the post-processing mechanism, which brings additional computing overhead. If the resolution of the input image to be recognized is very high, this will inevitably make the processing time longer and affect the use.

[0047] Figure 2 A schematic diagram of a text recognition method.

[0048] In the image recognition stage, the input text line image is normalized to 32 pixels in height, and then the width of the text line image is adjusted proportionally according to the original text line aspect ratio. The text line image is then slid in sequence using a 32*32 sliding window, and multiple text areas are selected and sent to the neural network for text content recognition. In this scheme, the sliding step size of the sliding window is set to 4 and 8 according to different languages. The step size of 4 is for languages ​​such as English where the height of a single character is usually greater than the width, and the step size of 8 is for languages ​​such as Chinese where the aspect ratio of a single character is basically 1. In order to take into account the edge text in the width direction, the sliding window will also be padded at the edge of the image. Figure 2As shown in the figure, the original input text line image contains the text "HOPE". After the sliding window slides for multiple cuts, each sliding window fills the edge of the image at the same time, and then it is sent to the classification layer of the neural network for classification. The transcription layer merges the classification results of the classification layer in order and finally outputs the result "HOPE". The network input is a cut N*32*32 dimensional array, where N is the number of sliding windows, and 32*32 is the image content contained in each sliding window. The network output in this scheme is 37 or 7357, which corresponds to the number of characters that can be processed by the English recognition model and the Chinese recognition model, respectively. In the training process, the connectionist temporal classification (CTC) based on the neural network is used as the loss function. This loss function has the ability to process sequence context related information, and is used here to guide the training of the recognition of text line sequences.

[0049] However, this solution still has several problems. First, a sliding window with a width of 32 is used on the original layer, and the step size is set to 4 or 8, so the amount of calculation sent to the neural network for recognition will result in 8 times or 4 times repeated calculations. As a result, when performing calculations in the terminal device, repeated calculations and multiple calls to the neural network chip NPU are required. Secondly, the sliding window is truncated on the original layer, and all the sliding windows of an image are spliced ​​together, and CTC is used to calculate the loss of the input sequence and the sample's true content (ground truth, gt) sequence, so that the effectiveness of CTC is not truly utilized. Finally, for a picture, since the sliding window operation needs to be performed on each text line, the NPU at each moment can only process one line of text at a time, the parallel ability between text lines is extremely poor, and multiple calls to the NPU are required for processing.

[0050] Above Figure 1 , Figure 2 In the scheme, the image is segmented by sliding windows at different stages of text recognition, which increases the amount of subsequent calculations and causes serious time consumption. At the same time, the NPU is called multiple times, resulting in huge computing overhead and low efficiency. In response to the problems in the above scheme, the embodiment of the present application provides a text recognition method that can effectively reduce the time consumption of text recognition without calling the NPU multiple times.

[0051] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0052] Figure 3 A flow chart of a text recognition method provided in an embodiment of the present application.

[0053] like Figure 3As shown, a method for recognizing text is provided, the method comprising: step 3010, scaling the image to be recognized according to a first scaling ratio.

[0054] By scaling the input image to be recognized according to the first scaling ratio, the image to be recognized is reduced or enlarged according to the first scaling ratio. By scaling, the text line area in the input image to be recognized can be detected at one time, without the need to divide the larger image with recognition into multiple small sub-images through a sliding window for detection one by one. In one embodiment, the first scaling ratio can be 960*960. At the same time, the length and width scaling ratios can be recorded separately, such as the length scaling ratio sh and the width scaling ratio sw.

[0055] Step 3020: Determine first coordinate information corresponding to the text line area in the image to be recognized after scaling.

[0056] By detecting the scaled image with recognition, the region coordinates corresponding to all text line regions are found. The text line region can be one or more. It should be noted by those skilled in the art that the first coordinate information generally refers to the coordinate information of the text line region in the scaled image to be recognized, and does not specifically refer to the coordinate information corresponding to a certain text line region.

[0057] Step 3030: determine second coordinate information corresponding to the first coordinate information according to the first scaling ratio, where the second coordinate information is coordinate information of a text line area in the image to be identified.

[0058] After detecting the scaled image to be identified and obtaining the coordinates of all text line areas, the coordinate information is restored to the corresponding coordinate information in the original image to be identified according to the first scaling ratio, thereby completing the detection of the text line area of ​​the image to be identified. In one example, the coordinate information of the corresponding text line area in the original image to be identified is calculated according to the scaling ratios sh and sw recorded during the previous scaling, and the corresponding text line image in the image to be identified is obtained through the coordinate information of the corresponding text line area in the original image to be identified. By scaling the image to be identified, it is possible to avoid using a sliding window to divide the image to be identified into multiple sub-images, so that only one call to the NPU is required, which solves the limitation of multiple calls to the NPU, and at the same time completely solves the overlapping areas between sub-images and the extra computing overhead caused by the merging mechanism during post-processing, thus overcoming the efficiency bottleneck problem.

[0059] Step 3040, performing text recognition on the text line image corresponding to the second coordinate information through a recognition model to determine the text line content corresponding to the text line image; the image to be recognized includes the text line image.

[0060] In one embodiment, the text line image corresponding to the second coordinate information can be adjusted according to the second pixel ratio, and the adjusted text line image can be subjected to text recognition. The adjustment method may include: geometric scaling, equal width scaling, equal length scaling, tile scaling, and zero padding scaling. Personnel in the art should note that the adjustment method is not limited to the above-mentioned methods, and may also include any equivalent adjustment method.

[0061] In one embodiment, during the construction phase of the recognition model, the operators involved in the model do not include sliding window and recurrent neural network operator layers other than convolution operations, so the processing time of the model on the artificial intelligence chip of the terminal device can be short and efficient. Therefore, the processing advantages of the artificial intelligence chip in the terminal device can be brought into play without additional CPU operations and data transmission between the CPU and NPU.

[0062] The recognition model involved in the embodiment of the present application can be a full convolutional network model. Among them, the parameters output by the full convolutional network model can be [batch_num, word_num, 1, channel]. Batch_num is the number of samples processed in a batch. If multiple lines of text lines and images are to be processed in parallel, batch_num will be greater than 1. Word_num is the number of different characters in a language. Channel is the number of channels. We assume that A*A areas are the size of a character. If a text line is of A*H dimensions, then assuming that the average spacing between characters is stride, the number of channels is (HA) / Stride+1.

[0063] Then the text line image is input into the full convolutional network model for processing and obtaining the return result. When the terminal device processes, all operations of the full convolutional network model are completed in the artificial intelligence chip. When building a full convolutional network model, it is necessary to add a CTC loss layer after the network output layer so that the network weights can be learned through the maximum likelihood algorithm; in the prediction stage, it is only necessary to connect to the CTC Beam Search to obtain the best result in the cluster.

[0064] The recognition model in the embodiment of the present application solves the repeated calculations caused by using sliding windows on the original layer, and also solves the limitation that sequence targets and learning targets caused by the CTC mechanism cannot integrate contextual information on the network structure. At the same time, the recognition model can also process text lines in parallel, eliminate the width limit on the input, eliminate redundant calculations in the network structure, and improve NPU utilization.

[0065] The embodiment of the present application proposes for the first time a single-scale restoration detection method in a terminal device to solve the common high-resolution text line detection task in the terminal device. The text line area is detected by scaling the image to be recognized without sliding windows or multi-scale calculations. At the same time, high-resolution images can also be processed at one time, and only one NPU is called to run. Through the restoration mechanism, the first scaling ratio is retained. After the text position is detected, the coordinate information is restored to the original high-resolution image to be recognized, which effectively ensures the high accuracy of the recognition part, greatly optimizes the prediction speed of the terminal-side model, and improves the NPU utilization rate.

[0066] Figure 4 A flowchart of another text recognition method provided in an embodiment of the present application.

[0067] like Figure 4 After step 3010, the method further includes: step 3021, performing text area detection on the scaled image to be identified by using a neural network to obtain the confidence of at least one selected area.

[0068] In one example, the neural network may be a text region detection neural network. The neural network takes an image as input and the corner points of the text box region in the image as output. The text box region may be in the form of a convex quadrilateral. For the scaled image to be identified, the text region detection neural network extracts features from the image and detects the corner points of the text region. In the text region detection neural network, the feature layers are extracted by a fully connected layer on the feature layers with scaled sizes of 1 / 4 and 1 / 8, and the feature layers are spliced ​​back to the network backbone structure by an eltwise operator in the following two deconvolution layers. The feature layers with scaled sizes of 1 / 4 and 1 / 8 enable the text region detection neural network to detect texts of small and medium sizes, while the feature layers with scaled sizes of 1 / 16 in the network backbone ensure that the text region detection neural network detects texts of larger sizes. On the spliced ​​feature layers, a classification task and a regression task may be performed respectively. The classification task is used to determine whether the region in the original image to be identified mapped by the feature layer is a text region. The regression task is used to identify the four corner points of the text boundary of the region in the original image to be identified mapped by the feature layer. And the confidence of at least one detected region to be selected is calculated, and the confidence is the probability value of at least one region to be selected containing a text line.

[0069] The embodiment of the present application only needs to call the NPU once in this step, and use the text area detection neural network to perform text area detection on the scaled image to be identified.

[0070] Step 3022: Delete at least one candidate region whose confidence is less than the first confidence threshold, sort the confidence of the remaining at least one candidate region, and select the candidate region with the highest confidence.

[0071] The confidence of one or more candidate regions is obtained, and then at least one candidate region whose confidence is less than a first confidence threshold is deleted. The purpose is to delete regions with lower confidence and reduce the subsequent calculation burden. Then the confidence of the remaining one or more candidate regions is sorted, and the candidate region with the highest confidence is selected.

[0072] Step 3023: when the ratio of the overlapping areas of the unselected candidate regions to the selected candidate regions is greater than the first area ratio threshold, the candidate region with low confidence is deleted from the two candidate regions by using a non-maximum suppression algorithm.

[0073] In one example, the area ratio value may be a decimal or a percentage value, etc. For example, the area ratio value may be 0.8 or 80%, etc.

[0074] Step 3024: When the unselected candidate regions are adjacent to the selected candidate regions in the long side direction or the overlapping area in the long side direction is smaller than the first area ratio threshold, the unselected candidate regions and the selected candidate regions are merged into one region.

[0075] The unselected candidate areas are combined with the selected candidate areas, which may be suppressed or fused in one embodiment, wherein the suppression may be non-maximum suppression (NMS). By comparing the overlapping area ratio of the selected candidate areas and the unselected candidate areas, when there is a large overlap between the selected candidate areas and the unselected candidate areas, the text line areas with lower confidence are suppressed, that is, the text line areas with lower confidence are deleted. In one example, a first area ratio threshold may be set, and when the overlapping area ratio is greater than or equal to the first area ratio threshold, the text line areas with lower confidence are suppressed.

[0076] When the ratio of the overlapped edge areas of the selected candidate area and the unselected candidate area in the long side direction is less than the first area ratio threshold or when the edges are adjacent in the long side direction, the two candidate areas are merged into one area. In an example, it can also be determined whether the slopes of the selected candidate area and the unselected candidate area are the same. If they are the same, it can be determined that the selected candidate area and the unselected candidate area are on the same text line.

[0077] Step 3025, determine whether all the candidate areas have been combined. That is, determine whether all the candidate areas have been suppressed or fused. If there are still candidate areas that have not been suppressed or fused, continue to execute step 3022, sort the confidences of the remaining candidate areas that have not been suppressed or fused, and select the candidate area with the highest confidence to continue to suppress or merge with the unselected candidate areas. Until all the candidate areas are combined.

[0078] Step 3026, determining first coordinate information of at least one combined text line area.

[0079] By combining the selected regions, i.e., suppressing and fusing, the region coordinate information of one or more combined text line regions is finally obtained, and each text line is perfectly framed out from the image to be recognized. In one example, one or more text lines do not overlap with each other and are independent of each other.

[0080] In one example, the scaled image to be identified is subjected to text area detection, and the coordinate information of the text line area is calculated by the artificial neural network dedicated processing chip NPU computing unit. Then, one or more text line areas are screened by a preset first confidence threshold to screen out text line areas with high confidence. Then, according to the overlapping area of ​​different text line areas, suppression is performed, a part of the overlapping text line areas with low confidence is removed, and the adjacent text line areas are merged, and finally one or more relatively independent text line areas are obtained.

[0081] Figure 5 A flowchart of another text recognition method provided in an embodiment of the present application.

[0082] exist Figure 3 Before performing text recognition on the text line image corresponding to the second coordinate information by using the recognition model in step 3040, Figure 5 As shown, the following steps are also included:

[0083] Step 3050: Acquire a text line image corresponding to the second coordinate information from the image to be recognized.

[0084] The text line image of the corresponding text line area is selected from the original image to be recognized according to the second coordinate information.

[0085] Step 3060: grayscale the text line image. By grayscale the text line image, data information of the text line image can be effectively reduced, and the amount of subsequent calculations on the text line image can be reduced.

[0086] Step 3070, classifying the grayscaled text line image into three categories: horizontal direction, vertical direction, and non-horizontal and non-vertical direction.

[0087] The grayscaled text line images are classified into three categories: horizontal, vertical, and non-horizontal and non-vertical.

[0088] Step 3080: Perform affine transformation on text line images in non-horizontal and non-vertical directions, so that all text line images are in the horizontal direction or the vertical direction.

[0089] By using preset corresponding functions in different directions, the text line image in the non-horizontal and non-vertical directions is radially transformed to map the text line image in the non-horizontal and non-vertical directions to the horizontal direction or the vertical direction. In one example, the inclination angle between the text line image in the non-horizontal and non-vertical directions and the horizontal direction can be determined. If it is greater than a certain angle threshold, it is mapped to the vertical direction, otherwise it is mapped to the horizontal direction.

[0090] Step 3090, traverse all vertical text line images, split each vertical text line image into multiple horizontal text line images and annotate them.

[0091] Due to the framework limitations of the chips in most current terminal devices, the input dimension of the network cannot be adapted in the running state. Therefore, if horizontal and vertical texts are processed, two network models with different input dimensions are usually required to be preset for processing, which greatly increases the Rom space overhead of the terminal device. Therefore, the embodiment of the present application proposes to reduce the vertical text line recognition to the horizontal text line recognition.

[0092] For all vertical text line images, each character is horizontally annotated and split into multiple single-character horizontal text line images, and each split single-character horizontal text line image is annotated.

[0093] By using the above method, all vertical text line images are changed into multiple horizontal text line images, thus saving the ROM space overhead in the terminal device.

[0094] Step 3100: adjust the text line image according to the second pixel ratio.

[0095] To facilitate the subsequent recognition of the text line image, the pixel ratio of all text line images can be adjusted to a uniform second pixel ratio. In one example, the pixel ratio can be adjusted to a high-ratio size of 32*512. If the original text line image is not wide enough, it can be supplemented with 0.

[0096] Step 3110, sending the text line image to the recognition model for text recognition.

[0097] In one example, all text line images are grayed to reduce the amount of image data. Affine transformation is performed on the non-horizontal and non-vertical text line images in the grayed text line images to obtain grayed vertical or horizontal text line images. The vertical text line images are then split to obtain multiple horizontal text line images. At this point, all text line images are in the horizontal direction. In order to ensure faster recognition of text areas in the recognition model, all text line images in the horizontal direction can be uniformly adjusted to a high-ratio size of 32*512, and insufficient length can be compensated with 0 at the edges.

[0098] By using the above method, a text line image in a vertical direction can be converted into a text line image in a horizontal direction, which can greatly save the ROM space cost on the terminal device.

[0099] Figure 6 A flowchart of another text recognition method provided in an embodiment of the present application.

[0100] When Chinese and English are mixed, the space problem is particularly prominent, because there are natural spaces between Chinese characters, but they are narrower than those in English. The spacing between spaces is also different at different scales. Moreover, spaces do not appear in the font library. Some existing solutions are segmented by word segmentation models, some are based on char recognition models, and some ensure that there are no spaces in each line of text in annotation.

[0101] like Figure 6 As shown, the recognition model involved in the embodiment of the present application can also recognize spaces. In the recognition model training stage, the following steps can be used to enhance the sensitivity of the recognition model to space recognition. The steps include:

[0102] Step 6010, obtaining multiple text line data.

[0103] In the recognition model training stage, a large amount of text line data is needed to train the model. By obtaining multiple text line data, the model can be fully trained and learned.

[0104] Step 6020: Mark the space characters in the multiple text line data using preset labels.

[0105] When marking the above-mentioned multiple text line data, the space character is marked by a preset label, wherein the preset label indicates a space class.

[0106] Step 6030: update the recognition model according to the annotated text line data, and the updated recognition model is also used to recognize space characters.

[0107] During the recognition model training phase, the CTC loss layer continuously learns and learns the weights, making the recognition model more sensitive to space characters so that the space characters can be correctly identified in subsequent recognition model use.

[0108] The embodiment of the present application proposes for the first time to directly use CTC loss to solve the problem of space recognition in Chinese-English mixed text line detection. Through the above mechanism, the NPU chip on the terminal device can maximize its performance advantage in OCR tasks.

[0109] Figure 7 A schematic diagram of a terminal for recognizing text provided in an embodiment of the present application.

[0110] like Figure 7 As shown, the embodiment of the present application further provides a terminal device 700 , which includes: a processor 710 , a detector 720 , and a text recognizer 730 .

[0111] The processor 710 is configured to scale the image to be recognized according to a first scaling ratio.

[0112] The detector 720 is used to determine the first coordinate information corresponding to the text line area in the image to be recognized after scaling. The second coordinate information corresponding to the first coordinate information is determined according to the first scaling ratio, and the second coordinate information is the coordinate information of the text line area in the image to be recognized.

[0113] The text recognizer 730 is used to perform text recognition on the text line image corresponding to the second coordinate information through a recognition model to determine the text line content corresponding to the text line image. The image to be recognized includes the text line image.

[0114] In one example, the image to be identified includes at least one text line area. The detector 720 is also used to: use a neural network to perform text area detection on the scaled image to be identified, and obtain the confidence of at least one selected area. The confidence is the probability value of at least one selected area containing a text line. Delete at least one selected area whose confidence is less than a first confidence threshold, and sort the confidences of the remaining at least one selected area to select the selected area with the highest confidence. Combine the selected selected area with the unselected selected area, sort the confidences of at least one uncombined selected area, select the selected area with the highest confidence, combine the selected selected area with the unselected selected area, until all the selected areas are combined, and determine the first coordinate information of the at least one combined text line area.

[0115] In one example, the selected candidate area is combined with the unselected candidate area, including: when the ratio of the overlapping area between the unselected candidate area and the selected candidate area is greater than or equal to a first area ratio threshold, the text line area with low confidence in the two candidate areas is deleted by a non-maximum suppression algorithm. When the unselected candidate area is adjacent to the selected candidate area in the long side direction or the overlapping area in the long side direction is less than the first area ratio threshold, the unselected candidate area and the selected candidate area are merged into one area.

[0116] In one example, the detector 720 is further used to: obtain a text line image corresponding to the second coordinate information from the image to be recognized, grayscale the text line image, and send the grayscaled text line image to a text recognizer for text recognition.

[0117] In one example, before the grayscaled text line image is subjected to text recognition through the recognition model, the detector 720 is further used to: classify the grayscaled text line image into three categories: horizontal direction, vertical direction, and non-horizontal and non-vertical direction. Perform affine transformation on the text line image in the non-horizontal and non-vertical direction so that all text line images are in the horizontal direction or the vertical direction. Traverse all the text line images in the vertical direction, split each text line image in the vertical direction into multiple text line images in the horizontal direction and annotate them.

[0118] In one example, each vertical text line image is split into multiple horizontal text line images and annotated, including: using horizontal annotation, each vertical text line image is split into multiple single-character horizontal text line images and annotated.

[0119] In one example, the detector 720 is further configured to: scale the text line image corresponding to the second coordinate information according to a second scaling ratio, and perform text recognition on the scaled text line image.

[0120] In one example, the manner of scaling the text line image to the second pixel ratio includes: proportional scaling, equal-width scaling, equal-length scaling, tiled scaling, and zero-filling scaling.

[0121] In one example, the recognition model is also used to recognize space characters.

[0122] In one example, the operators within the recognition model do not include sliding window and recurrent neural network operator layers outside of convolution operations.

[0123] Figure 8 A schematic diagram of another terminal for recognizing text provided in an embodiment of the present application.

[0124] like Figure 8As shown, an embodiment of the present application also provides a terminal device 800, including: a convolutional network model trainer 810.

[0125] The convolutional network model trainer 810 is used to obtain multiple text line data; mark the space characters in the multiple text line data with preset labels; update the recognition model according to the marked text line data, and the updated recognition model is also used to recognize space characters.

[0126] In the text area detection of the embodiment of the present application, the input image is scaled to 960×960 and then sent to the text area model for detection. The embodiment of the present application only needs to call the NPU once. In the actual data test set, in the scheme involved in the embodiment of the present application, one input image takes about 280ms, which is effectively reduced compared to the existing scheme of about 920ms. In the text content detection of the embodiment of the present application, the input text line image is resized and directly sent to the recognition model for recognition operation. Compared with the sliding window mechanism with a step size of 8 and a size of 32×32, which takes about 120ms to recognize a typical line of book text, the scheme of the embodiment of the present application only takes about 35ms to recognize text lines of the same specification. It can be seen that the scheme involved in the embodiment of the present application can effectively improve the recognition speed.

[0127] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0128] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0129] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the specific implementation methods of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present application should be included in the scope of protection of the present application.

Claims

1. A method for recognizing text, characterized in that: The method comprises: scaling the image to be recognized according to a first scaling ratio; Determining first coordinate information corresponding to the text line area in the image to be recognized after scaling, specifically includes: Use a neural network to detect text regions in the scaled image to be recognized, and obtain the confidence of at least one region to be selected. Deleting the at least one candidate region whose confidence is less than the first confidence threshold, and sorting the confidences of the remaining at least one candidate region to select, and selecting the candidate region with the highest confidence; combining the selected candidate region with the unselected candidate region, sorting the confidences of the at least one candidate region that has not been combined, selecting the candidate region with the highest confidence, and combining the selected candidate region with the unselected candidate region until all the candidate regions are combined, and determining the first coordinate information of the at least one combined text line region; Determine second coordinate information corresponding to the first coordinate information according to the first scaling ratio, where the second coordinate information is coordinate information of a text line area in the image to be recognized; The text line image corresponding to the second coordinate information is subjected to text recognition through a recognition model to determine the text line content corresponding to the text line image; the image to be recognized includes the text line image.

2. The method according to claim 1, characterized in that The confidence level is a probability value that the at least one to-be-selected region contains a text line.

3. The method according to claim 2, characterized in that The combining the selected candidate area with the unselected candidate area includes: When the ratio of the overlapping areas of the unselected candidate areas and the selected candidate areas is greater than or equal to a first area ratio threshold, the candidate area with low confidence in the two candidate areas is deleted by a non-maximum suppression algorithm; When the unselected candidate regions are adjacent to the selected candidate regions in the long side direction or the ratio of the overlapping areas in the long side direction is less than a first area ratio threshold, the unselected candidate regions and the selected candidate regions are merged into one region.

4. The method according to claim 1, characterized in that: Before the step of identifying the text line image corresponding to the second coordinate information by using a recognition model, the method further includes: Acquire a text line image corresponding to the second coordinate information from the image to be recognized; Performing grayscale processing on the text line image; The text line image after grayscale processing is sent to the recognition model for text recognition.

5. The method according to claim 4, characterized in that Before performing text recognition on the grayscaled text line image through a recognition model, the method further includes: The grayscaled text line images are divided into three categories: horizontal, vertical, and non-horizontal and non-vertical; Performing affine transformation on text line images in non-horizontal and non-vertical directions so that all text line images are in the horizontal direction or the vertical direction; All vertical text line images are traversed, and each vertical text line image is split into multiple horizontal text line images and annotated.

6. The method according to claim 5, characterized in that The step of splitting each vertical text image line into a plurality of horizontal text line images and labeling them includes: Using the horizontal annotation form, each vertical text line image is split into multiple single-character horizontal text line images and annotated.

7. The method according to claim 1, characterized in that The performing text recognition on the text line image corresponding to the second coordinate information by using a recognition model also includes: scaling the text line image corresponding to the second coordinate information according to a second scaling ratio, and performing text recognition on the scaled text line image.

8. The method according to claim 7, characterized in that The manner of scaling the text line image to the second pixel ratio includes: proportional scaling, equal width scaling, equal length scaling, tile scaling and zero-filling scaling.

9. The method according to claim 1, characterized in that: The recognition model is also used to recognize space characters.

10. The method according to claim 9, characterized in that The method further comprises: Get multiple text line data; Marking the space characters in the plurality of text line data by using a preset label; The recognition model is updated according to the annotated text line data, and the updated recognition model is also used to recognize space characters.

11. A terminal device, characterized in that: The terminal device includes: a processor, a detector, and a text recognizer; The processor is used to scale the image to be recognized according to a first scaling ratio; The detector is used to determine the first coordinate information corresponding to the text line area in the image to be recognized after scaling, specifically including: Use a neural network to detect text regions in the scaled image to be recognized, and obtain the confidence of at least one region to be selected. Deleting the at least one candidate region whose confidence is less than the first confidence threshold, and sorting the confidences of the remaining at least one candidate region to select, and selecting the candidate region with the highest confidence; combining the selected candidate region with the unselected candidate region, sorting the confidences of the at least one candidate region that has not been combined, selecting the candidate region with the highest confidence, and combining the selected candidate region with the unselected candidate region until all the candidate regions are combined, and determining the first coordinate information of the at least one combined text line region; Determine second coordinate information corresponding to the first coordinate information according to the first scaling ratio, where the second coordinate information is coordinate information of a text line area in the image to be recognized; The text recognizer is used to perform text recognition on the text line image corresponding to the second coordinate information through a recognition model to determine the text line content corresponding to the text line image; the image to be recognized includes the text line image.

12. The terminal device according to claim 11, characterized in that: The image to be recognized includes at least one text line area; The confidence level is a probability value that the at least one to-be-selected region contains a text line.

13. The terminal device according to claim 12, characterized in that: The combining the selected candidate area with the unselected candidate area includes: When the ratio of the overlapping areas of the unselected candidate areas and the selected candidate areas is greater than or equal to a first area ratio threshold, the candidate area with low confidence in the two candidate areas is deleted by a non-maximum suppression algorithm; When the unselected candidate regions are adjacent to the selected candidate regions in the long side direction or the overlapping area in the long side direction is smaller than the first area ratio threshold, the unselected candidate regions and the selected candidate regions are merged into one region.

14. The terminal device according to claim 11, characterized in that: The detector is also used for: Acquire a text line image corresponding to the second coordinate information from the image to be recognized; Performing grayscale processing on the text line image; The text line image after grayscale processing is sent to the text recognizer for text recognition.

15. The terminal device according to claim 14, characterized in that: Before the grayscaled text line image is subjected to text recognition through a recognition model, the detector is further used to: The grayscaled text line images are divided into three categories: horizontal, vertical, and non-horizontal and non-vertical; Performing affine transformation on text line images in non-horizontal and non-vertical directions so that all text line images are in the horizontal direction or the vertical direction; All vertical text line images are traversed, and each vertical text line image is split into multiple horizontal text line images and annotated.

16. The terminal device according to claim 15, characterized in that: The step of splitting each vertical text image line into a plurality of horizontal text line images and labeling them includes: Using the horizontal annotation form, each vertical text line image is split into multiple single-character horizontal text line images and annotated.

17. The terminal device according to claim 11, characterized in that: The detector is further used to: scale the text line image corresponding to the second coordinate information according to a second scaling ratio, and perform text recognition on the scaled text line image.

18. The terminal device according to claim 17, characterized in that: The manner of scaling the text line image to the second pixel ratio includes: proportional scaling, equal width scaling, equal length scaling, tile scaling and zero-filling scaling.

19. The terminal device according to claim 11, characterized in that: The recognition model is also used to recognize space characters.

20. The terminal device according to claim 19, characterized in that: The device also includes: a convolutional network model trainer; The convolutional network model trainer is used to obtain multiple text line data; Marking the space characters in the plurality of text line data by using a preset label; The recognition model is updated according to the annotated text line data, and the updated recognition model is also used to recognize space characters.

21. A computer-readable storage medium comprising instructions, which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 9.

22. A computer-readable storage medium comprising instructions, which, when executed on a computer, cause the computer to perform the method according to claim 10.

23. A computer program product comprising instructions, which, when executed on a computer, causes the computer to perform the method according to any one of claims 1 to 9.

24. A computer program product comprising instructions, which, when executed on a computer, causes the computer to perform the method as claimed in claim 10.

Images