Character recognition method, device and equipment and computer medium

Abstract

The invention provides a character recognition method, device and equipment and a computer readable medium, and the method comprises the steps: obtaining a to-be-recognized image, then carrying out the detection of characters in the to-be-recognized image based on a pre-trained character detection model, and obtaining at least one character box, wherein the character detection model comprises a first feature detection head and a second feature detection head which are parallel, the first feature detection head is used for detecting conventional characters, the second feature detection head is used for detecting overlapped characters, and the first feature detection head and the second feature detection head share other network layers of the character detection model. And finally, character content recognition is performed based on the detected character box to obtain a character recognition result, so that character recognition in a character overlapping scene can be realized.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a text recognition method, device, equipment and computer-readable medium. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the application that are recited in the claims. The description herein is not admitted to be prior art by inclusion in this section.

[0003] With the development of computer vision technology, OCR (Optical Character Recognition) has been widely used in many fields such as image and text machine review, digital documents, book scanning, license plate recognition, and daily office work, greatly improving the efficiency of information processing. OCR is a technology that enables computers to read text in images and convert it into machine-encoded text. It uses computer vision and machine learning, especially deep learning technology, to convert text in images into editable and searchable text formats.

[0004] However, in practical applications, there are often overlapping texts in the images to be recognized, which poses a challenge to the optical character recognition (OCR) system, because traditional OCR technology usually assumes that each character or word is independent and non-overlapping. When the text overlaps, the traditional segmentation and recognition methods may fail, resulting in an increase in the error rate.

[0005] In view of the above problems, it is necessary to provide a technical solution that can effectively realize text recognition in text overlapping scenarios. Summary of the invention

[0006] Multiple aspects of the present application provide a text recognition method, apparatus, device and computer-readable storage medium for realizing text recognition in text overlapping scenarios.

[0007] In one aspect of the present application, a method for character recognition is provided, wherein the method comprises:

[0008] Obtain an image to be recognized;

[0009] Detecting the text in the image to be recognized based on a pre-trained text detection model to obtain at least one text frame, wherein the text detection model includes a first feature detection head and a second feature detection head in parallel, the first feature detection head is used to detect regular text, the second feature detection head is used to detect overlapping text, and the first feature detection head and the second feature detection head share other network layers of the text detection model;

[0010] The text content is recognized based on the detected text frame to obtain a text recognition result.

[0011] Another aspect of the present application provides a text recognition device, wherein the device comprises:

[0012] An image acquisition module, used for acquiring an image to be identified;

[0013] A text detection module, used to detect text in the image to be recognized based on a pre-trained text detection model to obtain at least one text frame, wherein the text detection model includes a first feature detection head and a second feature detection head in parallel, the first feature detection head is used to detect regular text, the second feature detection head is used to detect overlapping text, and the first feature detection head and the second feature detection head share other network layers of the text detection model;

[0014] The text recognition module is used to perform text content recognition based on the detected text frame to obtain a text recognition result.

[0015] Another aspect of the present application provides an electronic device, the electronic device comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor so that the at least one processor can perform the text recognition method as described above.

[0019] In another aspect of the present application, a computer-readable storage medium is provided, on which computer program instructions are stored. The computer program instructions can be executed by a processor to implement the text recognition method as described above.

[0020] In the solution provided in the embodiment of the present application, an image to be recognized is acquired, and then the text in the image to be recognized is detected based on a pre-trained text detection model to obtain at least one text frame, wherein the text detection model includes a first feature detection head and a second feature detection head in parallel, the first feature detection head is used to detect regular text, and the second feature detection head is used to detect overlapping text, the first feature detection head and the second feature detection head share other network layers of the text detection model, and finally text content recognition is performed based on the detected text frame to obtain a text recognition result, thereby realizing text recognition in a text overlapping scenario. On this basis, since the present application adopts a first feature detection head and a second feature detection head in parallel, The detection heads are used to detect regular text and overlapping text, so that each detection head can be more focused on its specific task, thereby significantly improving the recognition accuracy of text in scenarios with overlapping text. Moreover, since the first feature detection head and the second feature detection head share other network layers of the text detection model, this enables the present application solution to effectively process different types of text while maintaining a low computing cost, thereby enhancing robustness. In addition, since the above two detection heads can work in parallel, the processing time can be reduced when processing images containing regular text and overlapping text, thereby improving the overall efficiency of text recognition. Therefore, the present application solution can effectively improve the accuracy of text detection, enhance the robustness of text recognition, and improve the overall efficiency of text recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0022] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0023] Figure 1 A flowchart of a text recognition method provided in one embodiment of the present application;

[0024] Figure 2 A reasoning logic diagram of a text detection model provided in one embodiment of the present application;

[0025] Figure 3 A training logic diagram of a text detection model provided in one embodiment of the present application;

[0026] Figure 4 A structural diagram of a text recognition device provided in one embodiment of the present application;

[0027] Figure 5 A schematic diagram of the structure of an electronic device suitable for implementing the solution in the embodiments of the present application;

[0028] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0030] In a typical configuration of the present application, the terminal and the equipment of the service network each include one or more processors (CPU), input / output interface, network interface and memory.

[0031] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0032] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer program instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0033] Taking into account the challenges faced by traditional optical character recognition (OCR) technology when dealing with text overlap, the embodiments of the present application provide a text recognition method, apparatus, device and computer-readable storage medium to realize text recognition in text overlapping scenarios, which obtains an image to be recognized, and then detects the text in the image to be recognized based on a pre-trained text detection model to obtain at least one text frame, wherein the text detection model includes a first feature detection head and a second feature detection head in parallel, the first feature detection head is used to detect regular text, and the second feature detection head is used to detect overlapping text, the first feature detection head and the second feature detection head share other network layers of the text detection model, and finally, the text content is recognized based on the detected text frame to obtain a text recognition result, thereby realizing text overlapping scenarios. On this basis, since the present application adopts a parallel first feature detection head and a second feature detection head to detect regular text and overlapping text, each detection head can be more focused on its specific task, thereby significantly improving the recognition accuracy of text in scenarios with overlapping text, and since the first feature detection head and the second feature detection head share other network layers of the text detection model, this enables the present application solution to effectively process different types of text while maintaining a low computing cost, thereby enhancing robustness. In addition, since the above two detection heads can work in parallel, the processing time can be reduced when processing images containing regular text and overlapping text, thereby improving the overall efficiency of text recognition. Therefore, the present application solution can effectively improve the accuracy of text detection, enhance the robustness of text recognition, and improve the overall efficiency of text recognition.

[0034] The text recognition method, apparatus, device and computer-readable storage medium provided in the present application are described below in an exemplary manner.

[0035] In actual scenarios, the execution subject of the text recognition method provided in the embodiment of the present application can be a user device, or a device formed by integrating a user device and a network device through a network, or it can also be an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablet computers, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, which can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0036] Figure 1The processing flow of a text recognition method provided by an embodiment of the present application is shown, and the method at least includes the following processing steps:

[0037] Step S101, obtaining an image to be recognized.

[0038] Step S102: Detect the text in the image to be recognized based on a pre-trained text detection model to obtain at least one text frame, wherein the text detection model includes a first feature detection head and a second feature detection head in parallel, the first feature detection head is used to detect regular text, and the second feature detection head is used to detect overlapping text, and the first feature detection head and the second feature detection head share other network layers of the text detection model.

[0039] Step S103, performing text content recognition based on the detected text frame to obtain a text recognition result.

[0040] The above-mentioned text recognition method provided in the embodiment of the present application obtains an image to be recognized, and then detects the text in the image to be recognized based on a pre-trained text detection model to obtain at least one text frame, wherein the text detection model includes a parallel first feature detection head and a second feature detection head, the first feature detection head is used to detect regular text, and the second feature detection head is used to detect overlapping text, the first feature detection head and the second feature detection head share other network layers of the text detection model, and finally, the text content is recognized based on the detected text frame to obtain a text recognition result, thereby realizing text recognition in a text overlapping scene. On this basis, since the present application adopts a parallel first feature detection head and a second feature detection head The feature detection head is used to detect regular text and overlapping text, so that each detection head can focus more on its specific task, thereby significantly improving the recognition accuracy of text in scenarios with overlapping text. Since the first feature detection head and the second feature detection head share other network layers of the text detection model, this enables the present application solution to effectively process different types of text while maintaining a low computing cost, thereby enhancing robustness. In addition, since the above two detection heads can work in parallel, the processing time can be reduced when processing images containing regular text and overlapping text, thereby improving the overall efficiency of text recognition. Therefore, the present application solution can effectively improve the accuracy of text detection, enhance the robustness of text recognition, and improve the overall efficiency of text recognition.

[0041] In some modified implementations, the text detection model includes a backbone network, a feature detection layer, and a post-processing layer connected in sequence;

[0042] The backbone network is used to extract a multi-layer feature map of the image to be identified;

[0043] The feature detection layer includes a first feature detection head and a second feature detection head, wherein the first feature detection head is used to detect a regular text area according to the multi-layer feature map and generate a corresponding thermal value segmentation map and a threshold map, and the second feature detection head is used to detect an overlapping text area according to the multi-layer feature map and generate a corresponding thermal value segmentation map and a threshold map;

[0044] The post-processing layer is used to perform binarization processing on the corresponding heat value segmentation map according to the threshold map output by the feature detection layer to generate a binary image, and generate a text frame according to the edge of the binary image.

[0045] In the embodiment of the present application, the backbone network is one of the core components of the text detection model. Its main responsibility is to obtain multi-layer feature maps from the previous backbone network and use these multi-layer feature maps to perform specific task predictions. For example, text detection is performed in the present application. It is based on a pre-trained convolutional neural network. Through a series of convolution, pooling and other operations, multi-level features are extracted from the input image, and then feature maps of different scales are generated through feature pyramid technology, that is, multi-layer feature maps. These feature maps contain different levels of abstract information. These feature maps can be used to effectively detect text of different sizes and resolutions at different scales.

[0046] The feature detection head is another core component in the text detection model. In computer vision tasks, the feature detection head is specifically used to detect specific targets or objects, such as text areas, from the multi-layer feature map extracted from the backbone network. It generates a thermal value segmentation map and a threshold map by analyzing the information in the feature map. The former reflects the probability that each pixel belongs to text, and the latter is used to convert the thermal value segmentation map into a binary image to determine the boundary of the text area. By using the first feature detection head and the second feature detection head in parallel, the text recognition method provided in the embodiment of the present application can efficiently detect regular text and overlapping text respectively, thereby improving the accuracy and robustness of text detection.

[0047] The first feature detection head and the second feature detection head provided in the embodiment of the present application can adopt the same architecture. For example, they can both be composed of a convolution layer, an upsampling layer, a classifier and a regressor, wherein the convolution layer is used to further extract high-level features in the multi-layer feature map; the upsampling layer is used to restore the resolution by upsampling when the resolution of the feature map is lower than that of the original image, so as to generate an output of the same size as the original image; the classifier is used to predict the category probability of each position; the regressor is used to predict the position of the bounding box or the position of the key point.

[0048] Heatmap Segmentation Map is a probability distribution map used to represent different regions in an image belonging to a certain category, where the probability value of each pixel reflects the possibility that the pixel belongs to a certain category (such as text). Heatmap segmentation map is generated by the output layer of the feature detection head, which is usually a convolutional layer, and its purpose is to predict the possibility that each pixel in the image belongs to a specific category. In the embodiment of the present application, the above-mentioned specific category can be regular text and overlapping text.

[0049] The threshold image is a grayscale image generated corresponding to the thermal value segmentation map, in which the value of each pixel represents the threshold of the probability that the position belongs to a specific category. The threshold image can be used to binarize the thermal value segmentation map. For example, if the pixel value of the thermal value segmentation map is higher than the threshold in the threshold image of the corresponding position, the pixel is marked as white (usually representing the foreground), otherwise it is marked as black (usually representing the background). In order to obtain a better text edge detection effect, in an embodiment of the present application, the feature detection head can use an adaptive threshold algorithm to generate a threshold map. The adaptive threshold algorithm can calculate a suitable threshold for each pixel position, so that the threshold can be dynamically adjusted according to different areas of the image. Specifically, the average grayscale value or weighted average grayscale value of each pixel and the pixels around it can be calculated, and then a fixed offset (or an offset) is subtracted from the average value by applying the offset method to obtain the threshold corresponding to the pixel.

[0050] After obtaining the thermal value segmentation map and the threshold map, the post-processing layer can binarize the corresponding thermal value segmentation map according to the threshold map to generate a binary image. For example, for each pixel in the thermal value segmentation map, if the value of the pixel is higher than the corresponding threshold, it is set to white (usually 255), and if the value of the pixel is lower than the corresponding threshold, it is set to black (usually 0), thereby generating a binary image in which white pixels represent the parts recognized as text, and black pixels represent the background or other non-text parts.

[0051] After the binary image is generated, edge detection can be performed on the binary image to generate a text box surrounding the text. Specifically, the edge detection algorithm provided by a computer vision library such as OpenCV (Open Source Computer Vision Library) can be used to detect the edge of the text, thereby accurately generating a text box based on the detected text edge.

[0052] This embodiment, by introducing a backbone network and a feature detection layer, can better capture the features in the image, and by generating a thermal value segmentation map and an adaptive threshold map, can improve the recognition accuracy of the text area. The post-processing layer can further improve the accuracy of text box positioning, making subsequent text content recognition more accurate.

[0053] It should be noted that the other network layers mentioned in the above step S102, namely the above-mentioned backbone network and / or post-processing layer, can ensure that the first feature detection head and the second feature detection head receive the same multi-layer feature map as input from the backbone network by making the first feature detection head and the second feature detection head share the backbone network, so that the two can use the same underlying feature information for text detection, which helps to maintain the consistency of the model and avoid unnecessary repeated calculations. In addition, it also helps to reduce the number of model parameters and speed up training and inference speeds.

[0054] By making the first feature detection head and the second feature detection head share the same post-processing layer, the post-processing layer can use the same post-processing technology to process the data output by the first feature detection head and the second feature detection head, such as binarization method, edge detection algorithm, etc., which can also simplify the model structure, reduce model parameters, and improve the training speed and inference speed of the model.

[0055] Based on the above implementation, in some modified implementations, the post-processing layer includes a first post-processing branch corresponding to the first feature detection head, a second post-processing branch corresponding to the second feature detection head, and an integration node;

[0056] The first post-processing branch is used to perform binarization processing on the corresponding thermal value segmentation map according to the threshold map output by the first feature detection head to generate a binary image, and generate a regular text frame corresponding to the regular text according to the edge of the binary image;

[0057] The second post-processing branch is used to perform binarization processing on the corresponding thermal value segmentation map according to the threshold map output by the second feature detection head to generate a binary image, and generate an overlapping text frame corresponding to the overlapping text according to the edge of the binary image;

[0058] The input end of the integration node is connected to the first post-processing branch and the second post-processing branch, and is used to perform deduplication processing on the regular text frame and the overlapping text frame, and output non-overlapping text frames.

[0059] For reference Figure 2 The reasoning logic diagram of the text detection model provided by an embodiment of the present application is shown to understand, Figure 2The text detection head in is the first feature detection head, which is followed by a first post-processing branch for binarizing the corresponding thermal value segmentation map according to the threshold map output by the first feature detection head to generate a binary image, and using the OpenCV edge detection method to perform edge detection on the binarized image, and generate a regular text box corresponding to the regular text, as shown in the red rectangular box corresponding to the periphery of the character "hello" in the figure; the overlapping detection head is the second feature detection head, which is followed by a second post-processing branch for binarizing the corresponding thermal value segmentation map according to the threshold map output by the second feature detection head to generate a binary image, and using the OpenCV edge detection method to perform edge detection on the binarized image, and generate an overlapping text box corresponding to the overlapping text, as shown in the red rectangular box corresponding to the periphery of the character "hello" in the figure; the integration node is not shown in the figure.

[0060] The first post-processing branch and the second post-processing branch may be consistent in structure and parameters, or the first post-processing branch and the second post-processing branch may be only logically distinguished and may be the same post-processing branch within the model. It is only necessary to be able to distinguish between the regular text boxes and the overlapping text boxes in the output, and the integration node may be used for deduplication processing.

[0061] This implementation method can effectively process the detection results of regular text and overlapping text through the design of dual post-processing branches and the processing of integrated nodes, ensuring that the output text box will not have overlapping content, thereby improving the accuracy of text recognition.

[0062] In addition, the integration node can also handle some special cases. For example, when two feature detection heads detect the same text area, appropriate strategies can be adopted to select the best text box.

[0063] It should be noted that the overlapping text in this embodiment may refer to the part of text that overlaps on the regular text. The detected overlapping text is independent text that can be used for text content recognition. Correspondingly, the overlapping text box is a text box without overlap. For regular text, there are two possibilities in actual application. One is independent text without overlap, and the other is overlapped text. For the former, the text content can be directly recognized, and the corresponding regular text box is a text box without overlap. For the latter, the overlapping text superimposed on it needs to be removed in order to output a text box without overlap. For details, please refer to Figure 2 The reasoning logic diagram of the text detection model shown in the figure can be understood. Figure 2The "Hello" in it is regular text, on which the text "hello" is superimposed. By using the first feature detection head (i.e., the text detection head in the figure) and combining the subsequent binarization and edge detection steps, a regular text box can be detected. In addition to the regular text "Hello" in this regular text box, there is still the overlapping text "hello", and further deduplication processing is required to obtain a non-overlapping text box "Hello". By using the second feature detection head (i.e., the overlapping detection head in the figure) and combining the subsequent binarization and edge detection steps, an overlapping text box "hello" can be detected. This overlapping text box is also a non-overlapping text box and can be directly output.

[0064] Based on the above implementation manner, in some modified implementation manners, the integration node is specifically configured to perform deduplication processing by removing the overlapping text in the regular text box corresponding to the overlapping text box according to the position information of the overlapping text box.

[0065] Among them, the deduplication processing means the process of removing the overlapping part of two text boxes. It can compare the position information of the overlapping text box and the regular text box. If overlap is found, the overlapping part is removed from the regular text box.

[0066] Although Figure 2 the text box is represented in the form of a red rectangular box, in actual applications, the shape of the text box is not limited to a rectangular box. It can also be an irregular shape adjacent to the text. Since the position information of each pixel point has been inherited since the threshold map and heat value segmentation map are output from the feature detection head, according to the position information of the overlapping text box, the position of the overlapping text can be found in the regular text box corresponding to the overlapping text box and removed, thereby achieving deduplication processing.

[0067] Through this implementation manner, it can be ensured that the finally output text box is clear and non-overlapping, thereby avoiding confusion in the subsequent text content recognition process and improving the overall quality of text recognition.

[0068] Based on any of the foregoing implementation manners, in some modified implementation manners, before detecting the text in the image to be recognized based on a pre-trained text detection model, it further includes:

[0069] Using a sample image containing overlapping text to jointly train the first feature detection head and the second feature detection head to obtain a pre-trained text detection model.

[0070] The sample images containing overlapping text refer to images with overlapping text elements. Joint training refers to the technology of optimizing multiple model components or tasks simultaneously in a training process. In the embodiment of the present application, joint training refers to training the first feature detection head and the second feature detection head simultaneously so that they can be optimized together to solve the text detection task.

[0071] In this embodiment, by using sample images containing overlapping text to jointly train the first feature detection head and the second feature detection head, the following beneficial effects can be achieved:

[0072] On the one hand, by using sample images containing overlapping text to jointly train the first feature detection head and the second feature detection head, the training process can be simplified to ensure that the two detection heads are optimized in the same training process, which can reduce the resources and time required to train each detection head individually. Since the optimization is performed under the same framework, computing resources can be used more efficiently. In addition, since the first feature detection head and the second feature detection head share the same backbone network for feature extraction, repeated calculation of feature maps can be avoided, thereby saving computing costs.

[0073] On the other hand, sample images containing overlapping text increase the diversity of the training set, allowing the model to be trained on more types of inputs. This diversity helps improve the model's ability to generalize to unknown data. And by containing images with overlapping text, the model can be exposed to more difficult samples, which helps balance the impact of easy samples and difficult samples during training, and avoids the model overfitting simple examples and ignoring complex situations.

[0074] On the other hand, joint training ensures that the two detection heads have the same objective function, which means that they are optimized in the same direction, thereby reducing conflicts within the model and helping to achieve more consistent and coordinated prediction behaviors. The two detection heads influence each other during the training process, which can enable them to work better together. For example, the first feature detection head may focus on locating text areas, while the second feature detection head is responsible for further refining boundaries or dealing with overlapping problems. This synergistic effect helps to improve the overall detection accuracy.

[0075] In addition, by jointly training the model, the model can better learn how to distinguish between regular text and overlapping text, thereby improving the accuracy and robustness of text detection.

[0076] Based on the above implementation, in some modified implementations, the joint training of the first feature detection head and the second feature detection head using the sample image containing overlapping texts includes:

[0077] Acquire a training data set, the training data set including sample images and corresponding annotation information, the sample images including regular text and overlapping text, the annotation information including position information and text category of each text in the sample images, the text category including regular text and overlapping text;

[0078] The training data set is used to train a preset text detection model, wherein the first feature detection head is trained to recognize regular text in the sample image according to the text category, and the second feature detection head is trained to recognize overlapping text in the sample image according to the text category.

[0079] In this embodiment, the training data set includes sample images and their corresponding annotation information. The sample images contain regular text and overlapping text, and the annotation information includes the position information (such as bounding box coordinates) and text categories (such as regular text and overlapping text) of each text in the sample image. Based on the above position information and text categories, the above training data set can be used to jointly train the first feature detection head and the second feature detection head. For example, the first feature detection head is trained to recognize regular text in the sample image according to the text category, and the second feature detection head is trained to recognize overlapping text in the sample image according to the text category. During implementation, the parameters of the model can be adjusted by using a back propagation algorithm to minimize the difference between the model prediction results and the actual annotation information.

[0080] This embodiment, by using a training data set containing regular text and overlapping text, can ensure that the model can accurately classify and detect text in various situations, thereby improving the versatility and accuracy of text recognition.

[0081] Based on the above implementation, in some modified implementations, the using the training data set to train a preset text detection model includes:

[0082] Calculating loss functions of the first feature detection head and the second feature detection head respectively according to the text category, wherein the loss function is used to measure the difference between the prediction result corresponding to the sample image and the annotation information;

[0083] The model parameters of the first feature detection head and the second feature detection head are updated respectively according to the calculated value of the loss function until the preset training stop condition is met.

[0084] This implementation method can refer to Figure 3 The training logic diagram of the text detection model provided by an embodiment of the present application is shown as follows: Figure 3 As shown, Figure 3The sample image in it is an image composed of the regular text "你好" and the overlapping text "hello". After feature extraction using the backbone network, the first feature detection head (i.e., the text detection head in the figure) can detect the regular text box, such as Figure 3 the large red rectangular box with blank above the middle position. In addition, there is corresponding annotation information for the regular text "你好", which is shown in graphical form in the figure for easy understanding, such as Figure 3 shown by the large red rectangular box with text above the right position. In the embodiment of the present application, the difference between the blank red rectangular box and the red rectangular box with text can be calculated through the first loss function, and the model parameters of the first feature detection head can be adjusted accordingly. Similarly, the second feature detection head (i.e., the overlapping detection head in the figure) can detect the overlapping text box, such as Figure 3 the small red rectangular box with blank below the middle position. In addition, there is corresponding annotation information for the overlapping text "hello", which is shown in graphical form in the figure for easy understanding, such as Figure 3 shown by the small red rectangular box with text below the right position. In the embodiment of the present application, the difference between the blank red small rectangular box and the red small rectangular box with text can be calculated through the second loss function, and the model parameters of the second feature detection head can be adjusted accordingly. Until the preset training stop condition is met, the trained text recognition model can be obtained.

[0085] Among them, the selection of the loss function is crucial for the training process. The embodiment of the present application can adopt at least one function such as the cross-entropy loss function, the mean square error loss function or the bounding box regression loss function to measure the difference between the prediction result corresponding to the sample image and the annotation information. Those skilled in the art can flexibly select and implement according to actual needs, and the embodiment of the present application does not make a limitation.

[0086] During the training process, the model parameters of the feature detection head will be updated according to the calculated value of the loss function to minimize the loss function, which can be achieved through the backpropagation algorithm. Using the backpropagation algorithm, the model parameters can be updated according to the gradient of the loss function. This process will be repeated multiple times, and the model parameters will be updated each iteration until the preset training stop condition is reached.

[0087] In addition, the training stop condition can also be set according to specific training objectives, such as reaching the preset accuracy rate, the change of the loss function being less than the preset threshold or completing the preset number of iterations, etc. The embodiment of the present application also does not make a limitation. Once these conditions are met, the training process will stop.

[0088] This implementation method can continuously optimize the performance of the model by calculating the loss function and updating the model parameters, so that it can more accurately recognize regular text and overlapping text, and improve the overall efficiency and quality of text recognition. In addition, the parameter update during the training process is carried out gradually, so that the model can gradually approach the optimal solution and improve the generalization ability of the model.

[0089] After obtaining at least one non-overlapping text frame, text content recognition can be performed based on the detected text frame to obtain a text recognition result. Specifically, text content recognition is a key link in an optical character recognition (OCR) system. It follows text detection and aims to recognize specific text content from a text frame extracted from an image. It can be achieved by the following steps:

[0090] First, each text box is cropped from the original image. This operation can be done by using the detected text box coordinates to effectively reduce interference factors in subsequent processing and improve the accuracy of text recognition. Subsequently, in order to improve the image quality and prepare for subsequent text recognition, the cropped text image needs to be preprocessed. The preprocessing steps include converting the image to a grayscale image, using binarization to enhance the contrast between the text and the background, using filters to remove noise from the image, and adjusting the image size to meet the requirements of the text recognition model. These preprocessing measures can significantly improve the accuracy of text recognition.

[0091] Then, the text recognition model is used to identify the text content in the text box. This can be achieved using methods based on template matching and deep learning. The former uses a known text template to compare with the input image, while the latter uses a deep learning model, especially a convolutional neural network (CNN), to directly identify the characters in the text image. It is worth noting that the end-to-end recognition model can directly recognize the complete text string from the image without first performing character segmentation; in contrast, the character segmentation-based method first divides the text image into individual characters and then recognizes each character separately. Regardless of which method is used, the ultimate goal is to identify the specific content in the text box.

[0092] After the text recognition is completed, post-processing is required to improve the quality of the recognition results. This process may include spell checking to correct spelling errors in the recognized text, using language models to correct grammatical errors or words that do not make sense in the context, and optimizing the order of recognition results based on contextual relationships.

[0093] Through the above steps, accurate recognition of text content can be achieved based on the text frame, thereby achieving accurate recognition of text in a text overlapping scenario.

[0094] In addition, the embodiments of the present application also provide a simplified implementation method. For example, the text detection model provided by any of the aforementioned implementation methods of the present application can be implemented on the basis of the text detection model based on the DB (Differentiable Binarization) neural network provided by the prior art by adding an additional overlapping text feature detection head, namely, a second feature detection head. The second feature detection head works in parallel with the original first feature detection head for detecting conventional text, but its learning goal and task is to identify and mark overlapping areas. Specifically, the backbone network of the model remains unchanged for extracting features of the image; and after the backbone network, there was originally only one detection head (i.e., the first feature detection head) for detecting text, and now an additional second feature detection head is added for learning and identifying overlapping text.

[0095] During the training phase, the model receives both sample images of normal text and sample images containing overlapping text. For sample images containing overlapping text, an additional second feature detection head is trained to identify which parts are overlapping. This training process does not require additional datasets or training costs. It only requires adding sample images containing overlapping text to the existing dataset. When the model is training, the original first feature detection head continues to learn how to recognize the presence of regular text, while the newly added second feature detection head learns how to recognize overlapping text. This dual-detection head architecture allows the model to retain its original detection capabilities while gaining new skills in recognizing overlapping text.

[0096] In the inference phase, the model uses a small amount of additional inference resources to handle the recognition of overlapping text. This means that in practical applications, the model can accurately detect overlapping text and separate the overlapping parts through post-processing algorithms. For example, the original first feature detection head will output a heat value segmentation map, which is then binarized by a pre-set threshold, and then the binarized segmentation map is detected using the OpenCV border detection algorithm to obtain a regular text box. The newly added second feature detection head will also output a completely independent heat value segmentation map, but the segmented heat map is the heat map of the overlapping part. The same method is used to perform binarization and border detection on this overlapping heat map to obtain the border of the overlapping text. Extracting this part of the border separately is the overlapping text box. The above additional inference resources are reflected in a slight increase in computing time or hardware resources, but because the additional second feature detection head works in parallel with the original first feature detection head and only needs to focus on the overlapping area, it will not cause a significant resource burden overall.

[0097] For the newly added second feature detection head, its parameter setting can follow the parameters of the original first feature detection head, which not only simplifies the training and adjustment process of the model, but also ensures the stability of the model when processing overlapping text.

[0098] Based on the same inventive concept, a text recognition device is also provided in the embodiment of the present application. The method corresponding to the device can be the text recognition method in the aforementioned embodiment, and its principle of solving the problem is similar to that of the method. The text recognition device provided in the embodiment of the present application can implement the above-mentioned text recognition method, and the text recognition device can be implemented by software, hardware, or a combination of software and hardware. For example, the text recognition device may include integrated or separate functional modules or units to perform the corresponding steps in the above-mentioned methods. Since the device embodiment is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described below is merely illustrative. Please refer to Figure 4 , the text recognition device may include:

[0099] An image acquisition module 101 is used to acquire an image to be recognized;

[0100] A text detection module 102 is used to detect text in the image to be recognized based on a pre-trained text detection model to obtain at least one text frame, wherein the text detection model includes a first feature detection head and a second feature detection head in parallel, the first feature detection head is used to detect regular text, the second feature detection head is used to detect overlapping text, and the first feature detection head and the second feature detection head share other network layers of the text detection model;

[0101] The text recognition module 103 is used to perform text content recognition based on the detected text frame to obtain a text recognition result.

[0102] In some modified implementations, the text detection model includes a backbone network, a feature detection layer, and a post-processing layer connected in sequence;

[0103] The backbone network is used to extract a multi-layer feature map of the image to be identified;

[0104] The feature detection layer includes a first feature detection head and a second feature detection head, wherein the first feature detection head is used to detect a regular text area according to the multi-layer feature map and generate a corresponding thermal value segmentation map and a threshold map, and the second feature detection head is used to detect an overlapping text area according to the multi-layer feature map and generate a corresponding thermal value segmentation map and a threshold map;

[0105] The post-processing layer is used to perform binarization processing on the corresponding heat value segmentation map according to the threshold map output by the feature detection layer to generate a binary image, and generate a text frame according to the edge of the binary image.

[0106] Based on the above implementation, in some modified implementations, the post-processing layer includes a first post-processing branch corresponding to the first feature detection head, a second post-processing branch corresponding to the second feature detection head, and an integration node;

[0107] The first post-processing branch is used to perform binarization processing on the corresponding thermal value segmentation map according to the threshold map output by the first feature detection head to generate a binary image, and generate a regular text frame corresponding to the regular text according to the edge of the binary image;

[0108] The second post-processing branch is used to perform binarization processing on the corresponding thermal value segmentation map according to the threshold map output by the second feature detection head to generate a binary image, and generate an overlapping text frame corresponding to the overlapping text according to the edge of the binary image;

[0109] The input end of the integration node is connected to the first post-processing branch and the second post-processing branch, and is used to perform deduplication processing on the regular text frame and the overlapping text frame, and output non-overlapping text frames.

[0110] Based on the above implementation, in some modified implementations, the integration node is specifically used to implement deduplication processing by removing overlapping text in a regular text box corresponding to the overlapping text box according to the position information of the overlapping text box.

[0111] Based on any of the foregoing embodiments, in some modified embodiments, the device further includes:

[0112] The model training module is used to jointly train the first feature detection head and the second feature detection head using sample images containing overlapping text to obtain a pre-trained text detection model.

[0113] Based on the above implementation, in some modified implementations, the model training module includes:

[0114] A training data acquisition unit, used to acquire a training data set, wherein the training data set includes a sample image and corresponding annotation information, wherein the sample image includes regular text and overlapping text, and the annotation information includes position information and text category of each text in the sample image, wherein the text category includes regular text and overlapping text;

[0115] A model training unit is used to train a preset text detection model using the training data set, wherein the first feature detection head is trained to recognize regular text in the sample image according to the text category, and the second feature detection head is trained to recognize overlapping text in the sample image according to the text category.

[0116] Based on the above implementation, in some modified implementations, the model training unit includes:

[0117] A loss function calculation subunit, used to calculate the loss function of the first feature detection head and the second feature detection head respectively according to the text category, wherein the loss function is used to measure the difference between the prediction result corresponding to the sample image and the annotation information;

[0118] The parameter updating subunit is used to update the model parameters of the first feature detection head and the second feature detection head respectively according to the calculated value of the loss function until the preset training stop condition is met.

[0119] The text recognition device provided in the embodiment of the present application is based on the same inventive concept as the text recognition method provided in the aforementioned embodiment of the present application and has the same beneficial effects.

[0120] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present application, and the method corresponding to the electronic device may be the text recognition method in the aforementioned embodiment, and its principle of solving the problem is similar to that of the method. The electronic device provided in an embodiment of the present application includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the text recognition methods and / or technical solutions of the aforementioned multiple embodiments of the present application.

[0121] The electronic device may be a user device, or a device formed by integrating a user device and a network device through a network, or may be an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablet computers, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, which can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0122] Figure 5The structure of an electronic device suitable for implementing the method and / or technical solution in the embodiment of the present application is shown, and the electronic device 1200 includes a central processing unit (CPU, Central Processing Unit) 1201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM, Read Only Memory) 1202 or the program loaded from the storage part 1208 to the random access memory (RAM, Random Access Memory) 1203. In RAM 1203, various programs and data required for system operation are also stored. CPU 1201, ROM 1202 and RAM1203 are connected to each other through bus 1204. Input / output (I / O, Input / Output) interface 1205 is also connected to bus 1204.

[0123] The following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, a touch screen, a microphone, an infrared sensor, etc.; an output section 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), an LED display, an OLED display, etc., and a speaker, etc.; a storage section 1208 including one or more computer-readable media such as a hard disk, an optical disk, a magnetic disk, a semiconductor memory, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1209 performs communication processing via a network such as the Internet.

[0124] In particular, the methods and / or embodiments in the embodiments of the present application may be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 1201, the above functions defined in the method of the present application are executed.

[0125] Another embodiment of the present application further provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of the present application described above.

[0126] Specifically, the present embodiment may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination therewith.

[0127] Computer readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0128] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0129] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0130] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0131] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0132] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or page components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0133] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0134] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0135] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0137] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in a device claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

Claims

1. A method for character recognition, wherein: The method comprises: Obtain an image to be recognized; Detecting the text in the image to be recognized based on a pre-trained text detection model to obtain at least one text frame, wherein the text detection model includes a first feature detection head and a second feature detection head in parallel, the first feature detection head is used to detect regular text, the second feature detection head is used to detect overlapping text, and the first feature detection head and the second feature detection head share other network layers of the text detection model; The text content is recognized based on the detected text frame to obtain a text recognition result.

2. The text recognition method according to claim 1, wherein: The text detection model includes a backbone network, a feature detection layer and a post-processing layer connected in sequence; The backbone network is used to extract a multi-layer feature map of the image to be identified; The feature detection layer includes a first feature detection head and a second feature detection head, wherein the first feature detection head is used to detect a regular text area according to the multi-layer feature map and generate a corresponding thermal value segmentation map and a threshold map, and the second feature detection head is used to detect an overlapping text area according to the multi-layer feature map and generate a corresponding thermal value segmentation map and a threshold map; The post-processing layer is used to perform binarization processing on the corresponding heat value segmentation map according to the threshold map output by the feature detection layer to generate a binary image, and generate a text frame according to the edge of the binary image.

3. The text recognition method according to claim 2, wherein: The post-processing layer comprises a first post-processing branch corresponding to the first feature detection head, a second post-processing branch corresponding to the second feature detection head, and an integration node; The first post-processing branch is used to perform binarization processing on the corresponding thermal value segmentation map according to the threshold map output by the first feature detection head to generate a binary image, and generate a regular text frame corresponding to the regular text according to the edge of the binary image; The second post-processing branch is used to perform binarization processing on the corresponding thermal value segmentation map according to the threshold map output by the second feature detection head to generate a binary image, and generate an overlapping text frame corresponding to the overlapping text according to the edge of the binary image; The input end of the integration node is connected to the first post-processing branch and the second post-processing branch, and is used to perform deduplication processing on the regular text frame and the overlapping text frame, and output non-overlapping text frames.

4. The text recognition method according to claim 3, wherein: The integration node is specifically used to implement deduplication processing by removing overlapping text in a regular text box corresponding to the overlapping text box according to the position information of the overlapping text box.

5. The text recognition method according to claim 1, wherein: Before detecting the text in the image to be recognized based on the pre-trained text detection model, the method further includes: The first feature detection head and the second feature detection head are jointly trained using sample images containing overlapping text to obtain a pre-trained text detection model.

6. The text recognition method according to claim 5, wherein: The method of jointly training the first feature detection head and the second feature detection head by using a sample image containing overlapping texts includes: Acquire a training data set, the training data set including sample images and corresponding annotation information, the sample images including regular text and overlapping text, the annotation information including position information and text category of each text in the sample images, the text category including regular text and overlapping text; The training data set is used to train a preset text detection model, wherein the first feature detection head is trained to recognize regular text in the sample image according to the text category, and the second feature detection head is trained to recognize overlapping text in the sample image according to the text category.

7. The text recognition method according to claim 6, wherein: The adopting the training data set to train a preset text detection model includes: Calculating loss functions of the first feature detection head and the second feature detection head respectively according to the text category, wherein the loss function is used to measure the difference between the prediction result corresponding to the sample image and the annotation information; The model parameters of the first feature detection head and the second feature detection head are updated respectively according to the calculated value of the loss function until the preset training stop condition is met.

8. A text recognition device, wherein: The device comprises: An image acquisition module, used for acquiring an image to be identified; A text detection module, used to detect text in the image to be recognized based on a pre-trained text detection model to obtain at least one text frame, wherein the text detection model includes a first feature detection head and a second feature detection head in parallel, the first feature detection head is used to detect regular text, the second feature detection head is used to detect overlapping text, and the first feature detection head and the second feature detection head share other network layers of the text detection model; The text recognition module is used to perform text content recognition based on the detected text frame to obtain a text recognition result.

9. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A computer readable medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the method according to any one of claims 1 to 7.

Images