Text recognition method and apparatus

Abstract

The application provides a character recognition method and device, which can be used in the field of artificial intelligence technology, and the method comprises the following steps: obtaining a target multi-line character image; determining a plurality of predicted box positions in the target multi-line character image according to the target multi-line character image and a preset positioning model, wherein the preset positioning model is obtained by pre-training a target detection algorithm based on a batch of historical multi-line character images and their respective corresponding predicted box position labels; determining a score corresponding to each predicted box according to a preset weight matrix and each predicted box position; horizontally splicing each predicted box based on the score corresponding to each predicted box to obtain a new character image, performing character recognition on the new character image, and obtaining a character recognition result of the target multi-line character image. The application can improve the accuracy and efficiency of character recognition, thereby improving the customer service experience.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and apparatus for character recognition. Background Technology

[0002] Currently, the entry, verification, and review of bank counter service vouchers involve a large amount of content, a wide range of areas, and a heavy workload. When a single element in a service voucher has a lot of content, factors such as inconsistent order of handwritten areas by customers and different printing habits of different branches can lead to a heavy workload for staff in the entry, verification, and review processes, resulting in slow overall business processing time and a reduced customer experience.

[0003] With the advent of the intelligent era, models trained using artificial intelligence to locate and recognize existing data can identify text with relatively standardized layout or independent content. However, when recognizing content with irregular formatting and specific recognition order, problems arise such as only recognizing partial content or incorrect content order, failing to effectively reduce the workload of banking staff. Therefore, a method capable of detecting irregular multi-line text is urgently needed to quickly respond to the demand for intelligent recognition of counter service documents, accelerate the detection and recognition of irregular multi-line text, and reduce manual costs. Summary of the Invention

[0004] To address at least one problem in the prior art, this application proposes a character recognition method and apparatus that can improve the accuracy and efficiency of character recognition, thereby enhancing the customer's business experience.

[0005] To address the aforementioned technical problems, this application provides the following technical solution:

[0006] Acquire the target multi-line text image;

[0007] Based on the target multi-line text image and the preset localization model, the positions of multiple prediction boxes in the target multi-line text image are determined. The preset localization model is obtained by pre-training the target detection algorithm based on a batch of historical multi-line text images and their respective corresponding prediction box position labels.

[0008] Based on the preset weight matrix and the position of each prediction box, determine the score corresponding to each prediction box;

[0009] Based on the scores corresponding to each prediction box, the prediction boxes are horizontally stitched together to obtain a new text image. The new text image is then subjected to text recognition to obtain the text recognition result of the target multi-line text image.

[0010] In one embodiment, the character recognition method further includes:

[0011] Obtain batches of historical multi-line text images and their corresponding prediction box location labels;

[0012] The target detection algorithm is trained by applying a batch of historical multi-line text images and their corresponding prediction box position labels to obtain the preset localization model.

[0013] In one embodiment, determining the score corresponding to each prediction box based on a preset weight matrix and the position of each prediction box includes:

[0014] Based on the positions of each predicted bounding box, a position matrix is ​​constructed;

[0015] Multiply the preset weight matrix and the position matrix to obtain a score matrix, which includes the score corresponding to each prediction box.

[0016] In one embodiment, the step of horizontally stitching together the prediction boxes based on their scores to obtain a new text image includes:

[0017] The prediction boxes are sorted from largest to smallest based on their scores, and then horizontally stitched together based on the sorting results to obtain a new text image.

[0018] In one embodiment, before determining the positions of multiple predicted bounding boxes in the target multi-line text image based on the target multi-line text image and a preset localization model, the method further includes:

[0019] Retrieve batches of historical multi-line text images and their corresponding text tags;

[0020] The verification steps include: applying a batch of historical multi-line text images and the preset localization model to obtain the predicted bounding box positions of each historical multi-line text image; obtaining new text images corresponding to each historical multi-line text image based on the preset weight matrix and the predicted bounding box positions of each historical multi-line text image; performing text recognition on each new text image to obtain the text recognition results of each historical multi-line text image; and obtaining the recognition accuracy based on the text recognition results and text labels of each historical multi-line text image. If the recognition accuracy is greater than the accuracy threshold, then the preset localization model is determined to have passed verification.

[0021] In one embodiment, the character recognition method further includes:

[0022] If the recognition accuracy is less than or equal to the accuracy threshold, then the preset positioning model verification is determined to have failed.

[0023] Update the number of training rounds for the object detection algorithm;

[0024] The target detection algorithm is trained based on the updated training rounds, batch of historical multi-line text images and their corresponding prediction box position labels to obtain the retrained localization model;

[0025] The retrained localization model is applied to perform the verification step again until the preset localization model passes the verification.

[0026] In one embodiment, the step of performing character recognition on the new text image to obtain the character recognition result of the target multi-line text image includes:

[0027] By applying a preset character recognition model and the new character image, the character recognition result of the target multi-line character image is obtained;

[0028] The preset text recognition model is obtained by pre-training a convolutional recurrent neural network model based on a batch of text images and their corresponding text labels.

[0029] Secondly, this application provides a character recognition device, comprising:

[0030] The acquisition module is used to acquire the target multi-line text image;

[0031] The localization module is used to determine the positions of multiple prediction boxes in the target multi-line text image based on the target multi-line text image and a preset localization model. The preset localization model is obtained by pre-training the target detection algorithm based on a batch of historical multi-line text images and their respective corresponding prediction box position labels.

[0032] The determination module is used to determine the score corresponding to each prediction box based on the preset weight matrix and the position of each prediction box;

[0033] The text recognition module is used to horizontally stitch together the prediction boxes based on the scores corresponding to each prediction box to obtain a new text image, and to perform text recognition on the new text image to obtain the text recognition result of the target multi-line text image.

[0034] In one embodiment, the character recognition device further includes:

[0035] The first historical data acquisition module is used to acquire batches of historical multi-line text images and their corresponding prediction box position labels.

[0036] The training module is used to train the target detection algorithm by applying a batch of historical multi-line text images and their corresponding prediction box position labels to obtain the preset localization model.

[0037] In one embodiment, the determining module includes:

[0038] The building unit is used to construct a position matrix based on the positions of each prediction box;

[0039] A score matrix unit is determined, which is used to multiply the preset weight matrix and the position matrix to obtain a score matrix, wherein the score matrix includes the score corresponding to each prediction box.

[0040] In one embodiment, the text recognition module includes:

[0041] The stitching unit is used to sort the prediction boxes from largest to smallest based on the scores corresponding to each prediction box, and then stitch the prediction boxes horizontally based on the sorting results to obtain a new text image.

[0042] In one embodiment, the text recognition device further includes:

[0043] The second historical data acquisition module is used to acquire batches of historical multi-line text images and their corresponding text labels.

[0044] The first verification module is used to execute verification steps, which include: applying a batch of historical multi-line text images and the preset localization model to obtain the predicted bounding box positions of each historical multi-line text image; obtaining new text images corresponding to each historical multi-line text image based on the preset weight matrix and the predicted bounding box positions of each historical multi-line text image; performing text recognition on each new text image to obtain the text recognition results of each historical multi-line text image; obtaining the recognition accuracy based on the text recognition results and text labels of each historical multi-line text image; and determining that the preset localization model has passed verification if the recognition accuracy is greater than the accuracy threshold.

[0045] In one embodiment, the character recognition device further includes:

[0046] The second verification module is used to determine that the preset positioning model verification has failed if the recognition accuracy is less than or equal to the accuracy threshold.

[0047] An update module is used to update the number of training rounds of the object detection algorithm;

[0048] The retraining module is used to train the target detection algorithm based on the updated training rounds, batch of historical multi-line text images and their corresponding prediction box position labels, to obtain the retrained localization model;

[0049] The re-verification module is used to apply the retrained localization model to perform the verification steps again until the preset localization model passes the verification.

[0050] In one embodiment, the text recognition module includes:

[0051] The text recognition unit is used to apply a preset text recognition model and the new text image to obtain the text recognition result of the target multi-line text image;

[0052] The preset text recognition model is obtained by pre-training a convolutional recurrent neural network model based on a batch of text images and their corresponding text labels.

[0053] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the character recognition method.

[0054] Fourthly, this application provides a computer-readable storage medium storing computer instructions thereon, which, when executed, implement the text recognition method described above.

[0055] As can be seen from the above technical solution, this application provides a text recognition method and apparatus. The method includes: acquiring a target multi-line text image; determining the positions of multiple prediction boxes in the target multi-line text image based on the target multi-line text image and a preset localization model, wherein the preset localization model is obtained by pre-training a target detection algorithm based on batches of historical multi-line text images and their corresponding prediction box position labels; determining the score corresponding to each prediction box based on a preset weight matrix and each prediction box position; horizontally stitching the prediction boxes based on their scores to obtain a new text image; and performing text recognition on the new text image to obtain the text recognition result of the target multi-line text image. This method can improve the accuracy and efficiency of text recognition, thereby improving the customer's business processing experience. Specifically, through a series of calculations using YOLO, the position of irregular multi-line text in business documents is detected, and combined with the constructed position weight values, the order of horizontal stitching of each fragment is determined. This effectively solves the problem of recognizing irregular multi-line text, accelerates the process of intelligent recognition replacing manual labor, optimizes business processing time, and enhances the customer's business processing experience. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 This is a flowchart illustrating the character recognition method in the embodiments of this application;

[0058] Figure 2This is a schematic diagram of a multi-line text image fragment with irregular text order, as exemplified in this application;

[0059] Figure 3 This is a schematic diagram of a multi-line text image fragment with irregular text order, as another example of this application;

[0060] Figure 4 This is a schematic diagram of a fragment of a multi-line text image with normal text order, as exemplified in this application;

[0061] Figure 5 This is a flowchart illustrating steps 011 and 012 of the character recognition method in the embodiments of this application;

[0062] Figure 6 This is a flowchart illustrating steps 301 and 302 of the character recognition method in the embodiments of this application;

[0063] Figure 7 This is a novel text image illustration as an example in this application;

[0064] Figure 8 This is a flowchart illustrating steps 101 to 105 of the character recognition method in an application example of this application;

[0065] Figure 9 This is a flowchart illustrating steps 21 to 24 of the character recognition method in the application example of this application;

[0066] Figure 10 This is a flowchart illustrating steps 31 to 34 of the character recognition method in an application example of this application;

[0067] Figure 11 This is a schematic diagram of the structure of the character recognition device in the embodiments of this application;

[0068] Figure 12 This is a schematic block diagram illustrating the system configuration of an electronic device according to an embodiment of this application. Detailed Implementation

[0069] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0070] To address the problems existing in the prior art, this application considers uploading a multi-line text image; applying a pre-trained YOLO localization model to determine the predicted bounding box position of each text segment in the multi-line text image; using position weights to determine the weight value of the predicted bounding box corresponding to each character; horizontally stitching the predicted bounding boxes to obtain a new image with all text in one line; and based on the new image and the text recognition model, obtaining the text recognition result of the target image. This solves the problem of recognizing irregular multi-line text.

[0071] Therefore, in order to improve the accuracy and efficiency of text recognition and thus enhance the customer's business experience, this application provides a text recognition device. This device can be a server or a client device. The client device can include smartphones, tablet computers, network set-top boxes, portable computers, desktop computers, personal digital assistants (PDAs), in-vehicle devices, and smart wearable devices, etc. The smart wearable devices can include smart glasses, smartwatches, and smart bracelets, etc.

[0072] In practical applications, the text recognition portion can be executed on the server side as described above, or all operations can be completed on the client device. The choice can be made based on the processing power of the client device and the limitations of the user's usage scenario. This application does not impose any limitations on this. If all operations are completed on the client device, the client device may further include a processor.

[0073] The aforementioned client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side; in other implementation scenarios, it may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, a server cluster consisting of multiple servers, or a distributed server structure.

[0074] The server and the client device can communicate using any suitable network protocol, including network protocols not yet developed as of the date of this application. Such network protocols may include, for example, TCP / IP, UDP / IP, HTTP, HTTPS, etc. Of course, the network protocols may also include, for example, RPC (Remote Procedure Call Protocol) and REST (Representational State Transfer) protocols used on top of the above protocols.

[0075] It should be noted that the character recognition method and device disclosed in this application can be used in the field of financial technology, or in any field other than financial technology. The application field of the character recognition method and device disclosed in this application is not limited.

[0076] The following examples illustrate this in detail.

[0077] To improve the accuracy and efficiency of text recognition, thereby enhancing the customer experience, this embodiment provides a text recognition method whose execution subject is a text recognition device, including but not limited to a server, such as... Figure 1 As shown, this method specifically includes the following:

[0078] Step 100: Obtain the target multi-line text image.

[0079] Specifically, the target multi-line text image may be an image corresponding to a bank counter business voucher, which may have issues with non-standard text order; Figure 2 This is an illustration of a multi-line text image fragment with irregular text order, as shown in the example. Figure 3 This is another example of a multi-line text image fragment with irregular text order. Figure 4 This is a schematic diagram of a segment of a multi-line text image with the text in normal order, as exemplified by the example. Figures 2 to 4 In this text, 'x' represents text, dashed boxes represent test boxes, and numbers indicate the correct text order among the test boxes.

[0080] Step 200: Based on the target multi-line text image and the preset localization model, determine the positions of multiple prediction boxes in the target multi-line text image. The preset localization model is obtained by pre-training the target detection algorithm based on a batch of historical multi-line text images and their corresponding prediction box position labels.

[0081] Specifically, the predicted bounding box position can represent the coordinate information of the predicted bounding box within the target multi-line text image; the target detection algorithm can be the YOLO algorithm (You Only Look Once). The YOLO algorithm (You Only Look Once) is a highly efficient real-time target detection algorithm. Its core idea is to input an image to be detected into a convolutional network, and output multiple tensors in the fully connected layers, each tensor containing the target probability. The model's predicted bounding box (bbox) and label classification C are used to filter the target results by the intersection over union (IoU) with the ground truth bounding box (GT) and the non-maximum suppression (NMS) method.

[0082] Step 300: Determine the score corresponding to each prediction box based on the preset weight matrix and the position of each prediction box.

[0083] Step 400: Based on the scores corresponding to each prediction box, horizontally stitch the prediction boxes to obtain a new text image, perform text recognition on the new text image, and obtain the text recognition result of the target multi-line text image.

[0084] Specifically, the new text image is the text image obtained by rearranging the irregularly ordered characters in the target multi-line text image to a normal order. The text recognition result of the new text image can be equivalent to the text recognition result of the target multi-line text image.

[0085] Specifically, irregular multi-line text can be used as training samples, with each block and line treated as a region label to construct a ground truth (GT). Through a predetermined number of training epochs, this GT is fitted with a tensor to obtain a localization model. Based on the location information obtained from the localization model, a location matrix is ​​constructed. and a preset weight matrix And calculate the fraction matrix. This allows us to obtain the order in which the fragmented text is pieced together, and finally, we can use common recognition models to accurately identify the content of the text.

[0086] To further improve the reliability of the localization model, and thus apply a reliable localization model to improve the accuracy of text recognition, such as... Figure 5 As shown, in one embodiment, prior to step 200, the method further includes:

[0087] Step 011: Obtain a batch of historical multi-line text images and their corresponding prediction box position labels.

[0088] Step 012: Train the target detection algorithm using a batch of historical multi-line text images and their corresponding prediction box position labels to obtain the preset localization model.

[0089] To further improve the accuracy of determining the scores of each prediction box, such as Figure 6 As shown, in one embodiment, step 300 includes:

[0090] Step 301: Construct a position matrix based on the positions of each predicted bounding box.

[0091] Specifically, for any predicted bounding box, assume the x-coordinate of the top-left point is... The vertical axis is The prediction box width is The height of the prediction box is Then the x-coordinate of the lower right point of the prediction box is The ordinate of the bottom right point is .

[0092] Step 302: Multiply the preset weight matrix and the position matrix to obtain a score matrix, which includes the score corresponding to each prediction box.

[0093] Specifically, the preset weight matrix can be an n-dimensional vector, and the number of prediction boxes is n.

[0094] In one example, suppose there are four bounding boxes in the target multi-line text image, namely... , , , Then the position matrix is:

[0095]

[0096] in, Represents the prediction box The x-coordinate of the top-left point, Represents the prediction box The ordinate of the top left point, Represents the prediction box The x-coordinate of the lower right point, Represents the prediction box The ordinate of the bottom right point. One of the prediction boxes can be randomly selected as... Randomly select one of the remaining prediction boxes as... And so on...

[0097] The preset weight matrix is: The length L and height H of the target multi-line text image. When hour, , You can set it according to the actual situation. , , , to The prediction boxes are represented in sequence. to width, and This indicates the preset weight value of the top-left point. and This represents the preset weight value of the bottom right point; when hour, , , , to The prediction boxes are represented in sequence. to The height.

[0098] according to Thus, the derivation for this example can be obtained, where, Represents the prediction box The score:

[0099]

[0100] To improve the accuracy of the new text image, in one embodiment, step 400 includes: sorting the prediction boxes from largest to smallest based on the scores corresponding to each prediction box, and horizontally stitching the prediction boxes based on the sorting results to obtain a new text image.

[0101] Specifically, the horizontal stitching can refer to placing the first character of one prediction box after the last character of the previous prediction box, and so on, until all prediction boxes are stitched together. In one example, the new text image obtained after horizontal stitching is as follows: Figure 7 As shown, the numbers indicate the correct text order between the test boxes.

[0102] To further improve the reliability of the positioning model, in one embodiment, before step 200, the following steps are also included:

[0103] Step 021: Obtain a batch of historical multi-line text images and their corresponding text tags.

[0104] Step 022: Perform the verification step, which includes: applying a batch of historical multi-line text images and the preset localization model to obtain the predicted bounding box positions of each historical multi-line text image; obtaining new text images corresponding to each historical multi-line text image based on the preset weight matrix and the predicted bounding box positions of each historical multi-line text image; performing text recognition on each new text image to obtain the text recognition results of each historical multi-line text image; obtaining the recognition accuracy based on the text recognition results and text labels of each historical multi-line text image; if the recognition accuracy is greater than the accuracy threshold, then the preset localization model is determined to have passed verification.

[0105] To further improve the reliability of the localization model, in one embodiment, the text recognition method further includes:

[0106] Step 023: If the recognition accuracy is less than or equal to the accuracy threshold, then the preset positioning model verification is determined to have failed.

[0107] Step 024: Update the number of training rounds for the object detection algorithm.

[0108] Specifically, the number of training rounds of the target detection algorithm can be gradually increased according to a certain pattern, for example, by increasing the number of training rounds by 50 each time the number of training rounds is updated.

[0109] Step 025: Train the object detection algorithm based on the updated training rounds, batch of historical multi-line text images and their corresponding prediction box position labels to obtain the retrained localization model.

[0110] Step 026: Apply the retrained localization model to perform the verification step again until the preset localization model passes the verification.

[0111] To further improve the reliability of character recognition, in one embodiment, step 400 includes:

[0112] Step 401: Apply the preset text recognition model and the new text image to obtain the text recognition result of the target multi-line text image.

[0113] Step 402: The preset text recognition model is obtained by pre-training a convolutional recurrent neural network model based on a batch of text images and their corresponding text labels.

[0114] Specifically, the convolutional recurrent neural network model can be a CRNN model. Alternatively, existing text recognition methods can be applied to perform text recognition on the new text image.

[0115] To further illustrate this solution, this application provides an application example of a text detection method, which includes: determining the modeling scenario, selecting training samples, YOLO algorithm processing steps, constructing positional weight values, and performance evaluation, such as... Figure 8 As shown, the specific description is as follows:

[0116] Step 101. Determine the modeling scene:

[0117] Irregular multi-line text arises from the different writing and printing habits of various customers or business groups. Based on everyday perception, these different habits are easily recognizable, indicating a certain pattern in text recognition. By using YOLO to provide positional information and combining it with the order of weighted scores, accurate recognition of such irregular multi-line text can be achieved.

[0118] Step 102. Training Sample Selection:

[0119] Irregularly shaped multi-line text fragments were extracted from historical counter service vouchers and used as training sample data. After labeling the data, the sample data and labeled data were placed in a newly created dataset DSYw.

[0120] Step 103. YOLO algorithm processing steps. (As follows...) Figure 9 As shown, step 103 includes:

[0121] Step 21. Based on historical samples and business information, the dataset has n categories. Set labels for the categories. In the C category, 'n' does not represent order, but only the distinction between categories.

[0122] Step 22. Divide the input irregular multi-line text image into S×S cells. If the center point of a text block falls within a cell, then that cell is responsible for that text object. Using the left vertex of the cell as the initial coordinates (0,0), the x, y, w, and h coordinates of a text object's bounding box (bbox) (x and y being the coordinates of the left vertex of the bbox, and w and h being the width and height of the bbox) are taken with reference to the left vertex of the cell. This will give you the label for that cell. Then the network output dimension is: Where B (bbox predictor) defaults to 2, and 5 represents x, y, w, h, ... If the cell contains the target text, then , obtain coordinates , , , A certain piece of text Other tags If not included, then The values ​​that follow are not necessary to understand.

[0123] Step 23. Calculate confidence scores based on the bounding box and ground truth. ,Right now ,but Therefore, each bbox has one. By employing Non-Maximum Suppression (NMS), each label classification is filtered to obtain the best prediction result. Based on the loss calculation method given in the YOLO paper, the loss includes the coordinates of the bounding box and ground truth (GT) of the prediction result:

[0124] and (in Confidence loss:

[0125]

[0126] Category loss compared to the true classification:

[0127]

[0128] After the three calculations are added together, YOLO's neural network will be updated again.

[0129] Step 24. Through multiple training rounds and continuous optimization, the YOLO localization model is finally output.

[0130] It is particularly important to note that the C categories output by YOLO are also unordered.

[0131] Step 104. Construct the location weight value processing steps. For example... Figure 10 As shown, step 104 includes:

[0132] Step 31. Based on the position information predicted by the trained YOLO model, perform node transformation to obtain the position information of the top-left and bottom-right points of the predicted bounding box: , , , , , (where ex and ey are the position information of the lower right point of the prediction box).

[0133] Step 32. Taking a simple 2x2 text structure as an example, the sequential link could be left-right-left-right or left-left-right-right. Construct a position matrix based on the positional information of the four categories. ,

[0134] but Where C1-C4 represent four categories, x and y represent the top-left point of the predicted bounding box for each category, and ex and ey represent the bottom-right point of the predicted bounding box for each category.

[0135] Step 33. The preset weight matrix is: Detect the length L and height H of an irregular multi-line text image. When hour, , , , ;when hour, , , .in This represents the preset weight value of the top left point. The preset weight value represents the bottom right point, C1-C4 represent the four categories, and x and y represent the top left point.

[0136] Step 34. Calculate the fraction matrix, based on... From this, we can obtain the derivation formula for this case, where This represents the scores of the four prediction boxes arranged in a 2x2 grid:

[0137]

[0138] Step 105. Effectiveness Evaluation:

[0139] Take a new validation sample (left-right-left-right format), and set up a control group validation sample (left-left-right-right format). Analyze the score matrix obtained according to the above procedure. The system matches bounding boxes (bboxes) in descending order of scores, segments the predicted bounding boxes, and stitches them horizontally to output a new image. This image is then used to identify the corresponding text using existing general text recognition models, verifying that the identified content matches the label values ​​in the business transaction. If the model's prediction accuracy meets expectations, it can be deployed and applied. If the model's performance does not meet expectations, the number of YOLO training epochs and the preset weight matrix need to be adjusted. Then, retrain the model and verify its performance until the results meet expectations.

[0140] From a software perspective, in order to improve the accuracy and efficiency of text recognition and thus enhance the customer experience, this application provides an embodiment of a text recognition device for implementing all or part of the text recognition method, see [link to embodiment]. Figure 11 The text recognition device specifically includes the following components:

[0141] Module 01 is used to acquire the target multi-line text image;

[0142] The localization module 02 is used to determine the positions of multiple prediction boxes in the target multi-line text image based on the target multi-line text image and a preset localization model. The preset localization model is obtained by pre-training the target detection algorithm based on a batch of historical multi-line text images and their respective corresponding prediction box position labels.

[0143] The determination module 03 is used to determine the score corresponding to each prediction box based on the preset weight matrix and the position of each prediction box;

[0144] The text recognition module 04 is used to horizontally stitch together each prediction box based on the score corresponding to each prediction box to obtain a new text image, and to perform text recognition on the new text image to obtain the text recognition result of the target multi-line text image.

[0145] In one embodiment, the character recognition device further includes:

[0146] The first historical data acquisition module is used to acquire batches of historical multi-line text images and their corresponding prediction box position labels.

[0147] The training module is used to train the target detection algorithm by applying a batch of historical multi-line text images and their corresponding prediction box position labels to obtain the preset localization model.

[0148] In one embodiment, the determining module includes:

[0149] The building unit is used to construct a position matrix based on the positions of each prediction box;

[0150] A score matrix unit is determined, which is used to multiply the preset weight matrix and the position matrix to obtain a score matrix, wherein the score matrix includes the score corresponding to each prediction box.

[0151] In one embodiment, the text recognition module includes:

[0152] The stitching unit is used to sort the prediction boxes from largest to smallest based on the scores corresponding to each prediction box, and then stitch the prediction boxes horizontally based on the sorting results to obtain a new text image.

[0153] In one embodiment, the text recognition device further includes:

[0154] The second historical data acquisition module is used to acquire batches of historical multi-line text images and their corresponding text labels.

[0155] The first verification module is used to execute verification steps, which include: applying a batch of historical multi-line text images and the preset localization model to obtain the predicted bounding box positions of each historical multi-line text image; obtaining new text images corresponding to each historical multi-line text image based on the preset weight matrix and the predicted bounding box positions of each historical multi-line text image; performing text recognition on each new text image to obtain the text recognition results of each historical multi-line text image; obtaining the recognition accuracy based on the text recognition results and text labels of each historical multi-line text image; and determining that the preset localization model has passed verification if the recognition accuracy is greater than the accuracy threshold.

[0156] In one embodiment, the character recognition device further includes:

[0157] The second verification module is used to determine that the preset positioning model verification has failed if the recognition accuracy is less than or equal to the accuracy threshold.

[0158] An update module is used to update the number of training rounds of the object detection algorithm;

[0159] The retraining module is used to train the target detection algorithm based on the updated training rounds, batch of historical multi-line text images and their corresponding prediction box position labels, to obtain the retrained localization model;

[0160] The re-verification module is used to apply the retrained localization model to perform the verification steps again until the preset localization model passes the verification.

[0161] In one embodiment, the text recognition module includes:

[0162] The text recognition unit is used to apply a preset text recognition model and the new text image to obtain the text recognition result of the target multi-line text image;

[0163] The preset text recognition model is obtained by pre-training a convolutional recurrent neural network model based on a batch of text images and their corresponding text labels.

[0164] The embodiments of the character recognition device provided in this specification can be used to execute the processing flow of the embodiments of the above-described character recognition method. Its functions will not be repeated here, but can be referred to the detailed description of the embodiments of the above-described character recognition method.

[0165] From a hardware perspective, in order to improve the accuracy and efficiency of text recognition and thus enhance the customer's business experience, this application provides an embodiment of an electronic device for implementing all or part of the text recognition method. The electronic device specifically includes the following:

[0166] The device comprises a processor, memory, a communications interface, and a bus; wherein the processor, memory, and communications interface communicate with each other via the bus; the communications interface is used to realize information transmission between the character recognition device and related devices such as user terminals; the electronic device can be a desktop computer, tablet computer, or mobile terminal, etc., and this embodiment is not limited to these. In this embodiment, the electronic device can be implemented with reference to the embodiments for implementing the character recognition method and the embodiments for implementing the character recognition device, the content of which is incorporated herein, and repeated details will not be described again.

[0167] Figure 12 This is a schematic block diagram illustrating the system configuration of the electronic device 9600 according to an embodiment of this application. Figure 12 As shown, the electronic device 9600 may include a central processing unit 9100 and a memory 9140; the memory 9140 is coupled to the central processing unit 9100. It is worth noting that... Figure 12 This is an example; other types of structures can also be used to supplement or replace this structure to achieve telecommunications functions or other functions.

[0168] In one or more embodiments of this application, the text recognition function can be integrated into the central processing unit 9100. The central processing unit 9100 can be configured to perform the following control:

[0169] Step 100: Obtain the target multi-line text image.

[0170] Step 200: Based on the target multi-line text image and the preset localization model, determine the positions of multiple prediction boxes in the target multi-line text image. The preset localization model is obtained by pre-training the target detection algorithm based on a batch of historical multi-line text images and their corresponding prediction box position labels.

[0171] Step 300: Determine the score corresponding to each prediction box based on the preset weight matrix and the position of each prediction box.

[0172] Step 400: Based on the scores corresponding to each prediction box, horizontally stitch the prediction boxes to obtain a new text image, perform text recognition on the new text image, and obtain the text recognition result of the target multi-line text image.

[0173] As can be seen from the above description, the electronic device provided by the embodiments of this application can improve the accuracy and efficiency of text recognition, thereby improving the customer's business experience.

[0174] In another embodiment, the character recognition device can be configured separately from the central processing unit 9100. For example, the character recognition device can be configured as a chip connected to the central processing unit 9100, and the character recognition function can be implemented through the control of the central processing unit.

[0175] like Figure 12 As shown, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily need to include these components. Figure 12 All components shown; in addition, the electronic device 9600 may also include Figure 12 For components not shown, please refer to existing technologies.

[0176] like Figure 12 As shown, the central processing unit 9100, sometimes also referred to as a controller or operating control, may include a microprocessor or other processor device and / or logic device, which receives inputs and controls the operation of various components of the electronic device 9600.

[0177] The memory 9140 may be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It may store the aforementioned failure-related information, and also store a program for executing that information. The central processing unit 9100 may execute the program stored in the memory 9140 to perform information storage or processing, etc.

[0178] Input unit 9120 provides input to central processing unit 9100. Input unit 9120 may be, for example, a keypad or touch input device. Power supply 9170 provides power to electronic device 9600. Display 9160 displays images and text. Display may be, for example, an LCD display, but is not limited thereto.

[0179] The memory 9140 can be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), a SIM card, etc. It can also be a memory that retains information even when power is off, can be selectively erased, and contains more data; examples of this type of memory are sometimes referred to as EPROMs. The memory 9140 can also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142 for storing application programs and function programs or processes for executing the operation of the electronic device 9600 via the central processing unit 9100.

[0180] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for the electronic device for communication functions and / or for performing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0181] The communication module 9110 is a transmitter / receiver 9110 that transmits and receives signals via the antenna 9111. The communication module (transmitter / receiver) 9110 is coupled to the central processing unit 9100 to provide input signals and receive output signals, which can be the same as in a conventional mobile communication terminal.

[0182] Based on different communication technologies, multiple communication modules 9110 can be configured in the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module (transmitter / receiver) 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and receive audio input from the microphone 9132, thereby realizing typical telecommunications functions. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. Additionally, the audio processor 9130 is also coupled to a central processing unit 9100, enabling on-device recording via the microphone 9132 and on-device playback of stored audio via the speaker 9131.

[0183] As can be seen from the above description, the electronic device provided by the embodiments of this application can improve the accuracy and efficiency of text recognition, thereby improving the customer's business experience.

[0184] Embodiments of this application also provide a computer-readable storage medium capable of implementing all steps of the character recognition method in the above embodiments. The computer-readable storage medium stores a computer program that, when executed by a processor, implements all steps of the character recognition method in the above embodiments. For example, when the processor executes the computer program, it implements the following steps:

[0185] Step 100: Obtain the target multi-line text image.

[0186] Step 200: Based on the target multi-line text image and the preset localization model, determine the positions of multiple prediction boxes in the target multi-line text image. The preset localization model is obtained by pre-training the target detection algorithm based on a batch of historical multi-line text images and their corresponding prediction box position labels.

[0187] Step 300: Determine the score corresponding to each prediction box based on the preset weight matrix and the position of each prediction box.

[0188] Step 400: Based on the scores corresponding to each prediction box, horizontally stitch the prediction boxes to obtain a new text image, perform text recognition on the new text image, and obtain the text recognition result of the target multi-line text image.

[0189] As can be seen from the above description, the computer-readable storage medium provided in the embodiments of this application can improve the accuracy and efficiency of text recognition, thereby improving the customer's business experience.

[0190] The various embodiments of the methods described in this application are presented in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on explaining the differences from other embodiments. Relevant details can be found in the descriptions of the method embodiments.

[0191] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0192] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0193] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0194] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0195] This application uses specific embodiments to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A character recognition method characterized by comprising: include: Acquire the target multi-line text image; Based on the target multi-line text image and the preset localization model, the positions of multiple prediction boxes in the target multi-line text image are determined. The preset localization model is obtained by pre-training the target detection algorithm based on a batch of historical multi-line text images and their respective corresponding prediction box position labels. Based on the positions of each predicted bounding box, a position matrix is ​​constructed; Multiply the preset weight matrix and the position matrix to obtain the score matrix, which includes the score corresponding to each prediction box. Based on the scores corresponding to each prediction box, the prediction boxes are horizontally stitched together to obtain a new text image. The new text image is then subjected to text recognition to obtain the text recognition result of the target multi-line text image. The location matrix constructed based on the positions of each predicted bounding box includes: For each prediction box, the horizontal and vertical coordinates of the upper left and lower right points of the prediction box are determined according to the position of the prediction box. Based on the horizontal and vertical coordinates of the upper left point and the lower right point, construct the reciprocal position matrix of the prediction box. The position matrix is ​​obtained by concatenating the reciprocal position matrices of the coordinates of each predicted box; The preset weight matrix is ​​determined based on the image length and image height of the target multi-line text image: when the image length is greater than the image height, a first weight matrix is ​​used; wherein, in the first weight matrix, the weight value corresponding to the x-coordinate of the top left point is less than the weight value corresponding to the y-coordinate of the top left point, and the weight value corresponding to the x-coordinate of the bottom right point is the sum of the widths of each predicted box, and the weight value corresponding to the y-coordinate of the bottom right point is the reciprocal of the sum of the widths of each predicted box; when the image length is less than the image height, a second weight matrix is ​​used; wherein, in the second weight matrix, the weight value corresponding to the x-coordinate of the top left point is greater than the weight value corresponding to the y-coordinate of the top left point, and the weight value corresponding to the x-coordinate of the bottom right point is the sum of the heights of each predicted box, and the weight value corresponding to the y-coordinate of the bottom right point is the reciprocal of the sum of the heights of each predicted box.

2. The character recognition method according to claim 1, wherein Also includes: Obtain batches of historical multi-line text images and their corresponding prediction box location labels; The target detection algorithm is trained by applying a batch of historical multi-line text images and their corresponding prediction box position labels to obtain the preset localization model.

3. The character recognition method of claim 1, wherein, The step of horizontally stitching together the prediction boxes based on their scores to obtain a new text image includes: The prediction boxes are sorted from largest to smallest based on their scores, and then horizontally stitched together based on the sorting results to obtain a new text image.

4. The character recognition method of claim 2, wherein, Before determining the positions of multiple predicted bounding boxes in the target multi-line text image based on the target multi-line text image and a preset localization model, the method further includes: Retrieve batches of historical multi-line text images and their corresponding text tags; The verification steps include: applying a batch of historical multi-line text images and the preset localization model to obtain the predicted bounding box positions of each historical multi-line text image; obtaining new text images corresponding to each historical multi-line text image based on the preset weight matrix and the predicted bounding box positions of each historical multi-line text image; performing text recognition on each new text image to obtain the text recognition results of each historical multi-line text image; and obtaining the recognition accuracy based on the text recognition results and text labels of each historical multi-line text image. If the recognition accuracy is greater than the accuracy threshold, then the preset localization model is determined to have passed verification.

5. The character recognition method according to claim 4, characterized in that, Also includes: If the recognition accuracy is less than or equal to the accuracy threshold, then the preset positioning model verification is determined to have failed. Update the number of training rounds for the object detection algorithm; The target detection algorithm is trained based on the updated training rounds, batch of historical multi-line text images and their corresponding prediction box position labels to obtain the retrained localization model; The retrained localization model is applied to perform the verification step again until the preset localization model passes the verification.

6. The character recognition method according to claim 1, characterized in that, The step of performing character recognition on the new text image to obtain the character recognition result of the target multi-line text image includes: By applying a preset character recognition model and the new character image, the character recognition result of the target multi-line character image is obtained; The preset text recognition model is obtained by pre-training a convolutional recurrent neural network model based on a batch of text images and their corresponding text labels.

7. A character recognition device, characterized in that, include: The acquisition module is used to acquire the target multi-line text image; The localization module is used to determine the positions of multiple prediction boxes in the target multi-line text image based on the target multi-line text image and a preset localization model. The preset localization model is obtained by pre-training the target detection algorithm based on a batch of historical multi-line text images and their respective corresponding prediction box position labels. The determination module is used to determine the score corresponding to each prediction box based on the preset weight matrix and the position of each prediction box; The text recognition module is used to horizontally stitch together each prediction box based on the score corresponding to each prediction box to obtain a new text image, and to perform text recognition on the new text image to obtain the text recognition result of the target multi-line text image. The determining module includes: The building unit is used to construct a position matrix based on the positions of each prediction box; A score matrix unit is determined, which is used to multiply the preset weight matrix and the position matrix to obtain a score matrix, wherein the score matrix includes the score corresponding to each prediction box; The location matrix constructed based on the positions of each predicted bounding box includes: For each prediction box, the horizontal and vertical coordinates of the upper left and lower right points of the prediction box are determined according to the position of the prediction box. Based on the horizontal and vertical coordinates of the upper left point and the lower right point, construct the reciprocal position matrix of the prediction box. The position matrix is ​​obtained by concatenating the reciprocal position matrices of the coordinates of each predicted box; The preset weight matrix is ​​determined based on the image length and image height of the target multi-line text image: when the image length is greater than the image height, a first weight matrix is ​​used; wherein, in the first weight matrix, the weight value corresponding to the x-coordinate of the top left point is less than the weight value corresponding to the y-coordinate of the top left point, and the weight value corresponding to the x-coordinate of the bottom right point is the sum of the widths of each predicted box, and the weight value corresponding to the y-coordinate of the bottom right point is the reciprocal of the sum of the widths of each predicted box; when the image length is less than the image height, a second weight matrix is ​​used; wherein, in the second weight matrix, the weight value corresponding to the x-coordinate of the top left point is greater than the weight value corresponding to the y-coordinate of the top left point, and the weight value corresponding to the x-coordinate of the bottom right point is the sum of the heights of each predicted box, and the weight value corresponding to the y-coordinate of the bottom right point is the reciprocal of the sum of the heights of each predicted box.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the character recognition method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing computer instructions thereon, characterized in that, When the instruction is executed, it implements the character recognition method according to any one of claims 1 to 6.

Images