Text recognition method and device based on double channels, equipment and storage medium

Abstract

The application provides a text recognition method and device based on a double channel, equipment and a storage medium, and relates to the technical field of machine learning. The method comprises the following steps: obtaining a first string output by a long short-term memory network channel of a double channel model and a second string output by a transformation channel of the double channel model; obtaining a proportion of difference characters; if the proportion of the difference characters is less than a preset threshold, comparing the first character score and the corresponding second character score one by one, and taking the character with a higher character score as a third character; and outputting a third string composed of multiple third characters. According to the application, the first string output by the long short-term memory network channel and the second string output by the transformation channel are complementary, and the third string with a higher character score is determined, so that the accuracy of text recognition is improved.

Description

Technical Field

[0001] This application relates to machine learning technology, and more particularly to a dual-channel text recognition method, apparatus, device, and storage medium. Background Technology

[0002] Optical Character Recognition (OCR) refers to the process of acquiring text on a medium (such as paper or screen) through electronic devices (such as scanners or digital cameras) and then using text recognition methods to recognize the text.

[0003] The mainstream text recognition method is the Long Short-Term Memory (LSTM) network, also known as the Convolutional Recurrent Neural Network (CRNN), which is based on Convolutional Neural Networks (CNNs). The advantages of CRNNs include the ability to learn directly from sequence labels, freedom from sequence length constraints, fewer parameters, and lower memory requirements. However, CRNNs also have disadvantages such as generally poor performance, limited generalization ability, and inaccurate recognition of rare characters, similar-looking characters, and punctuation marks.

[0004] Another text recognition method is transformation-based text recognition. The advantages of Transformer are that it is not restricted by position association operations, has strong modeling ability, strong versatility, strong scalability, and can perform parallel computing better. The disadvantages of Transformer are that it is not strong in obtaining local information, there are problems in the encoding of position information, and the vanishing gradient of the top layer.

[0005] The text recognition accuracy of CRNN and Transformer-based text recognition is insufficient to meet the requirements. Complementing CRNN and Transformer-based text recognition to improve the accuracy of text recognition is an urgent problem to be solved. Summary of the Invention

[0006] This application provides a dual-channel text recognition method, apparatus, device, and storage medium to address the problem that the text recognition accuracy of both convolutional neural networks and transformation-based text recognition cannot meet the usage requirements, and to improve the text recognition accuracy by using complementary convolutional neural networks and transformation-based text recognition.

[0007] Firstly, this application provides a dual-channel text recognition method, comprising:

[0008] Obtain the first string output by the long short-term memory network channel of the dual-channel model, and the second string output by the transform channel of the dual-channel model. The first string includes multiple first characters and a first character score corresponding to each first character. The second string includes multiple second characters and a second character score corresponding to each second character. The character score is used to indicate the accuracy of the character.

[0009] The percentage of differing characters is obtained. The differing characters are determined by whether the characters at the same position in the first string and the second string are the same.

[0010] If the proportion of the different characters is less than the preset threshold, the score of the first character and the corresponding score of the second character are compared one by one, and the character with the higher score is taken as the third character.

[0011] Output a third string consisting of multiple third characters.

[0012] In one possible design, the percentage of differing characters is obtained, including:

[0013] Get the difference characters;

[0014] Determine the total number of characters based on the first string and the second string;

[0015] The percentage of differing characters is determined based on the total number of characters and the number of differing characters.

[0016] In one possible design, the difference characters are obtained, including:

[0017] If the first target character in the first string is different from the second target character in the second string, then the first target character and the second target character are considered as different characters, and the first target character and the second target character are in the same position in their respective strings.

[0018] In one possible design, prior to obtaining the percentage of differing characters, the following steps are also included:

[0019] If the number of the first character is different from the number of the second character, then the position of the missing character is determined based on the first string and the second string. The missing character is determined by whether the position of the first character and the corresponding position of the second character are the same.

[0020] Set the character corresponding to the position of the missing character to an empty character, and set the score of the character corresponding to the empty character to zero.

[0021] In one possible design, the method also includes:

[0022] If the proportion of the different characters is not less than the preset threshold, then the average score of the first character of the first string and the average score of the second character of the second string are determined based on the scores of multiple first characters and multiple second characters.

[0023] The third string is determined based on the average score of the first character and the average score of the second character. The third string is the string with the higher average score between the first string and the second string.

[0024] In one possible design, the first string of the long short-term memory network channel output of the dual-channel model and the second string of the transform channel output of the dual-channel model are obtained, including:

[0025] Process the preset recognition images to obtain the first feature sequence;

[0026] The first feature sequence is sent simultaneously to the Long Short-Term Memory (LSTM) network channel and the transformation channel to obtain the first string output by the LSM network channel and the second string output by the transformation channel.

[0027] In one possible design, before obtaining the first string of the long short-term memory network channel output of the dual-channel model and the second string of the transformed channel output of the dual-channel model, the method further includes:

[0028] A dual-channel model is constructed, preset dual-channel model parameters are obtained, and preset training images are processed according to the dual-channel model parameters to obtain the second feature sequence. The dual-channel model includes a long short-term memory network channel and a transformation channel.

[0029] The second feature sequence is sent to both the long short-term memory network channel and the transformation channel simultaneously to obtain the normalized exponential loss function output by the long short-term memory network channel and the continuous temporal classification loss function output by the transformation channel.

[0030] The joint loss function is determined based on the normalized exponential loss function and the continuous time series classification loss function;

[0031] The parameters of the dual-channel model are adjusted according to the joint loss function to perform iterative training of the dual-channel model until the preset maximum number of iterations is reached.

[0032] Secondly, this application provides a dual-channel text recognition device, comprising:

[0033] The acquisition module is used to acquire the first string output by the long short-term memory network channel of the dual-channel model and the second string output by the transformation channel of the dual-channel model. The first string includes multiple first characters and a first character score corresponding to each first character. The second string includes multiple second characters and a second character score corresponding to each second character. The character score is used to indicate the accuracy of the character.

[0034] The acquisition module is also used to obtain the proportion of differing characters, which are determined by whether the characters in the same position in the first string and the second string are the same;

[0035] The comparison module is used to compare the score of the first character with the score of the corresponding second character one by one if the proportion of the different characters is less than a preset threshold, and to take the character with the higher score as the third character.

[0036] The output module is used to output a third string consisting of multiple third characters.

[0037] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0038] The memory stores the instructions that the computer executes;

[0039] The processor executes computer execution instructions stored in memory to implement a dual-channel text recognition method.

[0040] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement a dual-channel-based text recognition method.

[0041] Fifthly, this application provides a computer program product, including a computer program, which, when executed by a processor, is used to implement a dual-channel-based text recognition method.

[0042] This application provides a dual-channel text recognition method, apparatus, device, and storage medium. Based on the first string output by the complementary long short-term memory network channel and the second string output by the transformation channel, a third string with a higher character score is determined. This achieves the following technical effects: the complementary convolutional neural network and transformation-based text recognition improve the accuracy of text recognition; the character score indicates the accuracy of the characters, facilitating the comparison between the first and second characters and determining the third character; when the number of first and second characters is the same, and the proportion of differing characters is less than a preset threshold, the character with the higher character score is determined as the third character, thus improving the character score of each character and consequently increasing the accuracy of text recognition. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 A schematic diagram of the system architecture of the dual-channel text recognition method provided in the embodiments of this application;

[0045] Figure 2 An application scenario framework diagram for the dual-channel text recognition method provided in the embodiments of this application;

[0046] Figure 3 A flowchart illustrating the dual-channel text recognition method provided in this application embodiment. Figure 1 ;

[0047] Figure 4 A flowchart illustrating the dual-channel text recognition method provided in this application embodiment. Figure 2 ;

[0048] Figure 5 A schematic diagram illustrating the principle of missing character supplementation provided in the embodiments of this application. Figure 1 ;

[0049] Figure 6 A schematic diagram of the character comparison principle provided in the embodiments of this application. Figure 2 ;

[0050] Figure 7 A flowchart illustrating the dual-channel text recognition method provided in this application embodiment. Figure 3 ;

[0051] Figure 8 A schematic diagram of the structure of a dual-channel text recognition device provided in an embodiment of this application;

[0052] Figure 9 This is a schematic diagram of the structure of the electronic device hardware provided in the embodiments of this application.

[0053] Figure label:

[0054] 11-Carrier; 12-Optical equipment; 13-Server host;

[0055] 20 - Dual-channel text recognition device; 21 - Acquisition module; 22 - Comparison module; 23 - Output module;

[0056] 30 - Electronic device; 31 - Processor; 32 - Memory; 33 - Communication component; 34 - Bus. Detailed Implementation

[0057] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0058] In the embodiments of this application, terms such as "first" and "second" are used to distinguish identical or similar items with substantially the same function and effect. For example, "first chip" and "second chip" are merely used to distinguish different chips and do not limit their order. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different. It should be noted that in the embodiments of this application, words such as "exemplary" or "for example" are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being better or more advantageous than other embodiments or design schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner. In the embodiments of this application, "at least one" means one or more, and "more than one" means two or more.

[0059] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of the relevant data shall comply with relevant laws, regulations and standards, and corresponding operation entry points shall be provided for users to choose to authorize or refuse.

[0060] It should be noted that the phrase "at...time" in the embodiments of this application can refer to the instant at which a certain situation occurs, or to a period of time after the occurrence of a certain situation; the embodiments of this application do not specifically limit this. Furthermore, the dual-channel text recognition provided in the embodiments of this application is merely an example; dual-channel text recognition may also include more or less content.

[0061] To facilitate a clear description of the technical solutions in the embodiments of this application, some terms and technologies involved in the embodiments of this application will be briefly introduced below:

[0062] Convolutional Neural Networks (CNNs) are a class of feedforward neural networks that incorporate convolutional computations and have a deep structure. They are one of the representative algorithms of deep learning.

[0063] Long Short-Term Memory (LSTM) network: refers to a type of time-recurrent neural network, which was specifically designed to solve the long-term dependency problem of general recurrent neural networks (RNN). All RNNs have a chain-like form of repeating neural network modules.

[0064] Transformer: This refers to a model that utilizes attention mechanisms to improve training speed. The Transformer can be considered a deep learning model entirely based on self-attention mechanisms. Its suitability for parallel computation and the complexity of the model itself result in higher accuracy and performance than previously popular RNNs.

[0065] Normalized exponential loss (Softmax loss): It is a combination of Softmax and cross-entropy loss, and is widely used in image classification and segmentation tasks.

[0066] Continuous Temporal Classification Loss (CTC loss): This is a decoding method used to convert the output matrix of an activation function into a sequence.

[0067] Feature Sequence: Each feature vector in the feature sequence is arranged from left to right in the feature map.

[0068] Feature maps are two-dimensional images. In the input layer, grayscale images consist of one feature map, while color images consist of three feature maps (red, green, and blue).

[0069] The mainstream text recognition method is LSTM based on CNN, also known as CRNN, while another text recognition method is Transformer based text recognition. Both of these methods can recognize text in images and output strings.

[0070] On a self-built test set, tests showed that the accuracy of the string output by CRNN was 95.6%, while the accuracy of the string output by Transformer-based text recognition was 96.4%. Strings output using these two text recognition methods may contain one or more incorrectly recognized characters or be missing; in some cases, most characters may be incorrectly recognized, rendering the string unusable.

[0071] When recognizing identical images, the content of the string output by CRNN is largely the same as that output by Transformer-based text recognition, and the missing characters are generally not in the same position. Therefore, by extracting characters with higher accuracy at the same position from both the CRNN and Transformer-based text recognition strings, combining these characters into a new string, and then outputting that new string, the accuracy of text recognition can be improved.

[0072] Based on this, embodiments of this application provide a dual-channel text recognition method, apparatus, device, and storage medium, which can be used in the field of machine learning technology and aims to solve the above-mentioned technical problems of the prior art. In embodiments of this application, a third string with a higher character score is determined based on the first string output by the character score complementary long short-term memory network channel and the second string output by the transformation channel, thereby improving the accuracy of text recognition.

[0073] Figure 1 This is a schematic diagram of the system architecture for a dual-channel text recognition method provided in an embodiment of this application. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this application, in order to help those skilled in the art understand the technical content of this application, but do not mean that the embodiments of this application cannot be used in other devices, systems, environments or scenarios.

[0074] like Figure 1 As shown, the system architecture of this method includes a carrier 11, an optical device 12, and a server host 13. The carrier 11 contains text. The optical device 12 exports the area containing the text on the carrier 11 as a recognition image and sends it to the server host 13. The server host 13 processes the recognition image, recognizes, and outputs the text recorded on the carrier 11. The optical device 12 acquires the text recorded on the carrier 11 through various methods, such as scanning, photographing, or screenshotting. Various connection types can exist between the optical device 12 and the server host 13, such as wired or wireless communication links.

[0075] In this embodiment, the carrier 11 can be various text carriers, including but not limited to paper and screens. Text is written on the paper or screen and displayed as a string. The string consists of multiple characters, and the types of characters include, but are not limited to, text characters, English characters, numeric characters, and other American Standard Code for Information Interchange (ASCII) characters. Parameters other than the character type, such as font, spacing, and order, are not limited in this embodiment.

[0076] Optical device 12 can be various optical character recognition devices, including but not limited to scanners, digital cameras, personal computers, etc. A scanner acquires text written on paper by scanning, or a digital camera acquires text written on a screen by taking a picture, or a personal computer acquires text written on a screen by taking a screenshot. The text is then exported as an image and sent to the server host 13. Image formats include, but are not limited to, JPG, PNG, and IMG formats.

[0077] Server host 13 can be a server for training and running a dual-channel model and outputting strings (for example only). The server can acquire recognition images, perform text recognition based on the recognition images, and output the recognized text; it can also randomly acquire training images from a database and train the dual-channel model based on the training images. The database can be hosted on a local server or an external server.

[0078] For example, if there is text on a piece of paper, a scanner scans the paper, identifies the area containing the text, and exports it as an IMG-formatted image. The scanner then sends the image to a server. The server runs a dual-channel model to process the image and outputs a string, the content of which is the text on the paper.

[0079] Figure 2 This is a framework diagram illustrating the application scenario of the dual-channel text recognition method provided in the embodiments of this application. (For example...) Figure 2 As shown, an optical character recognition device (such as optical device 12) acquires the text recorded on a text carrier (such as carrier 11), exports it as a recognition image in IMG format, and sends the recognition image to a server (such as server host 13).

[0080] The server runs a dual-channel model to process the image for recognition, obtaining the first feature sequence corresponding to the image. This first feature sequence is then sent to the LSTM channel of the dual-channel model, resulting in the first string "CS" output by the LSTM channel. LSTM Simultaneously, the feature sequence is sent to the Transformer channel of the dual-channel model to obtain the second string CS output by the Transformer channel.Tr .

[0081] First string CS LSTM Including j1 first characters CH LSTM and each first character CH LSTM Corresponding character score FR LSTM The first character of the i-th character CH LSTM-i The score for the i-th first character is FR LSTM-i The second string is CS. Tr Includes j2 second characters CH Tr and each second character CH Tr Corresponding character score FR Tr The second character of the i-th character CH Tr-i The score for the i-th second character is FR Tr-i Where i, j1, and j2 are all positive integers, i ≤ j1, i ≤ j2.

[0082] Alternatively, the server (such as server host 13) randomly retrieves training images from the database, processes them to obtain the second feature sequence corresponding to the training images, and sends the second feature sequence to the LSTM channel of the dual-channel model to obtain the Softmax loss output by the LSTM channel; simultaneously, it sends the feature sequence to the Transformer channel of the dual-channel model to obtain the Ctc loss output by the Transformer channel. Based on the Softmax loss and Ctc loss, a joint loss function is determined, and the parameters of the convolutional neural network are adjusted to perform iterative training of the dual-channel model until the preset maximum number of iterations is reached.

[0083] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0084] Figure 3 A flowchart illustrating the dual-channel text recognition method provided in this application embodiment. Figure 1 .like Figure 3 As shown, the method includes:

[0085] S101. Obtain the first string of the long short-term memory network channel output of the dual-channel model, and the second string of the transformation channel output of the dual-channel model;

[0086] Specifically, the dual-channel model includes an LSTM channel and a Transformer channel. The LSTM channel is used to output the recognized text during CRNN text recognition, while the Transformer channel is used to output the recognized text during Transformer-based text recognition. Since the recognized images are the same, the first string CS... LSTM The content and the second string CS Tr The content is the same. However, due to accuracy issues, the first string CS... LSTM The content and the second string CS Tr The content may differ, therefore it is necessary to obtain the i-th first character CH. LSTM-i Corresponding character score FR LSTM-i and the second character CH of the i-th character Tr-i Corresponding character score FR Tr-i , to complement the first string CS LSTM Second string CS Tr .

[0087] S102, Obtain the percentage of differing characters;

[0088] Specifically, the difference character is determined by whether the characters at the same position in the first string and the second string are the same. The first string is CS. LSTM Second string CS Tr In the middle, the first character CH at the same position LSTM Second character CH Tr At the same time, the first character CH LSTM Second character CH Tr As differentiating characters, a set of different characters may exist at different positions in a string. Therefore, it is understandable that when j1 is not equal to j2, it is impossible to obtain the accurate proportion of differentiating characters.

[0089] S103. If the proportion of the different characters is less than the preset threshold, then compare the score of the first character with the score of the corresponding second character one by one, and take the character with the higher score as the third character.

[0090] Specifically, if the proportion of differing characters is less than a preset threshold, it indicates that the content deviation between the first and second strings is small, and both strings are usable. The output character can be determined by comparing character scores. The character score comparison process starts from the first character and continues until all characters have been compared. If j1 = j2 = j, then j third characters (CH) are obtained. OUT The process for comparing the scores of the i-th character is as follows:

[0091] First, obtain the i-th first character CH. LSTM-i The corresponding score FR for the i-th first characterLSTM-i and the i-th second character CH Tr-i The corresponding score FR for the second character of the i-th character Tr-i .

[0092] Secondly, compare the score FR of the i-th first character. LSTM-i And the score FR of the second character of the i-th character Tr-i ;

[0093] If FR LSTM-i >FR Tr-i Then the i-th first character CH LSTM-i As the i-th third character CH OUT-i ;

[0094] If FR LSTM-i <FR Tr-i Then the i-th second character CH Tr-i As the i-th third character CH OUT-i ;

[0095] If FR LSTM-i =FR Tr-i Then the i-th first character CH LSTM-i and the second character CH of the i-th character Tr-i All can be used as the i-th third character CH OUT-i .

[0096] S104. Output a third string composed of multiple third characters;

[0097] Specifically, after comparing the character scores, we obtain the j third characters CH in sequential order. OUT CH OUT-1 CH OUT-2 CH OUT-i ...CH OUT-j , j third characters CH OUT Forming the third string CS OUT Output the third string CS OUT The third string is CS. OUT It is composed of characters with high character scores; therefore, the third string CS OUT The accuracy is greater than that of the first string CS LSTM Accuracy and second string CS Tr The accuracy rate.

[0098] This embodiment provides a dual-channel text recognition method. Based on the first string output by the complementary long short-term memory network channel and the second string output by the transformation channel, a third string with a higher character score is determined. This achieves the following technical effects: the complementary convolutional neural network and transformation-based text recognition improve the accuracy of text recognition; the character score indicates the accuracy of the character, facilitating the comparison between the first and second characters and determining the third character; when the number of first and second characters is the same and the proportion of differing characters is less than a preset threshold, the character with the higher character score is determined as the third character, thus improving the character score of each character and consequently improving the accuracy of text recognition.

[0099] Figure 4 A flowchart illustrating the dual-channel text recognition method provided in this application embodiment. Figure 2 .like Figure 4 As shown, in this embodiment... Figure 3 Based on the embodiments, a dual-channel text recognition method is described in detail. This embodiment provides a dual-channel text recognition method, including:

[0100] S201. Process the preset recognition image to obtain the first feature sequence;

[0101] Specifically, the process involves acquiring recognition images sent by an optical character recognition (OCR) device. These images are derived from the text processed by the OCR device on a text carrier. The recognition images are then standardized in size and loaded into the network layers of the network to perform convolution, pooling, and batch normalization operations to continuously extract features and obtain the first feature sequence.

[0102] S202. The first feature sequence is sent to the Long Short-Term Memory Network channel and the Transformation Channel simultaneously to obtain the first string output by the Long Short-Term Memory Network channel and the second string output by the Transformation Channel.

[0103] Specifically, the first feature sequence is sent to the LSTM channel, and the LSTM channel outputs the first string CS. LSTM Simultaneously, the first feature sequence is sent to the Transformer channel, and the second string CS is output by the Transformer channel. Tr .

[0104] First string CS LSTM Including j1 first characters CH LSTM and each first character CH LSTM Corresponding character score FR LSTM The first character of the i-th character CH LSTM-i The score for the i-th first character is FR LSTM-iThe second string is CS. Tr Includes j2 second characters CH Tr and each second character CH Tr Corresponding character score FR Tr The second character of the i-th character CH Tr-i The score for the i-th second character is FR Tr-i .

[0105] S203. Determine whether the number of the first character is the same as the number of the second character;

[0106] If not, it means there are missing characters, and empty characters need to be filled in the positions of the missing characters. Therefore, S204 is executed.

[0107] If so, it means there are no missing characters, and the third character can be determined directly based on the character score. Therefore, S206 is executed.

[0108] S204. Determine the position of the missing character based on the first string and the second string;

[0109] Specifically, if j1 ≠ j2, the accurate proportion of the differing characters cannot be obtained. It is necessary to determine the position of the missing character and fill that position with another character so that j1 = j2 = j, in order to obtain the accurate proportion of the differing characters.

[0110] Missing characters are determined by whether the position of the first character matches the position of the corresponding second character. Typically, missing characters in two strings are determined using a programming language, and most mainstream programming languages ​​can perform this operation.

[0111] S205. Set the character corresponding to the position of the missing character to an empty character, and set the score of the character corresponding to the empty character to zero;

[0112] Specifically, after determining the location of the missing character, the character corresponding to that location is set to an empty character, ensuring that j1 = j2. At the same time, the score for the character corresponding to the empty character is set to zero. This is because an empty character is definitely not part of the recognized text; setting it to a value other than zero would affect the accuracy of the third string.

[0113] The following example will be used to explain this embodiment.

[0114] Figure 5 A schematic diagram illustrating the principle of missing character supplementation provided in the embodiments of this application. Figure 1 .like Figure 5 As shown:

[0115] First string CS LSTM The content is 'character type';

[0116] CH LSTM-1 For 'text', FR LSTM-1 It is '0.8';

[0117] CH LSTM-2 For 'word', FR LSTM-2 It is '0.9';

[0118] CH LSTM-3 For 'different', FR LSTM-3 It is '0.9';

[0119] The second string CS Tr The content is 'text recognition';

[0120] CH Tr-1 For 'text', FR Tr-1i It is '0.9';

[0121] CH Tr-2 For 'word', FR Tr-2 It is '0.8';

[0122] CH Tr-3 For 'knowledge', FR Tr-3 It is '0.9';

[0123] CH Tr-4 For 'different', FR Tr-4 It is '0.9';

[0124] Use Python's diff library to determine CH LSTM-2 and CH LSTM-3 The middle character is missing, so it is in CH. LSTM-2 and CH LSTM-3 Insert the first missing character CH of the first string between the two strings. LSTM-La1 CH LSTM-La1 Set it to 'null', and set CH LSTM-La1 The score for the first missing character in the corresponding first string is FR. LSTM-La1 Set to '0'.

[0125] After setting, the first string is CS LSTM The content has been changed to 'text_other'

[0126] CH LSTM-1 For 'text', FR LSTM-1 It is '0.8';

[0127] CH LSTM-2 For 'character', FR LSTM-2 It is '0.9';

[0128] CH LSTM-La1 For 'null', FRLSTM-La1 It is '0';

[0129] CH LSTM-3 For 'different', FR LSTM-3 It is '0.9'.

[0130] S206. Determine whether the first target character in the first string is the same as the second target character in the second string;

[0131] If so, it means that the first target character and the second target character are not different characters. The next first character is taken as the first target character, and the next second character is taken as the second target character, that is, S208 is executed;

[0132] If not, it means that the first target character and the second target character are different characters, then execute S207;

[0133] S207. Treat the first target character and the second target character as the difference character;

[0134] Specifically, the first target character and the second target character are in the same position within their respective strings. Therefore, the i-th first character CH is... LSTM-i As the first target character, and the i-th second character CH is used as the first target character. Tr-i As the second target character, determine CH LSTM-i and CH Tr-i Are they the same? If they are the same, do not include CH. LSTM-i and CH Tr-i As a difference character; if they are different, use CH. LSTM-i As a difference character CH DLSTM-i and CH Tr-i As a difference character CH DTr-i .

[0135] S208. Change the target character until all characters in all positions have been compared;

[0136] Specifically, the first first character is taken as the first target character and the first second character is taken as the second target character, and the target characters are compared. After the comparison is completed, the next first character is taken as the first target character and the next second character is taken as the second target character, until all first characters have been taken as first target characters and all second characters have been taken as second target characters.

[0137] S209. Determine the total number of characters based on the first string and the second string;

[0138] Specifically, starting from i=1 and proceeding up to i=j, we obtain k differing characters CH. D Where k is an even number, k min=0,k max =2j.

[0139] S210. Determine the percentage of differing characters based on the total number of characters and the number of differing characters.

[0140] Specifically, the first string CS LSTM The first character CH LSTM The quantity is j1 = j, the second string is CS Tr The second character CH Tr The number of characters is j2 = j, therefore the total number of characters is 2j.

[0141] Difference character CH D The number of characters is k, and the percentage of different characters is...

[0142] PID = j / k * 100%.

[0143] When k min When k = 0, the percentage of different characters (PID) is 0; when k max When PID = 2j, the percentage of different characters is PID = 1.

[0144] S211. Determine whether the proportion of different characters is less than a preset threshold;

[0145] If so, the third character is determined by comparing the score of the first character with the score of the corresponding second character one by one, i.e., S212 is executed;

[0146] If not, the third string is determined by comparing the average score of the first character with the average score of the second character, i.e., S213 is executed.

[0147] S212. Compare the scores of the first character with the corresponding scores of the second character one by one, and take the character with the higher score as the third character;

[0148] S212 is similar to S103, and the principle of character comparison will not be described again in this embodiment.

[0149] The following example will be used to explain this embodiment.

[0150] Figure 6 A schematic diagram of the character comparison principle provided in the embodiments of this application. Figure 2 .like Figure 6 As shown:

[0151] First string CS LSTM The content is 'textual distinction'

[0152] CH LSTM-1 For 'text', FR LSTM-1 It is '0.8';

[0153] CH LSTM-2 is 'character', FR LSTM-2 is '0.9';

[0154] CH LSTM-3 is 'blazing', FR LSTM-3 is '0.4';

[0155] CH LSTM-4 is 'distinguish', FR LSTM-4 is '0.9';

[0156] The second string CS Tr has the content 'character recognition';

[0157] CH Tr-1 is 'article', FR Tr-1 is '0.9';

[0158] CH Tr-2 is 'character', FR Tr-2 is '0.8';

[0159] CH Tr-3 is'recognize', FR Tr-3 is '0.9';

[0160] CH Tr-4 is 'distinguish', FR Tr-4 is '0.9'.

[0161] Set the preset threshold to 50%.

[0162] The first string CS LSTM and the second string CS Tr Among them, the number of different characters CH D is k = 2, the total number of characters is 8, and the proportion of different characters PID = j / k * 100% = 25%, which is less than 50% of the preset threshold. Therefore, compare FR LSTM-i and FR Tr-i , and take the character with the higher character score as the third character.

[0163] FR LSTM-1 <FR Tr-1 , so take CH Tr-1 as CH OUT-1 , CH OUT-1 is 'article';

[0164] FR LSTM-2 >FR Tr-2 , so take CH LSTM-2 as CH OUT-2 , CH OUT-2 is 'character';

[0165] FRLSTM-3 <FR Tr-3 , so CH Tr-3 is used as CH OUT-3 , CH OUT-3 is 'knowledge';

[0166] FR LSTM-4 = FR Tr-4 , so CH LSTM-4 is used as CH OUT-1 , CH OUT-1 is 'distinguish';

[0167] The content of the third string CS OUT is 'character recognition'.

[0168] After executing S212, continue to execute S215.

[0169] S213. Determine the average score of the first characters of the first string and the average score of the second characters of the second string according to multiple first character scores and multiple second character scores;

[0170] Specifically, when the proportion of different characters is not less than the preset threshold, it means that the content deviation between the first string and the second string is too large, and the string with a small accuracy rate may be unavailable. Therefore, the third string is determined by comparing the average character scores of the two strings.

[0171] S214. Determine the third string according to the average score of the first characters and the average score of the second characters;

[0172] Specifically, the string with a higher average character score is more accurate, so directly use this string as the third string.

[0173] The following will explain this embodiment through an example. As Figure 6 shown:

[0174] The content of the first string CS LSTM is 'character blazing distinction'

[0175] CH LSTM-1 is 'text', FR LSTM-1 is '0.8';

[0176] CH LSTM-2 is 'character', FR LSTM-2 is '0.9';

[0177] CH LSTM-3 is 'blazing', FR LSTM-3 is '0.4';

[0178] CH LSTM-4 is 'distinction', FR LSTM-4It is '0.9';

[0179] The second string CS Tr The content is 'text recognition';

[0180] CH Tr-1 For 'text', FR Tr-1 It is '0.9';

[0181] CH Tr-2 For 'character', FR Tr-2 It is '0.8';

[0182] CH Tr-3 For 'knowledge', FR Tr-3 It is '0.9';

[0183] CH Tr-4 For 'different', FR Tr-4 It is '0.9'.

[0184] Set the preset threshold to 20%.

[0185] First string CS LSTM Second string CS Tr In the text, the difference character CH D The number of characters k = 2, the total number of characters is 8, and the percentage of different characters PID = j / k * 100% = 25%, which is not less than the preset threshold of 20%. Therefore, compared with the first string CS... LSTM First character average score FR avLSTM Second string CS Tr The second character average score FR avTr .

[0186]

[0187] FR avLSTM <FR avTr Then the second string CS Tr As the third string CS OUT The third string CS OUT The content is 'text recognition'.

[0188] S215. Output a third string composed of multiple third characters;

[0189] Specifically, on a self-built test set, tests showed that the dual-channel model achieved a test set accuracy of 97.2% for the output strings, higher than the 95.6% accuracy of the CRNN output strings and the 96.4% accuracy of the Transformer-based text recognition output strings. The dual-channel text recognition method can effectively improve the accuracy of text recognition.

[0190] In one possible design, the dual-channel model is iteratively trained before text recognition is performed.

[0191] Figure 7 A flowchart illustrating the dual-channel text recognition method provided in this application embodiment. Figure 3 .like Figure 7 As shown, the method also includes:

[0192] S301. Construct a dual-channel model, obtain the preset dual-channel model parameters, and process the preset training images according to the dual-channel model parameters to obtain the second feature sequence.

[0193] Specifically, during model training, a large amount of training data is fed into the neural network. The training process is a supervised learning process. The neural network calculates a loss function based on the result of each recognition, determines correction coefficients based on the loss function, and then propagates these correction coefficients in reverse to adjust the model parameters. To maximize recognition accuracy, a dual-channel model consisting of LSTM and Transformer channels is introduced during training to jointly train and calculate a joint loss function to optimize the model parameters.

[0194] After constructing the dual-channel model, initialize the dual-channel model parameters and load the preset dual-channel model parameters into the convolutional layer. The dual-channel model parameters include the learning rate and the maximum number of iterations.

[0195] The principle of obtaining the second feature sequence in S301 is similar to that in S201, and will not be repeated in this embodiment.

[0196] S302. The second feature sequence is sent to the Long Short-Term Memory Network channel and the Transform channel at the same time to obtain the normalized exponential loss function output by the Long Short-Term Memory Network channel and the continuous temporal classification loss function output by the Transform channel.

[0197] S302 is similar to S202, and will not be described again in this embodiment.

[0198] S303. Determine the joint loss function based on the normalized exponential loss function and the continuous time series classification loss function;

[0199] Specifically, the normalized exponential loss function L of the LSTM channel output sm Transformer channel output continuous temporal classification loss function L ctc Adding the two loss functions together yields the joint loss function L. sum L ctc =L sm +L ctc .

[0200] S304. Adjust the parameters of the dual-channel model according to the joint loss function;

[0201] Specifically, after obtaining the new joint loss function, correction coefficients are determined based on the joint loss function to determine the new dual-channel model parameters. After adjusting the dual-channel model parameters in reverse at the convolutional layer, pooling layer, and batch normalization layer, the current iteration of training is completed, and the next iteration begins, until the preset maximum number of iterations is reached.

[0202] This embodiment provides a dual-channel text recognition method. Based on the first string output by the complementary long short-term memory network channel and the second string output by the transform channel, a third string with a higher character score is determined. This achieves the following technical effects: the complementary convolutional neural network and transform-based text recognition improve the accuracy of text recognition; the character score indicates the accuracy of each character, facilitating the comparison between the first and second characters and determining the third character; when the number of first and second characters is the same, and the proportion of differing characters is less than a preset threshold, the character with the higher character score is determined as the third character, thus improving the character score of each character and consequently enhancing the accuracy of text recognition. The accuracy rate is improved by determining the third string based on the average score of characters when the number of the first and second characters is the same and the proportion of differing characters is not less than a preset threshold. This avoids the situation where strings with low accuracy are unusable. When the number of the first and second characters is different, the character corresponding to the position of the missing character is set to an empty character, and the character score corresponding to the empty character is set to zero. This avoids the problem of not being able to obtain the accurate proportion of differing characters due to missing characters. The joint loss function is determined by the normalized exponential loss function and the continuous temporal classification loss function to adjust the parameters of the dual-channel model and perform iterative training of the dual-channel model, thereby improving the accuracy of text recognition of the dual-channel model.

[0203] In this embodiment of the invention, electronic devices or main control devices can be divided into functional modules according to the above method examples. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional module. It should be noted that the module division in this embodiment of the invention is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0204] Figure 8 This is a schematic diagram of the structure of a dual-channel text recognition device provided in an embodiment of this application. Figure 8 As shown in the figure, an embodiment of this application provides a dual-channel text recognition device 20, which includes: an acquisition module 21, a comparison module 22, and an output module 23.

[0205] The acquisition module 21 is used to acquire the first string output by the long short-term memory network channel of the dual-channel model and the second string output by the transformation channel of the dual-channel model. The first string includes multiple first characters and a first character score corresponding to each first character. The second string includes multiple second characters and a second character score corresponding to each second character. The character score is used to indicate the accuracy of the character.

[0206] The acquisition module 21 is also used to acquire the proportion of the difference characters, which are determined based on whether the characters in the same position in the first string and the second string are the same;

[0207] The comparison module 22 is used to compare the score of the first character with the score of the corresponding second character one by one if the proportion of the difference characters is less than a preset threshold, and to take the character with the higher score as the third character.

[0208] Output module 23 is used to output a third string composed of multiple third characters.

[0209] In one possible design, the acquisition module 21 includes: a first determining module and a second determining module;

[0210] The acquisition module is used to acquire the difference characters;

[0211] The first determining module is used to determine the total number of characters based on the first string and the second string;

[0212] The second determining module is used to determine the proportion of differing characters based on the total number of characters and the number of differing characters.

[0213] In one possible design, the acquisition module 21 is used to treat the first target character and the second target character as difference characters if the first target character in the first string is different from the second target character in the second string, and the first target character and the second target character are in the same position in their respective strings.

[0214] In one possible design, the dual-channel text recognition device 20 further includes: a third determining module and a setting module;

[0215] The second determining module is used to determine the position of the missing character based on the first string and the second string if the number of the first character is different from the number of the second character. The missing character is determined based on whether the position of the first character and the position of the corresponding second character are the same.

[0216] The setting module is used to set the character corresponding to the position of the missing character to an empty character, and set the score of the character corresponding to the empty character to zero.

[0217] In one possible design, the third determining module is used to determine the average score of the first character of the first string and the average score of the second character of the second string based on the scores of multiple first characters and multiple second characters if the proportion of the difference characters is not less than a preset threshold.

[0218] The comparison module 22 is used to determine the third string based on the average score of the first character and the average score of the second character. The third string is the string with the higher average score of the characters in the first string and the second string.

[0219] In one possible design, the dual-channel text recognition device 20 also includes a processing module and a transmission module;

[0220] The processing module is used to process the preset recognition images to obtain the first feature sequence;

[0221] The transmission module is used to simultaneously send the first feature sequence to the long short-term memory network channel and the transformation channel to obtain the first string output by the long short-term memory network channel and the second string output by the transformation channel.

[0222] In one possible design, the dual-channel text recognition device 20 further includes: a fourth determining module and an adjusting module;

[0223] The processing module is used to construct a dual-channel model, obtain preset dual-channel model parameters, and process preset training images according to the dual-channel model parameters to obtain the second feature sequence. The dual-channel model includes a long short-term memory network channel and a transformation channel.

[0224] The transmission module is used to simultaneously send the second feature sequence to the long short-term memory network channel and the transformation channel to obtain the normalized exponential loss function output by the long short-term memory network channel and the continuous temporal classification loss function output by the transformation channel.

[0225] The fourth determination module is used to determine the joint loss function based on the normalized exponential loss function and the continuous time series classification loss function;

[0226] The adjustment module is used to adjust the parameters of the dual-channel model according to the joint loss function in order to perform iterative training of the dual-channel model until the preset maximum number of iterations is reached.

[0227] This embodiment provides a dual-channel text recognition device that can execute a dual-channel text recognition method described in the above embodiment. Its implementation principle and technical effects are similar, and will not be repeated here.

[0228] In the aforementioned specific implementation of a dual-channel text recognition method, each module can be implemented as a processor. The processor can execute computer execution instructions stored in the memory, thereby enabling the processor to execute the aforementioned dual-channel text recognition method.

[0229] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application. For example... Figure 9 As shown, the electronic device 30 includes at least one processor 31 and a memory 32. The electronic device 30 also includes a communication component 33. The processor 31, memory 32, and communication component 33 are connected via a bus 34.

[0230] In the specific implementation process, at least one processor 31 executes computer execution instructions stored in memory 32, causing at least one processor 31 to execute a dual-channel-based text recognition method as executed on the electronic device side.

[0231] The specific implementation process of processor 31 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0232] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0233] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage.

[0234] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0235] The above description of the functions implemented by electronic devices and main control devices has introduced the solutions provided by the embodiments of the present invention. It is understood that, in order to implement the above functions, the electronic device or main control device includes hardware structures and / or software modules corresponding to the execution of each function. By combining the units and algorithm steps of the various examples described in the embodiments of the present invention, the embodiments of the present invention can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of the embodiments of the present invention.

[0236] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described dual-channel text recognition method.

[0237] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0238] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in an electronic device or a host device.

[0239] This application also provides a computer program product, comprising: a computer program stored in a readable storage medium, wherein at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to cause the electronic device to perform the scheme provided in any of the above embodiments.

[0240] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disk, or optical disk.

[0241] The technical solutions of this application have been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it is readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. The above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A text recognition method based on dual channels, characterized in that, The method includes: The method obtains a first string output by the Long Short-Term Memory (LSTM) network channel of a dual-channel model and a second string output by the transform channel of the dual-channel model. The dual-channel model is a heterogeneous encoder parallel architecture within a single model. The LSM network channel and the transform channel share the underlying convolutional feature sequence. The dual-channel model is iteratively trained and optimized based on a joint loss function, which is determined based on the loss function output by the LSM network channel and the transform channel. The first string includes multiple first characters and a first character score corresponding to each first character. The second string includes multiple second characters and a second character score corresponding to each second character. The character score is used to indicate the accuracy of the character. If the number of the first character is different from the number of the second character, then the position of the missing character is determined based on the first string and the second string. The missing character is determined based on whether the position of the first character and the corresponding position of the second character are the same. The character corresponding to the position of the missing character is set to an empty character, and the score of the character corresponding to the empty character is set to zero. The proportion of differing characters is obtained, wherein the differing characters are determined based on whether the characters in the same position in the first string and the second string are the same; If the proportion of the differing characters is less than a preset threshold, the scores of the first characters and the corresponding scores of the second characters are compared one by one, and the character with the higher score is taken as the third character; if the proportion of the differing characters is not less than the preset threshold, the average score of the first character of the first string and the average score of the second character of the second string are determined based on the scores of the multiple first characters and the multiple second characters; the third string is determined based on the average scores of the first characters and the average scores of the second characters, and the third string is the string with the higher average score of the first string and the second string; Output a third string consisting of multiple third characters.

2. The method according to claim 1, characterized in that, The percentage of differing characters obtained includes: Get the difference characters; Determine the total number of characters based on the first string and the second string; The proportion of the differing characters is determined based on the total number of characters and the number of differing characters.

3. The method according to claim 2, characterized in that, The acquisition of the difference characters includes: If the first target character in the first string is different from the second target character in the second string, then the first target character and the second target character are regarded as difference characters, and the first target character and the second target character are in the same position in their respective strings.

4. The method according to claim 1, characterized in that, The acquisition of the first string output by the Long Short-Term Memory (LSTM) channel of the dual-channel model and the second string output by the transform channel of the dual-channel model includes: Process the preset recognition images to obtain the first feature sequence; The first feature sequence is simultaneously sent to the long short-term memory network channel and the transformation channel to obtain a first string output by the long short-term memory network channel and a second string output by the transformation channel.

5. The method according to claim 1, characterized in that, Before obtaining the first string output by the Long Short-Term Memory network channel of the dual-channel model and the second string output by the transformed channel of the dual-channel model, the method further includes: A dual-channel model is constructed, preset dual-channel model parameters are obtained, and preset training images are processed according to the dual-channel model parameters to obtain a second feature sequence. The dual-channel model includes the long short-term memory network channel and the transformation channel. The second feature sequence is simultaneously sent to the long short-term memory network channel and the transformation channel to obtain the normalized exponential loss function output by the long short-term memory network channel and the continuous temporal classification loss function output by the transformation channel. The joint loss function is determined based on the normalized exponential loss function and the continuous time series classification loss function; The parameters of the dual-channel model are adjusted according to the joint loss function to perform iterative training of the dual-channel model until the number of iterations reaches the preset maximum number of iterations.

6. A text recognition device based on dual channels, characterized in that, include: The acquisition module is used to acquire the first string output by the Long Short-Term Memory (LSTM) network channel of the dual-channel model and the second string output by the transform channel of the dual-channel model. The dual-channel model is a heterogeneous encoder parallel architecture within a single model. The LSM network channel and the transform channel share the underlying convolutional feature sequence, and the dual-channel model is iteratively trained and optimized based on a joint loss function. The joint loss function is determined based on the loss function output by the LSM network channel and the transform channel. The first string includes multiple first characters and a first character score corresponding to each first character. The second string includes multiple second characters and a second character score corresponding to each second character. The character score is used to indicate the accuracy of the character. A determining module is used to determine the position of a missing character based on the first string and the second string if the number of the first character is different from the number of the second character. The missing character is determined based on whether the position of the first character and the corresponding position of the second character are the same. The setting module is used to set the character corresponding to the position of the missing character to an empty character, and set the score of the character corresponding to the empty character to zero; The acquisition module is also used to acquire the proportion of differing characters, wherein the differing characters are determined based on whether the characters in the same position in the first string and the second string are the same; The comparison module is used to compare the scores of the first character and the corresponding scores of the second character one by one if the proportion of the difference characters is less than a preset threshold, and to take the character with the higher score as the third character. The determining module is further configured to, if the proportion of the differing characters is not less than the preset threshold, determine the average score of the first character of the first string and the average score of the second character of the second string based on the scores of multiple first characters and multiple second characters; The comparison module is also used to determine a third string based on the average score of the first character and the average score of the second character, wherein the third string is the string with the higher average character score between the first string and the second string; The output module is used to output a third string consisting of multiple third characters.

7. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement a dual-channel text recognition method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement a dual-channel text recognition method as described in any one of claims 1 to 5.

Images