A text recognition method, apparatus, computer device, and storage medium

Abstract

Embodiments of the present application disclose a text recognition method and device, computer equipment and a storage medium, relating to the technical field of communication. The method comprises: obtaining a to-be-recognized text image, the to-be-recognized text image comprising at least two image regions; performing feature extraction on the to-be-recognized text image to obtain feature information of each image region; for each image region, calculating content similarity between the image region and an associated image region according to the feature information of the image region and the feature information of the associated image region; performing fusion processing on the feature information of the image region and the associated image region according to the content similarity to obtain attention feature information of attention context information; and performing text content recognition on the to-be-recognized image based on the attention feature information to obtain a recognition result. According to the embodiments, the text recognition is performed according to the attention feature information of the attention context information, which can realize parallel recognition of the image regions in the to-be-recognized image and improve the recognition speed of the to-be-recognized image.

Description

Technical Field

[0001] This application relates to the field of communication technology, and specifically to a method, apparatus, computer device, and storage medium for text recognition. Background Technology

[0002] Optical Character Recognition (OCR) refers to the process by which computer devices detect the shape of characters, such as characters printed on paper or characters contained in images, and then use character recognition methods to translate the detected shapes into computer text. This can be achieved using a Convolutional Recurrent Neural Network (CRNN) model based on the Connectionist Temporal Classification (CTC) algorithm.

[0003] The CTC-based CRNN model includes CNN and RNN network layers. The RNN network layer uses a Long Short Term Memory (LSTM) network. LSTM is a serial network structure that calculates the prediction data of the (T+1)th image region based on the prediction data of the previous T image regions, resulting in a long prediction time. Summary of the Invention

[0004] This application provides a text recognition method, apparatus, computer device, and storage medium, which can realize parallel recognition of image regions in an image to be recognized, obtain the recognition result of the image to be recognized, and improve the recognition speed of the image to be recognized.

[0005] This application provides a text recognition method, including:

[0006] Acquire a text image to be recognized, wherein the text image to be recognized includes at least two image regions;

[0007] Feature extraction is performed on the text image to be identified to obtain feature information of each image region in the text image to be identified;

[0008] For each image region, the content similarity between the image region and the associated image region is calculated based on the feature information of the image region and the feature information of the associated image region.

[0009] For each image region, the feature information of the image region and the associated image region is fused based on the content similarity between the image region and the associated image region to obtain attention feature information that focuses on context information;

[0010] Based on the attention feature information, text content recognition is performed on the image to be recognized to obtain the recognition result.

[0011] Accordingly, embodiments of this application also provide a text recognition device, comprising:

[0012] Acquisition unit: used to acquire a text image to be recognized, wherein the text image to be recognized includes at least two image regions;

[0013] Extraction unit: used to extract features from the text image to be identified, and obtain feature information of each image region in the text image to be identified;

[0014] Calculation unit: For each image region, based on the feature information of the image region and the feature information of the associated image region, calculates the content similarity between the image region and the associated image region;

[0015] Fusion unit: For each image region, based on the content similarity between the image region and the associated image region, the feature information of the image region and the associated image region are fused to obtain attention feature information that focuses on the context information;

[0016] Recognition unit: used to perform text content recognition on the image to be recognized based on the attention feature information, and obtain the recognition result.

[0017] Accordingly, this application also provides a computer device including a memory and a processor; the memory stores a computer program, and the processor is used to run the computer program in the memory to execute any of the text recognition methods provided in this application.

[0018] Accordingly, embodiments of this application also provide a storage medium for storing a computer program, which is loaded by a processor to execute any of the text recognition methods provided in embodiments of this application.

[0019] This application embodiment can acquire a text image to be recognized, which includes at least two image regions; perform feature extraction on the text image to be recognized to obtain feature information of each image region in the text image to be recognized; for each image region, calculate the content similarity between the image region and the associated image region based on the feature information of the image region and the feature information of the associated image region; for each image region, perform fusion processing on the feature information of the image region and the associated image region based on the content similarity between the image region and the associated image region to obtain attention feature information that focuses on contextual information; and perform text content recognition on the image to be recognized based on the attention feature information to obtain the recognition result. This scheme achieves parallel recognition of image regions in the image to be recognized by fusing the feature information of each image region and the corresponding associated image region to obtain attention feature information that focuses on contextual information for each image region, and performs text recognition on each image region based on the attention feature information that focuses on contextual information, thereby improving the recognition speed of the image to be recognized. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a scene diagram of the text recognition method provided in the embodiments of this application;

[0022] Figure 2 This is a flowchart of the text recognition method provided in the embodiments of this application;

[0023] Figure 3 This is a schematic diagram of path planning provided in an embodiment of this application;

[0024] Figure 4 This is another path planning diagram provided in an embodiment of this application;

[0025] Figure 5 This is another path planning diagram provided in the embodiments of this application;

[0026] Figure 6 This is another path planning diagram provided in the embodiments of this application;

[0027] Figure 7 This is another schematic diagram of the path planning provided in the embodiments of this application.

[0028] Figure 8 This is another flowchart of the text recognition method provided in the embodiments of this application;

[0029] Figure 9 This is a user interface diagram of the text recognition method provided in the embodiments of this application;

[0030] Figure 10 This is a schematic diagram of feature vector fusion provided in an embodiment of this application;

[0031] Figure 11 This is a schematic diagram of the text recognition device provided in the embodiments of this application;

[0032] Figure 12 This is a schematic diagram of the structure of the computer device provided in the embodiments of this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0034] This application provides a text recognition method, apparatus, computer device, and storage medium. The text recognition apparatus can be integrated into a computer device, which may be a server or a terminal, etc.

[0035] The terminal may include mobile phones, wearable smart devices, tablets, laptops, personal computers (PCs), and in-vehicle computers, etc.

[0036] The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0037] For example, such as Figure 1As shown, a computer device can extract features from a text content recognition (CCR) image to be recognized using a trained CCR model, obtaining a feature map of the text image. Based on the data required by the attention mechanism in the trained CCR model, features contained in the feature map are selected, and based on the feature information required by the attention mechanism, the selected features are embedded to obtain feature information for each image region in the image to be recognized. For each image region, the attention mechanism of the trained CCR model fuses the feature information of the image region and the associated image region based on the content similarity between the image region and the associated image region, obtaining attention feature information that focuses on contextual information. The trained CCR model then performs text content recognition on the image to be recognized based on the attention feature information to obtain an initial recognition result. The initial recognition result is then mapped based on a preset strategy (e.g., removing spaces from the initial recognition result using a preset strategy) to obtain the final recognition result. This scheme, by fusing the feature information of each image region and its corresponding associated image region to obtain attention feature information that focuses on contextual information for each image region, and performing text recognition on each image region based on the attention feature information that focuses on contextual information, can achieve parallel recognition of image regions in the image to be recognized, improving the recognition speed of the image to be recognized.

[0038] The trained text content recognition model can be a neural network model based on machine learning. Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and many other disciplines. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0039] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the preferred order of the embodiments.

[0040] This embodiment will be described from the perspective of a text recognition device, which can be integrated into a computer device, such as a server or a terminal. Figure 2 As shown, the process of this text recognition method is as follows;

[0041] 101. Obtain the text image to be recognized, which includes at least two image regions.

[0042] The text image to be recognized can be an image that requires text content recognition. The text image to be recognized can contain text content, such as text content in different languages ​​such as Chinese characters, English, Arabic numerals, and Japanese.

[0043] The image region can be a region obtained by dividing the text image to be recognized. For example, by dividing the text image to be recognized with a preset unit as the step size, multiple image regions can be obtained.

[0044] For example, it could be obtaining a text image to be recognized sent by the terminal, obtaining a text image to be recognized from a database, or obtaining a text image to be recognized stored in a blockchain.

[0045] 102. Perform feature extraction on the text image to be recognized to obtain the feature information of each image region in the text image to be recognized.

[0046] Feature extraction can be a process of analyzing and transforming the text image to be recognized in order to extract characteristic information of the text image. The characteristic information can be the information obtained through feature extraction.

[0047] For example, the specific method could be to analyze the text image to be recognized, such as obtaining the color value of each pixel in the text image under different color channels, which can yield at least one color value matrix for the text image to be recognized. The color value matrix can represent the color value of each pixel in the text image to be recognized. The color value matrix of the text image to be recognized can be transformed to perform feature extraction, thereby obtaining the feature information of each image region in the text image to be recognized.

[0048] In one embodiment, feature information of each image region in the text image to be recognized can be obtained by performing convolution processing on the text image to be recognized, i.e., the step "feature extraction of the text image to be recognized to obtain feature information of each image region in the text image to be recognized" can specifically include:

[0049] The image to be recognized is convolved to obtain the feature map of the image to be recognized;

[0050] Feature extraction is performed on the feature map to obtain feature information of each image region in the image to be identified.

[0051] The feature map can be the output of convolution processing, representing the data of a certain feature distribution in the text image to be recognized.

[0052] For example, a specific method could be to convolve the text image to be recognized using a convolution kernel, also known as a filter, to obtain a feature map of the text image.

[0053] Feature selection is performed on the feature map to obtain the feature information of the text image to be recognized. The feature information of the text image to be recognized is a set of feature information of each image region. Therefore, the feature information of each image region is obtained.

[0054] The process of extracting features from the text image to be recognized to obtain feature information of each image region in the text image can also be implemented by a trained text content recognition model. That is, in one embodiment, the step "extracting features from the text image to be recognized to obtain feature information of each image region in the text image" can specifically include:

[0055] The trained text content recognition model extracts features from the text image to be recognized, obtaining feature information for each image region of the text image to be recognized.

[0056] The trained text content recognition model is a network system formed by extensive interconnection of a large number of simple processing units (called neurons) used to recognize text contained in images.

[0057] For example, feature extraction can be performed using a convolutional neural network (CNN) in a trained text recognition model. A CNN can include at least one convolutional layer, at least one sampling layer, and a classification layer. The convolutional layer can extract features from the text image to be recognized, the sampling layer can select features from the extracted features, and the fully connected layer can classify the features to obtain the feature information of each image region in the text image to be recognized.

[0058] The trained text content recognition model can be a model obtained by training an initial text content recognition model. This involves continuously adjusting the network parameters (such as attention weights and convolutional kernels) of the initial model until a preset training termination condition is met. For example, the accuracy of the prediction results output by the initial model is greater than a preset accuracy, or the number of training iterations is greater than a preset number of iterations. This results in the trained model. In one embodiment, before the step "obtain the image to be recognized," the following steps are also included:

[0059] Obtain text image samples, which include at least two image regions;

[0060] The initial text content recognition model is used to extract features from the image regions of the text image sample to obtain the feature information of each image region of the text image sample.

[0061] For each image region, the content similarity between the image region and the associated image regions is calculated based on the feature information of the image region and the feature information of the associated image regions.

[0062] For each image region, the feature information of the image region and the associated image region is fused based on the content similarity between the image region and the associated image region to obtain attention feature information that focuses on context information;

[0063] Text content recognition is performed on the image to be recognized based on attention feature information to obtain prediction results;

[0064] Based on the prediction results and the sample labels of the text image samples, the initial text content recognition model is trained to obtain the trained text content recognition model.

[0065] The text image sample can be an image input used to train the initial text content recognition model.

[0066] For example, specifically, text image samples can be obtained. For each image region of the text image sample, the feature information of the image region and the associated image region can be fused by the initial text content recognition model based on the content similarity between the image region and the associated image region to obtain attention feature information that focuses on contextual information. Based on the attention feature information, text content recognition can be performed on the image to be recognized to obtain the prediction result. The specific process can be referred to the relevant description in the embodiments of this application, and will not be repeated here.

[0067] The error between the prediction result and the sample label of the text image sample is calculated. The network parameters in the initial text content recognition model are adjusted according to the calculated error to train the initial text content recognition model until the preset training termination condition is met, and the trained text content recognition model is obtained.

[0068] In text image samples, each image region does not necessarily correspond to a sample label. For example, if the sample label is "food," the image region may be a blank area containing no text content. If an image region does not have a corresponding sample label, effective training is impossible. Therefore, a loss function based on a time-series alignment algorithm (Connectionist Temporal Classification, CTC) can be used. This algorithm adds a blank symbol to the sample label, eliminating the need to align the image region with the characters in the sample label or to label each image region individually. As long as the sample label can be obtained through mapping, the prediction result is correct. This solves the problems of inconsistent lengths between text image samples, image regions, and sample labels (i.e., text results) and uncertain positions of sample labels and image regions (sample label and image region alignment). In one embodiment, the step "training the initial text content recognition model based on the prediction result and the sample labels of the text image samples to obtain the trained text content recognition model" can specifically be:

[0069] The corresponding character path set is determined based on the characters in the sample label. The character path set contains at least one character path. The character path is mapped to obtain the sample label through a preset mapping strategy.

[0070] The path probability of each character path in the character path set is calculated based on the predicted probability of each character in the prediction result.

[0071] The error value between the prediction result and the sample label is calculated based on the character path probability.

[0072] The initial text content recognition model is trained based on the error value to obtain the trained text content recognition model.

[0073] The sample label can be the text content contained in the text image sample. For example, if the text content contained in the text image sample is "Who am I", then the sample label corresponding to the text image sample is "Who am I".

[0074] The character path set can be a set containing all character paths that can be mapped to obtain sample labels through a preset mapping strategy.

[0075] The default mapping strategy is to map duplicate characters to a single character (i.e., merge them into one character) and to map blank characters to empty (i.e., delete blank characters).

[0076] The character path can represent the correct prediction result of each image region of the text image sample. If the characters corresponding to the image regions of the text image sample correspond one-to-one with the character path, the sample label can be obtained through mapping, and the initial text content recognition model can correctly recognize the text image sample.

[0077] For example, if the sample label is "C (for distinction, it can be referred to as the first character C) C (for distinction, it can be referred to as the second character C) AT", the corresponding sample sequence is {-C,-,C-,A,-,}. The text image sample has 5 image regions. The corresponding path planning (excluding character paths) diagram can be shown as follows: Figure 3 As shown, each character in the sample sequence corresponds to a different row in sequence, and each image region of the text image sample corresponds to a different column.

[0078] Because the text content in the text image samples has a certain order, such as from left to right, inputting results out of order will lead to misaligned prediction results. Correspondingly, in the path planning diagram, the character path can only proceed from left to right (e.g., ...). Figure 4 Path a in the image (jumping from the first character C in image region 1 to the first character C in image region b), and jumping from top to bottom right (e.g.) Figure 4 The path b in the image region jumps from the first character C in image region 1 to the blank character in image region b.

[0079] Because the preset mapping strategy merges duplicate characters, characters in the path planning can jump within themselves, that is, jump from the current area to an adjacent image area within the same line, for example... Figure 4 The path 'a' in the text; jumps between characters with the same meaning are not allowed (e.g., ...). Figure 4 The path c shown will be merged because the two consecutive characters obtained by a direct jump are the same character, according to the preset mapping strategy. The character path must pass through a whitespace character after the first character C to reach the second character C. Figure 4 The paths shown in paths b and d allow direct jumps between different characters (e.g., ...). Figure 4 Path e in the path.

[0080] In summary, the set of paths corresponding to the sample label "CCAT" can include Figure 5 The complete character path shown. Figure 5 The character paths shown can all be used to obtain sample labels through a preset mapping strategy.

[0081] For example, the sample label could be "ab". Based on the CTC algorithm, a loss function can be calculated by adding a blank character (denoted as "-") to the characters 'a' and 'b' in the sample label, resulting in the label sequence {-,a,-,b,-}. Assuming the text image sample contains three image regions, and the preset mapping strategy maps repeated characters to a single character (merging them into one character) and blank characters to empty characters (deleting blank characters), the character path set could contain paths such as "-ab", "ab-", "ab", "aab", and "abb". By using the preset mapping strategy to delete blank characters in these paths and merge identical characters in the paths, the sample label "ab" can be obtained.

[0082] The prediction result includes the predicted probability of each character in the preset dictionary. For example, if the preset dictionary contains 1k characters, the prediction result includes the predicted probability of each image region in the text image sample for each character in the preset dictionary. The path probability of each character path in the character path set is calculated based on the preset probability of each character in the preset dictionary corresponding to each image region. Based on the sum of the path probabilities of all character paths in the path set, the probability of the initial text content recognition model predicting the sample label can be obtained, that is, the error value between the prediction result of the initial text content recognition model and the sample label. The network parameters of the initial text content recognition model are adjusted based on the error value to train the initial text content recognition model and obtain the trained text content recognition model.

[0083] In one embodiment, this can be achieved by restricting characters from jumping within themselves when transitioning from the current image region to an adjacent image region (e.g., ...). Figure 6 Path a) allows direct jumps to the same characters (e.g., ...). Figure 6 In path c), when multiple consecutive identical characters exist in a text image sample, accurate prediction can be performed, avoiding the situation where multiple duplicate characters are predicted and merged. That is, in one embodiment, the step "determine the corresponding character path set based on the characters in the sample label" may specifically include:

[0084] The label sequence is determined based on the characters contained in the sample labels. The label sequence includes the first character, the second character, and the separator character.

[0085] The corresponding character path set is determined based on the label sequence. The character path set includes paths that jump from the first character to the second character and paths that jump from the first character to the spacer character.

[0086] The first character and the second character can be different characters in the label sequence. The predicted sample is "C (for distinction, it can be referred to as the first character C) C (for distinction, it can be referred to as the second character C) AT". The first character can be the first character C and the second character can be the second character C, or the first character can be the second character C and the second character can be A. The first character and the second character correspond to different rows in the path planning diagram.

[0087] The separator character can be a character that separates the characters in the sample label, such as a whitespace character or other characters.

[0088] Specifically, because characters cannot jump to themselves (e.g. Figure 6 Path a) allows direct jumps to the same characters (e.g., ...). Figure 6 Path c) in

[0089] In one embodiment, the character path set may include the path set corresponding to the sample label "CCAT". Figure 7 The entire character path is shown. Since character jumps are restricted to themselves, the default mapping strategy is to map the separator character to empty (i.e., delete the separator character), and not to map repeated characters to a single character (i.e., not to merge repeated characters into a single character). Figure 7 The character paths shown can all be used to obtain the sample label "CCAT" through a preset mapping strategy.

[0090] 103. For each image region, calculate the content similarity between the image region and the associated image regions based on the feature information of the image region and the feature information of the associated image regions.

[0091] Among them, the associated image region can be an image region associated with the image region, such as an adjacent image region of the image region. The associated image region can also be all the image regions in the text image to be identified, or other image regions in the text image to be identified.

[0092] Content similarity can be defined as the similarity between the content contained in an image region and the content contained in an associated image region.

[0093] For example, the feature information of the image region and the feature information of the associated image region can be mapped into the target feature space. The distance between the feature information of the image region and the feature information of the associated image region in the target feature space can be calculated. Based on the calculated distance, the content similarity between the image region and the associated image region can be determined. The smaller the distance, the more similar the image region and the associated image region are, and the higher the content similarity. The larger the distance, the less similar the image region and the associated image region are, and the lower the content similarity.

[0094] Feature information may include feature vectors. The content similarity between an image region and its associated image regions can be calculated based on the distance between their feature vectors. Specifically, in one embodiment, the step "for each image region, calculate the content similarity between the image region and its associated image regions based on the feature information of the image region and the feature information of the associated image regions that match the image region" may include:

[0095] Determine the target image region to be processed, as well as the associated image regions of the target image region;

[0096] Based on attention weight information, the target feature vector of the target image region is spatially mapped to obtain the first mapping vector corresponding to the target feature vector of the target image region in the feature space.

[0097] Based on attention weight information, the associated feature vectors of the associated image regions are spatially mapped to obtain the second mapping vector corresponding to the associated feature vectors of the associated image regions in the feature space.

[0098] The content similarity between the target image region and the associated image region is calculated based on the distance between the first mapping vector and the second mapping vector, thus obtaining the content similarity of each image region.

[0099] The target image region can be the current image region to be processed, and the feature vector corresponding to the target image region is the target feature vector.

[0100] The attention weight information can be information about spatial transformation of the feature vector; for example, the attention weight information can be information in the form of an attention weight matrix.

[0101] Among them, the associated feature vector can be the feature vector of the associated image region corresponding to the target image region.

[0102] The first mapping vector can be a vector obtained by mapping the target feature vector corresponding to the target image region to the feature space; the second mapping vector can be a vector obtained by mapping the associated feature vector corresponding to the associated image region to the feature space.

[0103] The feature space can be different from the space where the target feature vector is located, or it can be the same as the space where the target feature vector is located.

[0104] For example, it can be to determine the current image region to be processed, i.e. the target image region, and the associated image region corresponding to the target image region. Based on the attention weight information, the target feature vector is linearly transformed to map the target feature vector into the feature space, thereby obtaining the first mapping vector corresponding to the target feature vector in the feature space.

[0105] Based on attention weight information, the relevant feature vectors of the relevant image regions are linearly transformed and mapped to the feature space to obtain the corresponding second mapping vector in the feature space.

[0106] Calculate the distance between the first mapping vector and the second mapping vector. The distance between the first mapping vector and the second mapping vector can be Manhattan distance, Euclidean distance, Chebyshev distance, cosine similarity, or Hamming distance, etc. Calculate the content similarity between the target image region and the associated image region based on the distance between the first mapping vector and the second mapping vector. For example, normalize the distance to convert it into data between 0 and 1 to obtain the content similarity between the target image region and the associated image region.

[0107] For each image region in the text image to be identified, perform the above operation to obtain the content similarity between each image region and the associated region.

[0108] The feature vectors corresponding to image regions can also be mapped to multiple vectors, each reflecting different feature information of the feature vector. Specifically, the first mapping vector includes a first query vector, a first key vector, and a first content vector; the second mapping vector includes a second query vector, a second key vector, and a second content vector. Each vector has a specific function, improving the similarity calculation. The similarity between each image region and its associated image region is calculated based on the mapped vectors. In one embodiment, the step "calculating the content similarity between the target image region and its associated image region based on the distance between the first and second mapping vectors to obtain the content similarity of each image region" can specifically include:

[0109] Based on the distance between the first query vector corresponding to the target image region and the second key vector corresponding to the associated image region, the content similarity between the target image region and the associated image region is calculated to obtain the content similarity of each image region.

[0110] Among them, the first query vector, the first key vector, and the first content vector can be vectors obtained by linearly transforming the target feature vector of the target image region according to different attention weight matrices in the attention weight information.

[0111] The second query vector, the second key vector, and the second content vector can be vectors obtained by linearly transforming the relevant feature vectors of the relevant image regions according to different attention weight matrices in the attention weight information.

[0112] For example, the weight information can include a first weight matrix, a second weight matrix, and a third weight matrix. The target feature vector is mapped based on the first weight matrix to obtain the first query vector, denoted as Query1, or Q1 for short. The target feature vector is mapped based on the second weight matrix to obtain the first key vector, denoted as Key1, or K1 for short. The target feature vector is mapped based on the third weight matrix to obtain the first content vector, denoted as Value1, or V1 for short.

[0113] By performing the same mapping process on the associated image regions, we can obtain Q12, K2, and V2 corresponding to each associated image region.

[0114] Calculate the distance between the first query vector corresponding to the target image region and the second key vector corresponding to the associated image region. For example, the first query vector and the second key vector can be multiplied by a dot product, such as Q·K. Normalize the result of the dot product to obtain the content similarity between the target image region and the associated image region, and obtain the content similarity of each image region.

[0115] The same processing is performed on each image region in the text image to be identified, and the content similarity between each image region and its corresponding associated region is obtained.

[0116] The second content vector of the associated image region is weighted based on the obtained content similarity and then fused with the first content vector. This allows the feature information of the associated image region and the target image region to be fused to obtain attention feature information that focuses on contextual information. Specifically, in one embodiment, the step "for each image region, based on the content similarity between the image region and the associated image region, the feature information of the image region and the associated image region are fused to obtain attention feature information that focuses on contextual information" can be:

[0117] For each image region, the content feature vector corresponding to the associated image region is weighted according to the content similarity between the image region and the associated image region to obtain the weighted content feature vector corresponding to the associated image region.

[0118] For each image region, the feature vector of the image region is fused with the weighted content feature vector of the associated image region to obtain attention feature information with contextual information for the image to be identified.

[0119] For example, specifically for each image region, the content similarity between the feature vector of the image region and the feature vector of the associated image region can be used as the weight of the associated image region. The content similarity can be multiplied by the content feature vector of the associated image region to obtain the weighted content feature vector of each associated image region.

[0120] The feature vectors of the image regions are fused with the weighted content feature vectors of the associated image regions. For example, the feature vectors of the image regions are added to the weighted content feature vectors of the associated image regions to obtain the attention feature information corresponding to the attention context information of each image region.

[0121] By performing the same processing on each image region, attention feature information corresponding to the attention context information of the text image to be recognized can be obtained.

[0122] The content similarity between image region i and its associated image region j is denoted as score. ij score ij =Qi·Kj, where the content similarity between image region j and its associated image region i is denoted as score. ij score ji =Qj·Ki, ideally, score ij If it is larger, then the score ji The score is also relatively high, and it can be adjusted by modifying the content similarity. ij and score ji Similarly, in one embodiment, the step "calculating the content similarity between the target image region and the associated image region based on the distance between the first query vector corresponding to the target image region and the second key vector corresponding to the associated image region, to obtain the content similarity of each image region" may specifically include:

[0123] Based on the distance between the first query vector corresponding to the target image region and the second key vector corresponding to the associated image region, the initial content similarity between the target image region and the associated image region is calculated to obtain the initial content similarity of each image region.

[0124] Based on the initial content similarity of each image region, the initial content similarity matrix of the image to be identified is obtained;

[0125] The initial content similarity matrix and its transpose are used to adjust the initial content similarity of each image region to obtain the content similarity between the target image region and the associated image region.

[0126] The initial content similarity can be obtained based on the distance between the first query vector and the second key vector, representing the degree of similarity between the image region and the associated image region.

[0127] Content similarity can be obtained by adjusting the initial content similarity.

[0128] The initial content similarity matrix can be obtained by calculating the initial content similarity between each image region and related image regions, and the transpose matrix can be obtained by interchanging the rows and columns of the initial content similarity matrix.

[0129] For example, specifically, the first query vector of the target image region and the second key vector of the associated image region can be multiplied to obtain the initial content similarity between the image region and the associated image region. Based on the initial content similarity between each image region and the associated image region, an initial content similarity matrix for the text image to be recognized can be obtained, denoted as SCORE0. The element 'score' in the i-th row and i-th column of the initial content similarity matrix is... ij The element "score" located in the j-th row and i-th column can represent the initial content similarity between image region i and image region j. ji This can represent the initial content similarity between image region j and image region i.

[0130] Interchange the rows and columns of the initial content similarity matrix to obtain the transpose matrix SCORE. T Add the transpose matrix to the initial content similarity matrix to obtain the content similarity matrix SCORE = SCORE0 + SCORE T The content similarity matrix is ​​a symmetric matrix, and the elements of the content similarity matrix are scores. ij =score ji .

[0131] The content similarity matrix can be used to determine the content similarity between the feature vector of each image region and the feature vector of the associated region. For example, the content similarity between image region i and image region j is the score of the content similarity matrix SCORE. ij .

[0132] During the training of the initial text content recognition model, the transpose matrix is ​​used to adjust the initial content similarity, resulting in a symmetric content similarity matrix. This can accelerate the convergence of the initial text content recognition model and speed up the training process.

[0133] The content similarity matrix contains the content similarity between each image region in the text image to be identified and itself, as well as other image regions. In an image, the text content contained in an image region is usually related to several adjacent image regions, while the relationship with image regions that are farther away is weaker. Therefore, when performing feature information fusion, it is possible to choose to fuse only several adjacent image regions. In one embodiment, a window matrix can be set on the content similarity matrix. The window matrix can shield other image regions that are unrelated to the image region. That is, the step "adjusting the initial content similarity matrix and the transpose of the initial content similarity matrix to obtain the content similarity between the image region and the associated image region" can specifically include:

[0134] The first content similarity matrix is ​​obtained by adding the initial content similarity matrix and its transpose.

[0135] For each image region, a corresponding window matrix is ​​set for the first content similarity matrix to obtain the first content similarity matrix;

[0136] The content similarity between the image region and the associated image region is determined based on the second content similarity matrix.

[0137] The window matrix can be a matrix of the same type as the content similarity matrix. It can be used to retain the content similarity within the region indicated by the window matrix (which can be called the window position) and to mask the content similarity at other positions, such as setting the content similarity at other positions to -∞. The window position can be set according to each image region. For example, for image region i, the window position can be score. ii score ii+1 and score ii+1 This means preserving the content similarity of three adjacent image regions while masking the content similarity with other image regions.

[0138] For example, the rows and columns of the initial content similarity matrix can be interchanged to obtain the transpose matrix. The transpose matrix is ​​then added to the initial content similarity matrix to obtain the first content similarity matrix. For each image region, a corresponding window matrix is ​​set for the obtained content similarity matrix to retain the content similarity of the image region within the area indicated by the window matrix, while masking the content similarity of other locations.

[0139] The window matrix can be set to 0 for the window position and -∞ or other very large negative numbers, such as 10, for other positions. -16The content similarity matrix is ​​then added to the window matrix. This causes the content similarity at non-window locations to be set to a large negative number due to the addition of a large negative number. After normalization, the content similarity at non-window locations is mapped to 0, resulting in a second content similarity matrix. Based on this second content similarity matrix, the content similarity between image regions and associated image regions can be determined.

[0140] Step 103 can be processed using a multi-head attention mechanism, which includes multiple attention mechanisms. An attention mechanism is a special structure embedded in a machine learning model to automatically learn and calculate the contribution of input data to output data. Each attention mechanism performs the same processing on each image region, and each attention mechanism can obtain the content similarity between the image region and the associated image region for each image region. Specifically, in one embodiment, the step "for each image region, calculate the content similarity between the image region and the associated image region based on the feature information of the image region and the feature information of the associated image region" can include:

[0141] For each image region, a multi-head attention mechanism is used to perform parallel calculations based on the feature information of the image region and the feature information of the associated image regions to obtain the content similarity between the image region and the associated image region under each attention mechanism.

[0142] The specific operations performed by each attention mechanism can be found in the description of the corresponding location in the embodiment, and will not be repeated here.

[0143] Each attention mechanism can output the content similarity between the corresponding image region and the associated image region for each image region.

[0144] By employing a multi-head attention mechanism, multiple sets of content similarity scores between each region and its associated image region can be obtained. Accordingly, step 104 requires processing these multiple sets of content similarity scores. Step 104 may specifically include:

[0145] For each image region, based on the multi-head attention mechanism, the feature information of the image region and the associated image region is fused according to the content similarity between the image region and the associated image region to obtain the attention feature information of the attention context under each attention mechanism;

[0146] The attention feature information under each attention mechanism is fused to obtain the attention feature information that focuses on the context information.

[0147] For example, specifically, it can be that each attention mechanism calculates the content similarity between the corresponding output image region and the associated image region, and then fuses the feature information of the image region and the associated image region (the specific implementation process can be referred to the description of the corresponding position in the embodiment, which will not be repeated here) to obtain the attention feature information under each attention mechanism.

[0148] The obtained attention feature information is concatenated to obtain concatenated feature information. This concatenated feature information is then processed to obtain feature information of the same dimension as the attention feature information under each attention mechanism, thus obtaining the attention feature information that focuses on the context information.

[0149] Step 103 can also iteratively process the feature information using a multi-layer attention mechanism. Each layer of attention mechanism can include at least one attention mechanism, and each layer performs the same processing on each image region. Each layer of attention mechanism can obtain the content similarity between the image region and the associated image region for each image region. That is, in one embodiment, the step "for each image region, calculate the content similarity between the image region and the associated image region based on the feature information of the image region and the feature information of the associated image region" can specifically include:

[0150] The target layer attention mechanism is determined from the multi-layer attention mechanism, and the feature information of each image region in the text image to be recognized is determined as the target input feature information of the target attention mechanism.

[0151] For each image region, the content similarity between the image region and the associated image region is calculated using the target layer attention mechanism based on the feature information of the image region and the feature information of the associated image region.

[0152] The target layer attention mechanism can be the attention mechanism layer currently used for text recognition. The target input feature information can be the feature information processed by the target layer attention mechanism.

[0153] For example, the target layer attention mechanism may include at least one attention mechanism. Each attention mechanism may calculate the content similarity between the image region and the associated image region for each image region based on the feature information of the image region and the feature information of the associated image region. The specific implementation process can be referred to the description of the corresponding position in the embodiment, and will not be repeated here.

[0154] Step 103 is processed through a multi-layer attention mechanism, so the corresponding step 104 may specifically include:

[0155] The target layer attention mechanism fuses the feature information of the image region and the associated image region based on the content similarity between the image region and the associated image region to obtain the processed attention feature information of the attention context.

[0156] When the target layer attention mechanism is not the predetermined layer attention mechanism, the associated layer attention mechanism of the target layer attention mechanism in the multi-layer attention mechanism is updated to the target attention mechanism, and the processed attention feature information is updated to the target input feature information. Then, the execution is returned to each image region. Based on the feature information of the image region and the feature information of the associated image region, the content similarity between the image region and the associated image region is calculated by the target layer attention mechanism.

[0157] When the target layer attention mechanism is the predetermined layer attention mechanism, the processed attention feature information is output to obtain the attention feature information of the text image to be recognized.

[0158] In this context, the association layer attention mechanism can be the next layer of attention mechanism after the target attention mechanism. The pre-defined layer attention mechanism can be the last layer in a multi-layer attention mechanism.

[0159] For example, specifically, the feature information of the image region and the associated image region can be fused based on the content similarity between the image region and the associated image region through the target layer attention mechanism to obtain the attention feature information after the attention context information is processed. The specific implementation process can be referred to the description of the corresponding position in the embodiment, and will not be repeated here.

[0160] When the target layer attention mechanism is not the predetermined layer attention mechanism, the processed attention feature information output by the target layer attention mechanism is used as the target input feature information of the next layer attention mechanism, and the next layer attention mechanism is used as the target layer attention mechanism. The same operation is performed, that is, the target feature information is processed through each layer attention mechanism in the multi-layer attention mechanism in sequence, and the processed attention feature information output by the previous layer attention mechanism is used as the target input feature information of its next layer attention mechanism, until the target layer attention mechanism is the predetermined layer attention mechanism, and the processed attention feature information is output to obtain the attention feature information.

[0161] 104. For each image region, based on the content similarity between the image region and the associated image region, the feature information of the image region and the associated image region is fused to obtain attention feature information that focuses on contextual information.

[0162] The attention feature information can be information that includes the feature information of each image region and its corresponding associated image region.

[0163] For example, specifically, for each image region, the similarity between the content of the image region and the associated image region can be used as the weight of the associated image region. The image region and the associated image region can be weighted and summed to obtain the attention feature information of the attention context information corresponding to each image region.

[0164] Attention feature information of the text image to be recognized can be obtained by using the attention feature information of the attention context information corresponding to each image region.

[0165] 105. Based on attention feature information, perform text content recognition on the image to be recognized to obtain the recognition result.

[0166] For example, text content recognition can be performed on the image to be recognized based on attention feature information. For each image region in the text image to be recognized, the predicted probability of the image region being each character in a preset dictionary is calculated. For example, if the preset dictionary includes four characters A, B, C, and D, and the text image to be recognized contains five image regions, the probability of image region 1 being A is 0.8, the probability of image region 1 being B is 0.05, the probability of image region 1 being C is 0.05, and the probability of image region 1 being D is 0.1. The same applies to the other image regions of the text image to be recognized.

[0167] The recognition result can be obtained based on the probability of each character for each image region. For example, the character with the highest probability is determined as the recognition result of that image region. For example, the recognition result of image region 1 can be A.

[0168] As described above, this application embodiment can acquire a text image to be recognized, which includes at least two image regions; perform feature extraction on the text image to be recognized to obtain feature information of each image region in the text image to be recognized; for each image region, calculate the content similarity between the image region and the associated image region based on the feature information of the image region and the feature information of the associated image region; for each image region, fuse the feature information of the image region and the associated image region based on the content similarity between the image region and the associated image region to obtain attention feature information that focuses on contextual information; and perform text content recognition on the image to be recognized based on the attention feature information to obtain the recognition result. This scheme achieves parallel recognition of image regions in the image to be recognized by fusing the feature information of each image region and the corresponding associated image region to obtain attention feature information that focuses on contextual information for each image region, and performs text recognition on each image region based on the attention feature information that focuses on contextual information, thereby improving the recognition speed of the image to be recognized.

[0169] Based on the above embodiments, the following examples will provide further detailed explanations.

[0170] This embodiment will be described from the perspective of a text recognition device, which can be integrated into a computer device, such as a server.

[0171] This application provides a text recognition method, such as... Figure 8 As shown, the specific process of this text recognition method can be as follows:

[0172] 201. The server obtains the text image to be recognized sent by the terminal.

[0173] For example, the terminal could provide a user interface that recognizes text and images, for instance. Figure 9 The user interface shown is designed to allow users to upload images of text to be recognized (i.e., ...). Figure 9 The method allows users to upload images and instruct the server to retrieve the text image to be recognized and perform text content recognition. The text recognition method provided in this application supports multi-scenario recognition, such as general character recognition, card text recognition, invoice and document recognition, automotive-related recognition, industry document recognition, and intelligent barcode scanning. It also supports multiple speech recognition languages, such as English, Korean, Japanese, and Spanish.

[0174] 202. The server extracts features from the text image to be recognized using the trained text recognition model, and obtains a feature vector sequence of the text image to be recognized. The feature vector sequence includes the feature vector of each image region in the text image to be recognized.

[0175] For example, feature extraction can be performed using a convolutional neural network (CNN) in a trained text recognition model. A CNN can include at least one convolutional layer, at least one sampling layer, and a classification layer. The convolution kernel of the convolutional layer, also known as a filter, can extract features from the text image to be recognized, thus obtaining the feature map of the text image to be recognized.

[0176] The sampling layer can perform feature selection on the feature map, and the fully connected layer can classify the features to obtain the feature vector sequence Γ of the text image to be recognized. Each feature vector in the feature vector sequence Γ corresponds to an image region in the text image to be recognized.

[0177] 203. The server uses the trained text recognition model to perform spatial mapping processing on the feature vector sequence of the text image to be recognized based on the attention mechanism, so as to obtain the query vector sequence, key vector sequence and content vector sequence corresponding to the feature vector sequence.

[0178] For example, the attention mechanism can be a self-attention mechanism, which is a neural network of the trained text recognition model. The attention mechanism includes attention weight information, which can include a first weight matrix, a second weight matrix, and a third weight matrix. The server maps the feature vector sequence Γ based on the first weight matrix of the attention mechanism using the trained text recognition model to obtain the query vector sequence, denoted as Query0, or Q0 for short. Based on the second weight matrix, it maps the feature vector sequence Γ to obtain the key vector sequence, denoted as Key0, or K0 for short. Based on the third weight matrix, it maps the feature vector sequence Γ to obtain the content vector sequence, denoted as Value0, or V0 for short.

[0179] The query vector sequence includes the query vector for each image region in the text image to be identified, the key vector sequence includes the key vector for each image region in the text image to be identified, and the content vector sequence includes the content vector for each image region in the text image to be identified.

[0180] 204. For each image region, the server calculates the content similarity between the image region and the associated image region based on the query vector of the image region and the key vector of the associated image region, and obtains the second content similarity matrix of the text image to be identified.

[0181] For example, specifically: the associated region can be all image regions of the text image to be identified. For each image region, the query vector corresponding to the image region and the key vector corresponding to the associated image region are multiplied by a dot product, such as Q·K. The result of the dot product is normalized to obtain the initial content similarity between each image region and the associated image region.

[0182] The initial content similarity matrix SCORE0 of the text image to be identified can be obtained based on the initial content similarity between each image region and its associated image regions.

[0183] The element in the i-th row and i-th column of the initial content similarity matrix, score ij The element "score" located in the j-th row and i-th column can represent the initial content similarity between image region i and image region j. ji This can represent the initial content similarity between image region j and image region i.

[0184] Interchange the rows and columns of the initial content similarity matrix to obtain the transpose matrix SCORE. T Add the transpose of the initial content similarity matrix to the initial content similarity matrix to obtain the first content similarity matrix SCORE = SCORE0 + SCORE TThe first content similarity matrix is ​​a symmetric matrix, and the elements of the first content similarity matrix are scores. ij = element score ji .

[0185] 205. For each image region, the server sets a window matrix for the first content similarity matrix to obtain the second content similarity matrix corresponding to each image region.

[0186] For example, the server could set the window matrix to 0 for the window position and -∞ or other very large negative numbers, such as 10, for other positions. -16 The first content similarity matrix is ​​obtained by adding a window matrix to the first content similarity matrix. This causes the content similarity of non-window positions in the first content similarity matrix to be set to a large negative number due to the addition of a large negative number. Through normalization, the content similarity of non-window positions can be mapped to 0, resulting in the second content similarity matrix. The content similarity between the image region and the associated image region can be determined based on the second content similarity matrix.

[0187] The window matrix can be used to select a target associated image region from the associated image regions. The content similarity of the window position is the content similarity between each image region and the corresponding target associated image region. The content similarity of the masked area is the content similarity between each image region and the non-target associated image region.

[0188] The target associated image region can be several adjacent image regions of each image region. For example, the target image region of image region 5 can be image region 3, image region 4, image region 5, image region 6, and image region 7, etc.

[0189] 206. For each image region, the server fuses the feature vectors of the image region and the associated image regions based on the second content similarity matrix to obtain the attention feature vector of the attention context information of each image region.

[0190] For example, specifically, the attention feature vector C of the text image to be recognized can be calculated based on the content vector sequence of the text image to be recognized and the second content similarity matrix, for example, C = V0·Γ. Due to the setting of the window matrix, the content similarity between the image region and the non-target associated image region is 0 for each image region. Therefore, during the calculation, the content vector of the non-target associated image region and the corresponding content similarity are calculated to obtain a zero vector. The feature vector of the image region plus the zero vector is still the original feature vector. That is, the feature vector of the image region is not fused with the feature vector of the non-target associated image region.

[0191] The current image region to be processed is defined as the target image region. For the target image region, the feature vectors of the target image region and its corresponding associated image regions are fused to obtain the feature vector. (See the schematic diagram below.) Figure 10 As shown, the self-attention mechanism can process each image region individually. The recognition result can be predicted based on the attention feature vector corresponding to each image region. It does not need to predict the current image region based on the recognition result of the previous image region, thus achieving parallel prediction of the text image to be recognized, improving the prediction speed, and eliminating the need to deploy multiple servers, which greatly reduces the server deployment cost.

[0192] Common prediction methods based on the LSTM mechanism require the output of the previous image region to predict the current image region, which makes the prediction process of the text image to be recognized time-consuming. When there are a large number of text images to be recognized, multiple servers need to be deployed to meet the needs of recognizing numerous text images, resulting in high service deployment costs.

[0193] By processing the feature vector sequence of the text image to be recognized through a multi-head, multi-layer self-attention mechanism, attention feature vectors that better reflect the features of the text image to be recognized can be obtained.

[0194] For example, each attention mechanism in the n-head attention mechanism can execute steps 203-206 above. Each attention mechanism can correspond to different attention weight information. Based on the different attention weight information, the attention feature vector Ci under each attention mechanism can be obtained, where i represents the attention feature vector obtained by the i-th attention mechanism. The n attention feature vectors Ci obtained by the n-head attention mechanism are concatenated to obtain C0 = concate{C1, C2, C3, ..., Cn-1, Cn}. The fully connected layer of the trained text recognition model is used to reduce the dimensionality of C0 so that the dimension of C0 is the same as that of Ci. For example, C = concate{C1, C2, C3, ..., Cn-1, Cn}·W, where W is a linear transformation that maps C0 to C.

[0195] The single-layer attention mechanism obtains attention feature information by executing steps 203-206 above, while the multi-layer attention mechanism repeats steps 203-206 above, with the output Ci of the i-th layer attention mechanism serving as the input Γi+1 of the i+1-th layer.

[0196] Right now:

[0197] 207. The server performs text content recognition on the image to be recognized based on the attention feature information, obtains the recognition result of the text image to be recognized, and sends the recognition result to the terminal.

[0198] For example, text content recognition can be performed on the image to be recognized based on attention feature information. For each image region in the text image to be recognized, the predicted probability of the image region being each character in a preset dictionary is calculated. For example, if the preset dictionary includes four characters A, B, C, and D, and the text image to be recognized contains five image regions, the probability of image region 1 being A is 0.8, the probability of image region 1 being B is 0.05, the probability of image region 1 being C is 0.05, and the probability of image region 1 being D is 0.1. The same applies to the other image regions of the text image to be recognized.

[0199] The recognition result can be obtained based on the probability of each character for each image region. For example, the character with the highest probability is determined as the recognition result of that image region. For example, the recognition result of image region 1 can be A.

[0200] The initial recognition result of the text image to be recognized can be obtained by recognizing the recognition result of each image region. The interval characters in the initial recognition result are removed to obtain the recognition result of the text image to be recognized, and the recognition result is returned to the terminal.

[0201] As can be seen from the above, in this embodiment, the server can acquire the text image to be recognized sent by the terminal; extract features from the text image to be recognized using a trained text recognition model to obtain a feature vector sequence of the text image to be recognized, the feature vector sequence including the feature vector of each image region in the text image to be recognized; perform spatial mapping processing on the feature vector sequence of the text image to be recognized using an attention mechanism based on the trained text recognition model to obtain a query vector sequence, a key vector sequence, and a content vector sequence corresponding to the feature vector sequence; for each image region, the server calculates the content similarity between the image region and the associated image region based on the query vector of the image region and the key vector of the associated image region to obtain a second content similarity matrix of the text image to be recognized; for each image region, the server sets a window matrix on the first content similarity matrix to obtain a second content similarity matrix corresponding to each image region; for each image region, the server performs fusion processing on the feature vectors of the image region and the associated image region based on the second content similarity matrix to obtain attention feature information of the attention context information of each image region; perform text content recognition on the image to be recognized based on the attention feature information to obtain the recognition result of the text image to be recognized, and send the recognition result to the terminal. This scheme fuses the feature information of each image region and its corresponding associated image regions to obtain the attention feature information of each image region that focuses on the context. Based on the attention feature information of the context, text recognition is performed on each image region, which can realize parallel recognition of image regions in the image to be recognized and improve the recognition speed of the image to be recognized.

[0202] To facilitate better implementation of the text recognition method provided in the embodiments of this application, a text recognition device is also provided in one embodiment. The meanings of the terms are the same as in the text recognition method described above, and specific implementation details can be found in the description of the method embodiments.

[0203] This text recognition device can be integrated into a computer device, such as... Figure 11 As shown, the text recognition device may include: an acquisition unit 301, an extraction unit 302, a calculation unit 303, a fusion unit 304, and a recognition unit 305, as detailed below:

[0204] Acquisition unit 301: used to acquire a text image to be recognized, the text image to be recognized includes at least two image regions.

[0205] For example, it could be obtaining a text image to be recognized sent by the terminal, obtaining a text image to be recognized from a database, or obtaining a text image to be recognized stored in a blockchain.

[0206] Extraction unit 302: used to extract features from the text image to be recognized, and obtain feature information of each image region in the text image to be recognized.

[0207] For example, the specific method could be to analyze the text image to be recognized, such as obtaining the color value of each pixel in the text image under different color channels, which can yield at least one color value matrix for the text image to be recognized. The color value matrix can represent the color value of each pixel in the text image to be recognized. The color value matrix of the text image to be recognized can be transformed to perform feature extraction, thereby obtaining the feature information of each image region in the text image to be recognized.

[0208] In one embodiment, the extraction unit 302 may include a convolution subunit and an extraction subunit, specifically:

[0209] Convolutional subunit: Used to perform convolution processing on the image to be recognized to obtain the feature map of the image to be recognized;

[0210] Extraction subunit: used to extract features from the feature map to obtain feature information of each image region in the image to be identified.

[0211] For example, a specific method could be to convolve the text image to be recognized using a convolution kernel, also known as a filter, to obtain a feature map of the text image.

[0212] Feature selection is performed on the feature map to obtain the feature information of the text image to be recognized. The feature information of the text image to be recognized is a set of feature information of each image region. Therefore, the feature information of each image region is obtained.

[0213] The process of extracting features from the text image to be recognized to obtain feature information of each image region in the text image can also be implemented by a trained text content recognition model. That is, in one embodiment, the extraction unit 302 may include a model extraction subunit, specifically:

[0214] Model extraction subunit: Used to extract features from image regions using a trained text content recognition model, obtaining feature information for each image region.

[0215] For example, feature extraction can be performed using a convolutional neural network (CNN) in a trained text recognition model. A CNN can include at least one convolutional layer, at least one sampling layer, and a classification layer. The convolutional layer can extract features from the text image to be recognized, the sampling layer can select features from the extracted features, and the fully connected layer can classify the features to obtain the feature information of each image region in the text image to be recognized.

[0216] The trained text content recognition model can be obtained by training an initial text content recognition model. This involves continuously adjusting the network parameters (such as attention weights and convolutional kernels) of the initial model until a preset training termination condition is met. For example, the accuracy of the prediction results output by the initial model is greater than a preset accuracy, or the number of training iterations is greater than a preset number of iterations. In one embodiment, the text recognition device further includes a sample acquisition unit, a sample feature extraction unit, a similarity calculation unit, a feature fusion unit, a sample recognition unit, and a training unit.

[0217] Sample acquisition unit: used to acquire text image samples, wherein the text image samples include at least two image regions;

[0218] Sample feature extraction unit: used to extract features from the image regions of the text image sample through the initial text content recognition model, and obtain the feature information of each image region of the text image sample;

[0219] Similarity calculation unit: For each image region, it calculates the content similarity between the image region and the associated image regions based on the feature information of the image region and the feature information of the associated image regions;

[0220] Feature fusion unit: For each image region, based on the content similarity between the image region and the associated image region, the feature information of the image region and the associated image region are fused to obtain attention feature information that focuses on context information;

[0221] Sample recognition unit: used to recognize text content in the image to be recognized based on attention feature information and obtain prediction results;

[0222] Training unit: Used to train the initial text content recognition model based on the prediction results and sample labels of text image samples, to obtain the trained text content recognition model.

[0223] For example, specifically, text image samples can be obtained. For each image region of the text image sample, the feature information of the image region and the associated image region can be fused by the initial text content recognition model based on the content similarity between the image region and the associated image region to obtain attention feature information that focuses on contextual information. Based on the attention feature information, text content recognition can be performed on the image to be recognized to obtain the prediction result. The specific process can be referred to the relevant description in the embodiments of this application, and will not be repeated here.

[0224] The error between the prediction result and the sample label of the text image sample is calculated. The network parameters in the initial text content recognition model are adjusted according to the calculated error to train the initial text content recognition model until the preset training termination condition is met, and the trained text content recognition model is obtained.

[0225] In one embodiment, the training unit may include a path determination subunit, a probability calculation subunit, an error calculation subunit, and a training subunit, specifically:

[0226] Path determination subunit: used to determine the corresponding set of character paths based on the characters in the sample label. The set of character paths contains at least one character path, and the character paths are mapped to obtain the sample label through a preset mapping strategy.

[0227] Probability calculation subunit: used to calculate the path probability of each character path in the character path set based on the prediction probability of each character in the prediction result;

[0228] Error calculation subunit: used to calculate the error value between the prediction result and the sample label based on the character path probability;

[0229] Training sub-unit: Used to train the initial text content recognition model based on the error value to obtain the trained text content recognition model.

[0230] The sample label can be the text content contained in the text image sample. For example, if the text content contained in the text image sample is "Who am I", then the sample label corresponding to the text image sample is "Who am I".

[0231] For example, if the sample label is "C (for distinction, it can be referred to as the first character C) C (for distinction, it can be referred to as the second character C) AT", the corresponding sample sequence is {-C,-,C-,A,-,}. The text image sample has 5 image regions. The corresponding path planning (excluding character paths) diagram can be shown as follows: Figure 3 As shown, each character in the sample sequence corresponds to a different row in sequence, and each image region of the text image sample corresponds to a different column.

[0232] Because the text content in the text image samples has a certain order, such as from left to right, inputting results out of order will lead to misaligned prediction results. Correspondingly, in the path planning diagram, the character path can only proceed from left to right (e.g., ...). Figure 4 Path a in the image (jumping from the first character C in image region 1 to the first character C in image region b), and jumping from top to bottom right (e.g.) Figure 4 The path b in the image region jumps from the first character C in image region 1 to the blank character in image region b.

[0233] Because the preset mapping strategy merges duplicate characters, characters in the path planning can jump within themselves, that is, jump from the current area to an adjacent image area within the same line, for example... Figure 4 The path 'a' in the text; jumps between characters with the same meaning are not allowed (e.g., ...). Figure 4 The path c shown will be merged because the two consecutive characters obtained by a direct jump are the same character (according to the preset mapping strategy). The character path must pass a whitespace character after the first character C to reach the second character C. Figure 4 The paths shown in paths b and d allow direct jumps between different characters (e.g., ...). Figure 4 Path e in the middle.

[0234] In summary, the set of paths corresponding to the sample label "CCAT" can include Figure 5 The complete character path shown. Figure 5 The character paths shown can all be used to obtain sample labels through a preset mapping strategy.

[0235] For example, the sample label could be "ab". Based on the CTC algorithm, a loss function can be calculated by adding a blank character (denoted as "-") to the characters 'a' and 'b' in the sample label, resulting in the label sequence {-,a,-,b,-}. Assuming the text image sample contains three image regions, and the preset mapping strategy maps repeated characters to a single character (merging them into one character) and blank characters to empty characters (deleting blank characters), the character path set could contain paths such as "-ab", "ab-", "ab", "aab", and "abb". By using the preset mapping strategy to delete blank characters in these paths and merge identical characters in the paths, the sample label "ab" can be obtained.

[0236] The prediction result includes the predicted probability of each character in the preset dictionary. For example, if the preset dictionary contains 1k characters, the prediction result includes the predicted probability of each image region in the text image sample for each character in the preset dictionary. The path probability of each character path in the character path set is calculated based on the preset probability of each character in the preset dictionary corresponding to each image region. Based on the sum of the path probabilities of all character paths in the path set, the probability of the initial text content recognition model predicting the sample label can be obtained, that is, the error value between the prediction result of the initial text content recognition model and the sample label. The network parameters of the initial text content recognition model are adjusted based on the error value to train the initial text content recognition model and obtain the trained text content recognition model.

[0237] In one embodiment, the path determination subunit may include a sequence determination module and a path set determination module, specifically:

[0238] Sequence Determination Module: Used to determine the label sequence based on the characters contained in the sample labels. The label sequence includes the first character, the second character, and the separator character.

[0239] Path set determination module: used to determine the corresponding character path set based on the label sequence. The character path set includes the path from the first character to the second character, and the path from the first character to the interval character.

[0240] Specifically, because characters cannot jump to themselves (e.g. Figure 6 Path a) allows direct jumps to the same characters (e.g., ...). Figure 6 Path c) in

[0241] In one embodiment, the character path set may include the path set corresponding to the sample label "CCAT". Figure 7 The entire character path is shown. Since character jumps are restricted to themselves, the default mapping strategy is to map the separator character to empty (i.e., delete the separator character), and not to map repeated characters to a single character (i.e., not to merge repeated characters into a single character). Figure 7 The character paths shown can all be used to obtain the sample label "CCAT" through a preset mapping strategy.

[0242] Calculation unit 303: For each image region, it calculates the content similarity between the image region and the associated image region based on the feature information of the image region and the feature information of the associated image region.

[0243] For example, the feature information of the image region and the feature information of the associated image region can be mapped into the target feature space. The distance between the feature information of the image region and the feature information of the associated image region in the target feature space can be calculated. Based on the calculated distance, the content similarity between the image region and the associated image region can be determined. The smaller the distance, the more similar the image region and the associated image region are, and the higher the content similarity. The larger the distance, the less similar the image region and the associated image region are, and the lower the content similarity.

[0244] In one embodiment, the calculation unit 303 may include a region determination subunit, a first mapping subunit, a second mapping subunit, and a calculation subunit, specifically:

[0245] Region determination subunit: used to determine the target image region to be processed, as well as the associated image regions of the target image region;

[0246] The first mapping subunit is used to perform spatial mapping processing on the target feature vector of the target image region based on attention weight information, so as to obtain the first mapping vector corresponding to the target feature vector of the target image region in the feature space.

[0247] The second mapping subunit is used to perform spatial mapping processing on the associated feature vectors of the associated image regions based on attention weight information, so as to obtain the second mapping vector corresponding to the associated feature vectors of the associated image regions in the feature space.

[0248] The calculation subunit is used to calculate the content similarity between the target image region and the associated image region based on the distance between the first mapping vector and the second mapping vector, and to obtain the content similarity of each image region.

[0249] For example, it can be to determine the current image region to be processed, i.e. the target image region, and the associated image region corresponding to the target image region. Based on the attention weight information, the target feature vector is linearly transformed to map the target feature vector into the feature space, thereby obtaining the first mapping vector corresponding to the target feature vector in the feature space.

[0250] Based on attention weight information, the relevant feature vectors of the relevant image regions are linearly transformed and mapped to the feature space to obtain the corresponding second mapping vector in the feature space.

[0251] Calculate the distance between the first mapping vector and the second mapping vector. The distance between the first mapping vector and the second mapping vector can be Manhattan distance, Euclidean distance, Chebyshev distance, cosine similarity, or Hamming distance, etc. Calculate the content similarity between the target image region and the associated image region based on the distance between the first mapping vector and the second mapping vector. For example, normalize the distance to convert it into data between 0 and 1 to obtain the content similarity between the target image region and the associated image region.

[0252] For each image region in the text image to be identified, perform the above operation to obtain the content similarity between each image region and the associated region.

[0253] In one embodiment, the computation subunit may specifically be used for:

[0254] Based on the distance between the first query vector corresponding to the target image region and the second key vector corresponding to the associated image region, the content similarity between the target image region and the associated image region is calculated to obtain the content similarity of each image region.

[0255] The fusion unit 304 may include a weighted subunit and a fusion subunit, specifically:

[0256] Weighted sub-unit: For each image region, the content feature vector corresponding to the associated image region is weighted according to the content similarity between the image region and the associated image region to obtain the weighted content feature vector corresponding to the associated image region.

[0257] Fusion subunit: For each image region, it performs fusion processing based on the feature vector of the image region and the weighted content feature vector of the associated image region to obtain attention feature information with contextual information corresponding to the image to be recognized.

[0258] For example, the weight information can include a first weight matrix, a second weight matrix, and a third weight matrix. The target feature vector is mapped based on the first weight matrix to obtain the first query vector, denoted as Query1, or Q1 for short. The target feature vector is mapped based on the second weight matrix to obtain the first key vector, denoted as Key1, or K1 for short. The target feature vector is mapped based on the third weight matrix to obtain the first content vector, denoted as Value1, or V1 for short.

[0259] By performing the same mapping process on the associated image regions, we can obtain Q12, K2, and V2 corresponding to each associated image region.

[0260] Calculate the distance between the first query vector corresponding to the target image region and the second key vector corresponding to the associated image region. For example, the first query vector and the second key vector can be multiplied by a dot product, such as Q·K. Normalize the result of the dot product to obtain the content similarity between the target image region and the associated image region, and obtain the content similarity of each image region.

[0261] The same processing is performed on each image region in the text image to be identified, and the content similarity between each image region and its corresponding associated region is obtained.

[0262] For each image region, the content similarity between the feature vector of the image region and the feature vector of the associated image region is used as the weight of the associated image region. The content similarity is multiplied by the content feature vector of the associated image region to obtain the weighted content feature vector of each associated image region.

[0263] The feature vectors of the image regions are fused with the weighted content feature vectors of the associated image regions. For example, the feature vectors of the image regions are added to the weighted content feature vectors of the associated image regions to obtain the attention feature information corresponding to the attention context information of each image region.

[0264] By performing the same processing on each image region, attention feature information corresponding to the attention context information of the text image to be recognized can be obtained.

[0265] In one embodiment, the calculation subunit may include an initial content similarity calculation module, an acquisition module, and an adjustment module, specifically:

[0266] Initial content similarity calculation module: It is used to calculate the initial content similarity between the target image region and the associated image region based on the distance between the first query vector corresponding to the target image region and the second key vector corresponding to the associated image region, so as to obtain the initial content similarity of each image region;

[0267] The module obtains the initial content similarity matrix of the image to be identified based on the initial content similarity of each image region.

[0268] Adjustment module: Used to adjust the initial content similarity matrix and its transpose for each image region to obtain the content similarity between the target image region and the associated image region.

[0269] For example, specifically, the first query vector of the target image region and the second key vector of the associated image region can be multiplied to obtain the initial content similarity between the image region and the associated image region. Based on the initial content similarity between each image region and the associated image region, an initial content similarity matrix for the text image to be recognized can be obtained, denoted as SCORE0. The element 'score' in the i-th row and i-th column of the initial content similarity matrix is... ij The element "score" located in the j-th row and i-th column can represent the initial content similarity between image region i and image region j. ji This can represent the initial content similarity between image region j and image region i.

[0270] Interchange the rows and columns of the initial content similarity matrix to obtain the transpose matrix SCORE. T Add the transpose matrix to the initial content similarity matrix to obtain the content similarity matrix SCORE = SCORE0 + SCORE T The content similarity matrix is ​​a symmetric matrix, and the elements of the content similarity matrix are scores. ij =score ji .

[0271] The content similarity matrix can be used to determine the content similarity between the feature vector of each image region and the feature vector of the associated region. For example, the content similarity between image region i and image region j is the score of the content similarity matrix SCORE. ij .

[0272] During the training of the initial text content recognition model, the transpose matrix is ​​used to adjust the initial content similarity, resulting in a symmetric content similarity matrix. This can accelerate the convergence of the initial text content recognition model and speed up the training process.

[0273] In one embodiment, a window matrix can be set on the content similarity matrix. The window matrix can mask other image regions that are unrelated to the image region. That is, the adjustment module may include a specific first sub-module, a second sub-module, and a determination sub-module.

[0274] The first submodule is used to add the initial content similarity matrix and its transpose to obtain the first content similarity matrix.

[0275] The second submodule is used to set a corresponding window matrix for the first content similarity matrix for each image region, so as to obtain the second content similarity matrix.

[0276] Determine the submodule: Used to determine the content similarity between an image region and its associated image regions based on the second content similarity matrix.

[0277] For example, the rows and columns of the initial content similarity matrix can be interchanged to obtain the transpose matrix. The transpose matrix is ​​then added to the initial content similarity matrix to obtain the first content similarity matrix. For each image region, a corresponding window matrix is ​​set for the obtained content similarity matrix to retain the content similarity of the image region within the area indicated by the window matrix, while masking the content similarity of other locations.

[0278] The window matrix can be set to 0 for the window position and -∞ or other very large negative numbers, such as 10, for other positions. -16 The content similarity matrix is ​​then added to the window matrix. This causes the content similarity at non-window locations to be set to a large negative number due to the addition of a large negative number. After normalization, the content similarity at non-window locations is mapped to 0, resulting in a second content similarity matrix. Based on this second content similarity matrix, the content similarity between image regions and associated image regions can be determined.

[0279] In one embodiment, the computing unit 303 may specifically be used for:

[0280] For each image region, a multi-head attention mechanism is used to perform parallel calculations based on the feature information of the image region and the feature information of the associated image regions, so as to obtain the content similarity between the image region and the associated image region under each attention mechanism.

[0281] The fusion unit 304 may include a first fusion subunit and a second fusion subunit, specifically:

[0282] The first fusion subunit is used to fuse the feature information of the image region and the associated image region based on the content similarity between the image region and the associated image region according to the multi-head attention mechanism for each image region, so as to obtain the attention feature information of the attention context under each attention mechanism.

[0283] The second fusion subunit is used to fuse the attention feature information under each attention mechanism to obtain the attention feature information of the attention context information.

[0284] The specific operations performed by each attention mechanism can be found in the description of the corresponding location in the embodiment, and will not be repeated here.

[0285] Each attention mechanism can output the content similarity between the corresponding image region and the associated image region for each image region.

[0286] Each attention mechanism calculates the content similarity between the corresponding output image region and the associated image region, and fuses the feature information of the image region and the associated image region (the specific implementation process can be referred to the description of the corresponding position in the embodiment, which will not be repeated here) to obtain the attention feature information under each attention mechanism.

[0287] The obtained attention feature information is concatenated to obtain concatenated feature information. This concatenated feature information is then processed to obtain feature information of the same dimension as the attention feature information under each attention mechanism, thus obtaining the attention feature information that focuses on the context information.

[0288] In one embodiment, the calculation unit 303 may include a mechanism determination subunit and a calculation subunit, specifically:

[0289] Mechanism Determination Subunit: Used to determine the target layer attention mechanism currently in use from the multi-layer attention mechanisms, and to determine the feature information of each image region in the text image to be recognized as the target input feature information of the target attention mechanism;

[0290] Computation subunit: For each image region, it calculates the content similarity between the image region and the associated image region based on the feature information of the image region and the feature information of the associated image region through the target layer attention mechanism.

[0291] The fusion unit 304 may include a third fusion subunit, an iteration subunit, and an output subunit, specifically:

[0292] The third fusion subunit is used to fuse the feature information of the image region and the associated image region based on the content similarity between the image region and the associated image region through the target layer attention mechanism, so as to obtain the processed attention feature information of the attention context.

[0293] Iterative subunit: When the target layer attention mechanism is not the predetermined layer attention mechanism, it updates the target attention mechanism with the associated layer attention mechanism of the target layer attention mechanism in the multi-layer attention mechanism, and updates the target input feature information with the processed attention feature information, and returns to execute for each image region, and calculates the content similarity between the image region and the associated image region through the target layer attention mechanism based on the feature information of the image region and the feature information of the associated image region.

[0294] Output sub-unit: When the target layer attention mechanism is a predetermined layer attention mechanism, it outputs the processed attention feature information to obtain the attention feature information of the text image to be recognized.

[0295] For example, the target layer attention mechanism may include at least one attention mechanism. Each attention mechanism may calculate the content similarity between the image region and the associated image region for each image region based on the feature information of the image region and the feature information of the associated image region. The specific implementation process can be referred to the description of the corresponding position in the embodiment, and will not be repeated here.

[0296] The target layer attention mechanism fuses the feature information of the image region and the associated image region based on the content similarity between the image region and the associated image region to obtain the attention feature information after the attention context information is processed. The specific implementation process can be referred to the description of the corresponding position in the embodiment, and will not be repeated here.

[0297] When the target layer attention mechanism is not the predetermined layer attention mechanism, the processed attention feature information output by the target layer attention mechanism is used as the target input feature information of the next layer attention mechanism, and the next layer attention mechanism is used as the target layer attention mechanism. The same operation is performed, that is, the target feature information is processed through each layer attention mechanism in the multi-layer attention mechanism in sequence, and the processed attention feature information output by the previous layer attention mechanism is used as the target input feature information of its next layer attention mechanism, until the target layer attention mechanism is the predetermined layer attention mechanism, and the processed attention feature information is output to obtain the attention feature information.

[0298] Fusion unit 304: For each image region, based on the content similarity between the image region and the associated image region, the feature information of the image region and the associated image region are fused to obtain attention feature information that focuses on context information.

[0299] For example, specifically, for each image region, the similarity between the content of the image region and the associated image region can be used as the weight of the associated image region. The image region and the associated image region can be weighted and summed to obtain the attention feature information of the attention context information corresponding to each image region.

[0300] Attention feature information of the text image to be recognized can be obtained by using the attention feature information of the attention context information corresponding to each image region.

[0301] Recognition unit 305: Based on attention feature information, it performs text content recognition on the image to be recognized and obtains the recognition result.

[0302] For example, text content recognition can be performed on the image to be recognized based on attention feature information. For each image region in the text image to be recognized, the predicted probability of the image region being each character in a preset dictionary is calculated. For example, if the preset dictionary includes four characters A, B, C, and D, and the text image to be recognized contains five image regions, the probability of image region 1 being A is 0.8, the probability of image region 1 being B is 0.05, the probability of image region 1 being C is 0.05, and the probability of image region 1 being D is 0.1. The same applies to the other image regions of the text image to be recognized.

[0303] The recognition result can be obtained based on the probability of each character for each image region. For example, the character with the highest probability is determined as the recognition result of that image region. For example, the recognition result of image region 1 can be A.

[0304] As can be seen from the above, the text recognition device in this embodiment can acquire a text image to be recognized by the acquisition unit 301, which includes at least two image regions; the extraction unit 302 extracts features from the text image to be recognized to obtain feature information of each image region in the text image to be recognized; the calculation unit 303 calculates the content similarity between the image region and the associated image region for each image region based on the feature information of the image region and the feature information of the associated image region; the fusion unit 304 fuses the feature information of the image region and the associated image region for each image region based on the content similarity between the image region and the associated image region to obtain attention feature information that focuses on contextual information; finally, the recognition unit 305 performs text content recognition on the image to be recognized based on the attention feature information to obtain the recognition result. This scheme achieves parallel recognition of image regions in the image to be recognized by fusing the feature information of each image region and the corresponding associated image region to obtain attention feature information that focuses on contextual information for each image region, and performs text recognition on each image region based on the attention feature information that focuses on contextual information, thereby improving the recognition speed of the image to be recognized.

[0305] This application also provides a computer device, which can be a terminal or a server, such as... Figure 12 As shown, it illustrates a structural schematic diagram of the computer device involved in the embodiments of this application, specifically:

[0306] The computer device may include components such as a processor 1001 with one or more processing cores, a memory 1002 with one or more computer-readable storage media (also referred to as storage media), a power supply 1003, and an input unit 1004. Those skilled in the art will understand that... Figure 12 The computer device structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0307] The processor 1001 is the control center of the computer device. It connects various parts of the computer device via various interfaces and lines, and performs various functions and processes data by running or executing software programs and / or modules stored in the memory 1002, and by calling data stored in the memory 1002, thereby providing overall control of the computer device. Optionally, the processor 1001 may include one or more processing cores; preferably, the processor 1001 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and computer programs, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 1001.

[0308] The memory 1002 can be used to store software programs and modules. The processor 1001 executes various functional applications and data processing by running the software programs and modules stored in the memory 1002. The memory 1002 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, computer programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 1002 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 1002 may also include a memory controller to provide the processor 1001 with access to the memory 1002.

[0309] The computer equipment also includes a power supply 1003 that supplies power to the various components. Preferably, the power supply 1003 can be logically connected to the processor 1001 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 1003 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0310] The computer device may also include an input unit 1004, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0311] Although not shown, the computer device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 1001 in the computer device loads the executable files corresponding to the processes of one or more computer programs into the memory 1002 according to the following instructions, and the processor 1001 runs the computer programs stored in the memory 1002 to realize various functions, as follows:

[0312] Acquire a text image to be recognized, wherein the text image to be recognized includes at least two image regions;

[0313] Feature extraction is performed on the text image to be identified to obtain feature information of each image region in the text image to be identified;

[0314] For each image region, the content similarity between the image region and the associated image region is calculated based on the feature information of the image region and the feature information of the associated image region.

[0315] For each image region, the feature information of the image region and the associated image region is fused based on the content similarity between the image region and the associated image region to obtain attention feature information that focuses on context information;

[0316] Based on the attention feature information, text content recognition is performed on the image to be recognized to obtain the recognition result.

[0317] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0318] As described above, the computer device in this embodiment can acquire a text image to be recognized, which includes at least two image regions; perform feature extraction on the text image to be recognized to obtain feature information of each image region in the text image to be recognized; for each image region, calculate the content similarity between the image region and the associated image region based on the feature information of the image region and the feature information of the associated image region; for each image region, fuse the feature information of the image region and the associated image region based on the content similarity between the image region and the associated image region to obtain attention feature information that focuses on contextual information; and perform text content recognition on the image to be recognized based on the attention feature information to obtain the recognition result. This scheme achieves parallel recognition of image regions in the image to be recognized by fusing the feature information of each image region and the corresponding associated image region to obtain attention feature information that focuses on contextual information for each image region, and performs text recognition on each image region based on the attention feature information that focuses on contextual information, thereby improving the recognition speed of the image to be recognized.

[0319] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations of the above embodiments.

[0320] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by a computer program, or by a computer program controlling related hardware. The computer program can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0321] Therefore, embodiments of this application provide a storage medium storing a computer program that can be loaded by a processor to execute any of the text recognition methods provided in embodiments of this application.

[0322] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0323] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0324] Since the computer program stored in the storage medium can execute any of the text recognition methods provided in the embodiments of this application, it can achieve the beneficial effects that any of the text recognition methods provided in the embodiments of this application can achieve, as detailed in the preceding embodiments, and will not be repeated here.

[0325] The foregoing has provided a detailed description of a text recognition method, apparatus, computer device, and storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A text recognition method, characterized by, The method comprises: obtaining a to-be-recognized text image, the to-be-recognized text image comprising at least two image regions; extracting features of the to-be-recognized text image by using a trained text content recognition model to obtain feature information of each image region of the to-be-recognized text image; for each image region, calculating an initial content similarity between the image region and an associated image region according to the feature information of the image region and the feature information of the associated image region; obtaining an initial content similarity matrix according to the initial content similarity of each image region, and adjusting the initial content similarity matrix based on a window matrix to obtain a second content similarity matrix; wherein the window matrix is used to retain the content similarity within the window position and mask the content similarity outside the window position; determining the content similarity between the image region and the associated image region according to the second content similarity matrix; for each image region, performing fusion processing on the feature information of the image region and the associated image region according to the content similarity between the image region and the associated image region to obtain attention feature information of the context of interest; performing text content recognition on the to-be-recognized text image based on the attention feature information to obtain a recognition result; wherein the text content recognition uses a preset path jump rule for decoding, and the path jump rule allows the same character to directly jump to establish a character path.

2. The method of claim 1, wherein, The feature information comprises a feature vector, and the calculation of the initial content similarity between the image region and the associated image region according to the feature information of the image region and the feature information of the associated image region comprises: determining a target image region to be processed and an associated image region of the target image region; performing spatial mapping processing on a target feature vector of the target image region based on attention weight information to obtain a first mapping vector corresponding to the target feature vector of the target image region in a feature space; performing spatial mapping processing on an associated feature vector of the associated image region based on attention weight information to obtain a second mapping vector corresponding to the associated feature vector of the associated image region in the feature space; calculating the content similarity between the target image region and the associated image region according to the distance between the first mapping vector and the second mapping vector to obtain the initial content similarity of each image region.

3. The method of claim 2, wherein, The first mapping vector comprises a first query vector, a first key vector, and a first content vector, the second mapping vector comprises a second query vector, a second key vector, and a second content vector, and the calculation of the content similarity between the target image region and the associated image region according to the distance between the first mapping vector and the second mapping vector to obtain the initial content similarity of each image region comprises: According to the distance between the first query vector corresponding to the target image region and the second key vector corresponding to the associated image region, the content similarity between the target image region and the associated image region is calculated, and the initial content similarity of each image region is obtained. According to the content similarity between the image region and the associated image region, the feature information of the image region and the associated image region is fused to obtain the attention feature information of the attention context information. According to the content similarity between the image region and the associated image region, the feature information of the image region and the associated image region is fused to obtain the attention feature information of the attention context information. According to the content similarity between the image region and the associated image region, the feature information of the image region and the associated image region is fused to obtain the attention feature information of the attention context information.

4. The method of claim 3, wherein, According to the distance between the first query vector corresponding to the target image region and the second key vector corresponding to the associated image region, the content similarity between the target image region and the associated image region is calculated, and the initial content similarity of each image region is obtained. According to the distance between the first query vector corresponding to the target image region and the second key vector corresponding to the associated image region, the content similarity between the target image region and the associated image region is calculated, and the initial content similarity of each image region is obtained.

5. The method of claim 4, wherein, According to the initial content similarity of each image region, an initial content similarity matrix is obtained, and the initial content similarity matrix is adjusted based on a window matrix to obtain a second content similarity matrix. The initial content similarity matrix and the transpose matrix of the initial content similarity matrix are added to obtain a first content similarity matrix. For each image region, a corresponding window matrix is set for the first content similarity matrix to obtain a second content similarity matrix.

6. The method of claim 1, wherein, According to the feature information of the image region and the feature information of the associated image region, the content similarity between the image region and the associated image region is calculated. According to the feature information of the image region and the feature information of the associated image region, the content similarity between the image region and the associated image region is calculated based on a multi-head attention mechanism in parallel to obtain the content similarity between the image region and the associated image region under each attention mechanism. According to the content similarity between the image region and the associated image region, the feature information of the image region and the associated image region is fused to obtain the attention feature information of the attention context information. The feature information of the image region and the associated image region is fused according to the content similarity between the image region and the associated image region based on the multi-head attention mechanism, to obtain attention feature information of the attention context information under each attention mechanism; The attention feature information under each attention mechanism is fused to obtain the attention feature information of the attention context information.

7. The method of claim 1, wherein, The content similarity between the image region and the associated image region is calculated according to the feature information of the image region and the feature information of the associated image region for each image region, including: The target layer attention mechanism currently used is determined from the multi-layer attention mechanism, and the feature information of each image region in the text image to be recognized is determined as the target input feature information of the target attention mechanism; The content similarity between the image region and the associated image region is calculated according to the feature information of the image region and the feature information of the associated image region for each image region through the target layer attention mechanism; The feature information of the image region and the associated image region is fused according to the content similarity between the image region and the associated image region based on the multi-head attention mechanism, to obtain attention feature information of the attention context information under each attention mechanism; The feature information of the image region and the associated image region is fused according to the content similarity between the image region and the associated image region based on the multi-head attention mechanism, to obtain attention feature information of the attention context information under each attention mechanism; When the target layer attention mechanism is not the predetermined layer attention mechanism, the associated layer attention mechanism of the target layer attention mechanism in the multi-layer attention mechanism is updated to the target layer attention mechanism, and the processed attention feature information is updated to the target input feature information, and the execution of the calculation of the content similarity between the image region and the associated image region according to the feature information of the image region and the feature information of the associated image region for each image region through the target layer attention mechanism is returned. When the target layer attention mechanism is the predetermined layer attention mechanism, the processed attention feature information is output to obtain the attention feature information of the text image to be recognized.

8. The method of claim 1, wherein, Before the text image to be recognized is obtained, the method further includes: Obtaining a text image sample, the text image sample including at least two image regions; Feature extraction is performed on the image regions of the text image sample through an initial text content recognition model to obtain the feature information of each image region of the text image sample; The content similarity between the image region and the associated image region is calculated according to the feature information of the image region and the feature information of the associated image region for each image region; The feature information of the image region and the associated image region is fused according to the content similarity between the image region and the associated image region based on the multi-head attention mechanism, to obtain attention feature information of the attention context information under each attention mechanism; perform text content recognition on the to-be-recognized image based on the attention feature information, to obtain a prediction result; perform training on the initial text content recognition model based on the prediction result and a sample label of the text image sample, to obtain a trained text content recognition model.

9. The method of claim 8, wherein, The prediction result includes characters and a prediction probability corresponding to each character. The training of the initial text content recognition model based on the prediction result and the sample label to obtain a trained text content recognition model includes: determining a corresponding character path set according to the characters in the sample label, the character path set containing at least one character path, the character path being mapped to the sample label through a preset mapping strategy; calculating a path probability of each character path in the character path set based on the prediction probability of each character in the prediction result; calculating an error value between the prediction result and the sample label according to the character path probability; training the initial text content recognition model based on the error value to obtain a trained text content recognition model.

10. The method of claim 9, wherein, The determination of the corresponding character path set according to the characters in the sample label includes: determining a label sequence according to the characters contained in the sample label, the label sequence including a first character, a second character and a gap character; determining a corresponding character path set based on the label sequence, the character path set including a path formed by jumping from the first character to the second character and jumping from the first character to the gap character.

11. The method of claim 1, wherein, The feature extraction of the to-be-recognized text image by the trained text content recognition model to obtain feature information of each image region of the to-be-recognized text image includes: performing convolution processing on the to-be-recognized text image by the trained text content recognition model to obtain a feature map of the to-be-recognized text image; performing feature extraction on the feature map to obtain feature information of each image region in the to-be-recognized text image.

12. A text recognition apparatus characterized by comprising: It includes: an acquisition unit configured to acquire a to-be-recognized text image, the to-be-recognized text image including at least two image regions; an extraction unit configured to perform feature extraction on the to-be-recognized text image by a trained text content recognition model to obtain feature information of each image region of the to-be-recognized text image; a calculation unit configured to, for each image region, calculate an initial content similarity between the image region and an associated image region according to feature information of the image region and feature information of the associated image region; obtain an initial content similarity matrix according to the initial content similarity of each image region, and adjust the initial content similarity matrix based on a window matrix to obtain a second content similarity matrix; wherein the window matrix is used to retain content similarity within a window position and mask content similarity outside the window position; determine a content similarity between an image region and an associated image region according to the second content similarity matrix; and The fusion unit is configured to fuse feature information of the image region and the associated image region according to content similarity between the image region and the associated image region, to obtain attention feature information of the attention context information for each image region. The recognition unit is configured to perform text content recognition on the to-be-recognized text image based on the attention feature information to obtain a recognition result, wherein the text content recognition adopts a preset path jump rule for decoding, and the path jump rule allows the same character to directly jump to establish a character path.

13. A computer device, comprising: The storage medium is configured to store a computer program, and the computer program is loaded by the processor to execute the text recognition method in any one of claims 1 to 11.

14. A storage medium, characterized by The storage medium is configured to store a computer program, and the computer program is loaded by the processor to execute the text recognition method in any one of claims 1 to 11.

Images