Expression recognition method and device, electronic equipment and storage medium

Abstract

The invention provides an expression recognition method and device, electronic equipment and a storage medium. The method comprises the steps that an expression image is acquired; performing symbol recognition on the expression image based on the image features of the expression image to obtain a symbol recognition result of the expression image; performing symbol correction and symbol connectivity prediction on the expression image based on the image features and the symbol recognition result to obtain a symbol correction result and a symbol connectivity relationship of the expression image; and determining an expression corresponding to the expression image based on the symbol correction result and the symbol connectivity relationship. According to the method and device, the electronic equipment and the storage medium provided by the invention, the overall visual characteristics of the expression image can be fully applied, so that the accuracy and reliability of expression recognition are ensured; moreover, symbol correction results and symbol connectivity relationships of all symbols in the expression image can be output at a time, and the non-autoregression form can greatly improve the recognition efficiency of the expression.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an expression recognition method, device, electronic equipment and storage medium. Background Art

[0002] Handwritten Mathematical Expression Recognition (HMER), as a basic task in the field of Optical Character Recognition (OCR) and pattern recognition, plays a wide range of roles in document understanding, teaching, and office automation.

[0003] Current HMER usually uses an encoder-decoder sequence modeling scheme to achieve the task of generating sequences from images. However, since such schemes lack the ability to flexibly utilize overall visual or language context features and need to recognize symbols in images through autoregression, the generated sequences have low reliability and slow generation speed. Summary of the invention

[0004] The present invention provides an expression recognition method, device, electronic device and storage medium, which are used to solve the defects of low reliability and slow speed of handwriting recognition in related technologies.

[0005] The present invention provides an expression recognition method, comprising: Get expression image; Based on the image features of the expression image, perform symbol recognition on the expression image to obtain a symbol recognition result of the expression image; Based on the image features and the symbol recognition results, symbol correction and symbol connectivity prediction are performed on the expression image to obtain a symbol correction result and a symbol connectivity relationship of the expression image; Based on the symbol correction result and the symbol connectivity relationship, an expression corresponding to the expression image is determined.

[0006] According to an expression recognition method provided by the present invention, the symbol recognition of the expression image based on the image features of the expression image includes: Based on a visual perception marker, applying image features of the expression image, performing symbol recognition on the expression image; The visual perception marker is trained by combining the expression recognition model with the reference symbol recognition results obtained by recognizing sample images.

[0007] According to an expression recognition method provided by the present invention, the training step of the visual perception marker comprises: Based on the initial perceptual marker, applying the image features of the sample image, performing symbol recognition on the sample image, and obtaining a test symbol recognition result of the sample image; Based on the expression recognition model, the sample image is decoded step by step to obtain a reference symbol of the sample image and an attention score of the reference symbol; Estimating the position of the reference symbol based on the attention score to obtain the reference symbol recognition result; Based on the difference between the test symbol recognition result and the reference symbol recognition result, the parameters of the initial perception marker are updated to obtain the visual perception marker.

[0008] According to an expression recognition method provided by the present invention, the symbol recognition includes visible symbol recognition and local relationship symbol recognition.

[0009] According to an expression recognition method provided by the present invention, the method of performing symbol correction and symbol connectivity prediction on the expression image based on the image features and the symbol recognition result comprises: Based on the local relationship symbol in the symbol recognition result, a virtual relationship identifier is added to the symbol recognition result to obtain a complete symbol recognition result; Based on the complete symbol recognition result and the image features, symbol correction and symbol connectivity prediction are performed on the expression image.

[0010] According to an expression recognition method provided by the present invention, the symbol correction and symbol connectivity prediction of the expression image based on the complete symbol recognition result and the image feature include: Determining the symbol features of each symbol in the complete symbol recognition result based on the image features; Based on the symbol features of each symbol and the image features, the expression image is symbol corrected, and based on the symbol features of each symbol and the image features, the expression image is respectively symbol-left connectivity predicted and symbol-right connectivity predicted.

[0011] According to an expression recognition method provided by the present invention, determining the expression corresponding to the expression image based on the symbol correction result and the symbol connectivity relationship includes: Based on the symbol correction result and the symbol connectivity relationship, construct a directed acyclic graph; Perform path selection on the directed acyclic graph to obtain an expression corresponding to the expression image.

[0012] The present invention also provides an expression recognition device, comprising: The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the expression recognition methods described above is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the expression recognition method described in any one of the above is implemented.

[0014] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned expression recognition methods.

[0015] The expression recognition method, device, electronic device and storage medium provided by the present invention perform symbol recognition on the expression image based on image features to obtain symbol recognition results, and then perform symbol correction and symbol connectivity prediction on the expression image in combination with the image features and the symbol recognition results, which can fully utilize the overall visual characteristics of the expression image, thereby ensuring the accuracy and reliability of expression recognition; and, based on the symbol recognition results, symbol correction and symbol connectivity prediction are performed, and the symbol correction results and symbol connectivity relationships of all symbols in the expression image can be output at one time. This non-autoregressive form can greatly improve the recognition efficiency of expressions. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a schematic diagram of HMER modeling in the related art.

[0018] Figure 2 This is one of the flow charts of the expression recognition method provided by the present invention.

[0019] Figure 3 It is a schematic diagram of the expression image provided by the present invention.

[0020] Figure 4 It is a schematic diagram of the training process of the visual perception marker provided by the present invention.

[0021] Figure 5 It is a schematic diagram of the flow of symbol correction and symbol connectivity prediction provided by the present invention.

[0022] Figure 6 This is the second flow chart of the expression recognition method provided by the present invention.

[0023] Figure 7 This is the third flow chart of the expression recognition method provided by the present invention.

[0024] Figure 8 It is a structural schematic diagram of the expression recognition device provided by the present invention.

[0025] Fig. 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] Handwritten mathematical expression recognition (HMER) is a basic task in the field of optical character recognition (OCR) and pattern recognition, and plays a wide range of roles in document understanding, teaching, and office automation. With the development of deep learning, sequence modeling, and text recognition, more and more HMER algorithms have been proposed, and their performance has been greatly improved.

[0028] Unlike text recognition and speech recognition, there are complex symbolic relationships in mathematical expressions, which makes the implementation of HMER more challenging. For example, there are three symbolic relationships in a mathematical expression “\frac{x^{y}}{z_{1}}”, namely fractions, exponents, and subscripts.

[0029] In response to the problem of complex symbolic relationships faced by HMER, researchers proposed to convert HMER into an image-to-label generation task. On this basis, many encoder-decoder based sequence modeling methods have been improved and successfully applied to HMER. Specifically, Figure 1 is a schematic diagram of HMER modeling in related technology, such as Figure 1 As shown in the figure, for a given image, we can first use an encoder to extract visual features, and then design an attention-based decoder to recognize LaTeX strings token by token in an autoregressive manner.

[0030] Despite the great improvement in HMER performance, current HMER methods still have some weaknesses. First, current HMER models lack the ability to flexibly exploit holistic visual or linguistic context features. Although bidirectional and covering modeling designs have improved the exploitation of context, current models are still prone to misclassifying ambiguous symbols that humans can easily correctly identify by using visual-linguistic context information. Second, since HMER models need to recognize symbols in an autoregressive manner, these models have slow inference speed.

[0031] In order to improve the utilization rate of the overall visual language information in the HMER scheme and optimize the recognition speed of the HMER scheme, an embodiment of the present invention provides an expression recognition method. Figure 2 It is one of the flow charts of the expression recognition method provided by the present invention, such as Figure 2 As shown, the method includes: Step 210, obtaining an expression image.

[0032] Specifically, the expression image is an image that needs to be subjected to expression recognition, in which the expression to be recognized is drawn. The expression here can be printed or handwritten, and the embodiment of the present invention does not specifically limit this.

[0033] Step 220: Based on the image features of the expression image, perform symbol recognition on the expression image to obtain a symbol recognition result of the expression image.

[0034] Specifically, after acquiring the expression image, feature extraction may be performed on the expression image to obtain image features of the expression image.

[0035] Here, the image feature of the expression image can be a feature used to identify information or attributes of the expression image, and can specifically be in the form of a two-dimensional spatial feature map. The image feature can be obtained by inputting the expression image into a pre-trained encoder, or by performing convolution, downsampling, and other operations on the expression image.

[0036] For example, for expression images, features at two different scales can be extracted, such as and , , ,in, , is the number of channels, , is the height and width of the expression image. and After that, two convolutional layers can be combined to obtain an 8-fold downsampled feature map, thereby effectively capturing smaller symbols in the expression image through higher-resolution features, which can be specifically expressed as the following formula:

[0037] In the formula, Indicates a specification of Conv-BN-Relu block structure. represents a bilinear 2x upsampling layer, Represents a feature concatenation operation. In the formula, That is, it can be used as the image feature of the expression image.

[0038] After obtaining the image features of the expression image, the expression image can be subjected to symbol recognition based on the image features of the expression image. Symbol recognition here is used to recognize the symbols of the expression contained in the expression image. The symbols referred to here can include symbols that are explicitly visible in the expression image, such as Figure 3 The "1", "S", "q", "2x", "+", etc. shown in the expression image may also include symbols that can reflect local relationships in the expression image, such as "\frac" representing a fraction, "\sqrt" representing a square root, and an implicit superscript "^", an implicit subscript "_", which may be Figure 3 The implicit superscript "^" is framed by a red rectangle, and the implicit subscript "_" is framed by a blue rectangle. It can be understood that the superscript "^" can reflect that the symbol is located in the upper right corner, and the subscript "_" can reflect that the symbol is located in the lower right corner.

[0039] Symbol prediction for expression images can be achieved through a pre-trained classifier. For example, the expression image can be input into a pre-trained classification head, thereby obtaining the probability of each type of symbol corresponding to each position in the expression image, and then obtaining the symbol recognition result of the expression image. It can be understood that the symbol recognition result here can include each symbol contained in the expression image and the position of each symbol in the expression image.

[0040] For example, for image features , the image features can be Input into a classification head, and the probability of each type of symbol at each position in the expression image is obtained. , that is, the symbol recognition result is obtained, which can be specifically expressed as the following formula:

[0041] In the formula, , is the total number of all symbols, Indicates all symbols and the case where it does not belong to any symbol.

[0042] Step 230, based on the image features and the symbol recognition results, perform symbol correction and symbol connectivity prediction on the expression image to obtain the symbol correction result and symbol connectivity relationship of the expression image.

[0043] Specifically, after obtaining the symbol recognition result of the expression image, the image features of the expression image and the symbol recognition result of the expression image can be combined to perform symbol correction and symbol connectivity prediction on the expression image. It should be noted that the symbol correction here is to correct the symbol recognition result obtained in step 220, so as to further ensure the accuracy and reliability of symbol recognition; the symbol connectivity prediction is to judge whether the symbols in the symbol recognition result obtained in step 220 are connected, thereby obtaining the positional relationship of each symbol in the expression, and then realizing the orderly arrangement of each symbol in the expression. In step 230, the symbol correction and the symbol connectivity prediction can be independent of each other, and the symbol correction and the symbol connectivity prediction can be performed in parallel or one after the other, and the embodiment of the present invention does not make specific limitations on this.

[0044] Different from the HMER scheme in the related art that recognizes symbols in an autoregressive manner, in the method provided in the embodiment of the present invention, after the symbol recognition result of the expression image has been obtained, the symbol correction and symbol connectivity prediction are respectively performed in combination with the image features and the symbol recognition result, that is, whether it is symbol correction or symbol connectivity prediction, the information of all symbols in the symbol recognition result that can reflect the global context can be obtained from the global perspective of the expression image. Therefore, in the process of symbol correction and symbol connectivity prediction, the overall visual characteristics of the expression image can be fully utilized to ensure the accuracy and reliability of symbol correction and symbol connectivity prediction.

[0045] Furthermore, in the method provided in the embodiment of the present invention, when the symbol recognition result of the expression image has been determined, whether it is symbol correction or symbol connectivity prediction, each symbol in the symbol recognition result can be executed in parallel with other symbols. In this way, through symbol correction and symbol connectivity prediction, the symbol correction results and symbol connectivity relationships of all symbols in the expression image can be output at one time. This non-autoregressive form greatly improves the expression recognition efficiency compared to the autoregressive decoding and symbol-by-symbol decoding form in the related art.

[0046] Among them, based on the image features and the symbol recognition results, the expression image is corrected by inputting the image features and the symbol recognition results into a pre-trained visual language model, and the visual language model corrects the symbol recognition results in combination with the image features; or the image features and the symbol recognition results are input into a pre-trained classification model, and the classification model outputs the result of reclassifying the symbols contained in the expression image in combination with the image features and the symbol recognition results. In an embodiment of the present invention, the result obtained by the symbol correction is recorded as the symbol correction result. It can be understood that the symbol correction result includes all the symbols contained in the expression image.

[0047] Based on the image features and the symbol recognition results, the symbol connectivity prediction of the expression image can be performed by inputting the image features and the symbol recognition results into a pre-trained layout model, and the layout model combines the image features to predict the connectivity of each symbol in the symbol recognition result with other symbols, thereby selecting the symbols arranged on the left and right of each symbol. In an embodiment of the present invention, the result of the symbol connectivity prediction is recorded as the symbol connectivity relationship. It can be understood that the symbol connectivity relationship is used to reflect whether there is a connectivity relationship between every two symbols contained in the expression image, that is, whether there is a left-right arrangement relationship between every two symbols contained in the expression image.

[0048] Step 240, determining the expression corresponding to the expression image based on the symbol correction result and the symbol connectivity relationship.

[0049] Specifically, after obtaining the symbol correction result and the symbol connectivity relationship, the expression contained in the expression image can be determined based on the two. That is, each symbol in the symbol correction result can be used as a symbol in the expression, and then the symbols in the expression are arranged through the symbol connectivity relationship, thereby obtaining the expression corresponding to the expression image.

[0050] The method provided by the embodiment of the present invention performs symbol recognition on the expression image based on image features to obtain symbol recognition results, and then combines the image features and the symbol recognition results to perform symbol correction and symbol connectivity prediction on the expression image, which can fully utilize the overall visual characteristics of the expression image, thereby ensuring the accuracy and reliability of expression recognition; and, based on the symbol recognition results, symbol correction and symbol connectivity prediction are performed, and the symbol correction results and symbol connectivity relationships of all symbols in the expression image can be output at one time. This non-autoregressive form can greatly improve the recognition efficiency of expressions.

[0051] Based on any of the above embodiments, in step 220, based on the image features of the expression image, symbol recognition is performed on the expression image, including: Based on a visual perception marker, applying image features of the expression image, performing symbol recognition on the expression image; The visual perception marker is trained by combining the expression recognition model with the reference symbol recognition results obtained by recognizing sample images.

[0052] Specifically, symbol recognition for expression images can be achieved through a pre-trained visually aware tokenizer. The visually aware tokenizer (VAT) here is a tokenizer that can capture local visual relationships in expressions. It analyzes expression images and segments expressions into a series of visually meaningful tokens, i.e., symbols. These tokens can reflect key information such as operators, numbers, and operation relationships in expression images. In other words, symbol recognition for expression images can be achieved based on the visually aware tokenizer.

[0053] Considering the reliability of symbol recognition, which depends on the reliability of the visual perception marker itself, it is necessary to train the visual perception marker before symbol recognition. When training the visual perception marker, it is necessary to apply a large number of symbol-level instances. Since it is difficult to obtain symbol-level instances, in the embodiment of the present invention, an expression recognition model is combined to assist the visual perception marker.

[0054] Here, the expression recognition model is the HMER model in the related art. Sample images, i.e. expression images as training samples, can be collected, and the expression recognition model is applied to perform expression recognition on the sample images, thereby obtaining each symbol contained in the sample images output by the expression recognition model, i.e., obtaining a reference symbol recognition result.

[0055] On this basis, the expression recognition model can be regarded as a teacher model, the sample images can be used as training samples, and the reference symbol recognition results can be used as labels to train the visual perception marker, thereby ensuring the reliability of the visual perception marker training.

[0056] Based on any of the above embodiments, the training step of the visual perception marker includes: Based on the initial perceptual marker, applying the image features of the sample image, performing symbol recognition on the sample image, and obtaining a test symbol recognition result of the sample image; Based on the expression recognition model, the sample image is decoded step by step to obtain a reference symbol of the sample image and an attention score of the reference symbol; Estimating the position of the reference symbol based on the attention score to obtain the reference symbol recognition result; Based on the difference between the test symbol recognition result and the reference symbol recognition result, the parameters of the initial perception marker are updated to obtain the visual perception marker.

[0057] Specifically, the initial perceptual marker can be understood as a model with the same structure as the visual perceptual marker and initialized parameters, that is, the trained initial perceptual marker is a visual perceptual marker. After obtaining the initial perceptual marker, the sample image can be input into the initial perceptual marker, and the initial perceptual marker extracts image features from the sample image, and then performs symbol recognition on the sample image based on the image features. Thus, the initial perceptual marker can output the symbol recognition result for the sample image, which is recorded as the test symbol recognition result.

[0058] In addition, the sample image can be input into a pre-trained expression recognition model, and the expression recognition model decodes the sample image step by step, thereby obtaining the attention score of each decoding step calculated by the expression recognition model during the decoding process, and the symbol decoded by each decoding step. Here, the symbol decoded by each decoding step is recorded as a reference symbol. In addition, the attention score of each decoding step is used as the attention score corresponding to the reference symbol obtained by decoding. It can be understood that the attention score reflects the degree of association between the output of the reference symbol and the various parts in the sample image as the input of the expression recognition model, that is, among the various parts of the sample image, the part with a higher attention score has a higher association with the reference symbol and is more likely to reflect the position of the reference symbol in the sample image.

[0059] After obtaining the reference symbol output by the expression recognition model and the attention score of the reference symbol, the attention score of the reference symbol can be applied to estimate the position of the parameter symbol in the sample image, thereby obtaining the position of the reference symbol in the sample image, and thus using the reference symbol and the position of the reference symbol in the sample image as the reference symbol recognition result.

[0060] Subsequently, the loss function of the initial perceptual marker can be calculated by comparing the difference between the test symbol recognition result and the reference symbol recognition result, and the parameters of the initial perceptual marker can be updated based on the loss function to realize the training of the visual perceptual marker.

[0061] For example, Figure 4 Schematic diagram of the training process of the visual perception marker provided by the present invention. Figure 4 As shown, visually perceptual markers may include Figure 4 In Conv-BN-Relu block structure, bilinear 2x upsampling layer , Conv-BN-Relu block structure, The Conv convolution layer and the Softmax layer. The Channel Concat in the figure represents channel concatenation. The dimension of Conv can be set to 256 or other values.

[0062] Figure 4 The DWAP (Detection with Weighted Attention Pooling) in the above example represents the expression recognition model, and DWAP is a fixed model with fixed parameters. Figure 4 is recorded as .

[0063] During the training of the visually aware labeler, forward propagation can be performed in teacher-forcing mode. . Where I is a given sample image, and y is a sequence of reference symbols of a given sample image. , we can get each decoding step Attention Score ,in is the number of decoding steps.

[0064] Then, the spatial position of the corresponding reference symbol can be estimated based on the maximum value of the attention score in each decoding step. ,in, , for The Locations, for The Group attention score.

[0065] Then, the probability of each type of symbol existing at each position of the initial perceptual marker based on the sample image I output is , and the spatial position matrix of the reference symbol obtained based on the expression recognition model , perform bipartite matching. In the bipartite matching process, the cost matrix Designed to Each element in Indicates the predicted probability Assign to location The Hungarian algorithm can be used for optimal allocation to find the matching result with the lowest cost and convert the matching result into the final training target. .

[0066] Finally, the loss function of the initial perceptual tagger can be It is expressed as the following formula:

[0067] In the formula, That is cross entropy.

[0068] After training, the visually aware tagger can be used to identify probabilities based on the detections. , select the symbol with the highest score at each position in the expression image and mark the unsigned The situation is filtered out, which can be specifically reflected in the following formula:

[0069] In the formula, That is, it can be used as a symbol recognition result.

[0070] Furthermore, in the process of bipartite graph matching, in order to achieve parallel matching, the given spatial position matrix can be is converted into a 3D matrix similar to the indicator function. Subsequently, a maximum pooling operation can be applied to achieve The prediction probability of the visual perception marker is calculated by matching the local window of The overall distance matrix between the transformed 3D matrix. Finally, the Hungarian algorithm performs the optimal allocation, thereby obtaining the final training target .

[0071] Based on any of the above embodiments, the symbol recognition includes visible symbol recognition and local relationship symbol recognition.

[0072] Among them, visible symbols refer to symbols that are explicitly visible in the expression image, such as Figure 3 The local relationship symbols refer to the symbols that can reflect the local relationship in the expression image, such as "\frac" representing fractions, "\sqrt" representing square roots, and the implicit superscript "^" and implicit subscript "_". Figure 3 The implied superscript “^” is framed by a red rectangle, and the implied subscript “_” is framed by a blue rectangle.

[0073] In step 220, through symbol recognition, both visible symbols in the expression image and local relationship symbols in the expression image can be identified, thereby ensuring the comprehensiveness of symbol recognition and providing conditions for subsequent symbol correction and symbol connectivity relationship prediction based on the symbol recognition results.

[0074] Based on any of the above embodiments, in step 230, performing symbol correction and symbol connectivity prediction on the expression image based on the image features and the symbol recognition result includes: Based on the local relationship symbol in the symbol recognition result, a virtual relationship identifier is added to the symbol recognition result to obtain a complete symbol recognition result; Based on the complete symbol recognition result and the image features, symbol correction and symbol connectivity prediction are performed on the expression image.

[0075] Specifically, in step 220, symbol recognition not only identifies visible symbols in the expression image, but also identifies local relationship symbols in the expression image. The symbol recognition result thus obtained includes visible symbols and local relationship symbols and their respective positions in the expression image.

[0076] Therefore, after obtaining the symbol recognition result, the symbol completion can be performed for the symbol recognition result based on the local relationship symbol in the symbol recognition result, that is, a virtual relationship symbol can be added. Here, the virtual relationship symbol corresponds to the local relationship symbol, and is used to indicate the end of the local relationship symbol. For example, for the local relationship symbol "^", a virtual relationship symbol "}" can be added to indicate the end of the superscript; for the local relationship symbol "\frac", two virtual relationship symbols can be added, which are used to indicate the end of the numerator and the end of the denominator respectively. It can be understood that adding virtual relationship symbols to the symbol recognition result makes the symbol recognition result more comprehensive and complete for the symbol representation in the expression image, thereby recording the symbol recognition result with the addition of virtual relationship symbols as a complete symbol recognition result.

[0077] Furthermore, the correspondence between local relation symbols and virtual relation symbols can be expressed in the following table:

[0078] After obtaining the complete symbol recognition result, the complete symbol recognition result and the image features of the expression image can be combined to perform symbol correction and symbol connectivity prediction on the expression image, thereby obtaining the symbol correction result and symbol connectivity relationship to determine the expression corresponding to the expression image.

[0079] Based on any of the above embodiments, in step 230, performing symbol correction and symbol connectivity prediction on the expression image based on the complete symbol recognition result and the image feature includes: Determining the symbol features of each symbol in the complete symbol recognition result based on the image features; Based on the symbol features of each symbol and the image features, the expression image is symbol corrected, and based on the symbol features of each symbol and the image features, the expression image is respectively symbol-left connectivity predicted and symbol-right connectivity predicted.

[0080] Specifically, in the process of symbol correction and symbol connectivity prediction for the expression image, the symbol features of each symbol contained in the complete symbol recognition result can be first extracted based on the image features of the expression image. Here, the symbol features of each symbol can be expressed as the image features of the area where each symbol is located in the expression image, and / or the image features of the context of the area where each symbol is located in the expression image, that is, the symbol features of each symbol can reflect the image information embodied by each symbol in the expression image locally and / or globally.

[0081] On this basis, the symbol features of each symbol and the image features of the expression image can be combined to perform symbol correction on the expression image; at the same time, the symbol features of each symbol and the image features of the expression image can be combined to predict the left-hand connectivity of the expression image from right to left to obtain the symbols connected on the left side of each symbol, and the right-hand connectivity of the expression image can be predicted from left to right to obtain the symbols connected on the right side of each symbol, thereby realizing the symbol connectivity prediction for the expression image.

[0082] For example, Figure 5 Schematic diagram of the process of symbol correction and symbol connectivity prediction provided by the present invention. Figure 5 As shown, symbol correction and symbol connectivity prediction can be implemented by a Parallel Graph Decoder (PGD).

[0083] exist Figure 5 middle, That is, the symbol recognition result, , are all image features. After being input into the parallel graph decoder, the parallel graph decoder can Add virtual relationship identifiers on the basis of Figure 5 Add Imaginary Tokens, for example, for symbol recognition results In the "\frac", you can add "\frac}1", "\frac}1" to mark the end of the numerator and denominator. For "^", you can add "}" to mark the end of the superscript. For "_", you can add "}" to mark the end of the subscript. In addition, you can add " <sos> ”、" <eos>" mark the beginning and end of an expression respectively.

[0084] For the symbol recognition results after adding virtual relationship identifiers, that is, the complete symbol recognition results, word vector encoding (Word Embedding) can be performed. In addition, the image features can be In the process, the feature sampling (Sample Feature) of each symbol in the complete symbol recognition result is performed. And, the position encoding (2D PositionEncoding) is performed for each symbol. Then, the result of word vector encoding, the result of feature sampling and the result of position encoding are element-wise summed (Elementwise Sum) to obtain the query input of the first layer in the parallel graph decoder. .

[0085] In addition, for image features , can be The Conv-BN-Relu block structure is further encoded, and the result of the further encoding is summed element by element with the result of the position encoding to obtain a new image feature , here Can be used as the question input and key-value input for attention transformation at each layer in the parallel decoder.

[0086] In the parallel graph decoder, the sign correction is implemented by the Self Correction Head. The left connectivity prediction of the sign is implemented by the Left Connectivity Prediction Head, and the right connectivity prediction of the sign is implemented by the Right Connectivity Prediction Head. Here, the Self Correction Head, the Left Connectivity Prediction Head, and the Right Connectivity Prediction Head can all be expressed in the form of an attention layer + a transformer layer. For each of these layers, the decoding process can be expressed as the following formula:

[0087] In the formula, To shift attention, For transformer conversion.

[0088] After decoding the self-node correction head, left connectivity prediction head, and right connectivity prediction head in parallel, the final features of each head can be obtained, namely, , and , J is the total number of layers. For example, the value of J can be 2, that is, a two-layer transformer structure can be set. In addition, the dimension of the transformer structure can be set to 256 or other values.

[0089] After obtaining the final features of each head, the symbol correction result can be calculated and symbolic connectivity .

[0090] For example, for the self-node correction head, the goal of the self-node correction head is to learn an implicit visual language model to correct misclassified symbols, using the input of image features to predict the symbols in the expression image and the approximate position of the symbols. The correction process includes modifying a symbol or deleting a symbol. The acquisition of the symbol correction result can be expressed as:

[0091] In the formula, is a trainable fully connected layer.

[0092] In addition, in order to achieve training for the self-node correction head, the loss function of the self-node correction head can be It is written as follows:

[0093] In the formula, It is the true label of the symbol contained in the sample image obtained based on bipartite graph matching. That is the cross entropy.

[0094] In addition, for the left connectivity prediction head and the right connectivity prediction head, explicit expression layouts can be learned in parallel to dynamically select the left and right symbols for each symbol. For example, for each symbol in the complete symbol recognition result, another symbol can be selected from the complete symbol recognition result as its left or right symbol, so that the parallel graph decoder can use this connectivity as an edge and build the overall directed graph. The selection process of the left or right symbol can be modeled as the query input A focus on the layout characteristics of computation.

[0095] On this basis, the symbolic connectivity can be determined by the following formula :

[0096] In the formula, Represents the score of connectivity between the i-th symbol and the j-th symbol, specifically the right connectivity score from the i-th symbol to the j-th symbol , and the left connectivity score from the jth symbol to the ith symbol sum.

[0097] It is understandable that If the value is greater than a preset threshold, it is considered that the ith symbol and the jth symbol are connected; otherwise, it is considered that the ith symbol and the jth symbol are not connected.

[0098] Correspondingly, the loss functions of the left connectivity prediction head and the right connectivity prediction head are and It can be expressed as follows:

[0099] In the formula, and The true labels of the symbols contained in the sample images are also obtained based on bipartite graph matching.

[0100] On this basis, the loss function applied for training the parallel graph decoder is It can be written as:

[0101] In addition, for the model constructed by the visually perceptual tagger and the parallel graph decoder for expression recognition, its loss function is It can be written as:

[0102] in, Hyperparameters for balancing the training of the visually aware tagger and the parallel graph decoder. For example, It can be set to 0.5, or other values ​​such as 0.4, 0.7, etc.

[0103] Based on any of the above embodiments, in step 240, determining the expression corresponding to the expression image based on the symbol correction result and the symbol connectivity relationship includes: Based on the symbol correction result and the symbol connectivity relationship, construct a directed acyclic graph; Perform path selection on the directed acyclic graph to obtain an expression corresponding to the expression image.

[0104] Specifically, the symbol correction result includes each symbol in the expression image, and the symbol connectivity relationship includes the characteristics of whether each symbol in the expression image is connected. Therefore, a directed acyclic graph (DAG) can be constructed with the symbols included in the symbol correction result as nodes and the connectivity relationships represented by the symbol connectivity relationship as the connecting edges between nodes. It can be understood that a directed acyclic graph is a directed graph that does not contain any rings.

[0105] After obtaining the directed acyclic graph, you can select a path for the directed acyclic graph. For example, you can find the symbol that identifies the beginning of the expression from the directed acyclic graph. <sos>To the symbol that identifies the end of the expression <eos>The longest path of can be realized by a dynamic programming method based on vertex topological sorting, for example. The path thus obtained can be understood as the expression corresponding to the expression image.

[0106] Based on any of the above embodiments, Figure 6 FIG. 2 is a flow chart of the expression recognition method provided by the present invention. Figure 6 As shown, in the expression recognition method provided by the present invention, a two-stage non-autoregressive recognition method is adopted. Among them, in the first stage, all symbols in the expression image can be predicted, including visible symbols and local relationship symbols, and the approximate positions of the above symbols in the expression image can be predicted, thereby providing a rough symbol-level recognition result of the expression image, that is, a symbol recognition result. In the second stage, the symbol recognition result can be corrected, that is, the rough symbol-level recognition result predicted in the first stage is refined. In addition, the connectivity relationship between symbols can also be predicted in parallel. Through the above two stages, a directed acyclic graph can be constructed, and the directed acyclic graph can be easily converted into an expression in LaTeX format.

[0107] That is, in the expression recognition method provided by the embodiment of the present invention, the non-autoregressive design can decode symbols in a non-sequential manner, thereby significantly improving the speed. In addition, the two-stage non-autoregressive design can more effectively utilize global visual and language context features, thereby improving the flexibility of correcting misclassified symbols and ensuring the reliability of reconstructing the expression structure from local to global.

[0108] In addition, in order to demonstrate the effectiveness of the expression recognition method provided by the present invention, test results based on the CROHME 2014, 2016, 2019 datasets and the HME100K dataset are provided in the embodiments of the present invention. On the CROHME dataset, the method provided by the present invention achieved an accuracy of 60.51%, 60.24% and 61.72% respectively without enhancement in CROHME 2014, 2016 and 2019, which is 1.93% higher than the CoMER method (58.57%, 57.89% and 59.71%) on average. In terms of reasoning speed, compared with the CoMER method, the method provided by the present invention improves the decoding time and the overall reasoning time by 13.7 times and 6.70 times respectively.

[0109] Furthermore, in order to evaluate the performance of the expression recognition method provided by the present invention, the expression recognition rate (ExpRate) can be used as an evaluation indicator. ExpRate is the percentage of correctly recognized expressions. And for the cases of ExpRate≤1 and ≤2, it can also be used as a supplementary reference to show that expression recognition can tolerate at most one or two symbol-level errors. In addition, GPU memory usage (MEM) and frames per second (FPS) are also used to measure the inference cost.

[0110] Based on any of the above embodiments, Figure 7 This is the third flow chart of the expression recognition method provided by the present invention. Figure 7 As shown, in the expression recognition method provided by the present invention, an encoder, a visual awareness tokenizer (VAT), and a parallel graph decoder (PGD) are applied. The encoder can adopt a DenseNet architecture.

[0111] In the expression recognition process, firstly given an expression image , the encoder first encodes the expression image into downsampled image features Subsequently, the visual perception tagger VAT predicts all visible symbols and local relationship symbols and their positions in parallel. Next, corresponding virtual relationship identifiers are added to the local relationship symbols, thereby constructing a rough symbol recognition result without considering the relationship between symbols.

[0112] Next, the symbol recognition results and the 2D position encoding of the symbol are input into the parallel graph decoder PGD. PGD consists of two main branches, one for correcting the input symbol recognition results and the other for predicting the directional connection between each symbol. Both branches are designed in a non-autoregressive manner.

[0113] Finally, we can use the symbol correction results and symbol connectivity relations output by PGD to construct a directed acyclic graph, and use the directed acyclic graph to trace the symbol " <sos>"Start to Symbol" <eos>"The path ends with the final recognition result.

[0114] The expression recognition device provided by the present invention is described below. The expression recognition device described below and the expression recognition method described above can be referenced to each other.

[0115] Figure 8 is a schematic diagram of the structure of the expression recognition device provided by the present invention, such as Figure 8 As shown, the device comprises: An image acquisition unit 810, used to acquire an expression image; A symbol recognition unit 820, configured to perform symbol recognition on the expression image based on the image features of the expression image, and obtain a symbol recognition result of the expression image; A correction prediction unit 830 is used to perform symbol correction and symbol connectivity prediction on the expression image based on the image features and the symbol recognition result, so as to obtain the symbol correction result and symbol connectivity relationship of the expression image; The expression determination unit 840 is used to determine the expression corresponding to the expression image based on the symbol correction result and the symbol connectivity relationship.

[0116] The device provided by the embodiment of the present invention performs symbol recognition on the expression image based on image features to obtain symbol recognition results, and then performs symbol correction and symbol connectivity prediction on the expression image in combination with the image features and the symbol recognition results, which can fully utilize the overall visual characteristics of the expression image, thereby ensuring the accuracy and reliability of expression recognition; and, by performing symbol correction and symbol connectivity prediction based on the symbol recognition results, the symbol correction results and symbol connectivity relationships of all symbols in the expression image can be output at one time, and this non-autoregressive form can greatly improve the recognition efficiency of expressions.

[0117] Based on any of the above embodiments, the symbol recognition unit is specifically used for: Based on a visual perception marker, applying image features of the expression image, performing symbol recognition on the expression image; The visual perception marker is trained by combining the expression recognition model with the reference symbol recognition results obtained by recognizing sample images.

[0118] Based on any of the above embodiments, the device further includes a training unit, which is used to: Based on the initial perceptual marker, applying the image features of the sample image, performing symbol recognition on the sample image, and obtaining a test symbol recognition result of the sample image; Based on the expression recognition model, the sample image is decoded step by step to obtain a reference symbol of the sample image and an attention score of the reference symbol; Estimating the position of the reference symbol based on the attention score to obtain the reference symbol recognition result; Based on the difference between the test symbol recognition result and the reference symbol recognition result, the parameters of the initial perception marker are updated to obtain the visual perception marker.

[0119] Based on any of the above embodiments, the symbol recognition includes visible symbol recognition and local relationship symbol recognition.

[0120] Based on any of the above embodiments, the correction prediction unit is specifically used for: Based on the local relationship symbol in the symbol recognition result, a virtual relationship identifier is added to the symbol recognition result to obtain a complete symbol recognition result; Based on the complete symbol recognition result and the image features, symbol correction and symbol connectivity prediction are performed on the expression image.

[0121] Based on any of the above embodiments, the correction prediction unit is specifically used for: Determining the symbol features of each symbol in the complete symbol recognition result based on the image features; Based on the symbol features of each symbol and the image features, the expression image is symbol corrected, and based on the symbol features of each symbol and the image features, the expression image is respectively symbol-left connectivity predicted and symbol-right connectivity predicted.

[0122] Based on any of the above embodiments, the expression determination unit is specifically used for: Based on the symbol correction result and the symbol connectivity relationship, construct a directed acyclic graph; Perform path selection on the directed acyclic graph to obtain an expression corresponding to the expression image.

[0123] Fig. 9 An example of a physical structure diagram of an electronic device is shown in FIG. Fig. 9 As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930 and a communication bus 940, wherein the processor 910, the communication interface 920 and the memory 930 communicate with each other through the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute the expression recognition method, which includes: Get expression image; Based on the image features of the expression image, perform symbol recognition on the expression image to obtain a symbol recognition result of the expression image; Based on the image features and the symbol recognition results, symbol correction and symbol connectivity prediction are performed on the expression image to obtain a symbol correction result and a symbol connectivity relationship of the expression image; Based on the symbol correction result and the symbol connectivity relationship, an expression corresponding to the expression image is determined.

[0124] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0125] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored in a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the expression recognition method provided by the above methods, the method includes: Get expression image; Based on the image features of the expression image, perform symbol recognition on the expression image to obtain a symbol recognition result of the expression image; Based on the image features and the symbol recognition results, symbol correction and symbol connectivity prediction are performed on the expression image to obtain a symbol correction result and a symbol connectivity relationship of the expression image; Based on the symbol correction result and the symbol connectivity relationship, an expression corresponding to the expression image is determined.

[0126] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the expression recognition method provided by the above methods, the method comprising: Get expression image; Based on the image features of the expression image, perform symbol recognition on the expression image to obtain a symbol recognition result of the expression image; Based on the image features and the symbol recognition results, symbol correction and symbol connectivity prediction are performed on the expression image to obtain a symbol correction result and a symbol connectivity relationship of the expression image; Based on the symbol correction result and the symbol connectivity relationship, an expression corresponding to the expression image is determined.

[0127] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0128] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0129] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.< / eos> < / sos> < / eos> < / sos> < / eos> < / sos>

Claims

1. An expression recognition method, characterized in that: include: Get expression image; Based on the image features of the expression image, perform symbol recognition on the expression image to obtain a symbol recognition result of the expression image; Based on the image features and the symbol recognition results, symbol correction and symbol connectivity prediction are performed on the expression image to obtain a symbol correction result and a symbol connectivity relationship of the expression image; Based on the symbol correction result and the symbol connectivity relationship, an expression corresponding to the expression image is determined.

2. The expression recognition method according to claim 1, characterized in that: The performing symbol recognition on the expression image based on the image feature of the expression image comprises: Based on a visual perception marker, applying image features of the expression image, performing symbol recognition on the expression image; The visual perception marker is trained by combining the expression recognition model with the reference symbol recognition results obtained by recognizing sample images.

3. The expression recognition method according to claim 2, characterized in that: The training steps of the visual perception marker include: Based on the initial perceptual marker, applying the image features of the sample image, performing symbol recognition on the sample image, and obtaining a test symbol recognition result of the sample image; Based on the expression recognition model, the sample image is decoded step by step to obtain a reference symbol of the sample image and an attention score of the reference symbol; Estimating the position of the reference symbol based on the attention score to obtain the reference symbol recognition result; Based on the difference between the test symbol recognition result and the reference symbol recognition result, the parameters of the initial perception marker are updated to obtain the visual perception marker.

4. The expression recognition method according to claim 1, characterized in that: The symbol recognition includes visible symbol recognition and local relationship symbol recognition.

5. The expression recognition method according to claim 4, characterized in that: The performing symbol correction and symbol connectivity prediction on the expression image based on the image features and the symbol recognition result comprises: Based on the local relationship symbol in the symbol recognition result, a virtual relationship identifier is added to the symbol recognition result to obtain a complete symbol recognition result; Based on the complete symbol recognition result and the image features, symbol correction and symbol connectivity prediction are performed on the expression image.

6. The expression recognition method according to claim 5, characterized in that: The performing symbol correction and symbol connectivity prediction on the expression image based on the complete symbol recognition result and the image feature comprises: Determining the symbol features of each symbol in the complete symbol recognition result based on the image features; Based on the symbol features of each symbol and the image features, the expression image is symbol corrected, and based on the symbol features of each symbol and the image features, the expression image is respectively symbol-left connectivity predicted and symbol-right connectivity predicted.

7. The expression recognition method according to any one of claims 1 to 6, characterized in that: The step of determining the expression corresponding to the expression image based on the symbol correction result and the symbol connectivity relationship comprises: Based on the symbol correction result and the symbol connectivity relationship, construct a directed acyclic graph; Perform path selection on the directed acyclic graph to obtain an expression corresponding to the expression image.

8. An expression recognition device, characterized in that: include: An image acquisition unit, used for acquiring an expression image; A symbol recognition unit, used for performing symbol recognition on the expression image based on the image features of the expression image to obtain a symbol recognition result of the expression image; A correction prediction unit, used for performing symbol correction and symbol connectivity prediction on the expression image based on the image features and the symbol recognition result, to obtain the symbol correction result and symbol connectivity relationship of the expression image; An expression determination unit is used to determine the expression corresponding to the expression image based on the symbol correction result and the symbol connectivity relationship.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements the expression recognition method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the expression recognition method as described in any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the expression recognition method as described in any one of claims 1 to 7 is implemented.

Images