Mathematical formula recognition method and apparatus, electronic device, and readable storage medium

Abstract

The present disclosure relates to a mathematical formula recognition method and apparatus, an electronic device, and a readable storage medium. The method comprises: acquiring an original image comprising a mathematical formula; inputting the original image into a formula recognition model to obtain a predicted character set outputted by the formula recognition model, wherein the predicted character set comprises character data and structural data; and, according to a preset formula format and the structural data, restoring the position of the character data in the mathematical formula to obtain the mathematical formula in the original image. In the present embodiments, a relative positional relationship between the character data can be represented by means of the provision of the structural data, thereby eliminating the problem of missing characters in a complex formula recognition process, improving the recognition accuracy.

Description

Mathematical formula recognition method, device, electronic device and readable storage medium Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a mathematical formula recognition method, device, electronic device, and readable storage medium. Background Art

[0002] Currently, handwritten formula recognition is typically implemented using an encoding and decoding architecture. For example, an image of a handwritten formula is fed into an encoder, which extracts an image feature map. This image feature map is then fed into a decoder, which recognizes the handwritten formula character by character, ultimately identifying the formula.

[0003] The existing encoding-decoding architecture relies on character-by-character prediction based on LaTeX tags. However, when the formula to be recognized contains complex nested structures such as fractions, radicals, exponents, and logarithms, it is very easy for "{" or "}" to be lost. This is especially true for long formulas, where some characters are easily lost during the recognition process, leading to recognition errors and reduced accuracy.

[0004] Summary of the Invention

[0005] The present disclosure provides a mathematical formula recognition method, device, electronic device and readable storage medium to address the deficiencies of related technologies.

[0006] According to a first aspect of an embodiment of the present disclosure, a mathematical formula recognition method is provided, the method comprising: acquiring an original image containing a mathematical formula; inputting the original image into a formula recognition model to obtain a predicted character set output by the formula recognition model; the predicted character set comprising character data and structural data; restoring the position of the character data in the mathematical formula according to a preset formula format and structural data to obtain the mathematical formula in the original image.

[0007] Optionally, the formula recognition model includes an encoder and a decoder; the encoder is used to obtain an image feature map corresponding to the original image; and the decoder is used to determine a predicted character set corresponding to the original image based on the image feature map.

[0008] Optionally, the encoder includes a DenseNet network or a Transformer network.

[0009] Optionally, the decoder includes a multi-scale counting module, a cyclic decoding module, a feature fusion module and a character prediction module; the multi-scale counting module is used to convert the image feature map into a plurality of count vectors of preset scales; the cyclic decoding module is used to obtain character latent vectors based on the image feature map; the feature fusion module is used to fuse the count vector, the character latent vector and the previous predicted character vector to obtain a target vector; the character prediction module is used to predict character data and structure data respectively based on the target vector.

[0010] Optionally, the cyclic decoding module is further used to obtain a context vector based on the image feature map; the feature fusion module is further used to fuse the count vector, the context vector, the character latent vector and the previous predicted character vector to obtain a target vector.

[0011] Optionally, the feature fusion module performs linear transformation processing on the count vector, the context vector, the character latent vector and the previous predicted character vector respectively to obtain a first linear vector, a second linear vector, a third linear vector and a fourth linear vector; and obtains the sum vector of the first linear vector, the second linear vector, the third linear vector and the fourth linear vector to obtain a target vector.

[0012] Optionally, the multi-scale counting module includes at least two sub-counting modules and a vector averaging sub-module; the at least two sub-counting modules contain convolution kernels of different sizes and are used to output sub-vectors of the same size, and each sub-vector is used to represent the number of times character data is predicted at different scales; the vector averaging sub-module is used to obtain the average vector of at least two sub-vectors to obtain a counting vector.

[0013] Optionally, the multi-scale counting module includes a first sub-counting module and a second sub-counting module; the first sub-counting module is used to identify feature information of a first preset scale in the image feature map to obtain a first sub-vector; the first sub-vector is used to represent the number of character data recognized under the convolution kernel of the first preset scale; the second sub-counting module is used to identify feature information of a second preset scale in the image feature map to obtain a second sub-vector; the second sub-vector is used to represent the number of character data recognized under the convolution kernel of the first preset scale.

[0014] Optionally, the first sub-counting module includes a first convolution unit, a channel unit, a conversion unit and a pooling unit; the first convolution unit is used to convert the image feature map into a first feature map using a convolution kernel of a first preset size; the channel unit is used to assign corresponding weights to different channels of the first feature map to obtain an attention feature map; and the attention feature map and the first feature map are multiplied to obtain a channel feature map; the conversion unit is used to perform scale transformation on the channel feature map to obtain a counting feature map; the pooling unit is used to perform pooling on the counting feature map to obtain a first sub-vector.

[0015] Optionally, the second sub-counting module includes a second convolution unit, a channel unit, a conversion unit and a pooling unit; the second convolution unit is used to convert the image feature map into a second feature map using a convolution kernel of a third preset size; the channel unit is used to assign corresponding weights to different channels of the second feature map to obtain an attention feature map; and the attention feature map and the second feature map are multiplied to obtain a channel feature map; the conversion unit is used to perform scale transformation on the channel feature map to obtain a count feature map; the pooling unit is used to perform pooling on the count feature map to obtain a second sub-vector.

[0016] Optionally, the multi-scale counting module further includes a third sub-counting module, and the third sub-counting module is used to identify feature information of a third preset scale in the image feature map to obtain a third sub-vector.

[0017] Optionally, the third sub-counting module includes a third convolution unit, a channel unit, a conversion unit and a pooling unit; the third convolution unit is used to convert the image feature map into a third feature map using a convolution kernel of a third preset size; the channel unit is used to assign corresponding weights to different channels of the third feature map to obtain an attention feature map; and the attention feature map and the third feature map are multiplied to obtain a channel feature map; the conversion unit is used to perform scale transformation on the channel feature map to obtain a count feature map; the pooling unit is used to perform pooling on the count feature map to obtain a third sub-vector.

[0018] Optionally, the cyclic decoding module includes a gated cyclic unit; the gated cyclic unit is used to generate the character latent vector based on the latent vector and the previous predicted character vector.

[0019] Optionally, the cyclic decoding module includes a first gated cyclic unit and a second gated cyclic unit; the first gated cyclic unit is used to generate a first gated vector based on the latent vector and the previous predicted character vector; the second gated cyclic unit is used to generate the character latent vector based on the first gated vector.

[0020] Optionally, the cyclic decoding module includes an attention submodule; the attention submodule is used to generate a context vector based on the image feature map, the all-zero tensor and the first gating vector; and the second gated cyclic unit is used to generate the character latent vector based on the first gating vector and the context vector.

[0021] Optionally, the attention submodule includes a weight acquisition submodule and a context acquisition submodule; the weight acquisition submodule is used to convolve and linearly process the all-zero tensor to obtain a first eigenvector, and to convolve the image feature map to obtain a second eigenvector, and to linearly process the first gating vector to obtain a third eigenvector; and to superimpose the first eigenvector, the second convolution vector and the third eigenvector to obtain a fourth eigenvector; the context acquisition submodule is used to perform a first activation process, a linear conversion process and a second activation process on the fourth eigenvector respectively to obtain an attention weight; and to multiply the image feature map and the attention weight, and then perform a summation process to obtain a context vector.

[0022] Optionally, the character prediction module includes a character prediction submodule and a structure prediction submodule; the character prediction submodule is used to process the target vector to obtain predicted character data; the structure prediction submodule is used to process the target vector to obtain predicted structure data.

[0023] Optionally, the character prediction submodule includes a linear conversion unit and an activation unit; the linear conversion unit is used to perform linear conversion on the target vector; the activation unit is used to perform activation processing on the linearly converted vector to obtain predicted character data.

[0024] Optionally, the structure prediction submodule includes a linear conversion unit and an activation unit; the linear conversion unit is used to perform linear conversion on the target vector; the activation unit is used to perform activation processing on the linearly converted vector to obtain predicted structure data.

[0025] Optionally, the formula recognition model is trained through the following steps, including: obtaining training sample data; the training sample data includes a data set of multiple mathematical formulas; each data set includes character data and structural data obtained after expansion according to a preset formula format; the structural data is used to represent the relative position relationship of some characters in the mathematical formula; using the training sample data to train the formula recognition model to be trained, and obtaining a predicted character set output by the formula recognition model; the predicted character set includes character data and structural data; when it is determined that the character data meets the preset conditions, the training is stopped to obtain a formula recognition model that has completed training.

[0026] Optionally, determining whether the character data meets a preset condition includes: obtaining the frequency of occurrence of each character data in the predicted character set to obtain the predicted frequency corresponding to each character data; obtaining the difference between the predicted frequency corresponding to each character data and the labeled frequency in the label of the training sample data to obtain the prediction error value corresponding to each character data; obtaining the average value of the prediction error values ​​corresponding to the character data in the predicted character set to obtain the error average value; when it is determined that the error average value is less than or equal to a preset average value threshold, determining that the character data meets the preset condition.

[0027] Optionally, the training sample data is obtained through the following steps, including: obtaining multiple original images containing mathematical formulas; obtaining character data corresponding to each original image; creating structural data according to a preset formula format and character data and arranging the character data to obtain training sample data corresponding to each original image.

[0028] According to a second aspect of an embodiment of the present disclosure, a mathematical formula recognition device is provided, which includes: an original image acquisition module for acquiring an original image containing a mathematical formula; a prediction set acquisition module for inputting the original image into a formula recognition model to obtain a predicted character set output by the formula recognition model; the predicted character set includes character data and structural data; and a mathematical formula acquisition module for restoring the position of the character data in the mathematical formula according to a preset formula format and structural data to obtain the mathematical formula in the original image.

[0029] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a processor; and a memory for storing a computer program executable by the processor; wherein the processor is configured to execute the computer program in the memory to implement the method described in the first aspect.

[0030] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, which, when an executable computer program in the storage medium is executed by a processor, can implement the method described in the first aspect.

[0031] The technical solutions provided by the embodiments of the present disclosure may have the following beneficial effects:

[0032] As can be seen from the above embodiments, the solution provided by the embodiments of the present disclosure can obtain an original image containing a mathematical formula; then, the original image is input into a formula recognition model to obtain a predicted character set output by the formula recognition model; the predicted character set includes character data and structural data; then, the position of the character data in the mathematical formula is restored based on a preset formula format and structural data, thereby obtaining the mathematical formula in the original image. Thus, by setting structural data, the relative positional relationship of the character data can be represented in this embodiment, eliminating the problem of missing characters during complex formula recognition, thereby improving recognition accuracy.

[0033] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0035] Fig. 1 is a flow chart showing a method for recognizing a mathematical formula according to an exemplary embodiment.

[0036] Fig. 2 is a block diagram showing a formula recognition model according to an exemplary embodiment.

[0037] Fig. 3 is a block diagram of a decoder according to an exemplary embodiment.

[0038] Fig. 4 is a block diagram of a multi-scale counting module according to an exemplary embodiment.

[0039] Fig. 5 is a block diagram showing a first sub-counting module according to an exemplary embodiment.

[0040] Fig. 6 is a block diagram showing a cyclic decoding module according to an exemplary embodiment.

[0041] FIG7 is a block diagram of an attention submodule according to an exemplary embodiment.

[0042] Fig. 8 is a block diagram of a character prediction module according to an exemplary embodiment.

[0043] Fig. 9 is a block diagram showing a formula recognition model according to an exemplary embodiment.

[0044] FIG10 is a flowchart showing a training formula recognition model according to an exemplary embodiment.

[0045] FIG11 is a schematic diagram showing the relationship between a mathematical formula, a preset formula format, and a latex tag according to an exemplary embodiment.

[0046] FIG12 is a flowchart showing a training formula recognition model according to an exemplary embodiment.

[0047] Fig. 13 is a block diagram showing a mathematical formula recognition device according to an exemplary embodiment. DETAILED DESCRIPTION

[0048] Exemplary embodiments will be described in detail herein, with examples shown in the accompanying drawings. When the following description refers to the drawings, identical numbers in different drawings represent identical or similar elements, unless otherwise indicated. The exemplary embodiments described below do not represent all embodiments consistent with the present disclosure. Rather, they are merely examples of devices consistent with certain aspects of the present disclosure, as detailed in the appended claims. It should be noted that, unless there is a conflict, the features of the following embodiments and implementations may be combined with each other.

[0049] Considering that in the related art, mathematical formulas with complex structures may lose "{" or "}" during the recognition process, especially when the formula to be recognized is long, some characters may be easily lost during the recognition process, which may cause mathematical formula recognition errors.

[0050] To solve the above technical problems, the present disclosure provides a mathematical formula recognition method, the inventive concept of which includes:

[0051] First, structural information is added to the training sample data. This structural information can represent the relative position relationship of some characters in the mathematical formula, making the mathematical formula clearer and more accurate. When the formula recognition model is trained, it can enable the formula recognition model to more accurately predict the position of each character data in the mathematical formula, thereby improving the accuracy of the formula recognition model.

[0052] Second, a multi-scale counting module is set in the decoder of the formula recognition model. The multi-scale counting module can obtain the frequency of occurrence of character data in the predicted character set. The above frequency can be used to construct a loss function to realize the training process of the supervised formula recognition model, which is beneficial to improving the accuracy of the formula recognition model in recognizing characters, and thus beneficial to improving the accuracy of the overall recognition of mathematical formulas.

[0053] The present disclosure provides a mathematical formula recognition method that can be applied to electronic devices, including but not limited to smartphones, tablet computers, smart displays, electronic whiteboards, etc. FIG1 is a flowchart of a mathematical formula recognition method according to an exemplary embodiment.

[0054] Referring to FIG1 , a mathematical formula recognition method includes steps 11 to 13:

[0055] In step 11, an original image containing a mathematical formula is obtained.

[0056] In this embodiment, the electronic device can obtain an original image containing a mathematical formula. In one example, the electronic device can be provided with an image acquisition module, such as a camera or an image sensor. When there is a need to capture the original image, for example, when there is an object in the preview area of ​​the image acquisition module, for example, the object can be a piece of paper with a mathematical formula written on it, the electronic device can control the image acquisition module to shoot the object, thereby obtaining the original image. In another example, the electronic device can be provided with a communication module, such as a WiFi module, an infrared module, a USB module, etc., and the electronic device can communicate with the opposite communication module of other devices through the above communication module, thereby reading the original image containing the mathematical formula from other devices.

[0057] It should be noted that the electronic device does not determine whether the image contains a mathematical formula when acquiring the image. In the subsequent embodiments, it is assumed that the image contains a mathematical formula for the convenience of description.

[0058] It should also be noted that in the subsequent embodiments, the original image can be a color image (i.e., an RGB image) or a grayscale image. In one example, the original image is implemented as a grayscale image, thereby reducing the data processing load of the formula recognition model and improving recognition efficiency.

[0059] In step 12, the original image is input into a formula recognition model to obtain a predicted character set output by the formula recognition model; the predicted character set includes character data and structure data.

[0060] In this embodiment, a formula recognition model may be stored in the electronic device. The formula recognition model may include an encoder and a decoder. Referring to FIG2 , the encoder 21 is configured to obtain an image feature map corresponding to the original image; the decoder 22 is configured to determine a predicted character set corresponding to the original image based on the image feature map.

[0061] In one example, encoder 21 can be implemented using a DenseNet network or a Transformer network. For example, if encoder 21 is implemented using a DenseNet network, the input data of the DenseNet network is an original image of size H*W*1, and its output data is an image feature map of size H / 16*W / 16*684. Here, H represents the height of the original image, and W represents the width of the original image.

[0062] In one example, the decoder 22 includes a multi-scale counting module, a cyclic decoding module, a feature fusion module, and a character prediction module. Referring to FIG3 , the multi-scale counting module (MSCM) 31 is used to convert the image feature map E(X) into counting vectors (CVs) of multiple preset scales.

[0063] The cyclic decoding module 32 obtains the character latent vector M and context vector Ω corresponding to the image feature map;

[0064] The feature fusion module 33 is used to combine the count vector CV, the character latent vector M and the previous predicted character vector Perform linear transformation to obtain the target vector LB;

[0065] The character prediction module 34 is used to predict character data and structure data according to the target vector LB, respectively. The character data and structure data constitute a predicted character set.

[0066] It should be noted that in one example, the target vector LB is the sum of the first, third, and fourth linear vectors. Specifically, the sum of the values ​​at the same position in each linear vector is used as the value of the sum vector at that position. Thus, after the summation, the target vector has the same dimensions as the individual linear vectors. For example, if the first, second, third, and fourth linear vectors are all 1*512 linear vectors, then the target vector is also a 1*512 linear vector.

[0067] In one example, the multi-scale counting module 31 includes at least two sub-counting modules and a vector averaging sub-module; the at least two sub-counting modules contain convolution kernels of different sizes and are used to output sub-vectors of the same size, and each sub-vector is used to represent the number of times character data is predicted at different scales; the vector averaging sub-module is used to obtain the average vector of at least two sub-vectors to obtain a counting vector.

[0068] In one example, the multi-scale counting module 31 includes a first sub-counting module and a second sub-counting module; the first sub-counting module is used to identify feature information of a first preset scale in the image feature map to obtain a first sub-vector; the second sub-counting module is used to identify feature information of a second preset scale in the image feature map to obtain a second sub-vector.

[0069] In another example, the multi-scale counting module further includes a third sub-counting module, which is configured to identify feature information of a third preset scale in the image feature map to obtain a third sub-vector. In this example, the multi-scale counting module 31 includes a first sub-counting module, a second sub-counting module, a third sub-counting module, and a vector averaging sub-module (Element Average). Referring to FIG4 , the first sub-counting module 41, the second sub-counting module 42, the third sub-counting module 43, and the vector averaging sub-module (Element Average) 44 are shown.

[0070] The first sub-counting module 41 is configured to identify feature information of a first preset scale in the image feature map E(X) to obtain a first sub-vector.

[0071] The second sub-counting module 42 is configured to identify feature information of a second preset scale in the image feature map E(X) to obtain a second sub-vector.

[0072] The third sub-counting module 43 is configured to identify feature information of a third preset scale in the image feature map E(X) to obtain a third sub-vector.

[0073] The vector average submodule (Element Average) 44 is used to obtain the average vector of the first subvector, the second subvector and the third subvector to obtain a counting vector CV (Counting Vector), wherein the counting vector CV is used to represent the number of each character.

[0074] In one embodiment, referring to FIG. 5 , the first sub-counting module 41 includes a first convolution unit 411 (Conv3*3+BN), a channel unit 412 , a conversion unit 413 (Conv1*1+Sigmoid) and a pooling unit 414 (Sum Pooling).

[0075] The first convolution unit 411 is used to convert the image feature map E(X) into a first feature map using a convolution kernel of a first preset size. The first convolution unit also includes a BN (Batch Normalization) layer for converting the image feature map E(X) into a feature map of a preset dimension. In addition, the first convolution unit includes a convolution kernel Conv. In one example, the size of the convolution kernel, i.e., the first preset size, is 3*3*684; and the number of convolution kernels is 512, so that the dimension of the first feature map output by the first convolution unit is H / 16*W / 16*512.

[0076] The channel unit 412 is used to assign corresponding weights to different channels of the first feature map to obtain an attention feature map; and multiply the attention feature map and the first feature map to obtain a channel feature map. In one example, the above-mentioned attention channel may include but is not limited to the following activation functions: ReLU and Sigmoid. Of course, the above-mentioned attention channel may also include functions such as GAP and Linear. Since different functions have different output value ranges, corresponding weights can be assigned to different channels of the first feature map to select the attention value range under different channels. Then, the channel unit 52 can multiply the weight value of each channel with the feature map of a preset dimension to obtain a channel feature map.

[0077] Taking the attention channel including the GAP function as an example, the GAP function can perform average pooling on the first feature map, that is, calculate the average value of the H / 16*W / 16 plane to obtain the value corresponding to the plane; after 512 times, a channel feature map of H / 16*W / 16*512 dimensions can be obtained.

[0078] The conversion unit 413 is used to scale the channel feature map to obtain a counting feature map. The conversion unit 413 is used to convert the channel feature map of different input dimensions into a feature map of another preset dimension, which is subsequently referred to as a counting feature map. In one example, the conversion unit includes a convolution kernel and a Sigmoid activation function. The size of the convolution kernel is 1*1*512, and the number of convolution kernels is K, so that K characters can be recognized, that is, the counting feature map is a 1*K dimensional vector.

[0079] The pooling unit 414 is configured to perform pooling processing on the count feature map to obtain a first sub-vector.

[0080] Continuing to refer to Figure 5, the first convolution unit 411 performs 3*3*684 convolution through the convolution kernel. After 512 convolution kernels, the number of channels of the image feature map can be adjusted from 684 dimensions to 512 dimensions. Then, the channel unit 412 can obtain the weight value of each of the 512 channels, and multiply the weight value of each channel with the first feature map output by the first convolution unit 411 for weighted addition to obtain a channel feature map of H / 16*W / 16*512 dimensions. Afterwards, the conversion unit 413 adjusts the number of channels of the first feature map from 512 dimensions to 208 dimensions through 1*1*208 convolution. It should be noted that the above-mentioned 208 dimensions are the character data of the recognition network, which can be adjusted according to the pattern. At this time, the dimension of the counting feature map is H*W*208. Afterwards, the pooling unit 414 may sum each H*W plane to obtain a first sub-vector of 1*208 dimension, where the pixel value of each of the 208 channels of the first sub-vector corresponds to the number of times the character appears in the current formula.

[0081] Continuing to refer to Figure 5, the second sub-counting module 42 includes a second convolution unit 421, a channel unit 422, a conversion unit 423, and a pooling unit 424. Among them, the second convolution unit 421 is used to convert the image feature map E(X) into a second feature map using a convolution kernel of a second preset size; the channel unit 422 is used to assign corresponding weights to different channels of the second feature map to obtain an attention feature map, and multiply the attention feature map and the second feature map to obtain a channel feature map; the conversion unit 423 is used to scale the channel feature map to obtain a count feature map; and the pooling unit 424 is used to pool the count feature map to obtain a second sub-vector.

[0082] It should be noted that the structure of the second sub-counting module 42 is identical to that of the first sub-counting module 41. The difference lies in the different convolution units. Specifically, the second sub-counting module 42 converts the image feature map to a second preset size rather than the first preset size. In one example, the second convolution unit 421 includes a convolution kernel of size 5*5*684, i.e., the second preset size is 5*5*684; and the number of convolution kernels is 512. As a result, the dimension of the second feature map output by the second convolution unit 421 is H / 16*W / 16*512. Furthermore, the dimension of the second sub-vector is H*W*208.

[0083] Continuing with FIG5 , the third sub-counting module 43 includes a third convolution unit 431, a channel unit 432, a conversion unit 433, and a pooling unit 434. The third convolution unit 431 is configured to convert the image feature map E(X) into a third feature map using a convolution kernel of a third preset size; the channel unit 432 is configured to assign corresponding weights to different channels of the third feature map to obtain an attention feature map; and the attention feature map is multiplied by the third feature map to obtain a channel feature map; the conversion unit 433 is configured to scale the channel feature map to obtain a count feature map; and the pooling unit 434 is configured to perform pooling on the count feature map to obtain a third sub-vector.

[0084] It should be noted that the structure of the third sub-counting module 43 is identical to that of the first sub-counting module 41, differing only in the convolution units used. In one example, the third convolution unit 431 includes a convolution kernel of size 7*7*684, i.e., the third preset size is 7*7*684. In other words, the dimension of the third sub-vector is H*W*208.

[0085] In one embodiment, the cyclic decoding module includes a gated cyclic unit for generating the character latent vector based on the latent vector and the previous predicted character vector. In one example, the cyclic decoding module includes a first gated cyclic unit and a second gated cyclic unit; the first gated cyclic unit is used to generate a first gated vector based on the latent vector and the previous predicted character vector; the second gated cyclic unit is used to generate the character latent vector based on the first gated vector. It should be noted that the number of gated cyclic units can be selected according to the specific scenario. When the recognition accuracy is met, the corresponding solution falls within the scope of protection of this disclosure.

[0086] In another example, the loop decoding module further includes an attention submodule (attention model). Referring to FIG6 , the loop decoding module 32 includes an attention submodule 61, a first gated loop unit 62, and a second gated loop unit 63; the first gated loop unit 62 is used to control the latent vector and the previous predicted character vector Generate the first gating vector The two input latent vectors of the first gated recurrent unit 62 and the previous predicted character vector The dimension is 256. The previous prediction character vector The generated initial value is the starting symbol " <sos>The 256-dimensional vector of " represents that the value after the first time is the result of the previous training process. The hidden vector The initial value is the linear transformation (dimension 1*256) of the average value (dimension 1*684) of each channel pixel of the image feature map E(X) output by the encoder, that is, the 1*256 dimension vector corresponding to the image feature map E(X); and when it is not the first cycle This M is the character latent vector M output by the second gated recurrent unit 63.

[0087] It should be noted that the previous predicted character vector After character prediction based on the character latent vector M, character data can be obtained; then, a table lookup operation is performed based on the character data to find the vector corresponding to the above character data, and the found vector is used as the previous predicted character vector The value of the previous prediction character vector effect.

[0088] Attention submodule 61 is used to calculate the image feature map E(X) and the zero tensor att α (X) and the first gating vector Generate context vector Ω; the input of attention submodule 61 is E(X), whose size is H / 16*W / 16*684. The output of the first gated recurrent unit 62 is Its dimension is 1*256. All-zero tensor att α The initialization dimension of (X) is 1*1*H / 16*W / 16, which is used to record the historical focus area.

[0089] It should be noted that in this example, the all-zero tensor att α (X) is to set the initialization value of the vector to all zeros, so as to facilitate the operation. Of course, you can also set its initialization value to other random numbers and get the corresponding results. α (X) The value of att in non-first loop α (X)=att α (X) + α, where α is the update vector of the attention submodule 61 after each cycle as shown in FIG7 .

[0090] The second gated loop unit 63 is configured to control the first gate vector The character latent vector M is generated by summing up the context vector Ω. The output of the second gated recurrent unit 63 is M, whose dimension is 1*256.

[0091] 7 , the attention submodule 61 includes a weight acquisition submodule 71 and a context acquisition submodule 72 .

[0092] The weight acquisition submodule 71 is used to convert the all-zero tensor att α (X) is convolved and linearly processed (Conv+linear) to obtain the first eigenvector, and the image feature map E(X) is convolved (Conv) to obtain the second eigenvector, and the first gate vector Perform linear processing (linear) to obtain the third eigenvector; and superimpose the first eigenvector, the second convolution vector and the third eigenvector ("+") to obtain the fourth eigenvector. Superposition processing refers to converting the all-zero tensor att α The H / 16*W / 16*512 dimension vector after processing (X) and the image feature map E(X) are added to obtain a H / 16*W / 16*512 dimension sum vector; then, the first gate vector Perform linear processing (linear) to obtain a vector of 1*512 dimension; then, continue to sum the sum vector of H / 16*W / 16*512 dimension and the vector of 1*512 dimension to obtain the fourth eigenvector of H / 16*W / 16*512 dimension.

[0093] The context acquisition submodule 72 is used to perform a first activation process (activation function is tanh), a linear transformation process (linear), and a second activation process (activation function is softmax) on the fourth eigenvector to obtain the attention weight α; and multiply the image feature map E(X) and the attention weight ("X"), and then perform a summation process (sum) to obtain the context vector Ω.

[0094] In this embodiment, the attention submodule 61 adopts the Coverage mechanism to avoid repeated recognition, and the context vector Ω=Σ h Σ w E(X)×α, input att in the next cycle α (X)=att α (X)+α. The first gate vector After linear processing, the number of channels is adjusted to 512 dimensions. α (X) undergoes a convolutional layer (kernel size is 11*11, padding=5, stride is 1), and linear transformation is performed to adjust the channel dimension to 512. The image feature map E(X) undergoes 1*1 convolution to adjust the number of channels to 512. Then, the first, second, and third sub-vectors after the three channel processing are added together, activated by the tanh activation function, and then linearly transformed to reduce the number of channels to 1. Finally, the softmax activation function is used to obtain the attention map weight α.

[0095] As shown in Figure 8 , the character prediction module includes a character prediction submodule 81 and a structure prediction submodule 82. Character prediction submodule 81 processes the target vector to obtain predicted character data, while structure prediction submodule 82 processes the target vector to obtain predicted structure data. The character data and structure data together constitute the predicted character set.

[0096] In one example, the character prediction submodule 81 includes a linear transformation unit (Linear) and an activation unit (sigmoid). The linear transformation unit 811 is used to perform linear transformation on the target vector; the activation unit 812 is used to perform activation processing on the linearly transformed vector to obtain predicted character data.

[0097] In one example, the structure prediction submodule 82 includes a linear transformation unit (Linear) and an activation unit (softmax). The linear transformation unit is used to perform linear transformation on the target vector; the activation unit is used to perform activation processing on the linearly transformed vector to obtain predicted character data.

[0098] 2 to 8 , in one possible embodiment, the structure of the formula recognition model may be as shown in FIG9 .

[0099] 9 , by setting a multi-scale counting module 31 and a cyclic decoding module 32 in the formula recognition model, features of regions of different sizes in the image feature map can be identified, which is beneficial to improving the detection accuracy of characters.

[0100] 10 , the training process of the formula recognition model includes steps 101 to 103 .

[0101] In step 101, training sample data is obtained; the training sample data includes a data set of multiple mathematical formulas; each data set includes character data and structural data obtained after expansion according to a preset formula format; the structural data is used to represent the relative position relationship of some characters in the mathematical formula.

[0102] In this step, the electronic device can obtain training sample data. It is understood that the training sample data includes a data set of multiple mathematical formulas, each of which includes character data and structure data obtained by expanding according to a preset formula format; the structure data is used to represent the relative positional relationship of some characters in the mathematical formula.

[0103] In this step, the mathematical formula includes at least one of the following structures: fraction, subscript, superscript, radical, overline, overarrow, triangle, limit, summation, etc. The symbols for each of these structures are \frac, _, ^, \sqrt, \overline, \overrightarrow, \widehat, \limits, and \sum, respectively. It is understood that the following structures in mathematical formulas, such as log, ln, lg, or exponents, are also applicable to the disclosed solutions.

[0104] Table 1: Mathematical formulas of different structures and their latex representations

[0105] As shown in Table 1, there are seven positional relationships in mathematical formulas, namely inside, right, above, below, left superscript (left_sup, such as square root), and ), subscript (sub), superscript (sup).

[0106] Take a mathematical formula including a fraction as an example, see Figure 11 to illustrate the relationship between the character position in the mathematical formula and the latex tag. Above (above) is character a, below (below) is character b, and to the right (right) is character c; when the preset formula format is \frac{above}{below}right, the above formula It can be written as \frac{a}{b}c.

[0107] By combining the positional relationship of the characters in the mathematical formula and the preset structure shown in FIG11 , training sample data can be formed in this embodiment.

[0108] 1. The mathematical formula is The training sample data is shown in Table 2.

[0109] Table 2: Training sample data

[0110] 2. The mathematical formula is The training sample data is shown in Table 3.

[0111] Table 3: Training sample data

[0112] It should be noted that the " <eos>" is the end character, the last one <eos>Indicates that the entire formula recognition is completed, the middle part <eos>Indicates the end of each structure.

[0113] It should also be noted that "struct" in Table 2 and Table 3 indicates the relative position relationship of the characters contained therein (7 types). If there is a position relationship, fill it in; if there is no position relationship, fill in "None".

[0114] Combining Tables 2 and 3, the first column (from top to bottom) shows the formulas using the latex tag and the added structure "struct," along with their relative positional relationships (seven types). Columns 2 through 7 show the relative positional relationships after the added structure. The entire table constitutes the training sample data. This shows that, by adding structural data to the training sample data, this embodiment allows for more accurate representation of mathematical formulas, facilitating subsequent accurate restoration of the mathematical formulas.

[0115] Based on the above principles, a data set of several mathematical formulas can be obtained to obtain training sample data.

[0116] In one embodiment, several users can handwrite mathematical formulas and take pictures to obtain several original images containing mathematical formulas. The electronic device stores a preset image recognition model, such as a convolutional neural network, and uses the image recognition model to recognize each original image to obtain character data corresponding to each original image. Alternatively, the character data corresponding to each original image can be manually annotated, and the electronic device can obtain the input character data. Finally, the electronic device can create structural data and arrange the character data according to the preset formula format and character data to obtain training sample data corresponding to each original image. This embodiment can not only improve the diversity of mathematical formulas, but also improve the efficiency of obtaining training sample data.

[0117] In step 102, the formula recognition model to be trained is trained using the training sample data to obtain a predicted character set output by the formula recognition model; the predicted character set includes character data and structure data.

[0118] In this step, the electronic device can sequentially input each piece of training sample data into the formula recognition model to be trained, and the formula recognition model can process the training sample data to obtain a predicted character set. It is understandable that the predicted character set includes character data and structure data.

[0119] In this step, the reasoning process of the formula recognition model is shown in Figure 12. Referring to Figure 12, it includes: first, the structure information pushed into the stack is [position information, parent_symbol], and the prediction result pre_string is initialized to '', which is used to store the recognized character data. It is the representation of the latex tag of the mathematical formula in Figure 11. Character by character recognition is performed to obtain the current character current_symbol, and parent_symbol=current_symbol, that is, the value of current_symbol is assigned to parent_symbol. Then, it is determined that the current character current_symbol is not equal to 'struct' and current_symbol is not the end character ' <eos>'. When current_symbol is not 'struct' and ' <eos>', pre_string + = current_symbol, that is, the current character is added to pre_string. Then, it is determined whether the current character current_symbol is 'struct'. When current_symbol is 'struct' or ' <eos>', determine whether the current character current_symbol is 'struct'.

[0120] To determine whether the current character current_symbol is 'struct', when current_symbol is 'struct', predict the structure information and push the structure data into the stack, pop the top element of the stack, and adjust the pre_string based on the top element of the stack and parent_symbol. When current_symbol is not 'struct', determine whether the stack is empty. If the stack is not empty, pop the top element of the stack, and adjust the pre_string based on the top element of the stack and parent_symbol. When the stack is empty, determine whether the number of "{" and "}" in pre_string is equal. If the number of "{" and "}" in pre_string is equal, exit the loop and return the recognition result. If the number of "{" and "}" in pre_string is not equal, add a "}" to pre_string, that is, pre_string + = "}", and the number of "{" and "}" in pre_string is equal, exit the loop and return the recognition result.

[0121] After adjusting pre_string, we can check whether the number of loops exceeds 100. If so, we exit the loop and return the recognition result. Otherwise, we recognize the character and update the current character current_symbol, then we loop again and return the recognition result directly.

[0122] In step 103, when it is determined that the character data meets the preset conditions, the training is stopped to obtain a trained formula recognition model.

[0123] In this step, in order to improve the recognition accuracy of the formula recognition model, a loss function is set in this embodiment. The loss function is constructed based on the frequency of occurrence of each character in the label data of the training sample data and the frequency of occurrence of each character in the prediction data set. In one example, the loss function includes three sub-loss functions, namely: the first sub-loss function is a sub-loss function for comparing the number of characters in the counting feature map with the training sample data, and the sub-loss function is a SmoothL1Loss function; the second sub-loss function is used to judge the structural information prediction result, and the sub-loss function adopts the cross entropy function; the third sub-loss function is used to judge the character information prediction result, and the sub-loss function adopts the cross entropy function.

[0124] In this step, the electronic device can determine whether the character data meets the preset conditions. For example, the electronic device can obtain the frequency of occurrence of each character data in the predicted character set to obtain the predicted frequency corresponding to each character data; then, the electronic device can obtain the difference between the predicted frequency corresponding to each character data and the labeled frequency in the label of the training sample data to obtain the prediction error value corresponding to each character data; thereafter, the electronic device can obtain the average value of the prediction error values ​​corresponding to the character data in the predicted character set to obtain the error average value; when it is determined that the error average value is less than or equal to the preset average value threshold (such as 1-3 times), the electronic device can determine that the character data meets the preset conditions and stop training the formula recognition model, that is, obtain a formula recognition model that has completed training. When it is determined that the above-mentioned error average value is greater than the preset average value threshold, the electronic device can determine that the character data does not meet the preset conditions and can return to step 102 to continue training until the preset conditions are met.

[0125] In step 13, the position of the character data in the mathematical formula is restored according to the preset formula format and structure data to obtain the mathematical formula in the original image.

[0126] In this step, considering that the predicted character set identified in step 12 is embodied in the structure shown in Table 2 and Table 3, the electronic device can restore the position of the character data in the mathematical formula according to the preset formula format and structural data to obtain the mathematical formula in the original image.

[0127] In this step, the comparison of the recognition results of the mathematical formulas obtained using the above formula recognition model is shown in Table 4.

[0128] Table 4 Recognition accuracy before and after using the formula recognition model

[0129] In Table 4, a check mark "√" indicates the use of this feature. As shown in Table 4, using only the counting module improves the accuracy of mathematical formula recognition by 45.49%. Using only structural information improves the accuracy by 50.25%. Using both the counting module and structural information improves the accuracy by 51.95%.

[0130] Thus, the solution provided by the disclosed embodiment can obtain an original image containing a mathematical formula; then, the original image is input into a formula recognition model to obtain a predicted character set output by the formula recognition model; the predicted character set includes character data and structural data; then, the position of the character data in the mathematical formula is restored according to a preset formula format and the structural data, thereby obtaining the mathematical formula in the original image. Thus, by providing structural data, the relative positional relationship of the character data can be represented in this embodiment, eliminating the problem of missing characters during complex formula recognition, thereby improving recognition accuracy. In other words, this embodiment uses a dual prediction approach of structural information and character counts to improve the model's recognition ability for complex formulas with multi-layer nested structures, reducing formula misrecognition caused by missing "{" or "}" characters in multi-layer nesting, and breaking complex formulas into components according to different structures for recognition, thereby improving recognition accuracy. A multi-scale counting module is added to the decoding stage, constructing a counting loss function based on the number of characters in the label, providing weak supervision for the training process, improving the detection rate of characters in each component, and improving recognition accuracy.

[0131] On the basis of a mathematical formula recognition method provided in an embodiment of the present disclosure, this embodiment also provides a mathematical formula recognition device, see Figure 13, the device includes: an original image acquisition module 131, used to acquire an original image containing a mathematical formula; a prediction set acquisition module 132, used to input the original image into a formula recognition model to obtain a predicted character set output by the formula recognition model; the predicted character set includes character data and structural data; a mathematical formula acquisition module 133, used to restore the position of the character data in the mathematical formula according to a preset formula format and structural data, and obtain the mathematical formula in the original image.

[0132] It should be noted that the device embodiment shown in this embodiment matches the content of the above-mentioned method embodiment. You can refer to the content of the above-mentioned method embodiment and will not repeat it here.

[0133] In an exemplary embodiment, an electronic device is further provided, comprising: a display screen; a processor; and a memory for storing a computer program executable by the processor, wherein the processor is configured to execute the computer program in the memory to implement the above method.

[0134] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including an executable computer program. The executable computer program can be executed by a processor to implement the method of the above embodiment. The computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0135] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the disclosure herein. This disclosure is intended to cover any variations, uses, or adaptations that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.< / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / sos>

Claims

1. A mathematical formula recognition method, It is characterized in that The method comprises: Get the original image containing the mathematical formula; Inputting the original image into a formula recognition model to obtain a predicted character set output by the formula recognition model; the predicted character set includes character data and structure data; The position of the character data in the mathematical formula is restored according to the preset formula format and structure data to obtain the mathematical formula in the original image.

2. The method according to claim 1, It is characterized in that The formula recognition model includes an encoder and a decoder; The encoder is used to obtain an image feature map corresponding to the original image; The decoder is used to determine a predicted character set corresponding to the original image according to the image feature map.

3. The method according to claim 2, It is characterized in that The encoder includes a DenseNet network or a Transformer network.

4. The method according to claim 2, It is characterized in that The decoder includes a multi-scale counting module, a cyclic decoding module, a feature fusion module and a character prediction module; The multi-scale counting module is used to convert the image feature map into counting vectors of multiple preset scales; The cyclic decoding module is used to obtain a character latent vector according to the image feature map; The feature fusion module is used to fuse the count vector, the character latent vector and the previous predicted character vector to obtain a target vector; The character prediction module is used to predict character data and structure data respectively according to the target vector.

5. The method according to claim 4, It is characterized in that The circular decoding module is also used to obtain a context vector based on the image feature map; the feature fusion module is also used to fuse the count vector, the context vector, the character latent vector and the previous predicted character vector to obtain a target vector.

6. The method according to claim 5, It is characterized in that The feature fusion module performs linear transformation processing on the count vector, the context vector, the character latent vector and the previous predicted character vector respectively to obtain a first linear vector, a second linear vector, a third linear vector and a fourth linear vector; and obtains the sum vector of the first linear vector, the second linear vector, the third linear vector and the fourth linear vector to obtain a target vector.

7. The method according to claim 4, It is characterized in that The multi-scale counting module includes at least two sub-counting modules and a vector averaging sub-module; the at least two sub-counting modules contain convolution kernels of different sizes and are used to output sub-vectors of the same size, and each sub-vector is used to represent the number of times character data is predicted at different scales; the vector averaging sub-module is used to obtain the average vector of at least two sub-vectors to obtain a counting vector.

8. The method according to claim 7, It is characterized in that The multi-scale counting module includes a first sub-counting module and a second sub-counting module; The first sub-counting module is used to identify feature information of a first preset scale in the image feature map to obtain a first sub-vector; the first sub-vector is used to represent the number of times character data is recognized under the convolution kernel of the first preset scale; The second sub-counting module is used to identify feature information of a second preset scale in the image feature map to obtain a second sub-vector; the second sub-vector is used to represent the number of times character data is recognized under a convolution kernel of a second preset scale.

9. The method according to claim 8, It is characterized in that The first sub-counting module includes a first convolution unit, a channel unit, a conversion unit and a pooling unit; The first convolution unit is used to convert the image feature map into a first feature map using a convolution kernel of a first preset size; The channel unit is used to assign corresponding weights to different channels of the first feature map to obtain an attention feature map; and obtaining a channel feature map by multiplying the attention feature map and the first feature map; The conversion unit is used to perform a scale transformation process on the channel feature map to obtain a count feature map; The pooling unit is used to perform pooling processing on the count feature map to obtain a first sub-vector.

10. The method according to claim 8, It is characterized in that The second sub-counting module includes a second convolution unit, a channel unit, a conversion unit and a pooling unit; The second convolution unit is used to convert the image feature map into a second feature map using a convolution kernel of a second preset size; The channel unit is used to assign corresponding weights to different channels of the second feature map to obtain an attention feature map; and obtaining a channel feature map by multiplying the attention feature map and the second feature map; The conversion unit is used to perform a scale transformation process on the channel feature map to obtain a count feature map; The pooling unit is used to perform pooling processing on the count feature map to obtain a second sub-vector.

11. The method according to claim 8, It is characterized in that The multi-scale counting module further includes a third sub-counting module, and the third sub-counting module is used to identify feature information of a third preset scale in the image feature map to obtain a third sub-vector.

12. The method according to claim 11, It is characterized in that The third sub-counting module includes a third convolution unit, a channel unit, a conversion unit and a pooling unit; The third convolution unit is used to convert the image feature map into a third feature map by using a convolution kernel of a third preset size; The channel unit is used to assign corresponding weights to different channels of the third feature map to obtain an attention feature map; and obtaining a channel feature map by multiplying the attention feature map and the third feature map; The conversion unit is used to perform a scale transformation process on the channel feature map to obtain a count feature map; The pooling unit is used to perform pooling processing on the count feature map to obtain a third sub-vector.

13. The method according to claim 4, It is characterized in that The loop decoding module includes a gated loop unit; The gated recurrent unit is used to generate the character latent vector according to the latent vector and the previous predicted character vector.

14. The method according to claim 13, It is characterized in that The loop decoding module includes a first gated loop unit and a second gated loop unit; The first gated recurrent unit is used to generate a first gated vector according to the latent vector and the previous predicted character vector; The second gated recurrent unit is configured to generate the character latent vector according to the first gating vector.

15. The method according to claim 14, It is characterized in that The loop decoding module also includes an attention submodule; The attention submodule is used to generate a context vector according to the image feature map, the all-zero tensor and the first gating vector; The second gated recurrent unit is configured to generate the character latent vector according to the first gating vector and the context vector.

16. The method according to claim 15, It is characterized in that The attention submodule includes a weight acquisition submodule and a context acquisition submodule; The weight acquisition submodule is used to perform convolution and linear processing on the all-zero tensor to obtain a first feature vector, and to perform convolution processing on the image feature map to obtain a second feature vector, and to perform convolution processing on the first gating vector Linear processing is performed to obtain a third eigenvector; and the first eigenvector, the second convolution vector and the third eigenvector are superimposed to obtain a fourth eigenvector; The context acquisition submodule is used to obtain an attention weight after performing a first activation process, a linear conversion process and a second activation process on the fourth eigenvector respectively; The image feature map and the attention weight are multiplied and then summed to obtain a context vector.

17. The method according to claim 4, It is characterized in that The character prediction module includes a character prediction submodule and a structure prediction submodule; The character prediction submodule is used to process the target vector to obtain predicted character data; The structure prediction submodule is used to process the target vector to obtain predicted structure data.

18. The method according to claim 17, It is characterized in that The character prediction submodule includes a linear conversion unit and an activation unit; The linear transformation unit is used to perform linear transformation on the target vector; The activation unit is used to perform activation processing on the linearly transformed vector to obtain predicted character data.

19. The method according to claim 17, It is characterized in that The structure prediction submodule includes a linear conversion unit and an activation unit; The linear transformation unit is used to perform linear transformation on the target vector; The activation unit is used to perform activation processing on the linearly transformed vector to obtain predicted structural data.

20. The method according to claim 1, It is characterized in that The formula recognition model is trained by the following steps, including: Acquire training sample data; the training sample data includes a data set of multiple mathematical formulas; each data set includes character data and structure data obtained after expansion according to a preset formula format; the structure data is used to represent the relative position relationship of some characters in the mathematical formula; Using the training sample data to train the formula recognition model to be trained, to obtain a predicted character set output by the formula recognition model; the predicted character set includes character data and structure data; When it is determined that the character data meets the preset conditions, the training is stopped to obtain a trained formula recognition model.

21. The method according to claim 20, It is characterized in that Determining that the character data meets a preset condition includes: Obtaining the frequency of occurrence of each character data in the predicted character set to obtain the predicted frequency corresponding to each character data; Obtaining the difference between the predicted frequency corresponding to each character data and the marked frequency in the label of the training sample data, and obtaining the prediction error value corresponding to each character data; Obtaining an average value of prediction error values ​​corresponding to character data in the predicted character set to obtain an error average value; When it is determined that the error average value is less than or equal to the preset average value threshold, it is determined that the character data meets the preset condition.

22. The method according to claim 20, It is characterized in that The training sample data is obtained through the following steps, including: Get multiple original images containing mathematical formulas; Obtain character data corresponding to each original image; Structural data is created according to a preset formula format and character data, and the character data is arranged to obtain training sample data corresponding to each original image.

23. A mathematical formula recognition device, It is characterized in that The device comprises: An original image acquisition module is used to acquire the original image containing the mathematical formula; A prediction set acquisition module, used for inputting the original image into a formula recognition model to obtain a predicted character set output by the formula recognition model; the predicted character set includes character data and structure data; The mathematical formula acquisition module is used to restore the position of the character data in the mathematical formula according to the preset formula format and structure data to obtain the mathematical formula in the original image.

24. An electronic device, It is characterized in that include processor; a memory for storing a computer program executable by the processor; The processor is configured to execute the computer program in the memory to implement the method according to any one of claims 1 to 22.

25. A computer-readable storage medium, It is characterized in that When the executable computer program in the storage medium is executed by a processor, the method according to any one of claims 1 to 22 can be implemented.

Images