Mathematical formula recognition method and device, electronic equipment and readable storage medium

Abstract

The present disclosure relates to a mathematical formula recognition method and device, electronic equipment and readable storage medium. The method comprises: obtaining an original image containing a mathematical formula; inputting the original image into a formula recognition model to obtain a predicted character set output by the formula recognition model; the predicted character set comprises character data and structure data; restoring the position of the character data in the mathematical formula according to a preset formula format and the structure data to obtain the mathematical formula in the original image. In this embodiment, by setting the structure data, the relative position relationship of the character data can be represented, the problem of missing characters in the complex formula recognition process can be eliminated, and the recognition accuracy can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a mathematical formula recognition method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] Currently, handwritten formula recognition typically employs an encoding and decoding architecture. For example, the handwritten formula image is input into an encoder, which extracts image feature maps; then, these feature maps are input into a decoder, which performs character-by-character recognition of the handwritten formula, ultimately resulting in the recognized handwritten formula.

[0003] In existing schemes, the encoding-decoding architecture is based on LaTeX tags for character-by-character prediction. However, when the formula to be recognized contains complex structures such as nested fractions, radicals, exponents, and logarithms, it is easy for "{" or "}" to be lost, especially when the formula to be recognized is long, which can easily lead to the loss of some characters during the recognition process, resulting in recognition errors and reduced recognition accuracy. Summary of the Invention

[0004] This disclosure provides a mathematical formula recognition method, apparatus, electronic device, and readable storage medium to address the shortcomings of related technologies.

[0005] According to a first aspect of the present disclosure, a mathematical formula recognition method is provided, the method comprising:

[0006] Obtain the original image containing mathematical formulas;

[0007] The original image is input into the formula recognition model to obtain the predicted character set output by the formula recognition model; the predicted character set includes character data and structural data.

[0008] The position of the character data in the mathematical formula is restored according to the preset formula format and structural data to obtain the mathematical formula in the original image.

[0009] Optionally, the formula recognition model includes an encoder and a decoder;

[0010] The encoder is used to obtain the image feature map corresponding to the original image;

[0011] The decoder is used to determine the set of predicted characters corresponding to the original image based on the image feature map.

[0012] Optionally, the encoder may include a DenseNet network or a Transformer network.

[0013] Optionally, the decoder includes a multi-scale counting module, a cyclic decoding module, a feature fusion module, and a character prediction module;

[0014] The multi-scale counting module is used to convert the image feature map into counting vectors of multiple preset scales;

[0015] The loop decoding module is used to obtain hidden characters based on the image feature map;

[0016] The feature fusion module is used to fuse the counting vector, the character latent vector, and the previous predicted character vector to obtain the target vector;

[0017] The character prediction module is used to predict character data and structural data respectively based on the target vector.

[0018] Optionally, the loop decoding module is further configured to obtain a context vector based on the image feature map; the feature fusion module is further configured to fuse the counting vector, the context vector, the character latent vector, and the previous predicted character vector to obtain a target vector.

[0019] Optionally, the feature fusion module performs linear transformation processing on the counting vector, the context vector, the character latent vector, and the previous predicted character vector to obtain a first linear vector, a second linear vector, a third linear vector, and a fourth linear vector; and obtains the sum vector of the first linear vector, the second linear vector, the third linear vector, and the fourth linear vector to obtain the target vector.

[0020] Optionally, the multi-scale counting module includes at least two sub-counting modules and a vector averaging sub-module; the at least two sub-counting modules contain convolution kernels of different sizes and are used to output sub-vectors of the same size, each sub-vector representing the number of times the predicted character data is used at different scales; the vector averaging sub-module is used to obtain the average vector of at least two sub-vectors to obtain the counting vector.

[0021] Optionally, the multi-scale counting module includes a first sub-counting module and a second sub-counting module;

[0022] The first sub-counting module is used to identify feature information of a first preset scale in the image feature map to obtain a first sub-vector; the first sub-vector is used to represent the number of character data identified under the convolution kernel of the first preset scale;

[0023] The second sub-counting module is used to identify feature information at a second preset scale in the image feature map to obtain a second sub-vector; the second sub-vector is used to represent the number of character data identified under the convolution kernel at the first preset scale.

[0024] Optionally, the first sub-counting module includes a first convolutional unit, a channel unit, a transformation unit, and a pooling unit;

[0025] The first convolutional unit is used to convert the image feature map into a first feature map using a convolutional kernel of a first preset size;

[0026] The channel unit is used to assign corresponding weights to different channels of the first feature map to obtain an attention feature map; and to multiply the attention feature map and the first feature map to obtain a channel feature map.

[0027] The conversion unit is used to perform a scaling transformation on the channel feature map to obtain a counting feature map;

[0028] The pooling unit is used to perform pooling processing on the counting feature map to obtain a first sub-vector.

[0029] Optionally, the second sub-counting module includes a second convolution unit, a channel unit, a transformation unit, and a pooling unit;

[0030] The second convolutional unit is used to convert the image feature map into a second feature map using a convolutional kernel of a third preset size;

[0031] The channel unit is used to assign corresponding weights to different channels of the second feature map to obtain an attention feature map; and to multiply the attention feature map and the second feature map to obtain a channel feature map.

[0032] The conversion unit is used to perform a scaling transformation on the channel feature map to obtain a counting feature map;

[0033] The pooling unit is used to perform pooling processing on the counting feature map to obtain a second sub-vector. Optionally, the multi-scale counting module further includes a third sub-counting module, which is used to identify feature information at a third preset scale in the image feature map to obtain a third sub-vector.

[0034] Optionally, the third sub-counting module includes a third convolution unit, a channel unit, a transformation unit, and a pooling unit;

[0035] The third convolutional unit is used to convert the image feature map into a third feature map using a convolutional kernel of a third preset size;

[0036] The channel unit is used to assign corresponding weights to different channels of the third feature map to obtain an attention feature map; and to multiply the attention feature map and the third feature map to obtain a channel feature map.

[0037] The conversion unit is used to perform a scaling transformation on the channel feature map to obtain a counting feature map;

[0038] The pooling unit is used to perform pooling processing on the counting feature map to obtain a third sub-vector.

[0039] Optionally, the loop decoding module includes a gated loop unit;

[0040] The gated loop unit is used to generate the character latent vector based on the latent vector and the previous predicted character vector.

[0041] Optionally, the loop decoding module includes a first gated loop unit and a second gated loop unit;

[0042] The first gated loop unit is used to generate a first gated vector based on the latent vector and the previous predicted character vector;

[0043] The second gated loop unit is used to generate the character latent vector based on the first gated vector.

[0044] Optionally, the loop decoding module includes an attention submodule;

[0045] The attention submodule is used to generate a context vector based on the image feature map, the all-zero tensor, and the first gating vector.

[0046] The second gated loop unit is used to generate the character latent vector based on the first gate vector and the context vector.

[0047] Optionally, the attention submodule includes a weight acquisition submodule and a context acquisition submodule;

[0048] The weight acquisition submodule is used to perform convolution and linear processing on the all-zero tensor to obtain a first feature vector, and to perform convolution processing on the image feature map to obtain a second feature vector, and to perform linear processing on the first gate vector to obtain a third feature vector; and to perform superposition processing on the first feature vector, the second convolution vector and the third feature vector to obtain a fourth feature vector.

[0049] The context acquisition submodule is used to perform a first activation process, a linear transformation process, and a second activation process on the fourth feature vector to obtain attention weights; and to multiply the image feature map and the attention weights, and then sum them to obtain a context vector.

[0050] Optionally, the character prediction module includes a character prediction submodule and a structure prediction submodule;

[0051] The character prediction submodule is used to process the target vector to obtain the predicted character data;

[0052] The structure prediction submodule is used to process the target vector to obtain the predicted structure data.

[0053] Optionally, the character prediction submodule includes a linear transformation unit and an activation unit;

[0054] The linear transformation unit is used to perform a linear transformation on the target vector;

[0055] The activation unit is used to activate the linearly transformed vector to obtain the predicted character data.

[0056] Optionally, the structure prediction submodule includes a linear transformation unit and an activation unit;

[0057] The linear transformation unit is used to perform a linear transformation on the target vector;

[0058] The activation unit is used to activate the linearly transformed vector to obtain the predicted structural data.

[0059] Optionally, the formula recognition model is trained through the following steps:

[0060] Acquire training sample data; the training sample data includes a set of data for multiple mathematical formulas; each set of data includes character data and structural data obtained after expanding according to a preset formula format; the structural data is used to represent the relative positional relationship of some characters in the mathematical formula;

[0061] The training sample data is used to train the formula recognition model to be trained, and the predicted character set output by the formula recognition model is obtained; the predicted character set includes character data and structural data.

[0062] Training stops when the character data meets the preset conditions, and the trained formula recognition model is obtained.

[0063] Optionally, determining that the character data meets preset conditions includes:

[0064] Obtain the frequency of occurrence of each character data in the predicted character set, and obtain the prediction frequency corresponding to each character data;

[0065] The difference between the predicted frequency corresponding to each character data and the labeled frequency in the label of the training sample data is obtained to obtain the prediction error value corresponding to each character data.

[0066] The average prediction error value corresponding to the character data in the predicted character set is obtained to obtain the average error value;

[0067] When the average error is determined to be less than or equal to a preset average threshold, the character data is determined to meet the preset condition.

[0068] Optionally, the training sample data is obtained through the following steps:

[0069] Obtain multiple raw images containing mathematical formulas;

[0070] Obtain the character data corresponding to each original image;

[0071] By creating structured data according to a preset formula format and character data, and arranging the character data, training sample data corresponding to each original image is obtained.

[0072] According to a second aspect of the present disclosure, a mathematical formula recognition device is provided, the device comprising:

[0073] The original image acquisition module is used to acquire original images containing mathematical formulas;

[0074] The prediction set acquisition module is used to input the original image into the formula recognition model to obtain the predicted character set output by the formula recognition model; the predicted character set includes character data and structural data.

[0075] The mathematical formula acquisition module is used to restore the position of the character data in the mathematical formula according to the preset formula format and structure data, so as to obtain the mathematical formula in the original image.

[0076] According to a third aspect of the present disclosure, an electronic device is provided, including...

[0077] processor;

[0078] Memory for storing computer programs executable by the processor;

[0079] The processor is configured to execute a computer program in the memory to implement the method described in the first aspect.

[0080] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that, when an executable computer program in the storage medium is executed by a processor, enables the implementation of the method described in the first aspect.

[0081] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0082] As can be seen from the above embodiments, the solution provided in this disclosure can acquire an original image containing mathematical formulas; then, the original image is input into a formula recognition model to obtain a predicted character set output by the formula recognition model; the predicted character set includes character data and structural data; subsequently, the position of the character data in the mathematical formula is restored according to a preset formula format and structural data to obtain the mathematical formula in the original image. Thus, by setting structural data in this embodiment, the relative positional relationship of character data can be characterized, eliminating the problem of missing characters during complex formula recognition and improving recognition accuracy.

[0083] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0084] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0085] Figure 1 This is a flowchart illustrating a mathematical formula recognition method according to an exemplary embodiment.

[0086] Figure 2 This is a block diagram illustrating a formula recognition model according to an exemplary embodiment.

[0087] Figure 3 This is a block diagram illustrating a decoder according to an exemplary embodiment.

[0088] Figure 4 This is a block diagram illustrating a multi-scale counting module according to an exemplary embodiment.

[0089] Figure 5 This is a block diagram illustrating a first sub-counting module according to an exemplary embodiment.

[0090] Figure 6 This is a block diagram illustrating a loop decoding module according to an exemplary embodiment.

[0091] Figure 7 This is a block diagram illustrating an attention submodule according to an exemplary embodiment.

[0092] Figure 8 This is a block diagram illustrating a character prediction module according to an exemplary embodiment.

[0093] Figure 9 This is a block diagram illustrating a formula recognition model according to an exemplary embodiment.

[0094] Figure 10This is a flowchart illustrating a training formula recognition model according to an exemplary embodiment.

[0095] Figure 11 This is a schematic diagram illustrating the relationship between a mathematical formula, a preset formula format, and a LaTeX tag, according to an exemplary embodiment.

[0096] Figure 12 This is a flowchart illustrating a training formula recognition model according to an exemplary embodiment.

[0097] Figure 13 This is a block diagram illustrating a mathematical formula recognition device according to an exemplary embodiment. Detailed Implementation

[0098] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described below by way of example do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatus consistent with some aspects of this disclosure as detailed in the appended claims. It should be noted that, without conflict, the following embodiments and features in the implementation methods can be combined with each other.

[0099] Considering that complex mathematical formulas in related technologies may lose "{" or "}" during the recognition process, especially when the formula to be recognized is long, some characters may be lost during the recognition process, which may lead to problems in the recognition of mathematical formulas.

[0100] To address the aforementioned technical problems, this disclosure provides a method for recognizing mathematical formulas, the inventive concept of which includes:

[0101] First, structural information is added to the training sample data. This structural information can represent the relative positional relationship of some characters in a mathematical formula, making the mathematical formula clearer and more accurate. When training the formula recognition model, it can enable the formula recognition model to more accurately predict the position of each character data in the mathematical formula, thereby improving the accuracy of the formula recognition model.

[0102] Second, the decoder of the formula recognition model is equipped with a multi-scale counting module. This multi-scale counting module can obtain the frequency of character data occurrence in the prediction character set. The loss function can be constructed using the above frequency to supervise the training process of the formula recognition model, which is conducive to improving the accuracy of the formula recognition model in recognizing characters, and thus to improving the overall accuracy of mathematical formula recognition.

[0103] This disclosure provides a mathematical formula recognition method that can be applied to electronic devices, including but not limited to smartphones, tablets, smart displays, or electronic whiteboards. Figure 1 This is a flowchart illustrating a mathematical formula recognition method according to an exemplary embodiment.

[0104] See Figure 1 A mathematical formula recognition method, comprising steps 11 to 13:

[0105] In step 11, the original image containing the mathematical formula is obtained.

[0106] In this embodiment, the electronic device can acquire a raw image containing mathematical formulas. In one example, the electronic device can be equipped with an image acquisition module, such as a camera or image sensor. When there is a need to acquire a raw image, for example, when there is an object in the preview area of ​​the image acquisition module (e.g., a piece of paper with mathematical formulas written on it), the electronic device can control the image acquisition module to photograph the object, thereby obtaining the raw image. In another example, the electronic device can be equipped with a communication module, such as a WiFi module, an infrared module, a USB module, etc. The electronic device can communicate with the peer communication module of other devices through the above communication module, thereby reading the raw image containing mathematical formulas from other devices.

[0107] It should be noted that electronic devices are not certain that images contain mathematical formulas when acquiring them. In subsequent embodiments, it will be assumed that images contain mathematical formulas for ease of description.

[0108] It should also be noted that in subsequent embodiments, the original image can be a color image (i.e., an RGB image) or a grayscale image. In one example, the original image is implemented using a grayscale image, thereby reducing the data processing volume of the formula recognition model and improving recognition efficiency.

[0109] In step 12, the original image is input into the formula recognition model to obtain the predicted character set output by the formula recognition model; the predicted character set includes character data and structural data.

[0110] In this embodiment, the electronic device may store a formula recognition model. This formula recognition model may include an encoder and a decoder. See also... Figure 2 The encoder 21 is used to obtain the image feature map corresponding to the original image; the decoder 22 is used to determine the predicted character set corresponding to the original image based on the image feature map.

[0111] In one example, encoder 21 may be implemented using either a DenseNet network or a Transformer network. Taking encoder 21 implemented using a DenseNet network as an example, the input data of this DenseNet network is an original image of size H*W*1, and its output data is an image feature map of size H / 16*W / 16*684. Here, H represents the height of the original image, and W represents the width of the original image.

[0112] In one example, decoder 22 includes a multi-scale counting module, a cyclic decoding module, a feature fusion module, and a character prediction module. See also Figure 3 The Multi-Scale Counting Module (MSCM) 31 is used to transform the image feature map E(X) into a counting vector (CV) of multiple preset scales.

[0113] The loop decoding module 32 obtains the character latent vector M and the context vector Ω corresponding to the image feature map;

[0114] Feature fusion module 33 is used to integrate the count vector CV, the character latent vector M, and the previously predicted character vector. Perform a linear transformation to obtain the target vector LB;

[0115] The character prediction module 34 is used to predict character data and structural data based on the target vector LB, and the character data and structural data constitute the predicted character set.

[0116] It should be noted that in one example, the target vector LB is the sum of the first, third, and fourth linear vectors. That is, the sum of the number of occurrences of each linear vector at the same position is used as the value of the sum vector at that position. After summation, the target vector has the same dimension as each of the individual linear vectors. For example, if the first, second, third, and fourth linear vectors are all 1*512 dimension linear vectors, then the target vector is also a 1*512 dimension linear vector.

[0117] In one example, the multi-scale counting module 31 includes at least two sub-counting modules and a vector averaging sub-module; the at least two sub-counting modules contain convolution kernels of different sizes and are used to output sub-vectors of the same size, each sub-vector representing the number of times the predicted character data is used at different scales; the vector averaging sub-module is used to obtain the average vector of the at least two sub-vectors to obtain the counting vector.

[0118] In one example, the multi-scale counting module 31 includes a first sub-counting module and a second sub-counting module; the first sub-counting module is used to identify feature information at a first preset scale in the image feature map to obtain a first sub-vector; the second sub-counting module is used to identify feature information at a second preset scale in the image feature map to obtain a second sub-vector.

[0119] In another example, the multi-scale counting module further includes a third sub-counting module, which is used to identify feature information at a third preset scale in the image feature map to obtain a third sub-vector. In this example, the multi-scale counting module 31 includes a first sub-counting module, a second sub-counting module, a third sub-counting module, and a vector averaging sub-module (Element Average). See also Figure 4 The first sub-counting module 41, the second sub-counting module 42, the third sub-counting module 43, and the vector averaging sub-module (Element Average) 44;

[0120] The first sub-counting module 41 is used to identify feature information of the first preset scale in the image feature map E(X) to obtain the first sub-vector;

[0121] The second sub-counting module 42 is used to identify feature information of the second preset scale in the image feature map E(X) to obtain the second sub-vector;

[0122] The third sub-counting module 43 is used to identify the feature information of the third preset scale in the image feature map E(X) and obtain the third sub-vector;

[0123] The element average submodule 44 is used to obtain the average vector of the first, second, and third subvectors, resulting in a counting vector CV. The counting vector CV represents the count of each character.

[0124] In one embodiment, see Figure 5 The first sub-counting module 41 includes a first convolution unit 411 (Conv3*3+BN), a channel unit 412, a conversion unit 413 (Conv1*1+Sigmoid), and a pooling unit 414 (Sum Pooling).

[0125] The first convolutional unit 411 is used to transform the image feature map E(X) into a first feature map using a convolutional kernel of a first preset size. This first convolutional unit also includes a BN (Batch Normalization) layer, used to transform the image feature map E(X) into a feature map of a preset dimension. Furthermore, the first convolutional unit includes a convolutional kernel Conv; in one example, the size of this convolutional kernel, i.e., the first preset size, is 3*3*684; and the number of convolutional kernels is 512, so the dimension of the first feature map output by the first convolutional unit is H / 16*W / 16*512.

[0126] Channel unit 412 is used to assign corresponding weights to different channels of the first feature map to obtain an attention feature map; and to multiply the attention feature map and the first feature map to obtain a channel feature map. In one example, the attention channels may include, but are not limited to, the following activation functions: ReLU and Sigmoid. Of course, the attention channels may also include functions such as GAP and Linear. Since different functions have different output value ranges, corresponding weights can be assigned to different channels of the first feature map to select the attention value range under different channels. Then, the channel unit 52 can multiply the weight values ​​of each channel with the feature map of a preset dimension to obtain the channel feature map.

[0127] Taking the GAP function as an example, the GAP function can perform average pooling on the first feature map, that is, calculate the average value of the H / 16*W / 16 plane to obtain the value corresponding to the plane; after 512 times, a channel feature map of H / 16*W / 16*512 dimensions can be obtained.

[0128] Transformation unit 413 is used to perform scaling transformation on the channel feature maps to obtain counting feature maps. This transformation unit 413 is used to transform channel feature maps of different input dimensions to a feature map of another preset dimension, hereinafter referred to as a counting map. In one example, this transformation unit includes a convolutional kernel and a sigmoid activation function. The size of the convolutional kernel is 1*1*512, and the number of convolutional kernels is K, thus enabling the recognition of K characters, i.e., the counting feature map is a 1*K dimensional vector.

[0129] Pooling unit 414 is used to perform pooling processing on the counting feature map to obtain the first sub-vector.

[0130] See also Figure 5The first convolutional unit 411 performs a 3*3*684 convolution using 512 kernels, reducing the number of channels in the image feature map from 684 to 512. Then, the channel unit 412 obtains the weight values ​​for each of the 512 channels and multiplies each channel's weight value with the first feature map output by the first convolutional unit 411, resulting in a weighted H / 16*W / 16*512 dimensional channel feature map. Next, the transformation unit 413 reduces the number of channels in the first feature map from 512 to 208 dimensions using a 1*1*208 convolution. It should be noted that the 208 dimensions refer to the character data of the recognition network and can be adjusted according to the pattern. At this point, the dimension of the counting feature map is H*W*208. Finally, the pooling unit 414 sums the values ​​across each H*W plane to obtain a 1*208 dimensional first sub-vector, where the pixel value of each of the 208 channels in the first sub-vector represents the number of times a character appears in the current formula.

[0131] See also Figure 5 The second sub-counting module 42 includes a second convolution unit 421, a channel unit 422, a transformation unit 423, and a pooling unit 424. Among them,

[0132] The second convolutional unit 421 is used to convert the image feature map E(X) into a second feature map using a convolutional kernel of a second preset size;

[0133] Channel unit 422 is used to assign corresponding weights to different channels of the second feature map to obtain an attention feature map; and to multiply the attention feature map and the second feature map to obtain a channel feature map;

[0134] The conversion unit 423 is used to perform a scaling transformation on the channel feature map to obtain a counting feature map;

[0135] Pooling unit 424 is used to perform pooling processing on the counting feature map to obtain the second sub-vector.

[0136] It should be noted that the structure of the second sub-counting module 42 is the same as that of the first sub-counting module 41, the difference being that their convolutional units are different. Specifically, the second sub-counting module 42 converts the image feature map to a second preset size instead of a first preset size. In one example, the second convolutional unit 421 includes convolutional kernels of size 5*5*684, i.e., the second preset size is 5*5*684; and the number of convolutional kernels is 512. Thus, the dimension of the second feature map output by the second convolutional unit 421 is H / 16*W / 16*512. Furthermore, the dimension of the second sub-vector is H*W*208.

[0137] See also Figure 5The third sub-counting module 43 includes a third convolution unit 431, a channel unit 432, a transformation unit 433, and a pooling unit 434. Among them,

[0138] The third convolutional unit 431 is used to convert the image feature map E(X) into a third feature map using a convolutional kernel of a third preset size;

[0139] Channel unit 432 is used to assign corresponding weights to different channels of the third feature map to obtain the attention feature map; and after multiplying the attention feature map and the third feature map, the channel feature map is obtained.

[0140] The conversion unit 433 is used to perform a scaling transformation on the channel feature map to obtain a counting feature map;

[0141] Pooling unit 434 is used to perform pooling processing on the counting feature map to obtain a third sub-vector.

[0142] It should be noted that the structure of the third sub-counting module 43 is the same as that of the first sub-counting module 41, the difference being that their convolutional units are different. In one example, the third convolutional unit 431 includes a convolutional kernel of size 7*7*684, that is, the third preset size is 7*7*684. In other words, the dimension of the third sub-vector is H*W*208.

[0143] In one embodiment, the loop decoding module includes a gated loop unit for generating the character latent vector based on the latent vector and the previous predicted character vector. In one example, the loop decoding module includes a first gated loop unit and a second gated loop unit; the first gated loop unit is used to generate a first gated vector based on the latent vector and the previous predicted character vector; the second gated loop unit is used to generate the character latent vector based on the first gated vector. It should be noted that the number of gated loop units can be selected according to the specific scenario, and the corresponding scheme falls within the protection scope of this disclosure while satisfying the recognition accuracy requirements.

[0144] In another example, the recurrent decoding module also includes an attention submodule (attention model), see [link to example]. Figure 6 The loop decoding module 32 includes an attention submodule 61, a first gated loop unit 62, and a second gated loop unit 63; the first gated loop unit 62 is used to determine the latent vector... and the previous predicted character vector Generate the first gating vector The two input latent vectors of the first gated recurrent unit 62 and the previous predicted character vector Both are 256-dimensional. Previous predicted character vector. The initial value generated is the start symbol "<sos>The 256-dimensional vector representation of "" indicates that the values ​​taken after the first iteration are the results of the previous training process. (Hidden vector) The initial value is a linear transformation (1*256) of the average pixel value (dimension 1*684) of each channel of the image feature map E(X) output by the encoder, i.e., a 1*256 dimensional vector corresponding to the image feature map E(X); and during non-first loops... M is the character latent vector M output by the second gated loop unit 63.

[0145] It should be noted that the previous predicted character vector The character data is obtained by predicting characters based on the latent vector M. Then, a lookup operation is performed on the character data to find the vector corresponding to the character data, and the retrieved vector is used as the previous predicted character vector. The value of is used to reach the previous predicted character vector. The effect.

[0146] Attention submodule 61 is used to determine the image feature map E(X) and the all-zero tensor att. α (X) and the first gating vector Generate the context vector Ω; the input to the attention submodule 61 is E(X), with dimensions H / 16*W / 16*684. The output of the first gated loop unit 62 is... Its dimension is 1*256. The all-zero tensor att α The initial dimension of (X) is 1*1*H / 16*W / 16, which is used to record the historical areas of interest.

[0147] It should be noted that the all-zero tensor at in this example α (X) initializes the vector to all zeros for easier computation. Of course, it can also be initialized with other random numbers, yielding the same result. This all-zero tensor att α (X) takes the value at in non-first loops. α (X) = att α (X)+α. Where α is composed of (X)+α. Figure 7 The attention submodule 61 shown in the example updates the vector after each loop.

[0148] The second gated loop unit 63 is used to determine the first gated vector. The character latent vector M is generated from the context vector Ω. The output of the second gated loop unit 63 is M, which has a dimension of 1*256.

[0149] See Figure 7 Attention submodule 61 includes weight acquisition submodule 71 and context acquisition submodule 72;

[0150] The weight acquisition submodule 71 is used to obtain the weights from the all-zero tensor att. α (X) undergoes convolution and linear processing (Conv+linear) to obtain the first feature vector, and the image feature map E(X) undergoes convolution (Conv) to obtain the second feature vector, and the first gating vector is... A linear processing is performed to obtain the third feature vector; and the first feature vector, the second convolution vector, and the third feature vector are superimposed ("+") to obtain the fourth feature vector; wherein, the superposition process refers to combining the all-zero tensor att α The processed vector of dimension H / 16*W / 16*512 and the image feature map E(X) are added together to obtain a sum vector of dimension H / 16*W / 16*512; then, the first gate vector... Perform linear processing to obtain a vector of 1*512 dimensions; then, sum the sum vector of dimensions H / 16*W / 16*512 with the vector of dimensions 1*512 to obtain the fourth feature vector of dimensions H / 16*W / 16*512.

[0151] The context acquisition submodule 72 is used to perform the first activation processing (activation function is tanh), linear transformation processing (linear), and second activation processing (activation function is softmax) on the fourth feature vector to obtain the attention weight α; and to multiply the image feature map E(X) and the attention weight ("X") and then perform summation processing (sum) to obtain the context vector Ω.

[0152] In this embodiment, the attention submodule 61 employs a Coverage mechanism to avoid repeated recognition, and the context vector Ω = ∑ h ∑ w E(X)×α, input att in the next loop α (X) = att α (X)+α. First gating vector. After linear processing, the number of channels was adjusted to 512 dimensions. α (X) is processed by a convolutional layer (kernel size 11*11, padding = 5, stride = 1), and then linearly transformed to adjust the number of channels to 512. The image feature map E(X) is then processed by a 1*1 convolution to adjust the number of channels to 512. Then, the first, second, and third sub-vectors processed from the above three channels are added together, then activated by the tanh activation function, then linearly transformed to compress the number of channels to 1, and finally activated by the softmax activation function to obtain the attention map weight α.

[0153] See Figure 8 The character prediction module includes a character prediction submodule 81 and a structure prediction submodule 82. The character prediction submodule 81 processes the target vector to obtain predicted character data; the structure prediction submodule 82 processes the target vector to obtain predicted structure data. The character data and structure data constitute the predicted character set.

[0154] In one example, the character prediction submodule 81 includes a linear transformation unit (Linear) and an activation unit (Sigmoid). The linear transformation unit 811 is used to perform a linear transformation on the target vector; the activation unit 812 is used to activate the linearly transformed vector to obtain the predicted character data.

[0155] In one example, the structure prediction submodule 82 includes a linear transformation unit and an activation unit (softmax). The linear transformation unit is used to perform a linear transformation on the target vector; the activation unit is used to activate the linearly transformed vector to obtain the predicted character data.

[0156] Combination Figures 2-8 In one possible embodiment, the structure of the formula recognition model can be as follows: Figure 9 As shown.

[0157] See Figure 9 In the formula recognition model, by setting up a multi-scale counting module 31 and a cyclic decoding module 32, the features of different sized regions in the image feature map can be identified, which is beneficial to improving the detection accuracy of characters.

[0158] See Figure 10 The training process of the formula recognition model includes steps 101 to 103.

[0159] In step 101, training sample data is obtained; the training sample data includes a set of data for multiple mathematical formulas; each set of data includes character data and structural data obtained after expanding according to a preset formula format; the structural data is used to represent the relative positional relationship of some characters in the mathematical formula.

[0160] In this step, the electronic device can acquire training sample data. It is understood that the training sample data includes a set of multiple mathematical formulas, each set comprising character data and structural data obtained by expanding the formula according to a preset format; the structural data is used to represent the relative positional relationships of some characters in the mathematical formula.

[0161] In this step, the mathematical formula includes at least one of the following structures: fraction, subscript, superscript, radical, overline, overarrow, trigonometric sign, limit, summation symbol, etc. The symbols for these structures are \frac, _, ^, \sqrt, \overline, \overrightarrow, \widehat, \limits, \sum, respectively. It is understood that the following structures in mathematical formulas, such as log, ln, lg, or exponents, also apply to the scheme disclosed herein.

[0162] Table 1. Mathematical formulas for different structures and their LaTeX representations.

[0163]

[0164] Referring to Table 1, there are seven positional relationships in mathematical formulas: inside, right, above, below, and left superscript (e.g., for square roots). ), subscript (sub), superscript (sup).

[0165] For example, in mathematical formulas that include fractions, see [link / reference]. Figure 11 This explains the relationship between character positions in mathematical formulas and LaTeX tags. See also... Figure 11 ,formula Above the character is 'a', below it is 'b', and to its right is 'c'. When the default formula format is \frac{above}{below}right, the above formula... It can be written as \frac{a}{b}c.

[0166] Combining the positional relationships of characters in mathematical formulas and Figure 11 The preset structure shown can be used to generate training sample data in this embodiment.

[0167] 1. The mathematical formula is The training sample data is shown in Table 2.

[0168] Table 2 Training Sample Data

[0169]

[0170]

[0171] 2. The mathematical formula is: The training sample data is shown in Table 3.

[0172] Table 3 Training Sample Data

[0173]

[0174] It should be noted that the "" in Tables 2 and 3 <eos>" is the end character, the last one <eos>This indicates the end of formula recognition; the middle part... <eos>This indicates the end of each structure.

[0175] It should also be noted that in Tables 2 and 3, "struct" indicates the relative positional relationship of the characters contained here (7 types). If there is a positional relationship, fill it in; otherwise, fill in "None".

[0176] Combining Tables 2 and 3, the first column (from top to bottom) shows the formulas using LaTeX tags and the addition of the "struct" structure, along with their relative positions (7 types). Columns 2 through 7 show the relative positions after adding the structures. The entire table constitutes the training sample data. It is evident that by adding structure data to the training sample data in this embodiment, mathematical formulas can be represented more accurately, facilitating accurate subsequent reconstruction of the mathematical formulas.

[0177] Based on the above principles, a dataset of several mathematical formulas can be obtained to get training sample data.

[0178] In one embodiment, several users can handwrite mathematical formulas and capture images to obtain several original images containing the mathematical formulas. An electronic device stores a preset image recognition model, such as a convolutional neural network, and uses this model to recognize each original image, obtaining the character data corresponding to each original image. Alternatively, the character data corresponding to each original image can be manually labeled, and the electronic device can acquire the input character data. Finally, the electronic device can create structured data according to the preset formula format and character data, and arrange the character data to obtain training sample data corresponding to each original image. This embodiment not only improves the diversity of mathematical formulas but also increases the efficiency of acquiring training sample data.

[0179] In step 102, the training sample data is used to train the formula recognition model to be trained, and the predicted character set output by the formula recognition model is obtained; the predicted character set includes character data and structural data.

[0180] In this step, the electronic device can sequentially input each training sample data into the formula recognition model to be trained. This formula recognition model can process the training sample data to obtain a set of predicted characters. It is understood that this set of predicted characters includes both character data and structural data.

[0181] In this step, the reasoning process of the formula recognition model is as follows: Figure 12 As shown, see Figure 12 This includes: First, the structure information pushed onto the stack is [position information, parent_symbol], the prediction result pre_string = '' is initialized, and it is used to store the recognized character data. Figure 11 The LaTeX tag representation for mathematical formulas. It identifies each character character by character, obtaining the current character `current_symbol`, and assigns the value of `current_symbol` to `parent_symbol`. Then, it checks if `current_symbol` is not equal to 'struct' and if `current_symbol` is not the end character ('). <eos>'. When current_symbol is not 'struct' and ' <eos>When '', `pre_string += current_symbol`, that is, add the current character to `pre_string`. Then, check if the current character `current_symbol` is 'struct'. If `current_symbol` is 'struct' or '', the character is added to `pre_string`. <eos>When ' is in the current character string, check if current_symbol is 'struct'.

[0182] To determine if the current character `current_symbol` is 'struct', if `current_symbol` is 'struct', predict the structure information and push the structure data onto the stack, pop the top element from the stack, and adjust `pre_string` based on the top element and `parent_symbol`. If `current_symbol` is not 'struct', check if the stack is empty. If the stack is not empty, pop the top element from the stack, and adjust `pre_string` based on the top element and `parent_symbol`. If the stack is empty, check if the number of curly braces "{" and curly braces "}" in `pre_string` is equal. If they are equal, exit the loop and return the recognition result. If they are not equal, add a curly brace "}" to `pre_string` (i.e., `pre_string += "}"`). If the number of curly braces "{" and curly braces "}" in `pre_string` is equal, exit the loop and return the recognition result.

[0183] After adjusting `pre_string`, you can check if the loop count exceeds 100. If it does, exit the loop and return the recognition result. Otherwise, recognize the character, update the current character's `current_symbol`, restart the loop, and return the recognition result directly.

[0184] In step 103, training stops when the character data is determined to meet the preset conditions, and the trained formula recognition model is obtained.

[0185] In this step, to improve the accuracy of the formula recognition model, a loss function is set in this embodiment. This loss function is constructed based on the frequency of each character in the label data of the training sample data and the frequency of each character in the prediction data set. In one example, this loss function includes three sub-loss functions: the first sub-loss function, which compares the number of characters in the count feature map with the number of characters in the training sample data, is the SmoothL1Loss function; the second sub-loss function, which judges the prediction result of structural information, uses the cross-entropy function; and the third sub-loss function, which judges the prediction result of character information, also uses the cross-entropy function.

[0186] In this step, the electronic device can determine that the character data meets preset conditions. For example, the electronic device can obtain the frequency of occurrence of each character data in the predicted character set, and obtain the prediction frequency corresponding to each character data; then, the electronic device can obtain the difference between the prediction frequency corresponding to each character data and the labeled frequency in the labels of the training sample data, and obtain the prediction error value corresponding to each character data; then, the electronic device can obtain the average value of the prediction error values ​​corresponding to the character data in the predicted character set, and obtain the average error; when it is determined that the average error is less than or equal to a preset average threshold (e.g., 1-3 times), the electronic device can determine that the character data meets the preset conditions and stop training the formula recognition model, that is, obtain the formula recognition model that has been trained. When it is determined that the above average error is greater than the preset average threshold, the electronic device can determine that the character data does not meet the preset conditions, and can return to step 102 to continue training until the preset conditions are met.

[0187] In step 13, the position of the character data in the mathematical formula is restored according to the preset formula format and structural data to obtain the mathematical formula in the original image.

[0188] In this step, considering that the predicted character set identified in step 12 is represented by the structure shown in Tables 2 and 3, the electronic device can restore the position of the character data in the mathematical formula according to the preset formula format and structural data, and obtain the mathematical formula in the original image.

[0189] In this step, the recognition results of the mathematical formulas obtained by using the above formula recognition model are shown in Table 4.

[0190] Table 4 shows the recognition accuracy before and after using the formula recognition model.

[0191]

[0192] In Table 4, a checkmark "√" indicates that the function was used. Referring to Table 4, using only the counting module improved the accuracy of mathematical formula recognition by 45.49%; using only structural information improved the accuracy by 50.25%; and using both the counting module and structural information improved the accuracy by 51.95%.

[0193] Thus, the solution provided in this embodiment can obtain an original image containing mathematical formulas; then, the original image is input into a formula recognition model to obtain a predicted character set output by the formula recognition model; the predicted character set includes character data and structural data; subsequently, the position of the character data in the mathematical formula is restored according to the preset formula format and structural data to obtain the mathematical formula in the original image. In this way, by setting structural data, the relative positional relationship of character data can be characterized, eliminating the problem of missing characters during complex formula recognition and improving recognition accuracy. Alternatively, this embodiment uses a dual prediction method of structural information and character counting to improve the model's ability to recognize complex formulas with multi-layered nested structures, reducing misidentification of formulas caused by the loss of "{" or "}" due to time constraints, and breaking down complex formulas into components for recognition according to different structures improves recognition accuracy; a multi-scale counting module is added in the decoding stage, constructing a counting loss function based on the number of characters in the label, providing weak supervision of the training process, improving the detection rate of characters in each component, and improving recognition accuracy.

[0194] Based on the mathematical formula recognition method provided in this disclosure, this embodiment also provides a mathematical formula recognition device, see [link to relevant documentation]. Figure 13 The device includes:

[0195] The original image acquisition module 131 is used to acquire an original image containing mathematical formulas;

[0196] The prediction set acquisition module 132 is used to input the original image into the formula recognition model to obtain the predicted character set output by the formula recognition model; the predicted character set includes character data and structural data.

[0197] The mathematical formula acquisition module 133 is used to restore the position of the character data in the mathematical formula according to the preset formula format and structural data, so as to obtain the mathematical formula in the original image.

[0198] It should be noted that the device embodiment shown in this embodiment matches the content of the above method embodiment, and the content of the above method embodiment can be referred to, and will not be repeated here.

[0199] In an exemplary embodiment, an electronic device is also provided, including...

[0200] Display screen;

[0201] processor;

[0202] Memory for storing computer programs executable by the processor;

[0203] The processor is configured to execute a computer program in the memory to implement the method described above.

[0204] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including an executable computer program that can be executed by a processor to implement the method as described in the above embodiments. The readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc.

[0205] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This disclosure is intended to cover any variations, uses, or adaptations that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0206] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.< / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / sos>

Claims

1. A mathematical formula recognition method, characterized by, The method comprises: obtaining an original image containing a mathematical formula; inputting the original image into a formula recognition model to obtain a predicted character set output by the formula recognition model; the predicted character set comprises character data and structure data; restoring the position of the character data in the mathematical formula according to a preset formula format and the structure data to obtain the mathematical formula in the original image; the formula recognition model comprises an encoder and a decoder; the encoder is used to obtain an image feature map corresponding to the original image; the decoder is used to determine a predicted character set corresponding to the original image according to the image feature map; wherein the decoder comprises a multi-scale counting module, a recurrent decoding module, a feature fusion module and a character prediction module; the multi-scale counting module is used to convert the image feature map into counting vectors of a plurality of preset scales; the recurrent decoding module is used to obtain a character hidden vector according to the image feature map; the feature fusion module is used to perform fusion processing on the counting vectors, the character hidden vector and a previous predicted character vector to obtain a target vector; the character prediction module is used to predict character data and structure data respectively according to the target vector.

2. The method of claim 1, wherein, The encoder comprises a DenseNet network or a Transformer network.

3. The method of claim 1, wherein, The recurrent decoding module is also used to obtain a context vector according to the image feature map; and the feature fusion module is also used to perform fusion processing on the counting vectors, the context vector, the character hidden vector and the previous predicted character vector to obtain a target vector.

4. The method of claim 3, wherein, The feature fusion module performs linear conversion processing on the counting vectors, the context vector, the character hidden vector and the previous predicted character vector respectively to obtain first, second, third and fourth linear vectors; and obtains a sum vector of the first, second, third and fourth linear vectors to obtain a target vector.

5. The method of claim 1, wherein, The multi-scale counting module comprises at least two sub-counting modules and a vector average sub-module; the at least two sub-counting modules contain convolution kernels of different sizes and are used to output sub-vectors of the same size, each sub-vector being used to represent the number of times of predicting character data at different scales; and the vector average sub-module is used to obtain an average vector of at least two sub-vectors to obtain a counting vector.

6. The method of claim 5, wherein, The multi-scale counting module comprises a first sub-counting module and a second sub-counting module; the first sub-counting module is used to identify feature information of a first preset scale in the image feature map to obtain a first sub-vector; the first sub-vector is used to represent the number of times of identifying character data under a convolution kernel of the first preset scale; the second sub-counting module is used to identify feature information of a second preset scale in the image feature map to obtain a second sub-vector; the second sub-vector is used to represent the number of times of identifying character data under a convolution kernel of the second preset scale.

7. The method of claim 6, wherein, The first sub-counting module comprises a first convolution unit, a channel unit, a conversion unit and a pooling unit; The first convolution unit is configured to convert the image feature map into a first feature map by using a convolution kernel of a first preset size. The channel unit is configured to assign corresponding weights to different channels of the first feature map to obtain an attention feature map. The attention feature map and the first feature map are subjected to multiplication processing to obtain a channel feature map. The conversion unit is configured to perform scale transformation processing on the channel feature map to obtain a count feature map. The pooling unit is configured to perform pooling processing on the count feature map to obtain a first sub-vector.

8. The method of claim 6, wherein, The second sub-count module includes a second convolution unit, a channel unit, a conversion unit and a pooling unit. The second convolution unit is configured to convert the image feature map into a second feature map by using a convolution kernel of a second preset size. The channel unit is configured to assign corresponding weights to different channels of the second feature map to obtain an attention feature map. The attention feature map and the second feature map are subjected to multiplication processing to obtain a channel feature map. The conversion unit is configured to perform scale transformation processing on the channel feature map to obtain a count feature map. The pooling unit is configured to perform pooling processing on the count feature map to obtain a second sub-vector.

9. The method of claim 6, wherein, The multi-scale count module further includes a third sub-count module.

10. The method of claim 9, wherein, The third sub-count module includes a third convolution unit, a channel unit, a conversion unit and a pooling unit. The third convolution unit is configured to convert the image feature map into a third feature map by using a convolution kernel of a third preset size. The channel unit is configured to assign corresponding weights to different channels of the third feature map to obtain an attention feature map. The attention feature map and the third feature map are subjected to multiplication processing to obtain a channel feature map. The conversion unit is configured to perform scale transformation processing on the channel feature map to obtain a count feature map. The pooling unit is configured to perform pooling processing on the count feature map to obtain a third sub-vector.

11. The method of claim 1, wherein, The recurrent decoding module includes a gated recurrent unit. The gated recurrent unit is configured to generate the character hidden vector according to the hidden vector and a previous predicted character vector.

12. The method of claim 11, wherein, The recurrent decoding module includes a first gated recurrent unit and a second gated recurrent unit. The first gated recurrent unit is configured to generate a first gating vector according to the hidden vector and a previous predicted character vector. The second gated recurrent unit is configured to generate the character hidden vector according to the first gating vector.

13. The method of claim 12, wherein, The recurrent decoding module further includes an attention sub-module. The attention sub-module is configured to generate a context vector according to the image feature map, an all-zero tensor and the first gating vector. The second gated recurrent unit is configured to generate the character hidden vector according to the first gating vector and the context vector.

14. The method of claim 13, wherein, The attention sub-module includes a weight acquisition sub-module and a context acquisition sub-module. The weight obtaining submodule is configured to perform convolution and linear processing on the all-zero tensor to obtain a first feature vector, perform convolution processing on the image feature map to obtain a second feature vector, perform linear processing on the first gate vector to obtain a third feature vector, and perform superposition processing on the first feature vector, the second convolution vector, and the third feature vector to obtain a fourth feature vector. The context obtaining submodule is configured to obtain attention weights by performing first activation processing, linear conversion processing, and second activation processing on the fourth feature vector. The image feature map and the attention weights are multiplied and then summed to obtain a context vector.

15. The method of claim 1, wherein, The character prediction module comprises a character prediction submodule and a structure prediction submodule. The character prediction submodule is configured to process the target vector to obtain predicted character data. The structure prediction submodule is configured to process the target vector to obtain predicted structure data.

16. The method of claim 15, wherein, The character prediction submodule comprises a linear conversion unit and an activation unit. The linear conversion unit is configured to perform linear conversion on the target vector. The activation unit is configured to perform activation processing on the linearly converted vector to obtain predicted character data.

17. The method of claim 15, wherein, The structure prediction submodule comprises a linear conversion unit and an activation unit. The linear conversion unit is configured to perform linear conversion on the target vector. The activation unit is configured to perform activation processing on the linearly converted vector to obtain predicted structure data.

18. The method of claim 1, wherein, The formula recognition model is trained by the following steps, comprising: obtaining training sample data; the training sample data comprises a data set of multiple mathematical formulas; each data set comprises character data and structure data obtained after expansion according to a preset formula format; the structure data is used to represent the relative position relationship of part of the characters in the mathematical formula; training the formula recognition model to be trained using the training sample data to obtain a predicted character set output by the formula recognition model; the predicted character set comprises character data and structure data; stopping training when it is determined that the character data satisfies a preset condition to obtain a trained formula recognition model.

19. The method of claim 18, wherein, Determining that the character data satisfies the preset condition comprises: obtaining the frequency of occurrence of each character data in the predicted character set to obtain a predicted frequency corresponding to each character data; obtaining the difference between the predicted frequency corresponding to each character data and the labeled frequency in the label of the training sample data to obtain a predicted error value corresponding to each character data; obtaining the average value of the predicted error values corresponding to the character data in the predicted character set to obtain an error average value; when it is determined that the error average value is less than or equal to a preset average value threshold, it is determined that the character data satisfies the preset condition.

20. The method of claim 18, wherein, The training sample data is obtained by the following steps, comprising: obtaining multiple original images containing mathematical formulas; obtaining character data corresponding to each original image; creating structure data according to a preset formula format and character data and arranging the character data to obtain training sample data corresponding to each original image.

21. A mathematical formula recognition apparatus characterized by comprising: The device comprises: An original image acquisition module, configured to acquire an original image containing a mathematical formula; A prediction set acquisition module, configured to input the original image into a formula recognition model to obtain a predicted character set output by the formula recognition model; the predicted character set comprises character data and structure data; A mathematical formula acquisition module, configured to restore the position of the character data in the mathematical formula according to a preset formula format and the structure data, and obtain a mathematical formula in the original image; The formula recognition model comprises an encoder and a decoder; the encoder is configured to acquire an image feature map corresponding to the original image; the decoder is configured to determine a predicted character set corresponding to the original image according to the image feature map; The decoder comprises a multi-scale counting module, a recurrent decoding module, a feature fusion module and a character prediction module; The multi-scale counting module is configured to convert the image feature map into counting vectors of a plurality of preset scales; The recurrent decoding module is configured to acquire a character hidden vector according to the image feature map; The feature fusion module is configured to perform fusion processing on the counting vectors, the character hidden vector and a previous predicted character vector to obtain a target vector; The character prediction module is configured to predict the character data and the structure data respectively according to the target vector.

22. An electronic device, comprising: It comprises A processor; A memory for storing a computer program executable by the processor; The processor is configured to execute the computer program in the memory to implement the method of any one of claims 1-20.

23. A computer-readable storage medium, characterized in that, When the executable computer program in the storage medium is executed by the processor, the method of any one of claims 1-20 can be implemented.

Images