Formula recognition method, device and equipment and storage medium

Abstract

Embodiments of the present application disclose a formula recognition method and device, equipment and a storage medium. A target image containing a target formula is subjected to feature extraction to obtain a feature map. The feature map is decoded to obtain a first token sequence, the first token sequence containing at least part of the tokens in a target token sequence representing the target formula. The tokens in the first token sequence represent formula symbols in the target formula that only have a first dimension of positional relationship. If the first token sequence contains at least one placeholder and part of the tokens in the target token sequence, at least based on the first token sequence, sub-features corresponding to each placeholder are extracted from the feature map. The sub-features corresponding to each placeholder are respectively decoded to obtain a sub-token sequence corresponding to each placeholder. The target token sequence is constructed based on the sub-token sequence corresponding to each placeholder and the first token sequence. The formula recognition efficiency is improved based on the present application.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and more specifically, to a formula recognition method, apparatus, device, and storage medium. Background Technology

[0002] With the increasing demand for digitized documents, there is a growing number of applications that use text recognition to transform paper content into interactive multimedia content. For example, text recognition in educational materials can help collect and store teaching materials and students' assignments, becoming an indispensable prerequisite for smart teaching such as personalized learning and precision teaching.

[0003] However, in text recognition scenarios involving formulas, the efficiency of formula recognition is relatively low. Summary of the Invention

[0004] In view of this, this application provides a formula recognition method, apparatus, device, and storage medium to improve the efficiency of formula recognition.

[0005] To achieve the above objectives, the following solution is proposed:

[0006] A formula recognition method, comprising:

[0007] Feature extraction is performed on the target image containing the target formula to obtain a feature map;

[0008] The feature map is decoded to obtain a first marker sequence, which contains at least a portion of the markers in the target marker sequence that characterize the target formula; the markers in the first marker sequence characterize the formula symbols in the target formula that have only a first-dimensional positional relationship.

[0009] If the first token sequence contains at least one placeholder and some tokens in the target token sequence, extract the sub-features corresponding to each placeholder in the feature map based at least on the first token sequence;

[0010] Decode the sub-features corresponding to each placeholder to obtain the sub-tag sequence corresponding to each placeholder;

[0011] Based on the sub-marker sequence corresponding to each placeholder and the first marker sequence, the target marker sequence is constructed as the formula recognition result.

[0012] In the above method, preferably, the step of extracting sub-features corresponding to each placeholder in the feature map based at least on the first marker sequence includes:

[0013] The attention weights of each feature point and each placeholder in the feature map are determined using the first marker sequence.

[0014] For any placeholder and each feature point in the feature map, the feature vector at that feature point is multiplied by the attention weight between that feature point and any placeholder to obtain the sub-feature corresponding to any placeholder.

[0015] The above method, preferably, involves determining the attention weights between each feature point and each placeholder in the feature map using the first marker sequence, including:

[0016] Encode each placeholder and token in the first token sequence to obtain the encoding result with context information corresponding to each placeholder and token;

[0017] Based on the feature vectors at each feature point in the feature map and the encoding result of any placeholder, the attention weight between each feature point in the feature map and any placeholder is determined.

[0018] Preferably, in the above method, decoding the feature map includes:

[0019] The feature map is subjected to non-autoregressive decoding to obtain the first token sequence.

[0020] In the above method, preferably, the step of extracting sub-features corresponding to each placeholder in the feature map based at least on the first marker sequence includes:

[0021] For any placeholder, based on the first marker sequence and the spike information corresponding to any placeholder formed when performing non-autoregressive decoding on the feature map, sub-features corresponding to each placeholder are extracted from the feature map.

[0022] The above method, preferably, involves extracting sub-features corresponding to each placeholder in the feature map based on the first marker sequence and the spike information corresponding to any placeholder formed during non-autoregressive decoding of the feature map, including:

[0023] Based on the spike information corresponding to any placeholder, determine the target region in the feature map corresponding to any placeholder;

[0024] Obtain the attention weights of each feature point of the target region in the feature map determined using the first marker sequence and any placeholder;

[0025] Multiply the feature vector of each feature point within the target area by the attention weight of that feature point and any placeholder, and set the feature points of non-target areas in the feature map to zero to obtain the sub-feature corresponding to any placeholder.

[0026] The above method, preferably, includes the following steps: determining the target region, obtaining the attention weights of each feature point in the target region and any placeholder, and obtaining the sub-features corresponding to any placeholder.

[0027] Based on the spike information corresponding to any placeholder, the target region corresponding to the spike information in the initial mask is set to 1, and a mask corresponding to any placeholder is generated; the mask has the same size as the feature map, and the value of each point in the initial mask is zero;

[0028] The attention weights of each feature point in the feature map and any placeholder are determined using the first marker sequence, thus obtaining the attention weight matrix corresponding to any placeholder.

[0029] Multiply the feature map, the attention weight matrix corresponding to any placeholder, and the value at the corresponding position of the mask to obtain the sub-feature corresponding to any placeholder.

[0030] In the above method, preferably, the step of decoding the sub-features corresponding to each placeholder includes:

[0031] Autoregressive decoding is performed on the sub-features corresponding to each placeholder to obtain the sub-tag sequence corresponding to each placeholder.

[0032] Of the above methods, preferably,

[0033] The sequence of sub-tags corresponding to any placeholder represents the formula structure with a second-dimensional positional relationship in the target formula and the positional relationship between the formula symbols constituting the formula structure;

[0034] or,

[0035] The formula structure in the target formula represented by the sub-marker sequence corresponding to any placeholder has a second-dimensional positional relationship with the formula symbols represented by the marks in the first marker sequence.

[0036] In the preferred embodiment of the above method, the formula recognition method is based on a formula recognition model, which is trained using images containing formulas as training samples and the main and sub-label sequences corresponding to the formulas in the training samples as labels.

[0037] Wherein, the formulas in the training samples correspond to a sequence of sample markers, and the main marker sequence includes at least some of the markers in the sequence of sample markers; the markers in the main marker sequence represent formula symbols in the formulas of the training samples that only have a positional relationship in the first dimension;

[0038] If the main token sequence contains all the tokens in the sample token sequence, the sub-token sequence is empty;

[0039] If the main marker sequence contains at least one placeholder and some markers from the sample marker sequence, each placeholder in the main marker sequence corresponds to a sub-marker sequence; each sub-marker sequence and the markers contained in the main marker sequence are used to construct the sample marker sequence.

[0040] Preferably, the formula recognition model is trained in the following manner:

[0041] The training samples are used to extract features using the formula recognition model to obtain sample feature maps;

[0042] The sample feature map is decoded by the formula recognition model to obtain the first token sequence corresponding to the training sample, wherein the first token sequence contains at least some of the tokens in the sample token sequence;

[0043] If the primary label sequence of the training sample contains at least one placeholder, the recognition model extracts the sample sub-features corresponding to each placeholder in the sample feature map based on the primary label sequence of the training sample using the formula.

[0044] The sample sub-features corresponding to each placeholder are decoded using the formula recognition model to obtain the sub-tag sequence corresponding to each placeholder; the sub-tag sequence corresponding to each placeholder and the first tag sequence are used to construct the sample tag sequence;

[0045] The parameters of the formula recognition model are updated with the goal of making the first token sequence corresponding to the training sample approach the main token sequence label corresponding to the training sample, and making the sub-token sequences corresponding to each placeholder approach the sub-token sequence label corresponding to the training sample.

[0046] A formula recognition device, the device comprising:

[0047] The first feature extraction module is used to extract features from the target image containing the target formula to obtain a feature map;

[0048] A first decoding module is used to decode the feature map to obtain a first marker sequence, wherein the first marker sequence contains at least a portion of the markers in the target marker sequence that characterize the target formula; the markers in the first marker sequence characterize the formula symbols in the target formula that have only a first-dimensional positional relationship.

[0049] The second feature extraction module is used to extract sub-features corresponding to each placeholder in the feature map based at least on the first marker sequence if the first marker sequence contains at least one placeholder and some markers in the target marker sequence.

[0050] The second decoding module is used to decode the sub-features corresponding to each placeholder to obtain the sub-tag sequence corresponding to each placeholder.

[0051] The construction module is used to construct the target marker sequence as the formula recognition result based on the sub-marker sequence corresponding to each placeholder and the first marker sequence.

[0052] A formula recognition device, including a memory and a processor;

[0053] The memory is used to store programs;

[0054] The processor is configured to execute the program to implement the various steps of the formula recognition method as described in any of the preceding claims.

[0055] A computer-readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements the steps of the formula recognition method as described in any of the preceding claims.

[0056] As can be seen from the above technical solutions, the formula recognition method, apparatus, device, and storage medium provided in this application extract features from a target image containing a target formula to obtain a feature map; decode the feature map to obtain a first marker sequence, wherein the first marker sequence contains at least some markers in a target marker sequence representing the target formula; the markers in the first marker sequence represent formula symbols in the target formula that only have a first-dimensional positional relationship; if the first marker sequence contains at least one placeholder and some markers in the target marker sequence, extract sub-features corresponding to each placeholder from the feature map based at least on the first marker sequence; decode the sub-features corresponding to each placeholder to obtain a sub-marker sequence corresponding to each placeholder; and construct a target marker sequence as the formula recognition result based on the sub-marker sequences corresponding to each placeholder and the first marker sequence. After obtaining the feature map of the target image, this application does not directly identify the target formula as a sequence of target markers based on the feature map. Instead, it first identifies the target formula as a first sequence of markers corresponding to the target formula based on the feature map. The markers in the first sequence of markers represent formula symbols in the target formula that only have a positional relationship in the first dimension. If the first sequence of markers contains at least one placeholder and some markers in the target sequence of markers, the sub-features corresponding to each placeholder are extracted from the feature map based on at least the first sequence of markers. The sub-features corresponding to each placeholder are decoded separately to obtain the sub-marker sequence corresponding to each placeholder. Since each placeholder is independent of the others, the sub-features corresponding to each placeholder can be decoded in parallel, thereby improving the efficiency of formula decoding and thus improving the efficiency of formula recognition. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0058] Figure 1 This is a flowchart illustrating one implementation of the formula recognition method disclosed in the embodiments of this application;

[0059] Figure 2 This is a flowchart of an implementation of extracting sub-features corresponding to each placeholder in a feature map based on at least a first marker sequence, as disclosed in an embodiment of this application.

[0060] Figure 3 This is a flowchart illustrating an implementation of determining the attention weights of each feature point and each placeholder in a feature map using a first marker sequence, as disclosed in an embodiment of this application.

[0061] Figure 4 This is an example diagram illustrating the relationship between the decoding result obtained by decoding the feature map of a target image having formula (3) and the feature map, as disclosed in an embodiment of this application.

[0062] Figure 5 This is a flowchart of an implementation of an embodiment of the present application, which describes how to extract a sub-feature corresponding to any placeholder in a feature map based on a first marker sequence and the spike information corresponding to any placeholder formed during non-autoregressive decoding of the feature map.

[0063] Figure 6 This is another implementation flowchart of extracting the sub-feature corresponding to any placeholder in the feature map based on the first marker sequence and the spike information corresponding to any placeholder, as disclosed in the embodiments of this application;

[0064] Figure 7 This is an example diagram of the pairing data for "image-formula labels" disclosed in an embodiment of this application;

[0065] Figure 8 This is a schematic diagram of the structure of the formula recognition model disclosed in an embodiment of this application;

[0066] Figure 9 The embodiments disclosed in this application are based on Figure 8 The formula recognition model shown is a principle example diagram for recognizing formulas in images containing formula (2);

[0067] Figure 10 This is a schematic diagram of the structure of the formula recognition device disclosed in an embodiment of this application;

[0068] Figure 11 This is a hardware structure block diagram of the formula recognition device disclosed in an embodiment of this application. Detailed Implementation

[0069] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0070] Existing formula recognition methods typically use the formula's LaTeX string as a label, employing a two-dimensional attention mechanism, a sequence-to-sequence model, and an autoregressive approach to extract the LaTeX characters one by one. This method can achieve good formula recognition results, but the autoregressive approach of extracting characters one by one has low utilization of computing power, making it unsuitable for scenarios with high efficiency requirements.

[0071] To improve decoding efficiency, conventional text recognition models (excluding formulas) employ non-autoregressive decoding methods. Non-autoregressive decoding can decode individual symbols in parallel and has gained widespread attention in machine translation, speech recognition, and speech synthesis. The input text in machine translation and speech synthesis, as well as the input audio in speech recognition, is typically a one-dimensional signal. This means that there is only a one-dimensional positional relationship between symbols in the text or between speech frames (i.e., a single-dimensional positional relationship), usually a left-right positional relationship. Ordinary text (excluding formulas) images, after being compressed to one dimension in the height direction, can also be treated as a one-dimensional signal. Therefore, ordinary text recognition can use non-autoregressive algorithms from machine translation and other fields to decode in parallel, thereby improving decoding efficiency with minimal impact on decoding accuracy. However, unlike ordinary text, the symbols in formulas often have two-dimensional positional relationships (i.e., positional relationships in two dimensions), usually left-right and up-down. Based on this, compressing an image containing formulas into one dimension in the height direction will result in the loss of some information in the height direction. Moreover, for formulas with two-dimensional positional relationships, the reading order of the formula will have complex jumps. Existing non-autoregressive methods are more suitable for processing samples with monotonous reading order and are difficult to handle samples with complex jumps (such as formulas with two-dimensional positional relationships). Therefore, directly applying existing non-autoregressive algorithms will introduce more decoding errors, resulting in a decrease in formula recognition accuracy.

[0072] Therefore, how to improve the efficiency of formula recognition without affecting the accuracy of formula recognition has become an urgent technical problem to be solved.

[0073] like Figure 1 The diagram shown is a flowchart of one implementation of the formula recognition method provided in this application, which may include:

[0074] Step S101: Extract features from the target image containing the target formula to obtain a feature map.

[0075] The target image can be obtained by image acquisition of the target formula, or it can be obtained by cropping the formula after recognizing it from the text, or it can be directly read from the memory.

[0076] The target formula can be a printed formula, a handwritten formula, or a combination of both.

[0077] Optionally, convolutional neural networks, such as ResNet and U-Net, can be used to extract features from the target image. Assuming the target image I has a size of H×W, and the extracted feature map is denoted as F... I F IThe size is H'×W'×C, where H'=H / s, W'=W / s, s is the downsampling factor when extracting features from the target image, which can be 8, 16, or 32, etc., and C is the number of channels in the feature map.

[0078] Step S102: Decode the above feature map to obtain a first marker sequence. The first marker sequence contains at least some of the markers in the target marker sequence that represent the target formula. The markers in the first marker sequence represent the formula symbols in the target formula that only have the positional relationship of the first dimension.

[0079] For ease of distinction, the symbols in the target formula will be denoted as formula symbols.

[0080] Optionally, the markup language used to mark up the formulas can be LaTeX or other markup languages; this application does not impose any specific limitations.

[0081] Taking LaTeX as the markup language, the target token sequence representing the target formula is a LaTeX symbol sequence, denoted as the target LaTeX symbol sequence; correspondingly, the first token sequence contains at least some of the LaTeX symbols in the target LaTeX symbol sequence.

[0082] In this application, the first marker sequence may contain all the markers in the target marker sequence, indicating that the target formula has only one-dimensional positional relationship, i.e., only a single-dimensional positional relationship. For ease of description, this single-dimensional positional relationship is referred to as the first-dimensional positional relationship. As an example, the first-dimensional positional relationship can be a left-right positional relationship.

[0083] The first marker sequence may also contain at least one placeholder and some markers from the target marker sequence, indicating that a two-dimensional positional relationship exists in the target formula, i.e., a positional relationship in two dimensions, denoted as the first-dimensional positional relationship and the second-dimensional positional relationship. For example, the first-dimensional positional relationship is a left-right positional relationship, and the second-dimensional positional relationship is a top-bottom positional relationship. Since the markers in the first marker sequence represent formula symbols in the target formula that only have a first-dimensional positional relationship, this application splits formulas with two-dimensional positional relationships into one-dimensional and two-dimensional structures during the identification process, recognizing the two-dimensional structure as a placeholder.

[0084] A one-dimensional structure includes formula symbols that have only a first-dimensional positional relationship in the formula, while a two-dimensional structure includes formula symbols that have a second-dimensional positional relationship with the formula symbols in the one-dimensional structure.

[0085] For example, suppose the target formula in the target image is formula (1):

[0086] y2 -2=2y (1)

[0087] The one-dimensional structure includes "y-2=2y" in the objective formula, and the two-dimensional structure includes y 2 The superscript "2" in the text has a positional relationship with "y" in both the first and second dimensions.

[0088] or,

[0089] A one-dimensional structure includes a portion of the formula symbols in the formula that have only a first-dimensional positional relationship. A two-dimensional structure includes another portion of the formula symbols in the formula that have only a first-dimensional positional relationship, and formula symbols that have a two-dimensional positional relationship with that other portion of the formula symbols. For example, assuming the target formula in the target image is formula (1), then the one-dimensional structure includes "-2=2y" in the target formula, and the two-dimensional structure includes y in the target formula. 2 .

[0090] In this application, the feature map can be decoded based on the objective that the formula symbols represented by the markers in the first marker sequence have only a first-dimensional positional relationship, so as to obtain the first marker sequence.

[0091] For example, assuming the target formula in the target image is the above formula (1), it can be represented in LaTeX markup language as: y^{2}-2=2y. Then:

[0092] The first token sequence obtained by decoding the feature map can be y\superscript-2=2y, where \superscript is a placeholder and the other symbols are tokens; \superscript indicates that the formula symbol 2 corresponding to its sub-token sequence belongs to the superscript, and the six tokens "y", "-", "2", "=", "2" and "y" respectively represent the formula symbols "y", "-", "2", "=", "2" and "y" in formula (1).

[0093] or,

[0094] The first token sequence obtained by decoding the feature map can be \superscript-2=2y, where \superscript is a placeholder and the other symbols are tokens; \superscript represents its corresponding sub-token sequence "y^{2}" which represents the formula structure with superscript relationship in the formula.

[0095] For example, suppose the target formula in the target image is formula (2):

[0096]

[0097] Then, the first token sequence obtained by decoding the feature map can be cos\alpha=\frac\numerator\denominator, where \numerator and \denominator are placeholders, and the other symbols are tokens; \numerator indicates that the formula structure 'a' represented by its corresponding sub-token sequence belongs to the numerator, and \denominator indicates that the formula structure represented by its corresponding sub-token sequence belongs to the numerator. Belonging to the denominator, the symbols “cos”, “alpha”, “=”, and “frac” respectively represent the formula symbols “cos”, “α”, “=”, and “——” in formula (2).

[0098] or,

[0099] The first token sequence obtained by decoding the feature map can be cos\alpha = \fraction, where \fraction is a placeholder and the other symbols are tokens; \fraction represents the formula structure represented by its corresponding sub-token sequence. It belongs to the fraction, and the symbols “cos”, “alpha” and “=" respectively represent the formula symbols “cos”, “α” and “=" in formula (2).

[0100] In summary, the sub-tag sequence corresponding to any placeholder in this application represents the formula structure with a second-dimensional positional relationship in the target formula and the positional relationship between the formula symbols constituting the formula structure. For example, if the target formula is the above formula (1), the sub-tag sequence "y^{2}" corresponding to the placeholder \superscript represents the formula structure y with a two-dimensional positional relationship. 2 The formula symbols “y” and “2” that constitute this formula structure have a two-dimensional positional relationship, specifically, formula symbol “2” is located to the upper right of formula symbol “y”. For example, if the target formula is formula (2) above, the sequence of sub-tags corresponding to the placeholder \fraction represents the formula structure with a two-dimensional positional relationship. The formula symbols that make up the formula structure (from top to bottom, from left to right: a, —, There are two-dimensional positional relationships between a, 2, +, b, 2), specifically including various positional relationships, such as the first 'a' being above '-'. Located below —, the first 2 is located to the upper right of the second 'a', the second 2 is located to the upper right of 'b', and so on, not to be listed here.

[0101] or,

[0102] The formula structure in the target formula represented by the sub-tag sequence corresponding to any placeholder has a second-dimensional positional relationship with the formula symbol represented by the tags in the first tag sequence. For example, if the target formula is the above formula (1), the formula structure "2" represented by the sub-tag sequence corresponding to the placeholder \superscript has a second-dimensional positional relationship with the formula symbol represented by the tags in the first tag sequence. 2 The formula symbols y in the formula have a two-dimensional positional relationship. For example, if the target formula is the above formula (2), the substructure sequence corresponding to the placeholder \numerator represents the formula structure a and the formula symbol "——" represented by the marker in the first marker sequence has a second-dimensional positional relationship, where the formula structure a is located above the formula symbol "——", and the substructure sequence corresponding to the placeholder \denominator represents the formula structure There is a second-dimensional positional relationship between the formula symbol "—" represented by the markers in the first marker sequence, where the formula structure... Located below the formula symbol "—".

[0103] In this application, since the sub-marker sequence represented by the placeholder may contain only one marker, and one marker corresponds to only one formula symbol in the formula, the formula structure represented by the sub-marker sequence corresponding to any placeholder contains at least one formula symbol.

[0104] Step S103: If the first token sequence contains at least one placeholder and some tokens in the target token sequence, extract the sub-features corresponding to each placeholder in the feature map based at least on the first token sequence.

[0105] When the first token sequence contains placeholders, for the i-th placeholder (i = 1, 2, 3, ..., M; M is the number of placeholders in the first token sequence) in the first token sequence, the sub-feature corresponding to the i-th placeholder can be extracted from the feature map based on the first token sequence. Alternatively,

[0106] The sub-feature corresponding to the i-th placeholder can be extracted from the feature map based on the first marker sequence and other information.

[0107] Step S104: Decode the sub-features corresponding to each placeholder to obtain the sub-tag sequence corresponding to each placeholder.

[0108] Since each placeholder is independent of the others, the sub-features corresponding to different placeholders can be decoded in parallel to obtain the sub-tag sequence corresponding to each placeholder, thereby improving decoding efficiency.

[0109] The method for decoding the sub-features corresponding to the placeholders can be the same as or different from the method for decoding the feature map described above.

[0110] For example, autoregressive decoding is performed on the sub-features corresponding to the feature map and placeholders.

[0111] Alternatively, non-autoregressive decoding can be performed on the feature map, while autoregressive decoding can be performed on the sub-features corresponding to the placeholders. Since non-autoregressive decoding is only used for decoding the markers and placeholders corresponding to formula symbols that only have a first-dimensional positional relationship, it can guarantee decoding accuracy. Furthermore, non-autoregressive decoding is more efficient than autoregressive decoding. Therefore, compared to performing autoregressive decoding on both the feature map and the sub-features corresponding to the placeholders, performing non-autoregressive decoding on the feature map and autoregressive decoding on the sub-features corresponding to the placeholders can further improve decoding efficiency while maintaining decoding accuracy.

[0112] Step S105: Based on the sub-marker sequence corresponding to each placeholder and the first marker sequence, construct the target marker sequence as the formula recognition result.

[0113] Optionally, for the i-th placeholder, the i-th placeholder in the first token sequence can be replaced with the sub-token sequence corresponding to the i-th placeholder based on the preset replacement rule corresponding to the i-th placeholder, so as to obtain the target token sequence.

[0114] For example, corresponding to formula (1), the decoded first token sequence is: y\superscript-2=2y. The decoded sub-token sequence of the placeholder \superscript is: 2. The replacement rule for the placeholder \superscript is: place the sub-token sequence into the {} within the virtual body ^{}, and replace the placeholder \superscript with the virtual body ^{2} containing the sub-token sequence of the placeholder \superscript. Based on this, replacing the placeholder \superscript with the sub-token sequence 2 results in the target token sequence: y^{2}-2=2y.

[0115] For example, corresponding to formula (2), the decoded first token sequence is: cos\alpha=\frac\numerator\denominator. The decoded sub-token sequence of the sub-feature corresponding to the placeholder \numerator is: a. The decoded sub-token sequence of the sub-feature corresponding to the placeholder \denominator is: \sqrt{a^{2}+b^{2}}. The replacement rule corresponding to the placeholder \numerator is: place the sub-token sequence into the virtual body {}, and replace \numerator in the first token sequence with the virtual body {a} of the sub-token sequence corresponding to the placeholder \numerator. The replacement rule corresponding to the placeholder \denominator is: place the sub-token sequence into the virtual body {}, and replace \denominator in the first token sequence with the virtual body {\sqrt{a^{2}+b^{2}}} of the sub-token sequence corresponding to the placeholder \denominator. Based on this, the target token sequence obtained by replacing \numerator with the sub-token sequence a and \denominator with the sub-token sequence \sqrt{a^{2}+b^{2}} is: cos\alpha=\frac{a}{\sqrt{a^{2}+b^{2}}}.

[0116] The formula decoding method provided in this application, after obtaining the feature map of the target image, does not directly identify the target formula as a target marker sequence based on the feature map. Instead, it first identifies the target formula as a first marker sequence corresponding to the target formula based on the feature map. The markers in the first marker sequence represent formula symbols in the target formula that only have a first-dimensional positional relationship. If the first marker sequence contains at least one placeholder and some markers in the target marker sequence, the sub-features corresponding to each placeholder are extracted from the feature map based at least on the first marker sequence. The sub-features corresponding to each placeholder are decoded separately to obtain the sub-marker sequence corresponding to each placeholder. Since each placeholder is independent of the others, the sub-features corresponding to each placeholder can be decoded in parallel, thereby improving the formula decoding efficiency and thus improving the formula recognition efficiency.

[0117] In an optional embodiment, one implementation of the above-described non-autoregressive decoding of the feature map can be:

[0118] Compressing the feature map into one dimension along the height direction yields a one-dimensional feature. For example, feature map F I If the size is H'×W'×C, then compressing the feature map into one dimension along the height direction yields a one-dimensional feature map of 1×W'×C, denoted as F. I ={f1, f2, ..., f W’}, ft (t=1,2,…,W') is the eigenvector of 1×C.

[0119] Optionally, the mean of feature points in the same column of the feature map can be calculated (i.e., the mean of the eigenvectors corresponding to feature points in the same column), and used as the eigenvector of the feature point corresponding to the same column in the one-dimensional feature, thereby compressing the feature map into one dimension in the height direction. Alternatively, the maximum value of feature points in the same column of the feature map can be taken (the value of a feature point can be the maximum value or the mean of the elements in the eigenvector corresponding to the feature point), and used as the eigenvector of the feature point corresponding to the same column in the one-dimensional feature, thereby compressing the feature map into one dimension in the height direction.

[0120] A fully connected layer is applied to one-dimensional features to obtain a fully connected result. Alternatively, the one-dimensional features can be input into two fully connected layers for fully connected processing.

[0121] The fully connected network results are classified using a classification function to obtain the classification result. The classification result includes f. t The probability of being classified into each character. The classification result can be denoted as: Y = {y1, y2, ..., y...} W’}, where y t The dimension of f is the same as the dimension of dictionary D, representing f. t The probability of being classified as each character in dictionary D, which contains all the markers (e.g., Latex symbols), spaces, and placeholders corresponding to all possible characters in the formula.

[0122] Optionally, the classification function can be the softmax function.

[0123] The classification results are mapped using a mapping function to obtain the recognition result. As an example, the sequence-to-sequence mapping function defined in the Connectionist Temporal Classification (CTC) criterion can be used to map the classification results, thereby removing redundant symbols (such as repeated whitespace) and obtaining the final recognition result.

[0124] In an optional embodiment, the flowchart of one method for extracting sub-features corresponding to each placeholder in the feature map based at least on the first token sequence is as follows: Figure 2 As shown, it may include:

[0125] Step S201: Use the first marker sequence to determine the attention weights of each feature point and each placeholder in the feature map.

[0126] Also using feature map F ITaking a feature map of size H'×W'×C as an example, there are a total of H'×W' feature points, and each feature point represents a 1×C feature vector. Each feature point in the feature map corresponds to a region in the target image, and different feature points correspond to different regions in the target image. Assuming that the j-th (j=1, 2, 3, ..., H'×W') feature point corresponds to the j-th region in the target image, then the feature vector at the j-th feature point represents the formula feature information extracted from the j-th region of the target image.

[0127] For the i-th placeholder in the first token sequence, this application obtains the attention weights of the i-th placeholder and each feature point in the feature map.

[0128] Step S202: For any placeholder and each feature point in the feature map, multiply the feature vector at the feature point by the attention weight between the feature point and any placeholder to obtain the sub-feature corresponding to any placeholder.

[0129] For the i-th placeholder and the j-th feature point in the feature map, this application multiplies the feature vector at the j-th feature point by the attention weight between the j-th feature point and the i-th placeholder to obtain the feature vector at the j-th feature point in the sub-feature corresponding to the i-th placeholder. That is, in this application, the size of the sub-feature corresponding to each placeholder is also H'×W'×C.

[0130] In an optional embodiment, the flowchart of one implementation of determining the attention weights of each feature point and each placeholder in the feature map using the first marker sequence is shown below. Figure 3 As shown, it may include:

[0131] Step S301: Encode each placeholder and token in the first token sequence to obtain the encoding result with context information corresponding to each placeholder and token.

[0132] The word vectors of each placeholder and token in the first token sequence can be obtained to obtain the representation information of the first token sequence; the representation information of the first token sequence is encoded based on context interaction to obtain the encoding results with context information corresponding to each placeholder and token.

[0133] As an example, a bidirectional LSTM (Bi-LSTM) network can be used to encode the representation information of the first token sequence to obtain the encoding results with context information corresponding to each placeholder and token.

[0134] As an example, a Transformer network can be used to encode the representation information of the first token sequence to obtain the encoding results with context information corresponding to each placeholder and token.

[0135] Step S302: Based on the feature vectors at each feature point in the feature map and the encoding result of any placeholder, determine the attention weight between each feature point in the feature map and any placeholder.

[0136] For the i-th placeholder and the j-th feature point in the feature map, the encoding result of the i-th placeholder can be used as the query vector, and the feature vector at the j-th feature point can be used as the key to calculate the attention weight between the j-th feature point and the i-th placeholder in the feature map. For specific calculation formulas, please refer to existing attention calculation methods; they will not be detailed here.

[0137] Figure 2 and Figure 3 The embodiment shown extracts the sub-features corresponding to each placeholder in the feature map based only on the first marker sequence. This method may lead to confusion between features. For example, suppose the target formula in the target image is formula (3):

[0138]

[0139] Formula (3) can be expressed in LaTeX markup language as:

[0140] \frac{a-2}{x}-\frac{1}{x–2}=0

[0141] Its corresponding first token sequence should be:

[0142] \frac\numerator\denominator-\frac\numerator\denominator=0

[0143] In this context, \numerator and \denominator are both placeholders, with \numerator representing the numerator and \denominator representing the denominator.

[0144] When extracting the corresponding sub-features based on the encoding result of the first \numerator in the feature map, the sub-features corresponding to the two \numerators will be extracted at the same time, which may result in the final decoding as frac{1}{x}-\frac{a-2}{x–2}=0.

[0145] To overcome the above problems and further improve decoding accuracy, in the case of non-autoregressive decoding of the feature map, this application provides another implementation method for extracting the sub-features corresponding to each placeholder in the feature map based at least on the first token sequence:

[0146] For any placeholder, based on the first marker sequence and the spike information corresponding to that placeholder formed when performing non-autoregressive decoding on the feature map, the sub-feature corresponding to that placeholder is extracted from the feature map.

[0147] In other words, for the i-th placeholder, based on the first marker sequence and the spike information corresponding to the i-th placeholder formed when performing non-autoregressive decoding on the feature map, the sub-feature corresponding to the i-th placeholder is extracted from the feature map.

[0148] In non-autoregressive decoding, for any symbol in dictionary D (denoted as S for ease of description; symbol S can be a Latex symbol, a placeholder, or a blank space, etc.), the classification result obtained by the classification function may contain blank spaces near symbol S. That is, at least one symbol decoded before symbol S, and / or at least one symbol decoded after symbol S, are blank spaces (i.e., without Latex symbols or placeholders). This phenomenon is called the spike phenomenon. Based on this, the spike information corresponding to a placeholder includes the spike-related position information, which can include: the spike position of the placeholder, and the positions of consecutive blank spaces near the spike position. Of course, if there are no blank spaces near a placeholder, the spike position of the placeholder only includes the spike position of the placeholder, that is, the position corresponding to the placeholder. Since each decoded symbol corresponds to a feature point in the one-dimensional feature, the peak position of the i-th placeholder refers to the position of the feature point of the decoded i-th placeholder in the one-dimensional feature. Furthermore, since each feature point in the one-dimensional feature corresponds to a column in the feature graph, the peak position of the i-th placeholder refers to the column position in the feature graph corresponding to the feature point of the decoded i-th placeholder. Similarly, the position corresponding to symbol S refers to the column position in the feature graph corresponding to the feature point of the decoded symbol S, or the position of the feature point of the decoded symbol S in the one-dimensional feature.

[0149] like Figure 4 The figure shown is an example of the relationship between the decoding result obtained by decoding the feature map of a target image with formula (3) according to an embodiment of this application and the feature map (please ignore the blue attention information in the figure). Wherein, n represents the placeholder numerator, b represents blank, d represents the placeholder denominator, and -, =, and 0 all represent regular LaTeX symbols. Figure 4 Each column of the feature map corresponds to a column in the feature map, which is a feature point in the one-dimensional feature.

[0150] Figure 4 In the first numerator of the first token sequence corresponding to formula (3), the spike information includes: Figure 4Looking from left to right, the position corresponding to the first n (i.e., the first column position of the feature map), and the position corresponding to the consecutive blanks adjacent to it, i.e., the positions corresponding to the three consecutive bs to the right of the first n. The spike information corresponding to the second \numerator in the first marker sequence corresponding to formula (3) includes: Figure 4 Looking from left to right, the position corresponding to the second n (i.e., the eighth column of the feature map), and the position corresponding to the consecutive blanks adjacent to it, that is, the position corresponding to the b to the left of the second n, and the position corresponding to the two consecutive bs to the right of the second n.

[0151] The spike information corresponding to the first denominator in the first denominator sequence of formula (3) includes: Figure 4 Looking from left to right, the position corresponding to the first d (i.e., the fifth column of the feature map), and the position corresponding to the consecutive blanks adjacent to it, i.e., the positions corresponding to the three consecutive bs to the left of the first d. The spike information corresponding to the second denominator in the first denominator sequence corresponding to formula (3) includes: Figure 4 Looking from left to right, the position corresponding to the second d (i.e., the eleventh column of the feature map), and the position corresponding to the consecutive blank space adjacent to it, i.e., the position corresponding to the two consecutive b to the left of the second d.

[0152] In an optional embodiment, the flowchart for extracting the sub-feature corresponding to any placeholder in the feature map based on the first marker sequence and the spike information corresponding to any placeholder formed during non-autoregressive decoding of the feature map is as follows: Figure 5 As shown, it may include:

[0153] Step S501: Determine the target region in the feature map corresponding to any placeholder based on the spike information corresponding to any placeholder.

[0154] For the i-th placeholder, the target region corresponding to the i-th placeholder in the feature map is determined based on the spike information corresponding to the i-th placeholder.

[0155] The target region is the region in the feature map that corresponds to the peak information, such as... Figure 4 As shown, the target region determined by the spike information corresponding to the first \numerator in the first denominator sequence corresponding to formula (3) is the 1st to 4th columns of the feature map, the target region determined by the spike information corresponding to the second \numerator is the 7th to 10th columns of the feature map, the target region determined by the spike information corresponding to the first \denominator is the 2nd to 5th columns of the feature map, and the target region determined by the spike information corresponding to the second \denominator is the 9th to 11th columns of the feature map.

[0156] Step S502: Obtain the attention weights of each feature point of the target region in the feature map determined by the first marker sequence and any placeholder.

[0157] For the i-th placeholder, obtain the attention weights of each feature point of the target region in the feature map determined by the first marker sequence and the i-th placeholder.

[0158] The specific calculation method for attention weights can be found in the aforementioned embodiments, and will not be repeated here.

[0159] Step S503: Multiply the feature vector of each feature point in the target area by the attention weight of the feature point and any placeholder, set the feature points of non-target areas in the feature map to zero, and obtain the sub-feature corresponding to any placeholder.

[0160] based on Figure 5 The illustrated embodiment provides another implementation flowchart for extracting the sub-feature corresponding to any placeholder in the feature map based on the first marker sequence and the spike information corresponding to any placeholder, as shown below. Figure 6 As shown, it may include:

[0161] Step S601: Based on the spike information corresponding to any placeholder, set the target region corresponding to the spike information in the initial mask to 1, and generate a mask corresponding to any placeholder; the mask is the same size as the feature map, and the value of each point in the initial mask is zero.

[0162] Since the size of the mask is the same as the size of the feature map, for example, both the mask and the feature map are H'×W', the target region corresponding to the spike information in the mask is the same as the target region corresponding to the spike information in the feature map.

[0163] For the i-th placeholder, based on the spike information corresponding to the i-th placeholder, set the target region corresponding to the spike information in the initial mask to 1, and generate the mask corresponding to the i-th placeholder.

[0164] Step S602: Use the first marker sequence to determine the attention weights of each feature point in the feature map and any placeholder, and obtain the attention weight matrix corresponding to any placeholder.

[0165] For the i-th placeholder, the attention weights between each feature point in the feature map and the i-th placeholder are determined using the first token sequence, resulting in the attention weight matrix corresponding to the i-th placeholder. The size of the attention weight matrix is ​​also H'×W'.

[0166] Step S603: Multiply the feature map, the attention weight matrix corresponding to any placeholder, and the value at the corresponding position of the mask to obtain the sub-feature corresponding to any placeholder.

[0167] For the i-th placeholder, multiply the feature map, the attention weight matrix corresponding to the i-th placeholder, and the corresponding values ​​in the mask to obtain the sub-feature corresponding to the i-th placeholder. This can be expressed by the formula:

[0168]

[0169] in, This represents the sub-feature corresponding to the i-th placeholder. This represents the attention weight matrix corresponding to the i-th placeholder. F represents the mask corresponding to the i-th placeholder. I Represents the feature map.

[0170] The formula recognition method provided in this application embodiment can be implemented based on a formula recognition model, which is trained using images containing formulas as training samples and the main and sub-label sequences corresponding to the formulas in the training samples as labels.

[0171] For ease of description, the sequence of markers corresponding to the formulas in the training samples is denoted as the sample marker sequence. The main marker sequence includes at least some of the markers in the sample marker sequence. The markers in the main marker sequence represent the formula symbols in the training samples that only have the positional relationship of the first dimension.

[0172] When there is only a one-dimensional positional relationship in the formula of the training samples, the main marker sequence contains all the markers in the sample marker sequence, and the sub-marker sequence is empty.

[0173] When there is a two-dimensional positional relationship in the formula of the training samples, the main marker sequence contains at least one placeholder and some markers in the sample marker sequence. Each placeholder in the main marker sequence corresponds to a sub-marker sequence. The sub-marker sequences and the markers contained in the main marker sequence are used to construct the sample marker sequence.

[0174] The training samples in this application can be obtained by processing paired data of "image-formula labels," with the formula labels represented using LaTeX symbols as an example. Figure 7The image shown is an example of image-formula label pairing data provided in this application embodiment. In this example, the left side shows three formula images, and the right side shows the LaTeX expressions corresponding to the formulas in the images. To enable the formula model to decode the one-dimensional and two-dimensional parts of the formulas separately, this application processes the formula labels into a one-dimensional structure (also called the main structure, i.e., the main label sequence) and a substructure (i.e., the sub-label sequence) containing placeholders. In this example, the content within the dotted "{}" in the formula label is defined as a substructure, which is represented by placeholders in the main structure. Depending on the shape of each substructure and the relationship between the substructure and the main structure, the placeholders of the substructures can be refined according to their respective formula symbols, making them easier to identify in the main structure. For example, a substructure belonging to a superscript (i.e., the content represented by the dotted body "^{}") is represented as "\superscript", and a substructure belonging to a subscript (i.e., the content represented by the dotted body "_{}") is represented as "\subscript"; a substructure belonging to the numerator in a fraction (i.e., the content represented by the first "{}" after \frac) is defined as "\numerator", and a substructure belonging to the denominator in a fraction (i.e., the content represented by the second "{}" after \frac) is defined as "\denominator". "\superscript", "\subscript", "\numerator", and "\denominator" are treated as a single symbol during the decoding of the main structure, used to replace the content within "{}"; the specific content within "{}" is obtained during the decoding of the substructure; other substructures are defined in a similar way and will not be described in detail here. As shown in Table 1, this application provides an example of a substructure for... Figure 7 The example shown is of the main structure and substructure obtained by splitting the formula labels.

[0175] Table 1

[0176]

[0177] As an example, a structural diagram of the formula recognition model provided in this application embodiment is shown below. Figure 8 As shown, it may include:

[0178] Feature encoder 801, first decoder 802, and second decoder 803; wherein,

[0179] The feature encoder 801 is used to extract features from the target image containing the target formula to obtain a feature map;

[0180] The first decoder 802 is used to decode the feature map to obtain a first marker sequence. The first marker sequence contains at least some markers from the target marker sequence representing the target formula. The markers in the first marker sequence represent formula symbols in the target formula that only have positional relationships in the first dimension. The first marker sequence is the main marker sequence corresponding to the target image.

[0181] The second decoder 803 is used to extract sub-features corresponding to each placeholder in the feature map based at least on the first token sequence if the first token sequence contains at least one placeholder and some tokens in the target token sequence; and to decode the sub-features corresponding to each placeholder to obtain the sub-token sequence corresponding to each placeholder.

[0182] The sub-tag sequences corresponding to each placeholder and the tags in the first tag sequence are used to construct the target tag sequence.

[0183] like Figure 9 As shown, this application provides an embodiment based on... Figure 8 The formula recognition model shown is a principle example diagram for recognizing formulas in images with formula (2).

[0184] Figure 9 In the example shown, after the image containing formula (2) is input into the formula recognition model, the feature encoder 801 extracts features from the image to obtain a feature map. The feature map is used as the input of the first decoder. The first decoder 802 decodes the feature map to obtain the first token sequence: cos\alpha=\frac\numerator\denominator. The first token sequence and the feature map are used as the input of the second decoder 803. The second decoder 803 extracts the first sub-feature corresponding to \numerator and the second sub-feature corresponding to \denominator from the feature map based at least on the first token sequence. Then, the first sub-feature is decoded to obtain the sub-token sequence: a; the second sub-feature is decoded to obtain the sub-token sequence: \sqrt{a^{2}+b^{2}}.

[0185] Then, based on the first marker sequence cos\alpha=\frac\numerator\denominator, the sub-marker sequence a corresponding to the placeholder \numerator, and the sub-marker sequence \sqrt{a^{2}+b^{2}} corresponding to the placeholder \denominator, the formula recognition result is constructed as: cos\alpha=\frac{a}{\sqrt{a^{2}+b^{2}}}.

[0186] Optionally, the formula recognition model provided in this application embodiment can be trained using the following method:

[0187] The training samples are used to extract features by using a formula recognition model to obtain sample feature maps.

[0188] The sample feature map is decoded by the formula recognition model to obtain the first token sequence corresponding to the training sample. The first token sequence contains at least some of the tokens in the sample token sequence.

[0189] If the primary label sequence of the training sample contains at least one placeholder, the formula recognition model extracts the sample sub-features corresponding to each placeholder from the sample feature map based on the primary label sequence of the training sample.

[0190] The sample sub-features corresponding to each placeholder are decoded by the formula recognition model to obtain the sub-tag sequence corresponding to each placeholder; the sub-tag sequence corresponding to each placeholder and the first tag sequence are used to construct the sample tag sequence.

[0191] The parameters of the formula recognition model are updated with the goal of making the first token sequence corresponding to the training sample approach the main token sequence label corresponding to the training sample, and making the sub-token sequences corresponding to each placeholder approach the sub-token sequence labels corresponding to the training sample. This mainly involves updating the parameters of the feature encoder 801, the first decoder 802, and the second decoder 803 in the formula recognition model.

[0192] As an example, the first difference between the first token sequence and the main token sequence label corresponding to the training sample can be calculated using the CTC loss function. The smaller this first difference, the closer the first token sequence is to the main token sequence label corresponding to the training sample. Similarly, the second difference between the sub-token sequences corresponding to each placeholder and the sub-token sequence labels corresponding to the training sample can be calculated using the cross-entropy loss function. The smaller this second difference, the closer the sub-token sequences corresponding to each placeholder are to the sub-token sequence labels corresponding to the training sample. The parameters of the formula recognition model can be updated with the goal of minimizing both the first and second differences.

[0193] Corresponding to the method embodiments, this application also provides a formula recognition device. A schematic diagram of the structure of the formula recognition device provided in this application embodiment is shown below. Figure 10 As shown, it may include:

[0194] The system comprises a first feature extraction module 1001, a first decoding module 1002, a second feature extraction module 1003, a second decoding module 1004, and a construction module 1005; wherein...

[0195] The first feature extraction module 1001 is used to extract features from the target image containing the target formula to obtain a feature map;

[0196] The first decoding module 1002 is used to decode the feature map to obtain a first marker sequence, wherein the first marker sequence contains at least a portion of the markers in the target marker sequence that characterize the target formula; the markers in the first marker sequence characterize the formula symbols in the target formula that only have a first-dimensional positional relationship;

[0197] The second feature extraction module 1003 is used to extract sub-features corresponding to each placeholder in the feature map, based at least on the first marker sequence, if the first marker sequence contains at least one placeholder and some markers in the target marker sequence;

[0198] The second decoding module 1004 is used to decode the sub-features corresponding to each placeholder to obtain the sub-tag sequence corresponding to each placeholder.

[0199] The construction module 1005 is used to construct the target marker sequence as the formula recognition result based on the sub-marker sequence corresponding to each placeholder and the first marker sequence.

[0200] The formula recognition device provided in this application, after obtaining the feature map of the target image, does not directly recognize the target formula as a target marker sequence based on the feature map. Instead, it first recognizes the target formula as a first marker sequence corresponding to the target formula based on the feature map. The markers in the first marker sequence represent formula symbols in the target formula that only have a first-dimensional positional relationship. If the first marker sequence contains at least one placeholder and some markers in the target marker sequence, the device extracts the sub-features corresponding to each placeholder in the feature map based at least on the first marker sequence. The device decodes the sub-features corresponding to each placeholder to obtain the sub-marker sequence corresponding to each placeholder. Since each placeholder is independent of the others, the sub-features corresponding to each placeholder can be decoded in parallel, thereby improving the formula decoding efficiency and thus improving the formula recognition efficiency.

[0201] In an optional embodiment, the second feature extraction module 1003 includes an attention module for:

[0202] The attention weights of each feature point and each placeholder in the feature map are determined using the first marker sequence.

[0203] For any placeholder and each feature point in the feature map, the feature vector at that feature point is multiplied by the attention weight between that feature point and any placeholder to obtain the sub-feature corresponding to any placeholder.

[0204] In an optional embodiment, when the attention module determines the attention weights of each feature point and each placeholder in the feature map using the first marker sequence, it is used to:

[0205] Encode each placeholder and token in the first token sequence to obtain the encoding result with context information corresponding to each placeholder and token;

[0206] Based on the feature vectors at each feature point in the feature map and the encoding result of any placeholder, the attention weight between each feature point in the feature map and any placeholder is determined.

[0207] In an optional embodiment, when the first decoding module 1002 decodes the feature map, it is used to:

[0208] The feature map is subjected to non-autoregressive decoding to obtain the first token sequence.

[0209] In an optional embodiment, the second feature extraction module 1003 is specifically used for:

[0210] For any placeholder, based on the first marker sequence and the spike information corresponding to any placeholder formed when performing non-autoregressive decoding on the feature map, sub-features corresponding to each placeholder are extracted from the feature map.

[0211] In an optional embodiment, the second feature extraction module 1003 is specifically used for:

[0212] Based on the spike information corresponding to any placeholder, determine the target region in the feature map corresponding to any placeholder;

[0213] Obtain the attention weights of each feature point of the target region in the feature map determined using the first marker sequence and any placeholder;

[0214] Multiply the feature vector of each feature point within the target area by the attention weight of that feature point and any placeholder, and set the feature points of non-target areas in the feature map to zero to obtain the sub-feature corresponding to any placeholder.

[0215] In an optional embodiment, the second feature extraction module 1003 is specifically used for:

[0216] Based on the spike information corresponding to any placeholder, the target region corresponding to the spike information in the initial mask is set to 1, and a mask corresponding to any placeholder is generated; the mask has the same size as the feature map, and the value of each point in the initial mask is zero;

[0217] The attention weights of each feature point in the feature map and any placeholder are determined using the first marker sequence, thus obtaining the attention weight matrix corresponding to any placeholder.

[0218] Multiply the feature map, the attention weight matrix corresponding to any placeholder, and the value at the corresponding position of the mask to obtain the sub-feature corresponding to any placeholder.

[0219] In an optional embodiment, when the second decoding module 1004 decodes the sub-features corresponding to each placeholder, it is used to:

[0220] Autoregressive decoding is performed on the sub-features corresponding to each placeholder to obtain the sub-tag sequence corresponding to each placeholder.

[0221] In an alternative embodiment, wherein,

[0222] The sequence of sub-tags corresponding to any placeholder represents the formula structure with a second-dimensional positional relationship in the target formula and the positional relationship between the formula symbols that constitute the formula structure;

[0223] or,

[0224] The formula structure in the target formula represented by the sub-marker sequence corresponding to any placeholder has a second-dimensional positional relationship with the formula symbols represented by the marks in the first marker sequence.

[0225] In an optional embodiment, the formula recognition device is implemented based on a formula recognition model, which is trained using images containing formulas as training samples and the main and sub-label sequences corresponding to the formulas in the training samples as labels.

[0226] Wherein, the formulas in the training samples correspond to a sequence of sample markers, and the main marker sequence includes at least some of the markers in the sequence of sample markers; the markers in the main marker sequence represent formula symbols in the formulas of the training samples that only have a positional relationship in the first dimension;

[0227] If the main token sequence contains all the tokens in the sample token sequence, the sub-token sequence is empty;

[0228] If the main marker sequence contains at least one placeholder and some markers from the sample marker sequence, each placeholder in the main marker sequence corresponds to a sub-marker sequence; each sub-marker sequence and the markers contained in the main marker sequence are used to construct the sample marker sequence.

[0229] In an optional embodiment, the formula recognition model is trained as follows:

[0230] The training samples are used to extract features using the formula recognition model to obtain sample feature maps;

[0231] The sample feature map is decoded by the formula recognition model to obtain the first token sequence corresponding to the training sample, wherein the first token sequence contains at least some of the tokens in the sample token sequence;

[0232] If the primary label sequence of the training sample contains at least one placeholder, the recognition model extracts the sample sub-features corresponding to each placeholder in the sample feature map based on the primary label sequence of the training sample using the formula.

[0233] The sample sub-features corresponding to each placeholder are decoded using the formula recognition model to obtain the sub-tag sequence corresponding to each placeholder; the sub-tag sequence corresponding to each placeholder and the first tag sequence are used to construct the sample tag sequence;

[0234] The parameters of the formula recognition model are updated with the goal of making the first token sequence corresponding to the training sample approach the main token sequence label corresponding to the training sample, and making the sub-token sequences corresponding to each placeholder approach the sub-token sequence label corresponding to the training sample.

[0235] The formula recognition device provided in this application embodiment can be applied to formula recognition equipment, such as PC terminals, cloud platforms, servers, and server clusters. Optionally, Figure 11 The hardware structure block diagram of the formula recognition device is shown below. Figure 11 The hardware structure of the formula recognition device may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4;

[0236] In this embodiment of the application, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4;

[0237] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0238] Memory 3 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;

[0239] The memory stores a program, which the processor can call. The program is used for:

[0240] Feature extraction is performed on the target image containing the target formula to obtain a feature map;

[0241] The feature map is decoded to obtain a first marker sequence, which contains at least a portion of the markers in the target marker sequence that characterize the target formula; the markers in the first marker sequence characterize the formula symbols in the target formula that have only a first-dimensional positional relationship.

[0242] If the first token sequence contains at least one placeholder and some tokens in the target token sequence, extract the sub-features corresponding to each placeholder in the feature map based at least on the first token sequence;

[0243] Decode the sub-features corresponding to each placeholder to obtain the sub-tag sequence corresponding to each placeholder;

[0244] Based on the sub-marker sequence corresponding to each placeholder and the first marker sequence, the target marker sequence is constructed as the formula recognition result.

[0245] Optionally, the refined and extended functions of the program can be found in the description above.

[0246] This application embodiment also provides a storage medium that can store a program suitable for execution by a processor, the program being used for:

[0247] Feature extraction is performed on the target image containing the target formula to obtain a feature map;

[0248] The feature map is decoded to obtain a first marker sequence, which contains at least a portion of the markers in the target marker sequence that characterize the target formula; the markers in the first marker sequence characterize the formula symbols in the target formula that have only a first-dimensional positional relationship.

[0249] If the first token sequence contains at least one placeholder and some tokens in the target token sequence, extract the sub-features corresponding to each placeholder in the feature map based at least on the first token sequence;

[0250] Decode the sub-features corresponding to each placeholder to obtain the sub-tag sequence corresponding to each placeholder;

[0251] Based on the sub-marker sequence corresponding to each placeholder and the first marker sequence, the target marker sequence is constructed as the formula recognition result.

[0252] Optionally, the refined and extended functions of the program can be found in the description above.

[0253] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0254] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0255] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0256] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0257] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0258] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0259] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0260] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A formula recognition method, characterized in that, include: Feature extraction is performed on the target image containing the target formula to obtain a feature map; The feature map is non-autoregressive decoded to obtain a first token sequence, which contains at least a portion of the tokens in the target token sequence that characterizes the target formula. The markers in the first marker sequence represent the formula symbols in the target formula that only have a positional relationship in the first dimension; If the first token sequence contains at least one placeholder and some tokens in the target token sequence, extract the sub-features corresponding to each placeholder in the feature map based at least on the first token sequence; Autoregressive decoding is performed on the sub-features corresponding to each placeholder in parallel to obtain the sub-tag sequence corresponding to each placeholder; Based on the sub-marker sequence corresponding to each placeholder and the first marker sequence, the target marker sequence is constructed as the formula recognition result; Wherein, the extraction of sub-features corresponding to each placeholder in the feature map based at least on the first marker sequence includes: The attention weights of each feature point and each placeholder in the feature map are determined using the first marker sequence; for any placeholder and each feature point in the feature map, the feature vector at the feature point is multiplied by the attention weight of the feature point and the placeholder to obtain the sub-feature corresponding to the placeholder. Alternatively, for any placeholder, based on the first marker sequence and the spike information corresponding to any placeholder formed when performing non-autoregressive decoding on the feature map, the sub-feature corresponding to any placeholder is extracted from the feature map.

2. The method according to claim 1, characterized in that, The step of determining the attention weights of each feature point and each placeholder in the feature map using the first marker sequence includes: Encode each placeholder and token in the first token sequence to obtain the encoding result with context information corresponding to each placeholder and token; Based on the feature vectors at each feature point in the feature map and the encoding result of any placeholder, the attention weight between each feature point in the feature map and any placeholder is determined.

3. The method according to claim 1, characterized in that, The step of extracting sub-features corresponding to each placeholder in the feature map based on the first marker sequence and the spike information corresponding to any placeholder formed during non-autoregressive decoding of the feature map includes: Based on the spike information corresponding to any placeholder, determine the target region in the feature map corresponding to any placeholder; Obtain the attention weights of each feature point of the target region in the feature map determined using the first marker sequence and any placeholder; Multiply the feature vector of each feature point within the target area by the attention weight of that feature point and any placeholder, and set the feature points of non-target areas in the feature map to zero to obtain the sub-feature corresponding to any placeholder.

4. The method according to claim 1, characterized in that, in, The sequence of sub-tags corresponding to any placeholder represents the formula structure with a second-dimensional positional relationship in the target formula and the positional relationship between the formula symbols constituting the formula structure; or, The formula structure in the target formula represented by the sub-marker sequence corresponding to any placeholder has a second-dimensional positional relationship with the formula symbol represented by the markers in the first marker sequence.

5. The method according to any one of claims 1-4, characterized in that, The formula recognition method is based on a formula recognition model, which is trained using images containing formulas as training samples and the main and sub-label sequences corresponding to the formulas in the training samples as labels. Wherein, the formulas in the training samples correspond to a sequence of sample markers, and the main marker sequence includes at least some of the markers in the sequence of sample markers; the markers in the main marker sequence represent formula symbols in the formulas of the training samples that only have a positional relationship in the first dimension; If the main token sequence contains all the tokens in the sample token sequence, the sub-token sequence is empty; If the main marker sequence contains at least one placeholder and some markers from the sample marker sequence, each placeholder in the main marker sequence corresponds to a sub-marker sequence; each sub-marker sequence and the markers contained in the main marker sequence are used to construct the sample marker sequence.

6. The method according to claim 5, characterized in that, The formula recognition model is trained in the following way: The training samples are used to extract features using the formula recognition model to obtain sample feature maps; The sample feature map is non-autoregressive decoded using the formula recognition model to obtain a first token sequence corresponding to the training sample, wherein the first token sequence contains at least some of the tokens in the sample token sequence; If the primary label sequence of the training sample contains at least one placeholder, the recognition model extracts the sample sub-features corresponding to each placeholder in the sample feature map based on the primary label sequence of the training sample using the formula. The formula-based recognition model performs autoregressive decoding on the sample sub-features corresponding to each placeholder in parallel to obtain the sub-label sequence corresponding to each placeholder; the sub-label sequence corresponding to each placeholder and the first label sequence are used to construct the sample label sequence; The parameters of the formula recognition model are updated with the goal of making the first token sequence corresponding to the training sample approach the main token sequence label corresponding to the training sample, and making the sub-token sequences corresponding to each placeholder approach the sub-token sequence label corresponding to the training sample.

7. A formula recognition device, characterized in that, The device includes: The first feature extraction module is used to extract features from the target image containing the target formula to obtain a feature map; The first decoding module is used to perform non-autoregressive decoding on the feature map to obtain a first marker sequence. The first marker sequence contains at least a portion of the markers in the target marker sequence that characterize the target formula. The markers in the first marker sequence characterize the formula symbols in the target formula that only have a first-dimensional positional relationship. The second feature extraction module is used to extract sub-features corresponding to each placeholder in the feature map based at least on the first marker sequence if the first marker sequence contains at least one placeholder and some markers in the target marker sequence. The second decoding module is used in parallel to perform autoregressive decoding on the sub-features corresponding to each placeholder, so as to obtain the sub-tag sequence corresponding to each placeholder. The construction module is used to construct the target marker sequence as the formula recognition result based on the sub-marker sequence corresponding to each placeholder and the first marker sequence; Wherein, when the second feature extraction module extracts the sub-features corresponding to each placeholder in the feature map based at least on the first marker sequence, it is used for: The attention weights of each feature point and each placeholder in the feature map are determined using the first marker sequence; for any placeholder and each feature point in the feature map, the feature vector at the feature point is multiplied by the attention weight of the feature point and the placeholder to obtain the sub-feature corresponding to the placeholder. Alternatively, for any placeholder, based on the first marker sequence and the spike information corresponding to any placeholder formed when performing non-autoregressive decoding on the feature map, the sub-feature corresponding to any placeholder is extracted from the feature map.

8. A formula recognition device, characterized in that, Including memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement each step of the formula recognition method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements each step of the formula recognition method as described in any one of claims 1-6.

Images