Formula recognition model training method and device

Abstract

The application discloses a training method and device of a formula recognition model, obtains multiple training instances, screens the training instances with image sizes in a preset range, fills the image sizes to obtain training instances with consistent image sizes, divides the training instances into the same cluster to obtain one or more clusters, divides multiple clusters to obtain multiple groups of batch data each containing the same number of training instances, adds batch data with consistent longest formula lengths into the same set to obtain one or more sets, determines a maximum likelihood loss function based on the training instances, determines a maximum margin loss function according to the maximum likelihood loss function, and the formula recognition model is successfully trained after the maximum likelihood loss function converges and the maximum margin loss function converges. The model has recognition ability after the maximum likelihood loss function converges, the model is adjusted through the maximum margin loss function, so that the model is accurate in recognition and the recognized formula meets the grammar rules.

Description

Technical Field

[0001] This application relates to the field of formula recognition, and in particular to a method and apparatus for training a formula recognition model. Background Technology

[0002] As an important part of image recognition, mathematical formula recognition has been widely studied and applied in scenarios such as smart education, photo recognition, document recognition, and test question input and retrieval.

[0003] Currently, mathematical formulas are typically identified using formula recognition models. Existing formula recognition models are generally based on an encoder-decoder structure, which extracts image features through convolutional neural networks and then decodes these features using neural networks and other structures to obtain the recognized formula. However, since the formula recognition model does not explicitly define the formula syntax rules, the recognized formula may contain syntax errors and cannot be used directly.

[0004] Therefore, how to train an accurate formula recognition model has become a problem that urgently needs to be solved in this field. Summary of the Invention

[0005] This application provides a method and apparatus for training a formula recognition model, with the aim of training a formula recognition model that can accurately recognize formulas.

[0006] To achieve the above objectives, this application provides the following technical solution:

[0007] A method for training a formula recognition model, comprising:

[0008] Multiple training instances are obtained; each training instance consists of an image-text sequence pair, the text sequence representing a formula marked by a markup language; the formula contained in the image is consistent with the text sequence in the training instance;

[0009] For each training instance, training instances with image sizes within a preset range are selected, and the image sizes of the images in the training instances are padded to obtain training instances with consistent image sizes;

[0010] The training instances with the same image size are grouped into the same cluster to obtain one or more clusters;

[0011] Divide the data into multiple clusters to obtain multiple batches of data, each containing the same number of training instances;

[0012] For each set of batch data, the batch data with the same longest formula length in the text sequence are added to the same set to obtain one or more sets;

[0013] Multiple sets of batch data are selected from the set, and the maximum likelihood loss function is calculated based on the training instances in the multiple sets of batch data.

[0014] Based on the maximum likelihood loss function, determine the maximum margin loss function;

[0015] Once the maximum likelihood loss function and the maximum margin loss function converge, the formula recognition model is considered to have been successfully trained.

[0016] Optionally, determining the maximum margin loss function based on the maximum likelihood loss function includes:

[0017] After the maximum likelihood loss function converges, it is determined that the initial formula recognition model has been successfully trained.

[0018] The image to be recognized is input into the initial formula recognition model to obtain the recognition result output by the initial formula recognition model; the recognition result includes at least positive sample data and negative sample data;

[0019] Calculate the negative likelihood function difference between the positive sample data and the negative sample data;

[0020] When the difference in the negative likelihood function is not greater than a first preset threshold, the difference in the negative likelihood function is determined as the maximum interval loss function.

[0021] Optionally, the step of dividing the data into multiple clusters to obtain multiple batches of data, each containing the same number of training instances, includes:

[0022] For each image in the training instance, the image is encoded using an encoder to obtain a feature map;

[0023] The computing resources are calculated based on the image size of the feature map, the proportional relationship between the formula length of the text sequence and the video memory; the computing resources include at least video memory.

[0024] The number of batches is determined based on the computing resources.

[0025] For each cluster, the cluster is divided according to the number of batches to obtain multiple batches of data, each containing the same number of training instances.

[0026] Optionally, determining the batch size based on the computing resources includes:

[0027] Set the preset multiplier of video memory to the second preset threshold;

[0028] The second preset threshold is adjusted to obtain the adjusted second preset threshold; the adjusted second preset threshold is close to the preset maximum video memory and not greater than the preset maximum video memory.

[0029] For each training instance, the image size of the image is multiplied by the training instance to obtain each first calculation result;

[0030] The first calculation result is summed to obtain the second calculation result;

[0031] When the second calculation result is greater than the adjusted second preset threshold and not greater than the third preset threshold, the number of training instances is determined as the batch number.

[0032] Optionally, for each group of batch data, the batch data with the longest formula length in the text sequence are added to the same set to obtain one or more sets, including:

[0033] For each set of batch data, the formula length of each formula is obtained from the text sequence of the batch data;

[0034] The longest formula length is selected from the formula lengths of each formula.

[0035] Add batches of data with the same longest formula length to the same set to obtain one or more sets.

[0036] Optionally, the step of calculating the maximum likelihood loss function based on the training instances from multiple sets of the batch data includes:

[0037] Each pre-calculated attention distribution is summed to obtain multiple visual coverage sets;

[0038] Multiple visual overlay sets are fused with each feature map using image fusion technology to obtain each target feature map; the feature map is obtained by pre-encoding the target image;

[0039] For each target feature map, replace the feature map in each pre-computed probability distribution with the target feature map;

[0040] The target parameters are obtained by summing up each of the probability distributions.

[0041] Based on the target parameters, the training instances in the multiple sets of the batch data, and the model parameters, each negative likelihood loss function is calculated.

[0042] The smallest negative likelihood loss function is selected from all the negative likelihood loss functions and identified as the target model parameter;

[0043] The maximum likelihood loss function is calculated based on the target parameters, the training instances in the multiple sets of batch data, and the target model parameters.

[0044] Optional, also includes:

[0045] An image containing a mathematical formula is input into the formula recognition model to obtain the formula recognition result input by the formula recognition model; the formula recognition result includes at least the mathematical formula.

[0046] A training device for a formula recognition model, comprising:

[0047] An acquisition unit is used to acquire multiple training instances; each training instance consists of an image-text sequence pair, the text sequence representing a formula marked by a markup language; the formula contained in the image is consistent with the text sequence in the training instance;

[0048] The filtering unit is used to filter training instances whose image size is within a preset range for each training instance, and to fill the image size of the images in the training instances to obtain training instances with consistent image sizes.

[0049] The first partitioning unit is used to partition the training instances with the same image size into the same cluster, thereby obtaining one or more clusters;

[0050] The second partitioning unit is used to partition the data into multiple clusters, resulting in multiple batches of data, each containing the same number of training instances.

[0051] An adding unit is used to add batches of data with the same longest formula length in the text sequence to the same set for each group of batch data, thereby obtaining one or more sets;

[0052] A selection unit is used to select multiple sets of the batch data from the set, and calculate the maximum likelihood loss function based on the training instances in the multiple sets of the batch data;

[0053] The first determining unit is used to determine the maximum interval loss function based on the maximum likelihood loss function;

[0054] The second determining unit is used to determine that the formula recognition model has been successfully trained after the maximum likelihood loss function converges and the maximum margin loss function converges.

[0055] Optionally, the first determining unit is specifically used for:

[0056] After the maximum likelihood loss function converges, it is determined that the initial formula recognition model has been successfully trained.

[0057] The image to be recognized is input into the initial formula recognition model to obtain the recognition result output by the initial formula recognition model; the recognition result includes at least positive sample data and negative sample data;

[0058] Calculate the negative likelihood function difference between the positive sample data and the negative sample data;

[0059] When the difference in the negative likelihood function is not greater than a first preset threshold, the difference in the negative likelihood function is determined as the maximum interval loss function.

[0060] Optionally, the second partitioning unit is specifically used for:

[0061] For each image in the training instance, the image is encoded using an encoder to obtain a feature map;

[0062] Based on the proportional relationship between the image size of the feature map and the video memory, the computing resources are calculated; the computing resources include at least video memory.

[0063] The number of batches is determined based on the computing resources.

[0064] For each cluster, the cluster is divided according to the number of batches to obtain multiple batches of data, each containing the same number of training instances.

[0065] The technical solution provided in this application involves: acquiring multiple training instances; filtering training instances whose image sizes are within a preset range; filling the image size to obtain training instances with consistent image sizes; dividing these training instances into the same cluster to obtain one or more clusters; dividing into multiple clusters to obtain multiple sets of batch data, each containing the same number of training instances; adding batch data with the longest formula of consistent length to the same set to obtain one or more sets; determining the maximum likelihood loss function based on the training instances; determining the maximum margin loss function based on the maximum likelihood loss function; and successfully training the formula recognition model after the maximum likelihood loss function and the maximum margin loss function converge. After the maximum likelihood loss function converges, the model has recognition capability. Adjusting the model using the maximum margin loss function ensures that the model accurately recognizes formulas that conform to grammatical rules. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1A flowchart illustrating a training method for a formula recognition model provided in this application embodiment;

[0068] Figure 2 A flowchart illustrating a method for partitioning batch data as provided in this application embodiment;

[0069] Figure 3 A schematic diagram illustrating a method for determining the batch size provided in an embodiment of this application;

[0070] Figure 4 A flowchart illustrating a method for determining the batch size provided in an embodiment of this application;

[0071] Figure 5 This is a schematic diagram illustrating the allocation of batch data by bucket, as provided in an embodiment of this application.

[0072] Figure 6 A flowchart illustrating a method for adding batch data as provided in this application embodiment;

[0073] Figure 7 A flowchart illustrating a method for calculating the maximum likelihood loss function provided in an embodiment of this application;

[0074] Figure 8 A schematic diagram illustrating the display effect of a visual overlay set provided in an embodiment of this application;

[0075] Figure 9 A flowchart illustrating a method for determining the maximum interval loss function provided in an embodiment of this application;

[0076] Figure 10 A schematic diagram illustrating a training method for a formula recognition model provided in an embodiment of this application;

[0077] Figure 11 This is a schematic diagram of the architecture of a training device for a formula recognition model provided in an embodiment of this application. Detailed Implementation

[0078] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0079] like Figure 1 The flowchart shown is a training method for a formula recognition model provided in an embodiment of this application, including the following steps:

[0080] S101: Obtain multiple training instances.

[0081] Each training instance consists of an image-text sequence pair, where the text sequence represents a formula labeled using a markup language; the formula contained in the image is consistent with the text sequence in the training instance.

[0082] Optionally, multiple training instances can be obtained from the Internet.

[0083] Training examples include, but are not limited to, the formula recognition public dataset IM2LATEX-100K.

[0084] Understandably, the text sequence can be rendered using a formula renderer to generate a corresponding image that is identical to the image in the training instance.

[0085] In this embodiment of the application, the formula in text form can be pre-segmented into a text sequence using a formula parser.

[0086] It should be noted that the text sequence in each training instance corresponds one-to-one with the formula in the image. For example, if the text sequence is: Formula 1, Formula 2, and Formula 3, then the formula contained in the image is: Formula 1, Formula 2, and Formula 3.

[0087] S102: For each training instance, select training instances whose image size is within a preset range, and fill in the image size of the images in the training instances to obtain training instances with consistent image size.

[0088] It is understandable that the process of selecting training instances whose image size is within a preset range means selecting training instances with similar image sizes and then filling the image size of the training instances to make them into instances with the same image size.

[0089] For example, suppose the image size of the first training instance is 15, the image size of the second training instance is 16, and the image size of the third training instance is 28, with a preset range of 15 to 20. The training instances whose image sizes are within the preset range are: the first training instance and the second training instance. The images in the first training instance and the second training instance are then filled with images of the same size.

[0090] It should be noted that there are multiple preset ranges, which can be modified according to the actual situation.

[0091] The specific implementation process of filling the image size of the training instance is as follows: the image size of the training instance is filled using formula (1), and the specific form of formula (1) is as follows.

[0092] (h,w)=(32*m,32*n),m,n∈N+ (1)

[0093] In formula (1), h is the height of the filled image, w is the width of the filled image, m refers to the height before filling, n refers to the width before filling, and N... + It refers to the set of positive integers.

[0094] S103: Divide training instances with consistent image sizes into the same cluster to obtain one or more clusters.

[0095] Specifically, training instances with consistent image sizes are grouped into the same cluster. If the image sizes of each training instance are consistent, then the training instances with consistent image sizes are grouped into the same cluster, resulting in one cluster. If the image sizes of multiple training instances are inconsistent, then the training instances with consistent image sizes are grouped into the same cluster, resulting in multiple clusters.

[0096] For example, the image size of the first training instance is 100, the image size of the second training instance is 150, the image size of the third training instance is 150, and the image size of the fourth training instance is 100. Training instances with the same image size are grouped into the same cluster to obtain multiple clusters. The first cluster includes the first training instance and the fourth training instance, and the second cluster includes the second training instance and the third training instance.

[0097] S104: Divide the data into multiple clusters to obtain multiple batches of data, each containing the same number of training instances.

[0098] It is understandable that dividing the data into multiple clusters results in multiple batches of data, each containing the same number of training instances. In other words, for each cluster, the cluster is divided into multiple batches of data, each containing the same number of training instances.

[0099] For example, there are two clusters, each containing six training instances. Dividing the data into multiple clusters results in four batches of data, each containing three training instances.

[0100] Optionally, in another embodiment of this application, one specific implementation of step S104 is as follows: Figure 2 As shown, it includes the following steps:

[0101] S201: For each training instance, the image is encoded using an encoder to obtain a feature map.

[0102] The images in the training instances can be represented as x∈R B×H×W×3 The image is encoded using an encoder, and the resulting feature map is... Where B is the number of training instances, H is the height of the image, W is the width of the image, H' is the height of the feature map, W' is the width of the feature map, and D is the dimension of the feature map.

[0103] S202: The computational resources are calculated based on the proportional relationship between the image size of the feature map, the formula length of the text sequence, and the video memory.

[0104] The computing resources include at least video memory.

[0105] Optionally, during the feature map decoding process, the decoder will perform the following calculations at each time step: formula (2), formula (3), formula (4), formula (5), formula (6) and formula (7).

[0106]

[0107] In formula (2), c t For the context vector, z t p(z) represents the probability corresponding to different parts of the feature map. t =(h,w)) represents the probability value corresponding to the coordinates (h,w).

[0108]

[0109] In formula (3), the Score function measures the correlation between two vectors, t is the current time step, and softmax is the softmax function.

[0110]

[0111] In formula (4), dot, general, and concat are multiple calculation methods, and h t Let represent the hidden state of the decoder at the current time step t, hs be the feature map of the encoder, and Wa be the trainable parameter matrix.

[0112] o t =tanh(W c [h t c t (5)

[0113] In formula (5), o t For vector o t W c is a trainable parameter matrix.

[0114] h t =RNN(h t-1 ,[y t-1 ;o t-1 (6)

[0115] In formula (6), h t-1 For the hidden state of the decoder at time step t-1, o t-1 For vector o t-1 y t-1 This is the text sequence at time step t-1.

[0116]

[0117] In formula (7), W o For a trainable parameter matrix, This is a sequence of text that has already been translated.

[0118] From the above formulas (2), (3), (4), (5), (6) and (7), it can be seen that the feature map has the greatest impact on the computation graph during the decoding process. This is because the size of the feature map is one to two dimensions larger than other variables. Therefore, in order to estimate the computational resources occupied during the decoding process, it can be approximately assumed that the increase in video memory usage at each time step is proportional to the size of the feature map vector. That is, the computational resources are calculated based on the proportional relationship between the image size of the feature map and the video memory. The proportional relationship between the image size of the feature map, the length of the formula in the text sequence and the video memory is shown in formula (8).

[0119]

[0120] In formula (8), GPU Memory refers to computing resources, L is the formula length of the training instances in the batch data, and i is the number of training instances.

[0121] It should be noted that since the batch data is selected from the same cluster, the image size of each image in the batch data is consistent, and the image size of the feature map has a fixed constant ratio with the image size. Therefore, they can be substituted for each other in formula (8). In other words, the computing resources can also be calculated by the proportional relationship between the image size and the video memory.

[0122] S203: Determine the number of batches based on computing resources.

[0123] Understandably, the batch size is determined based on computing resources. In other words, the batch size selected through optimization methods is close to, but does not exceed, the computing resources, based on the actual situation of computing resources.

[0124] It should be noted that, as Figure 3As shown, there are two methods for determining the number of batches. One is a fixed batch size, which determines the number of batches by a fixed batch size (for example, if the fixed batch size is 3, the determined number of batches is also 3). The other is a dynamic batch size. In this embodiment, the dynamic batch size method is used to determine the number of batches, and the number of batches can change instead of being a fixed value.

[0125] Optionally, in another embodiment of this application, one specific implementation of step S203 is as follows: Figure 4 As shown, it includes the following steps:

[0126] S401: Set the preset multiple of video memory to the second preset threshold.

[0127] The preset multiplier can be one-half.

[0128] S402: Adjust the second preset threshold to obtain the adjusted second preset threshold.

[0129] Among them, the adjusted second preset threshold is close to the preset maximum video memory and not greater than the preset maximum video memory.

[0130] It should be noted that the second preset threshold is adjusted so that the adjusted second preset threshold is close to the preset maximum video memory. That is, if the second preset threshold is not greater than the preset maximum video memory, the value of the second preset threshold is increased. If the increased second preset threshold is greater than the preset maximum video memory, the value of the second preset threshold is decreased so that it is close to the preset maximum video memory and not greater than the preset maximum video memory.

[0131] It is understandable that the second preset threshold is adjusted, and the most suitable value is found in the adjustment process, and the most suitable value is determined as the second preset threshold.

[0132] S403: For each training instance, multiply the image size of the image by the training instance to obtain the first calculation result.

[0133] For each training instance, the height of the image, the width of the image, and the training instance are multiplied together to obtain the first calculation result. The specific representation of the height of the image, the width of the image, and the training instance can be found in formula (9).

[0134] S404: Summate each of the first calculation results to obtain the second calculation result.

[0135] The specific implementation process of summing up each first calculation result can be found in formula (9).

[0136]

[0137] In formula (9), A is the second calculation result, H×W×L i These are the initial calculation results for each item.

[0138] S405: When the second calculation result is greater than the adjusted second preset threshold and not greater than the third preset threshold, the number of training instances is determined as the batch number.

[0139] The specific manifestation of the second calculation result being greater than the adjusted second preset threshold can be found in formula (10).

[0140]

[0141] In formula (10), M is the adjusted second preset threshold, and K is the number of training instances.

[0142] It should be noted that, to prevent small datasets from having excessively large batch sizes, we can also set a third preset threshold B. max When the second calculation result is greater than the adjusted second preset threshold and not greater than the third preset threshold, the number of training instances is determined as the batch number.

[0143] Optionally, if the second calculation result is not greater than the adjusted second preset threshold, it means that the number of training instances can be increased until the second calculation result is greater than the adjusted second preset threshold.

[0144] Optionally, if the second calculation result is greater than the third preset threshold, it indicates that the batch size is too large. In this case, the number of training instances cannot be determined as the batch size.

[0145] S204: For each cluster, the cluster is divided according to the number of batches to obtain multiple batches of data, each containing the same number of training instances.

[0146] Understandably, a cluster can be represented as D = { <x s ,y s >|s=1,...,S}, where x (s) For the image in the s-th training instance, y (s) Let S be the text sequence in the s-th training instance, and S be the total number of training instances. The clusters are divided according to the number of batches to obtain multiple batches of data, each containing the same number of training instances.

[0147] Specifically, suppose there are three clusters, each containing 10 batches of data, with 5 batches in total. For the three clusters, the clusters are divided according to the number of batches, resulting in six sets of batches, each containing 5 training instances.

[0148] Optionally, for multiple batches of data, the number of training instances in each batch can be the same or similar. This is because if the total number of batches in a cluster is not divisible by the number of batches, then the number of training instances in the last batch will definitely be less than the number of batches.

[0149] It should be noted that by estimating the memory usage during the recognition process and dynamically adjusting the batch size, efficient training is achieved with fixed computing resources.

[0150] S105: For each batch of data, add the batch data with the longest formula length in the text sequence to the same set to obtain one or more sets.

[0151] Each batch of data contains multiple training instances, which correspond to multiple text sequences. Each text sequence has a formula with a corresponding length. Batch data with the longest formula in the text sequences that have the same length are added to the same set to obtain one or more sets. If the longest formula in multiple text sequences has the same length, batch data with the longest formula in the text sequences that have the same length are added to the same set to obtain one set. If the longest formula in multiple text sequences has different lengths, batch data with the longest formula in the text sequences that have the same length are added to the same set to obtain multiple sets.

[0152] It is understandable that the size and quantity of the batch data affect the computational resources used during training, while the length of the longest formula in the batch affects the training speed of the formula recognition model. All existing formula recognition models based on the encoder-decoder structure use autoregressive decoding, that is, a label is generated at each time step, and they must be generated sequentially. Therefore, the training time of a batch of data in the recognition model depends on the length of the longest formula, that is, the training time is proportional to the length of the longest formula, as shown in formula (11).

[0153]

[0154] In formula (11), Training Time is the training time, and maxL is the maximum training time. i This is the longest formula length.

[0155] It should be emphasized that, see Figure 5Following the principle of bucket sort, several pre-divided batches of data (GPU0, GPU1, and GPU2 refer to the batch data) are placed into different sets according to the length of the longest formula within each batch. During parallel training, in each iteration, multiple batches of data are randomly selected from one set and sent to different GPUs, thereby ensuring that the different GPUs are synchronized, reducing the synchronization latency between different GPUs, and ultimately accelerating the overall training speed.

[0156] Optionally, in another embodiment of this application, one specific implementation of step S105 is as follows: Figure 6 As shown, it includes the following steps:

[0157] S601: For each batch of data, obtain the formula length of each formula from the text sequence of the batch data.

[0158] Optionally, the formula length can be represented in characters; for example, the formula length is 1024 characters.

[0159] S602: Select the longest formula from the formula lengths of each formula.

[0160] Specifically, suppose there are three formulas. The first formula has a length of 512 characters, the second formula has a length of 256 characters, and the third formula has a length of 1024 characters. The longest formula length selected from the three formula lengths is the third formula.

[0161] Understandably, the longest formula length is selected from the formula lengths of each formula so that certain operations can be performed based on the longest formula length in subsequent steps.

[0162] S603: Add batches of data with the same longest formula length to the same set to obtain one or more sets.

[0163] It is understandable that when the longest formula length of each batch of data is the same, the batch data with the same longest formula length are added to the same set to obtain one set; when the longest formula length of each batch of data is different, the batch data with the same longest formula length are added to the same set to obtain multiple sets.

[0164] S106: Select multiple batches of data from the set, and calculate the maximum likelihood loss function based on the training instances in the multiple batches of data.

[0165] Among them, the maximum likelihood loss function is a commonly used loss function in recognition models based on encoders and decoders.

[0166] Understandably, the maximum likelihood loss function is calculated based on training instances from multiple batches of data. This is the process of training the initial formula recognition model. If the maximum likelihood loss function converges, the initial formula recognition model is considered to have been successfully trained. If the maximum likelihood loss function does not converge, the initial formula recognition model is considered to have been unsuccessfully trained, and the training of the initial formula recognition model continues.

[0167] Optionally, in another embodiment of this application, one specific implementation of step S106 is as follows: Figure 7 As shown, it includes the following steps:

[0168] S701: Add up each pre-calculated attention distribution to obtain multiple visual coverage sets.

[0169] Understandably, visual overlay sets provide auxiliary two-dimensional positional information for recognizing the movement of a model in two-dimensional space by recording historical information without adding any model parameters.

[0170] The calculation process of attention distribution can be found in formula (3), where each attention distribution indicates the attention distribution at each time step.

[0171] It should be noted that the specific implementation process of summing up each pre-calculated attention distribution to obtain the visual coverage set can be found in formula (12).

[0172]

[0173] In formula (12), Let p(z) be the visual overlay set at the current time step t. t-1 = (h, w)) represents the attention distribution at time step t-1. For the visual overlay set at time step t-1, VC t ∈R H’×W’ It records all the regions that the model has focused on before the current time step t and the attention weights corresponding to these regions. This visual history information can help the model quickly locate the current two-dimensional position and determine which row and column it is in, thus enabling it to make better decisions. Here, R is a real number.

[0174] S702: Using image fusion technology, multiple visual overlay sets and feature maps are fused to obtain feature maps of each target.

[0175] The feature map is obtained by pre-coding the target image.

[0176] It should be noted that the specific implementation process of fusing multiple visual overlay sets with feature maps using image fusion technology can be found in formula (13).

[0177]

[0178] In formula (13), Let α be the target feature map, and α be a hyperparameter used to adjust the proportions of features and historical information in the newly generated feature map.

[0179] Understandably, without increasing the parameters of the task model, fusing multiple visual overlay sets means that at each time step t, the decoder no longer sees a fixed feature map, but rather a dynamically updated target feature map as the decoding process progresses; that is, the target feature map is given a temporal dimension. In this way, we can assist the decoder in better selecting regions and using these regions to make more accurate translations (i.e., formula recognition), such as... Figure 8 The diagram shown illustrates the visual coverage set. It can be seen that the formula recognition model distinguishes between the regions of interest (i.e., regions that the formula recognition model has already paid attention to before the current time step) and the regions of non-interest (i.e., regions that the formula recognition model has not paid attention to before the current time step).

[0180] S703: For each target feature map, replace the feature map in each pre-computed probability distribution with the target feature map.

[0181] It should be noted that since the space of multi-line formulas includes both rows and columns, the formula recognition model can no longer simply translate from left to right as in the case of single-line formula recognition. Instead, it needs to first determine the total number of rows and columns, and then translate them sequentially in the order of rows first and then columns. Therefore, the feature maps in each pre-calculated probability distribution need to be replaced with the target feature map so that the formula recognition model can know the current recognition position and which areas have been recognized during the subsequent decoding process. This prevents the formula recognition model from getting lost in two-dimensional space, and a clear understanding of the position can help the model better plan the recognition path.

[0182] S704: Summate each probability distribution to obtain the target parameter.

[0183] For the specific implementation process of summing up each probability distribution, please refer to formula (14).

[0184]

[0185] In formula (14), θ is the model parameter, and P(y|x;θ) is the target parameter, which is used to determine the probability that the model produces formula y given an image and model parameter θ. It represents a probability distribution.

[0186] Optionally, the Teacher Forcing technique can be used to multiply the probability distribution of the decoder output at each time step. For the specific implementation process, please refer to formula (14).

[0187] S705: Based on the target parameters, training instances in multiple batches of data, and model parameters, calculate each negative likelihood loss function.

[0188] The specific implementation process of each negative likelihood loss function is calculated based on the target parameters, training instances in multiple batches of data, and model parameters. See formula (15).

[0189]

[0190] In formula (15), <x (s) ,y (s) > For training instances in multiple batches of data, L NLL (θ) represents the model parameters θ given the training set D = { <x (s) ,y (s) The negative likelihood loss function calculated from |s=1,...,S}.

[0191] S706: Select the negative likelihood loss function with the smallest value from all negative likelihood loss functions and mark it as the target model parameter.

[0192] It should be noted that in order for the model to be able to identify the corresponding formula based on the image, the minimum negative likelihood loss function is determined as the target model parameter. That is, the target model parameter is equivalent to the minimum negative likelihood loss function (i.e., the target model parameter is the set of parameters that can minimize the negative likelihood loss function). The specific manifestation of the target model parameter being equivalent to the minimum negative likelihood loss function is shown in formula (16).

[0193]

[0194] In formula (16), For the target model parameters, It is the minimum negative likelihood loss function.

[0195] S707: The maximum likelihood loss function is calculated based on the target parameters, training instances in multiple batches of data, and the target model parameters.

[0196] The specific implementation process of calculating the maximum likelihood loss function based on the target parameters, training instances in multiple batches of data, and target model parameters can be found in formula (17).

[0197]

[0198] S107: Determine the maximum margin loss function based on the maximum likelihood loss function.

[0199] It should be noted that after the maximum likelihood loss function converges, the initial formula recognition model is considered to have been successfully trained. The image to be recognized is then input into the initial formula recognition model to obtain the recognition result. The maximum margin loss function is then determined based on the recognition result.

[0200] Optionally, in another embodiment of this application, one specific implementation of step S107 is as follows: Figure 9 As shown, it includes the following steps:

[0201] S901: After the maximum likelihood loss function converges, the initial formula recognition model is confirmed to have been successfully trained.

[0202] Understandably, once the maximum likelihood loss function converges, the initial formula recognition model is considered successfully trained. When the image to be recognized is input into the initial formula recognition model, we can obtain a relatively high accuracy recognition result.

[0203] S902: Input the image to be recognized into the initial formula recognition model to obtain the recognition result output by the initial formula recognition model.

[0204] The recognition results include at least positive and negative sample data. Positive sample data indicates grammatically correct recognition results; negative sample data indicates grammatically incorrect recognition results.

[0205] It should be noted that since no grammar rules were involved in the training process, negative sample data that violates grammar rules may appear in the recognition results.

[0206] Optionally, the initial formula recognition model is used to perform a beam search on the image to be recognized, and then the top-k (k is a hyperparameter) recognition results are selected. From the k recognition results, the recognition results containing grammatical errors are selected and marked as negative sample data. If there are no grammatical errors in the k recognition results, then the initial recognition model has learned the relevant grammatical rules, and the difference in negative likelihood function between positive sample data and negative sample data is 0.

[0207] Understandably, the top-k formulas retrieved using Beam Search are the ones that the initial formula recognition model is most likely to generate. If we reduce the probability of these formulas, the parameters of the initial formula recognition model will be significantly affected. Therefore, the initial formula recognition model will also be able to more quickly identify how to recognize formulas that conform to grammatical rules, thereby achieving the goal of embedding external grammatical rules.

[0208] S903: Calculate the difference in negative likelihood between positive and negative sample data.

[0209] The specific implementation process for calculating the difference in negative likelihood function between positive and negative sample data can be found in formula (18).

[0210]

[0211] In formula (18), L s CL The negative likelihood function difference between positive and negative sample data is logP(y). (s) x (s) ;θ) represents positive sample data, For negative sample data, η is a hyperparameter used to adjust the interval.

[0212] S904: When the difference in the negative likelihood function is not greater than the first preset threshold, the difference in the negative likelihood function is determined as the maximum interval loss function.

[0213] In combination with formula (18), it can be seen that if This indicates that the probability of negative sample data is much lower than that of positive sample data, and the distance between positive and negative sample data is greater than the first preset threshold. At this point, the formula recognition model has met the standard, let L s CL =0; This indicates that the probability of negative sample data is higher than that of positive sample data, or that the difference in the negative likelihood function between positive and negative sample data is not greater than the first preset threshold. In this case, the difference in the negative likelihood function is determined as the maximum margin loss function, and the optimizer is used to optimize it to widen the distance between the two as much as possible, so that the probability of positive sample data is much higher than that of negative sample data.

[0214] S108: After the maximum likelihood loss function converges and the maximum margin loss function converges, the formula recognition model is considered to have been successfully trained.

[0215] Since learning grammar rules is a fine-tuning process that significantly modifies the parameters of the formula recognition model, in order to prevent the formula recognition model from generating only grammatically correct formulas while ignoring the requirement for recognition accuracy, the maximum likelihood loss function and the maximum margin loss function are used as optimization objectives. This ensures that the formula recognition model can accurately identify formulas while also ensuring that the grammar of the recognized formulas is correct. The specific form of the optimization objective is shown in formula (19).

[0216]

[0217] In formula (19), To optimize the objective.

[0218] It should be noted that after the maximum likelihood loss function and the maximum margin loss function converge, that is, after the optimization objective converges and the formula recognition model is successfully trained, it is still necessary to test the formula recognition model. At this time, the test set of the IM2LATEX-100K dataset (that is, unlabeled multi-line formula images) is used to test the formula recognition model (that is, to verify the effectiveness of this framework by measuring the BLEU (bilingual evaluation understudy) metric, image edit distance, image matching rate, etc.). The testing process is the same as the training process shown in S101-S108 above, and will not be repeated here.

[0219] After the formula recognition model is successfully trained, it can be used to recognize images containing formulas. Therefore, in another embodiment of this application, a formula recognition method is provided, including the following steps:

[0220] An image containing a mathematical formula is input into a formula recognition model to obtain the formula recognition result.

[0221] The formula recognition results include at least mathematical formulas.

[0222] In this method, images containing mathematical formulas are input into a pre-trained formula recognition model. The resulting formula recognition is not only accurate but also grammatically correct.

[0223] In summary, multiple sets of batch data are selected from the dataset, and the maximum likelihood loss function is calculated based on the training instances in these sets of batch data. The maximum margin loss function is then determined based on the maximum likelihood loss function. Once both the maximum likelihood loss function and the maximum margin loss function converge, the formula recognition model is considered successfully trained. In this embodiment, after the maximum likelihood loss function converges, the formula recognition model possesses a certain recognition capability. Adjusting the formula recognition model using the maximum margin loss function allows it to learn grammatical rules, ensuring both accurate recognition and correct grammar of the recognized formulas.

[0224] To better explain the training method for a formula recognition model provided in this embodiment of the invention, see [link to relevant documentation]. Figure 10 The diagram illustrates the training method for the formula recognition model.

[0225] The following diagrams illustrating the training method of the formula recognition model, using processes A1 to A8, provide further explanation.

[0226] A1: Select training instances from multiple batches of data and input the training instances into the encoder to extract feature maps.

[0227] Among them, the training instances contain the correct recognition results y (s) .

[0228] A2: Feature map The input is fed into the decoder, and then Beam Search is used to obtain k formula recognition results.

[0229] A3: Select the results of formula recognition that contain grammatical errors through grammar checking.

[0230] A4: Input the correct and incorrect recognition results into the decoder, and calculate the negative likelihood loss function corresponding to the correct and incorrect recognition results respectively.

[0231] A5: When the negative likelihood loss function of the correct result is much higher than the negative likelihood loss function corresponding to the incorrect recognition result, then the formula recognition model is successfully trained.

[0232] A6: When the negative likelihood loss function corresponding to the incorrect identification result is higher than the negative likelihood loss function corresponding to the correct result, calculate the difference in negative likelihood functions between the negative likelihood loss function corresponding to the incorrect identification result and the negative likelihood function corresponding to the correct result.

[0233] A7: When the difference in the negative likelihood function is greater than the first preset threshold, the difference in the negative likelihood function is determined as the maximum interval loss function.

[0234] A8: After the maximum likelihood loss function converges and the maximum margin loss function converges, the formula recognition model is considered to have been successfully trained.

[0235] In summary, by selecting training instances from batch data and training the formula recognition model based on these instances, the formula recognition model in this embodiment has a certain recognition capability after the maximum likelihood loss function converges. By adjusting the formula recognition model using the maximum margin loss function, the model learns grammatical rules, ensuring both accurate recognition and correct grammar of the recognized formulas.

[0236] like Figure 11 The diagram shown is an architectural schematic of a training device for a formula recognition model provided in an embodiment of this application. The training device includes: an acquisition unit 100, a filtering unit 200, a first division unit 300, a second division unit 400, an addition unit 500, a selection unit 600, a first determination unit 700, and a second determination unit 800.

[0237] The acquisition unit 100 is used to acquire multiple training instances; each training instance consists of an image-text sequence pair, where the text sequence represents a formula marked by a markup language; the formula contained in the image is consistent with the text sequence in the training instance.

[0238] The filtering unit 200 is used to filter training instances whose image size is within a preset range for each training instance, and to fill the image size of the images in the training instances to obtain training instances with consistent image sizes.

[0239] The first partitioning unit 300 is used to partition training instances with the same image size into the same cluster, resulting in one or more clusters.

[0240] The second partitioning unit 400 is used to partition multiple clusters to obtain multiple batches of data, each containing the same number of training instances.

[0241] The second partitioning unit 400 is specifically used for: encoding the image in each training instance using an encoder to obtain a feature map; calculating the computing resources based on the proportional relationship between the image size of the feature map, the formula length of the text sequence, and the video memory; the computing resources include at least video memory; determining the number of batches based on the computing resources; and partitioning each cluster according to the number of batches to obtain multiple batches of data, each containing the same number of training instances.

[0242] The second partitioning unit 400 is specifically used for: setting a preset multiple of video memory as a second preset threshold; adjusting the second preset threshold to obtain an adjusted second preset threshold; the adjusted second preset threshold is close to a preset maximum video memory and not greater than the preset maximum video memory; for each training instance, multiplying the image size of the image with the training instance to obtain each first calculation result; summing each first calculation result to obtain a second calculation result; when the second calculation result is greater than the adjusted second preset threshold and not greater than a third preset threshold, determining the number of training instances as the batch size.

[0243] Add unit 500 to add batches of text sequences with the same longest formula length to the same set for each batch of data, resulting in one or more sets.

[0244] Adding unit 500 is specifically used for: for each batch of data, obtaining the formula length of each formula from the text sequence of the batch data; filtering out the longest formula length from the formula length of each formula; adding batch data with the same longest formula length to the same set to obtain one or more sets.

[0245] Unit 600 is selected to select multiple batches of data from the set and calculate the maximum likelihood loss function based on the training instances in the multiple batches of data.

[0246] Unit 600 is specifically used for: summing each pre-calculated attention distribution to obtain multiple visual coverage sets; fusing multiple visual coverage sets with feature maps using image fusion technology to obtain various target feature maps; the feature maps are obtained by pre-encoding the target images; for each target feature map, replacing the feature map in each pre-calculated probability distribution with the target feature map; summing each probability distribution to obtain target parameters; calculating various negative likelihood loss functions based on the target parameters, training instances in multiple batches of data, and model parameters; selecting the smallest negative likelihood loss function from among the various negative likelihood loss functions and identifying it as the target model parameter; and calculating the maximum likelihood loss function based on the target parameters, training instances in multiple batches of data, and target model parameters.

[0247] The first determining unit 700 is used to determine the maximum interval loss function based on the maximum likelihood loss function.

[0248] The first determining unit 700 is specifically used for: determining that the initial formula recognition model has been successfully trained after the maximum likelihood loss function converges; inputting the image to be recognized into the initial formula recognition model to obtain the recognition result output by the initial formula recognition model; the recognition result includes at least positive sample data and negative sample data; calculating the negative likelihood function difference between the positive sample data and the negative sample data; and determining the negative likelihood function difference as the maximum interval loss function when the negative likelihood function difference is not greater than a first preset threshold.

[0249] The second determining unit 800 is used to determine that the formula recognition model has been successfully trained after the maximum likelihood loss function converges and the maximum margin loss function converges.

[0250] In summary, multiple sets of batch data are selected from the dataset, and the maximum likelihood loss function is calculated based on the training instances in these sets of batch data. The maximum margin loss function is then determined based on the maximum likelihood loss function. Once both the maximum likelihood loss function and the maximum margin loss function converge, the formula recognition model is considered successfully trained. In this embodiment, after the maximum likelihood loss function converges, the formula recognition model possesses a certain recognition capability. Adjusting the formula recognition model using the maximum margin loss function allows it to learn grammatical rules, ensuring both accurate recognition and correct grammar of the recognized formulas.

[0251] Preferred, combined Figure 11 As shown, the interface testing device also includes: a recognition unit for inputting an image containing mathematical formulas into a formula recognition model to obtain the formula recognition result input by the formula recognition model; the formula recognition result includes at least a mathematical formula.

[0252] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0253] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for training a formula recognition model, characterized in that, The method comprises the following steps: obtaining a plurality of training instances; each of the training instances is composed of an image-text sequence pair, and the text sequence is used to represent a formula marked by a markup language; the formula contained in the image is consistent with the text sequence in the training instance; for each of the training instances, screening the training instances with image sizes in a preset range, and padding the image sizes of the images in the training instances to obtain training instances with consistent image sizes; dividing the training instances with consistent image sizes into the same cluster to obtain one or more clusters; dividing a plurality of the clusters to obtain a plurality of batch data groups each containing the same number of training instances; for each of the batch data groups, adding the batch data groups with consistent longest formula lengths in the text sequence to the same set to obtain one or more sets; selecting a plurality of the batch data groups from the sets and calculating a maximum likelihood loss function based on the training instances in the plurality of the batch data groups; determining a maximum margin loss function according to the maximum likelihood loss function; after the maximum likelihood loss function converges and the maximum margin loss function converges, determining that the formula recognition model training is successful; wherein the step of dividing a plurality of the clusters to obtain a plurality of batch data groups each containing the same number of training instances comprises: for each of the images in the training instances, encoding the image using an encoder to obtain a feature map; based on the proportional relationship between the image size of the feature map and the formula length of the text sequence and the video memory, calculating a computing resource; the computing resource at least includes video memory; determining a batch number according to the computing resource; for each of the clusters, dividing the cluster according to the batch number to obtain a plurality of batch data groups each containing the same number of training instances; wherein the step of determining a batch number according to the computing resource comprises: setting a preset multiple of video memory as a second preset threshold; adjusting the second preset threshold to obtain an adjusted second preset threshold; the adjusted second preset threshold tends to a preset maximum video memory and is not greater than the preset maximum video memory; for each training instance, multiplying the image size of the image by the training instance to obtain each first calculation result; summing each of the first calculation results to obtain a second calculation result; when the second calculation result is greater than the adjusted second preset threshold and not greater than a third preset threshold, the number of training instances is determined as the batch number.

2. The method of claim 1, wherein, the step of determining a maximum margin loss function according to the maximum likelihood loss function comprises: after the maximum likelihood loss function converges, determining that the initial formula recognition model training is successful; inputting the image to be recognized into the initial formula recognition model to obtain a recognition result output by the initial formula recognition model; the recognition result at least includes positive sample data and negative sample data; calculating the negative likelihood function difference between the positive sample data and the negative sample data; when the negative likelihood function difference is not greater than a first preset threshold, the negative likelihood function difference is determined as the maximum margin loss function.

3. The method of claim 1, wherein, The batch data with the same longest formula length in the text sequence is added to the same set for each group of batch data, and one or more sets are obtained, including: For each group of batch data, the formula length of each formula is obtained from the text sequence of the batch data; The longest formula length is selected from the formula length of each formula; The batch data with the same longest formula length is added to the same set, and one or more sets are obtained.

4. The method of claim 1, wherein, The maximum likelihood loss function is calculated based on the training instances in the multiple groups of batch data, including: Add each pre-calculated attention distribution to obtain multiple visual coverage sets; Use image fusion technology to fuse multiple visual coverage sets with feature maps to obtain each target feature map; The feature map is obtained by pre-encoding the target image; For each target feature map, replace the feature map in each pre-calculated probability distribution with the target feature map; Each probability distribution is accumulated and summed to obtain a target parameter; Based on the target parameter, the training instances in the multiple groups of batch data, and the model parameter, each negative likelihood loss function is calculated; The smallest negative likelihood loss function is selected from each negative likelihood loss function and identified as a target model parameter; Based on the target parameter, the training instances in the multiple groups of batch data, and the target model parameter, the maximum likelihood loss function is calculated.

5. The method of claim 1, wherein, Also includes: Input the picture containing the mathematical formula into the formula recognition model to obtain the formula recognition result input by the formula recognition model; The formula recognition result at least includes the mathematical formula.

6. An apparatus for training a formula recognition model, comprising: Includes: An acquisition unit is configured to acquire a plurality of training instances; Each training instance is composed of an image-text sequence pair, and the text sequence is used to represent a formula marked by a markup language; The formula contained in the image is consistent with the text sequence in the training instance; A screening unit is configured to, for each training instance, screen training instances with image sizes within a preset range, and pad the image size of the image in the training instance to obtain training instances with consistent image sizes; A first division unit is configured to divide the training instances with consistent image sizes into the same cluster to obtain one or more clusters; A second division unit is configured to divide the multiple clusters to obtain multiple groups of batch data each containing the same number of training instances; An adding unit is configured to, for each group of batch data, add batch data with the same longest formula length in the text sequence to the same set to obtain one or more sets; A selecting unit is configured to select multiple groups of batch data from the sets and calculate a maximum likelihood loss function based on the training instances in the multiple groups of batch data; A first determining unit is configured to determine a maximum interval loss function according to the maximum likelihood loss function; A second determining unit is configured to determine that the formula recognition model training is successful after the maximum likelihood loss function converges and the maximum interval loss function converges. The second division unit is specifically configured to: For the image in each of the training instances, the image is encoded by using an encoder to obtain a feature map; Based on a proportional relationship between an image size of the feature map and a formula length of the text sequence and a video memory, a computing resource is calculated; the computing resource at least includes the video memory; According to the computing resource, a batch number is determined; For each of the clusters, the clusters are divided according to the batch number to obtain multiple batch data groups each containing the same number of training instances; The method further includes: The video memory of the preset multiple is set as a second preset threshold; The second preset threshold is adjusted to obtain an adjusted second preset threshold; the adjusted second preset threshold approaches a preset maximum video memory and is not greater than the preset maximum video memory; For each training instance, the image size of the image is multiplied by the training instance to obtain each first calculation result; Each of the first calculation results is accumulated and summed to obtain a second calculation result; When the second calculation result is greater than the adjusted second preset threshold and not greater than a third preset threshold, the number of the training instances is determined as the batch number.

7. The apparatus of claim 6, wherein, The first determining unit is specifically configured to: After the maximum likelihood loss function converges, it is determined that the initial formula recognition model training is successful; The to-be-recognized image is input into the initial formula recognition model to obtain a recognition result output by the initial formula recognition model; the recognition result at least includes positive sample data and negative sample data; A negative likelihood function difference value between the positive sample data and the negative sample data is calculated; When the negative likelihood function difference value is not greater than a first preset threshold, the negative likelihood function difference value is determined as a maximum interval loss function.

8. The apparatus of claim 6, wherein, The second dividing unit is specifically configured to: For the image in each of the training instances, the image is encoded by using an encoder to obtain a feature map; Based on a proportional relationship between an image size of the feature map and a formula length of the text sequence and a video memory, a computing resource is calculated; the computing resource at least includes the video memory; According to the computing resource, a batch number is determined; For each of the clusters, the clusters are divided according to the batch number to obtain multiple batch data groups each containing the same number of training instances.

Images