Training method and device of offline handwritten formula recognition model, equipment and medium

Abstract

A training method and device of an offline handwritten formula recognition model, equipment and a medium, comprising receiving a handwritten formula image containing a handwritten formula, determining a LaTeX label of the handwritten formula, and converting the handwritten formula into a printed formula image according to the LaTeX label; the printed formula image and the handwritten formula image are paired and input into a pre-constructed neural network model for training, the contrast learning loss of the trained neural network model is calculated according to the preset contrast learning loss function, and when it is determined that the contrast learning loss meets the preset condition, the trained neural network model is used as the offline handwritten formula recognition model, thereby using the paired input mode, and guiding the offline handwritten formula recognition model to learn semantic invariant features through the contrast learning loss function, thereby alleviating the influence of large writing style difference on performance and improving the overall recognition accuracy.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a training method, apparatus, device and medium for an offline handwritten formula recognition model. Background Technology

[0002] Handwritten formula recognition has attracted much attention due to its potential applications in scientific literature, education, office automation, and conferencing systems. Based on the type of input pattern, formula recognition methods can be divided into two categories: online formula recognition and offline formula recognition. Online formulas are data sequences of stroke coordinates recorded in real time from devices such as touchpads or styluses that can record trajectory information; they are a series of trajectory points. Offline formulas, on the other hand, generally refer to formula images captured from scanners or cameras.

[0003] In recent years, researchers have treated offline formula recognition as a special case of the image-to-sequence problem and have used offline handwritten formula recognition models with attention mechanisms to solve it. However, due to the local ambiguity of handwritten characters and the large differences in writing styles, current offline handwritten formula recognition models still have relatively low accuracy in formula recognition. For example, in the case of handwriting, many characters have a high degree of similarity, such as "z" and "2", "1" and "(", uppercase and lowercase "o" and "O", "s" and "S", etc., or the same symbol in the same formula may have large differences in writing style among different writers. This requires the model to learn the diversity of writing styles of characters and their semantic invariance. Summary of the Invention

[0004] The purpose of this application is to provide a training method, apparatus, device, and medium for an offline handwritten formula recognition model, so as to improve the accuracy of the offline handwritten formula recognition model in formula recognition.

[0005] To achieve the above objectives, this application provides a training method for an offline handwritten formula recognition model, comprising the following steps:

[0006] Receive images containing handwritten formulas;

[0007] Determine the LaTeX tags of the handwritten formula, and convert the handwritten formula into a printed formula image based on the LaTeX tags;

[0008] The printed formula image and the handwritten formula image are input in pairs into a pre-built neural network model for training;

[0009] The contrastive learning loss of the trained neural network model is calculated according to the preset contrastive learning loss function.

[0010] When the contrastive learning loss is determined to meet the preset conditions, the trained neural network model is used as an offline handwritten formula recognition model.

[0011] This application also provides a training device for an offline handwritten formula recognition model, including:

[0012] The receiving module is used to receive images of handwritten formulas containing handwritten formulas;

[0013] A determination module is used to determine the LaTeX tags of the handwritten formula and convert the handwritten formula into a printed formula image based on the LaTeX tags.

[0014] The training module is used to input the printed formula image and the handwritten formula image in pairs into a pre-built neural network model for training.

[0015] The calculation module is used to calculate the contrastive learning loss of the trained neural network model according to a preset contrastive learning loss function.

[0016] The determination module is used to use the trained neural network model as an offline handwritten formula recognition model when the contrastive learning loss is determined to meet the preset conditions.

[0017] This application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the training method for an offline handwritten formula recognition model as described in any of the above claims.

[0018] This application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the training method for an offline handwritten formula recognition model as described in any of the preceding claims.

[0019] The training method, apparatus, device, and medium for the offline handwritten formula recognition model disclosed in this application involve receiving handwritten formula images containing handwritten formulas, determining the LaTeX tags of the handwritten formulas, and converting the handwritten formulas into printed formula images based on the LaTeX tags. Utilizing the characteristic that many characters have high similarity in handwritten form but are easily distinguishable in printed form, the corresponding printed formula image is determined. The printed formula image and the handwritten formula image are input in pairs into a pre-constructed neural network model for training. The contrastive learning loss of the trained neural network model is calculated according to a preset contrastive learning loss function. When the contrastive learning loss meets a preset condition, the trained neural network model is used as the offline handwritten formula recognition model. This paired input mode can easily and quickly multiply the amount of training data, alleviating the problem of insufficient training data for offline handwritten formula recognition models. Furthermore, the contrastive learning loss function guides the offline handwritten formula recognition model to learn semantically invariant features, thereby mitigating the impact of large differences in handwriting style on performance and improving the overall recognition accuracy. Since only the condition of whether the contrastive learning loss meets the preset condition needs to be determined, no specific training strategy needs to be designed, resulting in low training difficulty and stable convergence. Attached Figure Description

[0020] Figure 1 A schematic diagram illustrating the steps of training an offline handwritten formula recognition model;

[0021] Figure 2 This is a flowchart illustrating a training method for an offline handwritten formula recognition model, mainly showing the conversion process of printed formula images.

[0022] Figure 3 A schematic block diagram of the structure of a training device for an offline handwritten formula recognition model;

[0023] Figure 4 This is a schematic block diagram of the structure of a computer device.

[0024] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0026] Reference Figure 1 This is a schematic diagram illustrating the training steps of an offline handwritten formula recognition model according to an embodiment of this application, using a computer device as the execution entity. The method includes:

[0027] S11. Receive a handwritten formula image containing the handwritten formula;

[0028] S12. Determine the LaTeX tags of the handwritten formula, and convert the handwritten formula into a printed formula image based on the LaTeX tags;

[0029] S13. Input the printed formula image and the handwritten formula image into a pre-built neural network model for training.

[0030] S14. Calculate the contrastive learning loss of the trained neural network model according to the preset contrastive learning loss function;

[0031] S15. When it is determined that the contrastive learning loss meets the preset conditions, the trained neural network model is used as an offline handwritten formula recognition model.

[0032] As described in step S11 above, the handwritten formula includes mathematical formulas, chemical formulas, or physical formulas, etc. The computer device can use a camera to capture the user's handwritten formula, obtaining an image of the handwritten formula; this image can also be obtained directly from a website or image library. After receiving the handwritten formula image, the computer device can also preprocess the image, including noise reduction and adjusting the contrast or brightness of the image.

[0033] As described in step S12 above, LaTeX can easily generate various special symbols using commands, such as the LaTeX tag for "\sqrt{3}" in mathematical formulas. The "\3" in mathematical formulas ^{2} The LaTeX tag for "3" is "3". 2 This embodiment determines the LaTeX tag corresponding to each character in the handwritten formula by querying a database. This database pre-stores the LaTeX tags corresponding to each character. Then, based on the LaTeX tags, the handwritten formula is converted into a printed formula image. In the printed formula image, "\sqrt{3}" is presented as... "\3 ^{2} Presented as “3” 2 ".

[0034] Since different LaTeX labels can express the same mathematical formula, such as "\frac 1 2a^2_1" and "\frac{1}{2}a_1{2}", different styles of training labels can make it difficult for the network to learn the association between images and LaTeX strings. Therefore, the LaTeX labels of existing data can be normalized.

[0035] As described in step S13 above, the neural network model can include a sequence-to-sequence model, which is an Encoder-Decoder neural network with an input sequence and an output sequence. In the Encoder, a variable-length sequence is transformed into a fixed-length vector representation. The Decoder then transforms this fixed-length vector representation into a variable-length target signal sequence, thus achieving a variable-length input to a variable-length output. Sequence-to-sequence models can include various types, such as seq2seq models based on Recurrent Neural Networks (RNNs) and seq2seq models based on Convolution (CONV) operations. This embodiment does not limit the specific type of neural network model.

[0036] Optionally, the neural network model can also be an encoder-decoder network structure based on the attention mechanism. That is, the network mainly consists of two parts: an encoder and a decoder. The encoder generally uses a convolutional neural network (CNN) to extract image features, while the decoder weights the features obtained by the encoder according to the weights obtained by the attention mechanism at each time step, and uses a recurrent neural network (RNN) to decode the weighted features at the current time step. Through decoding at each time step, the conversion of handwritten formula images and printed formula images into LaTeX tags is gradually completed.

[0037] This embodiment can also use CNN and Bidirectional Long Short-term Memory Neural Networks (BLSTM) to encode handwritten formula images and printed formula images, and use Long Short-term Memory Neural Networks (LSTM) as a decoder to decode the encoded features, achieving good results in the handwritten formula problem.

[0038] This embodiment uses paired input patterns to train the neural network model, which can easily and quickly increase the amount of training data exponentially, alleviating the problem of insufficient training data for offline handwritten formula recognition models.

[0039] As described in steps S14-S15 above, the contrastive learning loss function is used to update the encoder parameters. This contrastive loss function can be used to train the encoder of the neural network model, thereby bringing similar characters closer together in their high-dimensional representations after encoding, while simultaneously widening the gap between the high-dimensional representations of unrelated characters. In other words, after training the encoder using the contrastive learning loss function, two originally similar input characters will still have similar features in the feature space after encoding; conversely, two originally dissimilar input characters will still have dissimilar features in the feature space after encoding. This ensures the recognition performance of the trained offline handwritten formula recognition model.

[0040] Specifically, the optimization objective of the contrastive learning loss function is that when the characters in a pair of input printed formula images are similar to the corresponding characters in a handwritten formula image, the vector representations of the two input characters after encoding should also be similar; conversely, when the input characters are dissimilar, the vector representations of the two input characters after encoding should also be dissimilar. Therefore, the computer device can calculate the contrastive learning loss of the trained neural network model based on a preset contrastive learning loss function. When the contrastive learning loss does not meet the preset conditions, the encoder parameters are updated, and the updated encoder is retrained until the contrastive learning loss meets the preset conditions. When the contrastive learning loss is determined to meet the preset conditions, the trained neural network model is used as an offline handwritten formula recognition model.

[0041] The training method for the offline handwritten formula recognition model in this application involves determining the LaTeX tags of the handwritten formulas, converting the handwritten formulas into printed formula images based on the LaTeX tags, and utilizing the characteristic that many characters have high similarity in handwriting but are easily distinguishable in printed form to determine the corresponding printed formula image. The printed formula image and the handwritten formula image are then input in pairs into a pre-built neural network model for training. The contrastive learning loss of the trained neural network model is calculated according to a preset contrastive learning loss function. When the contrastive learning loss meets a preset condition, the trained neural network model is used as the offline handwritten formula recognition model. This paired input mode can easily and quickly increase the amount of training data exponentially, alleviating the problem of insufficient training data for offline handwritten formula recognition models. Furthermore, the contrastive learning loss function guides the offline handwritten formula recognition model to learn semantically invariant features, thereby mitigating the impact of large differences in handwriting style on performance and improving the overall recognition accuracy. Since it only requires determining whether the contrastive learning loss meets the preset condition, no specific training strategy needs to be designed, resulting in low training difficulty and stable convergence.

[0042] In one embodiment, the step of calculating the contrastive learning loss of the neural network model according to a preset contrastive learning loss function may specifically include the following steps:

[0043] Obtain multiple first context vectors of the handwritten formula image and multiple second context vectors of the printed formula image generated during the training of the neural network model;

[0044] Determine the second context vector corresponding to each of the first context vectors;

[0045] The loss value for each first context vector and its corresponding second context vector is calculated using the noise contrast estimation loss function;

[0046] The contrastive learning loss is obtained by summing the loss values ​​of each first context vector and its corresponding second context vector.

[0047] In this embodiment, the computer device can treat the recognition process of each character in the handwritten formula image during neural network model training as one step, and also treat the recognition process of each character in the printed formula image during neural network model training as one step. For the step-by-step decoding process, multiple first context vectors of the T-step handwritten formula image and multiple second context vectors of the printed formula image are obtained, which can be represented as follows: and Then determine the second context vector corresponding to each first context vector, such as... corresponding corresponding corresponding Wait a minute. Then, the contrastive learning loss function is used to make the two feature sets C... p and C h The corresponding features (positive pairs) are closer to each other and farther away from other misaligned features (negative pairs). The loss value of each first context vector and its corresponding second context vector is calculated using the noise contrastive estimation loss function, resulting in multiple loss values. The loss values ​​of each first context vector and its corresponding second context vector are accumulated to obtain the contrastive learning loss.

[0048] In one embodiment, the step of calculating the loss value of each first context vector and its corresponding second context vector using the noise contrast estimation loss function may specifically include the following formula:

[0049]

[0050] Wherein, the u a As the first context vector, u b For u aThe corresponding second context vector, sim(u a ,u b ) is the similarity function, and τ is the temperature parameter. In this embodiment, cosine distance can be used as the similarity function:

[0051]

[0052] In one embodiment, the step of summing the loss values ​​of each first context vector and its corresponding second context vector to obtain the contrastive learning loss may specifically include the following formula:

[0053]

[0054] In one embodiment, the step of determining that the contrastive learning loss meets a preset condition may specifically include the following steps:

[0055] Calculate the first loss value corresponding to the printed formula image and the second loss value corresponding to the handwritten formula image;

[0056] Obtain the hyperparameters pre-set for the contrastive learning loss;

[0057] The total loss is obtained by summing the product of the contrastive learning loss and the hyperparameter, the first loss value and the second loss value.

[0058] When the total loss is determined to be lower than a preset loss threshold, the contrastive learning loss is determined to meet the preset condition.

[0059] In this embodiment, printed formula images and handwritten formula images are input in pairs into a neural network model for training. A preset loss function can be used to calculate the first loss value corresponding to the printed formula image. And the second loss value corresponding to the handwritten formula image. This loss function is used to evaluate the degree to which the model's predictions differ from the actual values. A better loss function generally indicates better model performance. Different models typically use different loss functions.

[0060] Obtain the hyperparameters pre-set for the contrastive learning loss. These hyperparameters are parameters set before the neural network model begins training, not parameters obtained through training data. Typically, hyperparameters need to be optimized to select an optimal set for the neural network model to improve learning performance and effectiveness.

[0061] The contrastive learning loss is then multiplied by the hyperparameters to obtain the product of the contrastive learning loss and the hyperparameters. This product, along with the first and second loss values, is summed to obtain the total loss. If the total loss is determined to be lower than a preset loss threshold, the contrastive learning loss is considered to meet the preset condition, resulting in a trained offline handwritten formula recognition model. The preset loss threshold can be customized, for example, set to 0.8. It should be noted that the lower the total loss, the better the performance of the trained offline handwritten formula recognition model and the higher the accuracy of recognizing handwritten formulas.

[0062] The step of obtaining the total loss by multiplying the contrastive learning loss by the hyperparameter and summing the first loss value and the second loss value includes the following formula:

[0063]

[0064] Among them, the The first loss value, The second loss value, λL cons This is the product of the contrastive learning loss and the hyperparameter. Here, λ is a hyperparameter used to balance the recognition loss (first loss value and second loss value) and the contrastive learning loss, and can be set to 0.1.

[0065] In one embodiment, the step of inputting the printed formula image and the handwritten formula image in pairs into a pre-built neural network model for training may specifically include the following steps:

[0066] The handwritten formula image is subjected to data augmentation processing to obtain a standard handwritten formula image;

[0067] The printed formula image and the standard handwritten formula image are input in pairs into a pre-built neural network model for training.

[0068] The data processing in this embodiment can be to perform data augmentation on the handwritten formula image to obtain a standard handwritten formula image. For example, data augmentation can be performed by using conventional data augmentation methods such as rotation, tilting, stretching, distortion, adding white noise, or color space transformation with a certain probability. It can be one of these data augmentation methods or a combination of two or more data augmentation methods.

[0069] In addition, the dataset can be expanded by adding handwritten formula images to increase training samples. By combining the paired input patterns of printed formula images and handwritten formula images, the amount of training data can be increased exponentially in a simple and quick manner, alleviating the problem of insufficient training data for offline handwritten formula recognition models.

[0070] The dataset augmentation strategy mainly utilizes the stroke information of each character in the online data. During the process of converting the online data into offline data, new handwritten formula image samples are generated by randomly replacing characters and deforming the characters and the entire handwritten formula. These samples have different LaTeX labels than the original samples.

[0071] In one embodiment, the step of inputting the printed formula image and the handwritten formula image in pairs into a pre-built neural network model for training may specifically include the following steps:

[0072] Identify the blank areas of the printed formula image, remove the blank areas, and obtain the first printed formula image;

[0073] The size of the first printed formula image is adjusted to be the same as the size of the handwritten formula image to obtain the second printed formula image;

[0074] The second printed formula image and the handwritten formula image are input in pairs into a pre-built neural network model for training.

[0075] refer to Figure 2 As shown, this embodiment can render a printed formula image based on the LaTeX tags of a handwritten formula. Specifically, the rendering method involves using the matplotlib or pdflatex library to render the LaTeX tags into a printed image. This image contains a large number of blank areas, so it is necessary to determine these blank areas, remove them, and obtain a first printed formula image. Then, the accurate character area image of the first printed formula image is obtained. The size of the first printed formula image is adjusted to be the same as the size of the handwritten formula image to obtain a second printed formula image. For example, if the size of the first printed formula image is smaller than the size of the handwritten formula image, the first printed formula image is enlarged until the two images are the same size. If the size of the first printed formula image is larger than the size of the handwritten formula image, the first printed formula image is reduced until the two images are the same size.

[0076] In one embodiment, the step of converting the handwritten formula into a printed formula image based on the LaTeX tag may specifically include the following steps:

[0077] The LaTeX tags of the handwritten formula are rendered using the matplotlib or pdflatex library to obtain the printed formula image.

[0078] This embodiment can render a printed formula image from the LaTeX tags of a handwritten formula. Specifically, the rendering method involves using the matplotlib or pdflatex library to render the LaTeX tags of the handwritten formula, resulting in a printed formula image. Matplotlib is a third-party library in Python specifically designed for data visualization and is also the most popular plotting library.

[0079] Reference Figure 3 This is a structural block diagram of a training device for an offline handwritten formula recognition model disclosed in this application. The device includes:

[0080] Receiver module 11 is used to receive images of handwritten formulas containing handwritten formulas;

[0081] The determining module 12 is used to determine the LaTeX tags of the handwritten formula and convert the handwritten formula into a printed formula image based on the LaTeX tags;

[0082] Training module 13 is used to input the printed formula image and the handwritten formula image in pairs into a pre-built neural network model for training;

[0083] Calculation module 14 is used to calculate the contrastive learning loss of the trained neural network model according to a preset contrastive learning loss function;

[0084] The determination module 15 is used to use the trained neural network model as an offline handwritten formula recognition model when the contrastive learning loss is determined to meet the preset conditions.

[0085] In one embodiment, the computing module 14 is specifically configured as follows:

[0086] Obtain multiple first context vectors of the handwritten formula image and multiple second context vectors of the printed formula image generated during the training of the neural network model;

[0087] Determine the second context vector corresponding to each of the first context vectors;

[0088] The loss value for each first context vector and its corresponding second context vector is calculated using the noise contrast estimation loss function;

[0089] The contrastive learning loss is obtained by summing the loss values ​​of each first context vector and its corresponding second context vector.

[0090] In one embodiment, the computing module 14 is specifically configured as follows:

[0091]

[0092] Wherein, the u a As the first context vector, u b For u a The corresponding second context vector, sim(u a ,u b ) is the similarity function, and τ is the temperature parameter.

[0093] In one embodiment, the determination module 15 is specifically configured as follows:

[0094] Calculate the first loss value corresponding to the printed formula image and the second loss value corresponding to the handwritten formula image;

[0095] Obtain the hyperparameters pre-set for the contrastive learning loss;

[0096] The total loss is obtained by summing the product of the contrastive learning loss and the hyperparameter, the first loss value and the second loss value.

[0097] When the total loss is determined to be lower than a preset loss threshold, the contrastive learning loss is determined to meet the preset condition.

[0098] In one embodiment, the training module 13 is specifically configured as follows:

[0099] The handwritten formula image is subjected to data augmentation processing to obtain a standard handwritten formula image;

[0100] The printed formula image and the standard handwritten formula image are input in pairs into a pre-built neural network model for training.

[0101] In one embodiment, the training module 13 is specifically configured as follows:

[0102] Identify the blank areas of the printed formula image, remove the blank areas, and obtain the first printed formula image;

[0103] The size of the first printed formula image is adjusted to be the same as the size of the handwritten formula image to obtain the second printed formula image;

[0104] The second printed formula image and the handwritten formula image are input in pairs into a pre-built neural network model for training.

[0105] In one embodiment, the determining module 12 is specifically configured as follows:

[0106] The LaTeX tags of the handwritten formula are rendered using the matplotlib or pdflatex library to obtain the printed formula image.

[0107] As described above, it is understood that each component of the training device for the offline handwritten formula recognition model proposed in this application can realize the function of any of the above-mentioned training methods for the offline handwritten formula recognition model, and the specific structure will not be described in detail.

[0108] Reference Figure 4 This application also provides a computer device whose internal structure can be as follows: Figure 4 As shown, the computer device includes a processor, memory, and network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a training method for an offline handwritten formula recognition model. The training method includes: receiving a handwritten formula image containing a handwritten formula; determining the LaTeX tag of the handwritten formula; converting the handwritten formula into a printed formula image based on the LaTeX tag; inputting the printed formula image and the handwritten formula image in pairs into a pre-constructed neural network model for training; calculating the contrastive learning loss of the trained neural network model according to a preset contrastive learning loss function; and when the contrastive learning loss meets a preset condition, using the trained neural network model as an offline handwritten formula recognition model.

[0109] One embodiment of this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements a training method for an offline handwritten formula recognition model. It is understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium. The training method for the offline handwritten formula recognition model includes: receiving a handwritten formula image containing a handwritten formula; determining the LaTeX tag of the handwritten formula; converting the handwritten formula into a printed formula image based on the LaTeX tag; inputting the printed formula image and the handwritten formula image in pairs into a pre-constructed neural network model for training; calculating the contrastive learning loss of the trained neural network model according to a preset contrastive learning loss function; and when the contrastive learning loss satisfies a preset condition, using the trained neural network model as an offline handwritten formula recognition model.

[0110] In summary, the training method, apparatus, device, and medium for the offline handwritten formula recognition model of this application receive handwritten formula images containing handwritten formulas, determine the LaTeX tags of the handwritten formulas, convert the handwritten formulas into printed formula images based on the LaTeX tags, and utilize the characteristic that many characters have high similarity in handwritten form but are easily distinguishable in printed form to determine the corresponding printed formula image for the handwritten formula image; the printed formula image and the handwritten formula image are input in pairs into a pre-constructed neural network model for training, and the trained neural network model is calculated according to a preset contrastive learning loss function. The contrastive learning loss model is used to train a neural network model as an offline handwritten formula recognition model when the contrastive learning loss meets the preset conditions. This allows for the use of paired input patterns, which can quickly and easily increase the amount of training data, alleviating the problem of insufficient training data for offline handwritten formula recognition models. Furthermore, the contrastive learning loss function guides the offline handwritten formula recognition model to learn semantically invariant features, thereby mitigating the impact of large differences in handwriting styles on performance and improving the overall recognition accuracy. Since it is only necessary to determine whether the contrastive learning loss meets the preset conditions, no specific training strategy needs to be designed, resulting in low training difficulty and stable convergence.

[0111] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media provided in this application and in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-speed SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0112] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0113] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for training an offline handwritten formula recognition model, characterized in that, The method comprises the following steps: receiving a handwritten formula image containing a handwritten formula; determining a LaTeX label of the handwritten formula, and converting the handwritten formula into a printed formula image according to the LaTeX label; inputting the printed formula image and the handwritten formula image into a pre-constructed neural network model for training; calculating a contrastive learning loss of the trained neural network model according to a preset contrastive learning loss function; when it is determined that the contrastive learning loss meets a preset condition, taking the trained neural network model as an offline handwritten formula recognition model; the step of calculating the contrastive learning loss of the neural network model according to the preset contrastive learning loss function comprises the following steps: obtaining a plurality of first context vectors of the handwritten formula image and a plurality of second context vectors of the printed formula image generated during training of the neural network model; determining a second context vector corresponding to each first context vector; calculating a loss value of each first context vector and the corresponding second context vector by using a noise contrast estimation loss function; after accumulating the loss values of each first context vector and the corresponding second context vector, a contrastive learning loss is obtained. 2.The method of Claim 1, wherein, the step of calculating the loss value of each first context vector and the corresponding second context vector by using the noise contrast estimation loss function comprises the following formula: Wherein, the u a As the first context vector, u b For u a The corresponding second context vector, sim(u a ,u b ) is the similarity function, and τ is the temperature parameter. 3.The method of Claim 1, wherein, the step of determining that the contrastive learning loss meets the preset condition comprises the following steps: calculating a first loss value corresponding to the printed formula image and a second loss value corresponding to the handwritten formula image; obtaining a hyperparameter preset for the contrastive learning loss; after accumulating the product of the contrastive learning loss and the hyperparameter and the first loss value and the second loss value, a total loss is obtained; when it is determined that the total loss is lower than a preset loss threshold, it is determined that the contrastive learning loss meets the preset condition. 4.The method of Claim 1, wherein the step of inputting the printed formula image and the handwritten formula image into the pre-constructed neural network model for training comprises the following steps: performing data enhancement processing on the handwritten formula image to obtain a standard handwritten formula image; inputting the printed formula image and the standard handwritten formula image into the pre-constructed neural network model for training. 5.The method of Claim 1, wherein, the step of inputting the printed formula image and the handwritten formula image into the pre-constructed neural network model for training comprises the following steps: determining a blank area of the printed formula image, removing the blank area, and obtaining a first printed formula image; adjusting the size of the first printed formula image to be the same as the size of the handwritten formula image to obtain a second printed formula image; inputting the second printed formula image and the handwritten formula image into the pre-constructed neural network model for training. 6.The method of Claim 1, wherein the step of converting the handwritten formula into a printed formula image according to the LaTeX label comprises the following steps: rendering the LaTeX label of the handwritten formula by using a matplotlib or pdflatex library to obtain the printed formula image.

7. An apparatus for training an offline handwritten formula recognition model, configured to implement the method of any one of claims 1-6. The method comprises the steps of: receiving a handwritten formula image containing a handwritten formula; determining a LaTeX label of the handwritten formula, and converting the handwritten formula into a printed formula image according to the LaTeX label; training a pre-constructed neural network model by inputting the printed formula image and the handwritten formula image into the neural network model; calculating a contrastive learning loss of the trained neural network model according to a preset contrastive learning loss function; determining that the trained neural network model is an offline handwritten formula recognition model when the contrastive learning loss meets a preset condition.

8. A computer device comprising a memory and a processor, the memory having stored therein a computer program, characterized in that, The processor executes the computer program to implement the steps of the training method of the offline handwritten formula recognition model according to any one of claims 1 to 6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the training method of the offline handwritten formula recognition model according to any one of claims 1 to 6.

Images