Training method and device for formula recognition model and device for model training
By using iterative training methods of encoder and decoder in the formula recognition model, the problems of error accumulation and texture information ignorance in the prior art are solved, and the accuracy and accuracy of formula recognition are improved.
Patent Information
- Application Number
- CN202111156295.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-29
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-09-29
AI Technical Summary
In the existing formula recognition algorithm, the accumulation of errors of the staged recognition algorithm leads to low accuracy, while the end-to-end algorithm ignores the underlying texture information, resulting in poor recognition effect.
The formula recognition model including an encoder and a decoder is adopted. Through iterative training and joint loss value adjustment, the feature extraction function of each encoding module is fully considered, and feature information of each level of the formula recognition sample is introduced for training.
It improves the recognition accuracy and accuracy of the formula recognition model, provides more comprehensive feature information processing, and improves the accuracy of the recognition results.
Smart Images

Figure CN113822369B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of intelligent control technology, and in particular to a training method and device for a formula recognition model and a device for model training. Background Art
[0002] Formula recognition technology has a wide range of application needs in education fields such as automatic examination paper grading, photo search, and test digitization. Since the formula itself may have a complex two-dimensional structure, numerous symbols, similar symbols, and diverse meanings, the accuracy of formula recognition is not high.
[0003] In recent years, with the significant improvement of computer computing power and the advent of the big data era, neural network-based formula recognition algorithms have been widely used in various formula recognition scenarios. At present, there are two common formula recognition algorithms: one is a phased recognition algorithm based on the "segmentation-recognition-reprocessing" process, and the other is an end-to-end formula recognition algorithm.
[0004] However, in the staged recognition algorithm, each stage will cause errors, and the errors in each stage will accumulate, resulting in poor final accuracy; in the end-to-end formula recognition algorithm, usually only high-level semantic information is used for formula recognition, ignoring the underlying texture and other information, resulting in poor recognition effect. Summary of the invention
[0005] The embodiments of the present invention provide a training method and device for a formula recognition model and a device for model training, which can improve the recognition precision and accuracy of the formula recognition model.
[0006] In order to solve the above problems, an embodiment of the present invention discloses a training method for a formula recognition model, the method comprising:
[0007] Obtaining a formula recognition sample and a target recognition result corresponding to the formula recognition sample;
[0008] Based on the formula recognition samples, iteratively training a formula recognition model, wherein the formula recognition model includes an encoder and a decoder, and the encoder includes at least one encoding module;
[0009] In each round of training, according to the formula recognition results corresponding to each encoding module output by the decoder, a joint loss value of the formula recognition model is calculated, and model parameters of the formula recognition model are adjusted according to the joint loss value;
[0010] When the joint loss value meets the convergence condition, a trained formula recognition model is obtained.
[0011] Optionally, the formula recognition model includes N serially connected encoding modules, where N is a positive integer greater than 1, and in each round of training, according to the formula recognition result corresponding to each encoding module output by the decoder, calculating the joint loss value of the formula recognition model includes:
[0012] In each round of training, the formula recognition sample is input into the encoder of the formula recognition model, and the formula recognition sample is encoded in turn by the N encoding modules of the encoder to obtain N encoding features corresponding to the formula recognition sample;
[0013] According to the formula, N coding features corresponding to the sample are identified to determine M target coding features; wherein M is a positive integer less than or equal to N;
[0014] Inputting the M target coding features corresponding to the formula recognition sample into the decoder of the formula recognition model respectively, and decoding the M target coding features by the decoder to obtain M formula recognition results corresponding to the formula recognition sample;
[0015] The joint loss value of the formula recognition model is calculated according to the M formula recognition results corresponding to the formula recognition sample and the target recognition result.
[0016] Optionally, calculating the joint loss value of the formula recognition model according to the M formula recognition results corresponding to the formula recognition sample and the target recognition result includes:
[0017] According to each formula recognition result of the M formula recognition results corresponding to the formula recognition sample and the target recognition result, respectively calculating the loss value corresponding to each formula recognition result;
[0018] According to the loss value corresponding to each formula recognition result, the joint loss value of the formula recognition model is calculated.
[0019] Optionally, determining M target coding features by identifying N coding features corresponding to the sample according to the formula includes:
[0020] Fusing the coding feature output by the Nth coding module of the encoder and the coding feature output by the N-1th coding module to obtain a fused coding feature;
[0021] The fused coding features and the coding features output by the first N-2 coding modules of the encoder are used as target coding features.
[0022] Optionally, fusing the coding feature output by the Nth coding module of the encoder and the coding feature output by the N-1th coding module to obtain the fused coding feature includes:
[0023] Performing upsampling processing on the coding feature output by the Nth coding module of the encoder to obtain a first coding feature, where the dimension of the first coding feature is the same as the dimension of the coding feature output by the N-1th coding module of the encoder;
[0024] The first coding feature and the coding feature output by the N-1th coding module are fused to obtain a fused coding feature.
[0025] Optionally, fusing the coding feature output by the Nth coding module of the encoder and the coding feature output by the N-1th coding module to obtain the fused coding feature includes:
[0026] Downsampling the encoding feature output by the N-1th encoding module of the encoder to obtain a second encoding feature, where the dimension of the second encoding feature is the same as the dimension of the encoding feature output by the Nth encoding module of the encoder;
[0027] The second coding feature and the coding feature output by the Nth coding module are fused to obtain a fused coding feature.
[0028] Optionally, the method further comprises:
[0029] Acquire a target image, wherein the target image contains a formula to be recognized;
[0030] The target image is input into the trained formula recognition model for formula recognition processing to obtain a formula recognition result corresponding to the target image.
[0031] On the other hand, an embodiment of the present invention discloses a training device for a formula recognition model, the device comprising:
[0032] A training data acquisition module, used to acquire formula recognition samples and target recognition results corresponding to the formula recognition samples;
[0033] An iterative training module, used for iteratively training a formula recognition model based on the formula recognition samples, wherein the formula recognition model includes an encoder and a decoder, and the encoder includes at least one encoding module;
[0034] A joint loss value calculation module, used to calculate the joint loss value of the formula recognition model according to the formula recognition result corresponding to each encoding module output by the decoder in each round of training, and adjust the model parameters of the formula recognition model according to the joint loss value;
[0035] The training completion condition determination module is used to obtain a training completed formula recognition model when the joint loss value meets the convergence condition.
[0036] Optionally, the formula recognition model includes N serially connected encoding modules, where N is a positive integer greater than 1, and the joint loss value calculation module includes:
[0037] The encoding processing submodule is used to input the formula recognition sample into the encoder of the formula recognition model in each round of training, and sequentially encode the formula recognition sample through the N encoding modules of the encoder to obtain N encoding features corresponding to the formula recognition sample;
[0038] A target coding feature determination submodule is used to determine M target coding features by identifying N coding features corresponding to the sample according to the formula; wherein M is a positive integer less than or equal to N;
[0039] A decoding processing submodule, used for inputting the M target coding features corresponding to the formula recognition sample into the decoder of the formula recognition model respectively, and performing decoding processing on the M target coding features by the decoder to obtain M formula recognition results corresponding to the formula recognition sample;
[0040] The joint loss value calculation submodule is used to calculate the joint loss value of the formula recognition model according to the M formula recognition results corresponding to the formula recognition sample and the target recognition result.
[0041] Optionally, the joint loss value calculation submodule includes:
[0042] A first loss value calculation unit, configured to calculate a loss value corresponding to each formula recognition result according to each formula recognition result of the M formula recognition results corresponding to the formula recognition sample and the target recognition result;
[0043] The second loss value calculation unit is used to calculate the joint loss value of the formula recognition model according to the loss value corresponding to each formula recognition result.
[0044] Optionally, the target coding feature determination submodule includes:
[0045] A coding feature fusion unit, used to fuse the coding feature output by the Nth coding module of the encoder and the coding feature output by the N-1th coding module to obtain a fused coding feature;
[0046] A target coding feature determination unit is used to use the fused coding feature and the coding features output by the first N-2 coding modules of the encoder as the target coding feature.
[0047] Optionally, the coding feature fusion unit includes:
[0048] an upsampling processing subunit, configured to perform upsampling processing on the coding feature output by the Nth coding module of the encoder to obtain a first coding feature, wherein the dimension of the first coding feature is the same as the dimension of the coding feature output by the N-1th coding module of the encoder;
[0049] The first coding feature fusion subunit is used to fuse the first coding feature with the coding feature output by the N-1th coding module to obtain a fused coding feature.
[0050] Optionally, the coding feature fusion unit includes:
[0051] a downsampling processing subunit, configured to perform downsampling processing on the coding feature output by the N-1th coding module of the encoder to obtain a second coding feature, wherein the dimension of the second coding feature is the same as the dimension of the coding feature output by the Nth coding module of the encoder;
[0052] The second coding feature fusion subunit is used to fuse the second coding feature with the coding feature output by the Nth coding module to obtain a fused coding feature.
[0053] Optionally, the device further comprises:
[0054] A target image acquisition module, used to acquire a target image, wherein the target image contains a formula to be identified;
[0055] The formula recognition processing module is used to input the target image into the trained formula recognition model for formula recognition processing to obtain the formula recognition result corresponding to the target image.
[0056] On the other hand, an embodiment of the present invention discloses a device for model training, which includes a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors. The one or more programs include instructions for executing a training method for a formula recognition model as described above.
[0057] On the other hand, an embodiment of the present invention discloses a machine-readable medium having instructions stored thereon, which, when executed by one or more processors, enables the device to perform one or more of the aforementioned methods for training a formula recognition model.
[0058] The embodiments of the present invention include the following advantages:
[0059] The embodiment of the present invention obtains formula recognition samples and target recognition results corresponding to the formula recognition samples; then, based on the formula recognition samples, iteratively trains a formula recognition model, the formula recognition model includes an encoder and a decoder, and the encoder includes at least one encoding module; in each round of training, according to the formula recognition results corresponding to each encoding module output by the decoder, the joint loss value of the formula recognition model is calculated, and the model parameters of the formula recognition model are adjusted according to the joint loss value; when the joint loss value meets the convergence condition, a trained formula recognition model is obtained.
[0060] Compared with the prior art, for a formula recognition sample, the encoder of the formula recognition model only outputs one coding feature, and only one formula recognition result is obtained through the decoding process of the decoder. The embodiment of the present invention extracts the coding features output by each coding module, decodes the coding features output by each coding module, obtains the formula recognition result corresponding to each coding module, and calculates the joint loss value of the formula recognition model according to the formula recognition result corresponding to each coding module. Different coding modules output different coding features, and the corresponding formula recognition results are also different. Therefore, the coding features extracted by each coding module have an impact on the joint loss value of the formula recognition model. The formula recognition model is trained based on the joint loss value, and the feature extraction function of each coding module can be fully considered in the training process of the formula recognition model. The feature information of each level of the formula recognition sample is introduced to train the formula recognition model. The trained formula recognition model has higher recognition accuracy and more accurate recognition results. BRIEF DESCRIPTION OF THE DRAWINGS
[0061] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0062] Figure 1 is a schematic diagram of the structure of a computer system of the present invention;
[0063] Figure 2 It is a flowchart of the steps of an embodiment of a training method of a formula recognition model of the present invention;
[0064] Figure 3 It is a structural schematic diagram of a formula recognition model of the present invention;
[0065] Figure 4 It is a structural block diagram of an embodiment of a training device for a formula recognition model of the present invention;
[0066] Figure 5 is a block diagram of a device 800 for model training of the present invention;
[0067] Figure 6 It is a schematic diagram of the structure of the server in some embodiments of the present invention. DETAILED DESCRIPTION
[0068] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0069] Method Embodiment
[0070] The embodiment of the present invention provides a method for training a formula recognition model, which can be applied to a computer device with data processing capabilities. In an optional embodiment, the method for training a formula recognition model provided by the embodiment of the present invention can be applied to a personal computer, a workstation or a server, that is, formula recognition and training of a formula recognition model can be implemented by a personal computer, a workstation or a server.
[0071] The trained formula recognition model can become a part of the application and be installed in the terminal, so that the terminal can output the formula recognition result when receiving the content to be recognized; or, the trained formula recognition model is set in the background server of the application, so that the terminal with the application installed can realize the formula recognition function with the help of the background server.
[0072] Reference Figure 1 , shows a schematic diagram of the structure of a computer system provided by an embodiment of the present invention, the computer system includes a terminal 110 and a server 120. The terminal 110 and the server 120 perform data communication via a communication network. Optionally, the communication network may be a wired network or a wireless network, and the communication network may be at least one of a local area network, a metropolitan area network, and a wide area network.
[0073] The terminal 110 is installed with an application supporting the formula recognition function, which may be an e-book reading application, a web browsing application, a paper marking application, a photo search application, a social application, etc., which is not specifically limited in the embodiment of the present invention.
[0074] Optionally, the terminal 110 may be a mobile terminal such as a smart phone, a smart watch, a tablet computer, a laptop computer, an intelligent robot, or the like, or a terminal such as a desktop computer or a projection computer. The embodiment of the present invention does not limit the type of terminal.
[0075] The server 120 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, cloud communications, network services, middleware services, content delivery networks (CDN), and big data and artificial intelligence platforms. In an optional embodiment, the server 120 may be a background server of an application in the terminal 110.
[0076] In some embodiments, the terminal 110 includes a camera component; the terminal 110 obtains picture content through the camera component, and the picture content includes the formula to be recognized, and the terminal 110 sends the picture to the server 120. The server 120 includes a formula recognition module, and the formula recognition module includes a trained formula recognition model; the server 120 receives the picture sent by the terminal 110, recognizes the formula in the picture through the formula recognition model, and sends the recognition result to the terminal 110 for display.
[0077] Alternatively, the terminal 110 includes a formula recognition module, which includes a trained formula recognition model. After acquiring the picture, the terminal recognizes the formula in the picture through the formula recognition model to obtain a recognition result, and displays the recognition result.
[0078] It should be noted that, in the above embodiments, the display method of the terminal includes picture form or text form.
[0079] For the convenience of description, the following embodiments are described by taking the training method of the formula recognition model executed by the server as an example.
[0080] Reference Figure 2 , shows a flowchart of a training method embodiment of a formula recognition model of the present invention, and the method may specifically include the following steps:
[0081] Step 201: Obtain a formula recognition sample and a target recognition result corresponding to the formula recognition sample.
[0082] Step 202: iteratively train a formula recognition model based on the formula recognition samples, wherein the formula recognition model includes an encoder and a decoder, and the encoder includes at least one encoding module.
[0083] Step 203: In each round of training, the joint loss value of the formula recognition model is calculated according to the formula recognition results corresponding to each encoding module output by the decoder, and the model parameters of the formula recognition model are adjusted according to the joint loss value.
[0084] Step 204: When the joint loss value meets the convergence condition, a trained formula recognition model is obtained.
[0085] The formula recognition sample is an image used for training, and the image contains the formula to be recognized. The target recognition result corresponding to the formula recognition sample is the real recognition result of the formula recognition sample, which is used to verify the accuracy of the output result of the formula recognition model.
[0086] There are many ways to obtain formula recognition samples and target labels corresponding to the formula recognition samples. For example, existing formula images and formula recognition results on the Internet, formula images and recognition results stored on computer devices, etc. can all be used as formula recognition samples in embodiments of the present invention.
[0087] It should be noted that the embodiment of the present invention does not specifically limit the size of the formula recognition sample. In order to speed up the model training speed, the formula recognition sample can be preprocessed in advance. Exemplarily, the image height of each formula recognition sample can be uniformly adjusted to a preset height, for example, the image height of each formula recognition sample is fixed to 64 pixels; then, based on the height adjustment ratio of each formula recognition sample, the image width is scaled proportionally. For example, for formula recognition sample A, its image height before preprocessing is 32 pixels and its image width is 128 pixels. The image height of formula recognition sample A is adjusted to 64 pixels, and the image height of formula recognition sample A is stretched by 2 times. Therefore, its image width is also stretched by 2 times and adjusted to 256 pixels. In order to avoid the large difference in the size of the formula recognition samples during the training process, which increases the difficulty of training, the scaled formula recognition samples can be sorted in order from small to large or from large to small according to the image width, and the adjacent N formula recognition samples are taken as a sample group, and the formula recognition samples of the same sample group are synchronously processed.
[0088] The formula recognition model is used to identify the formula content in the input image data. In an embodiment of the present invention, the formula recognition model can be constructed using an end-to-end network structure. Exemplarily, the formula recognition model can adopt an Encoder-Decoder structure, including an encoder and a decoder, wherein the encoder may include at least one encoding module, and the encoding module is used to encode the input data. Moreover, each encoding module of the encoder can adopt the same network structure. The embodiment of the present invention does not specifically limit the network structure of the encoding module.
[0089] In each round of training, the formula recognition sample to be processed is input into the encoder of the formula recognition model, and each encoding module in the encoder performs joint encoding processing on the formula recognition sample to obtain the encoding features output by each encoding module. The decoder is used to decode the encoding features output by each encoding module to obtain the formula recognition result corresponding to each encoding module. According to the formula recognition result corresponding to each encoding module and the target recognition result corresponding to the formula recognition sample, the joint loss value of the formula recognition model is calculated, and the model parameters of the formula recognition model are adjusted according to the joint loss value until the convergence condition is met to obtain the trained formula recognition model. Among them, the joint loss value can be determined based on the cross entropy of the formula recognition result output by the formula recognition model and the target recognition result. The convergence condition can be that in multiple rounds of training, the joint loss value of the formula recognition model is less than a preset threshold, or the error between the joint loss values is less than a preset value.
[0090] In the prior art, for a formula recognition sample, the encoder only outputs one encoding result, which is the output result of the last encoding layer of the encoder and contains high-level semantic information of the formula recognition sample, while the underlying texture and other information of the formula recognition sample has been ignored after layer-by-layer encoding processing. This results in the feature information referenced by the formula recognition result obtained by the decoder for a coding result output by the encoder being incomplete and the recognition accuracy being poor.
[0091] Compared with the prior art, the present invention extracts the coding features output by each coding module, decodes the coding features output by each coding module, obtains the formula recognition result corresponding to each coding module, and calculates the joint loss value of the formula recognition model according to the formula recognition result corresponding to each coding module. Different coding modules output different coding features, and the corresponding formula recognition results are also different. Therefore, the coding features extracted by each coding module have an impact on the joint loss value of the formula recognition model. The formula recognition model trained based on the joint loss value makes full use of the feature extraction function of each coding module, and can simultaneously use the feature information extracted by each coding module, thereby improving the integrity of the feature information used by the formula recognition model, which is conducive to improving the accuracy of formula recognition.
[0092] In an optional embodiment of the present invention, the formula recognition model in step 203 includes N serially connected encoding modules, where N is a positive integer greater than 1, and in each round of training, according to the formula recognition result corresponding to each encoding module output by the decoder, the joint loss value of the formula recognition model is calculated, including:
[0093] Step S11: In each round of training, the formula recognition sample is input into the encoder of the formula recognition model, and the formula recognition sample is encoded in turn by the N encoding modules of the encoder to obtain N encoding features corresponding to the formula recognition sample;
[0094] Step S12, determining M target coding features by identifying N coding features corresponding to the sample according to the formula; wherein M is a positive integer less than or equal to N;
[0095] Step S13, inputting the M target coding features corresponding to the formula recognition sample into the decoder of the formula recognition model respectively, and decoding the M target coding features by the decoder to obtain M formula recognition results corresponding to the formula recognition sample;
[0096] Step S14: Calculate the joint loss value of the formula recognition model according to the M formula recognition results corresponding to the formula recognition sample and the target recognition result.
[0097] In an embodiment of the present invention, when the number of coding modules in the encoder is greater than 1, the coding modules in the encoder can be connected in series, and the formula recognition samples input to the encoder are jointly processed based on the series-connected coding modules to obtain the coding features output by each coding module. Then, the decoder based on the formula recognition model decodes the coding features output by each coding module to obtain the formula recognition results corresponding to each coding module.
[0098] It should be noted that the essence of recognizing the input formula image through the formula recognition model is to recognize the semantics of the formula image. The semantics of the image are divided into the visual layer, the object layer and the conceptual layer. Among them, the visual layer contains the underlying feature semantics such as image color, texture, shape, etc.; the middle layer of the object layer usually contains attribute features, that is, the state of an object at a certain moment; the conceptual layer contains high-level semantic features, which reflect the semantic information expressed by the image as a whole. Taking the formula image as an example, the underlying feature semantics can accurately reflect the contours, positional relationships and other information of each character in the formula, and the high-level feature semantics can reflect the meaning expressed by each character that makes up the formula, that is, the mathematical concept expressed by the formula.
[0099] The encoding module encodes the input data, which is essentially to extract features from the input data. Taking the encoding module using CNN (Convolution nNeural Networks) as an example, the encoding module extracts multiple features in the input data into one feature by convolution processing on the input data, and the extracted features form a feature map of the input data. In an embodiment of the present invention, the first encoding module convolves the input formula recognition sample to obtain a feature map, and the other encoding modules convolve the feature map output by the previous encoding module to obtain the next feature map. The more times the convolution is performed, the deeper the level of feature information reflected by the feature map, and the smaller the size of the feature map.
[0100] Assume that the encoder includes 5 encoding modules, that is, N=5, refer to Figure 3 , shows a schematic diagram of the structure of a formula recognition model according to an embodiment of the present invention. Figure 3 As shown, five encoding modules are connected in series, the first encoding module B1 encodes the input formula recognition sample, and encoding modules B2-B5 re-encode the output data of the previous encoding module.
[0101] Assuming that the height of the input formula recognition sample is h and the width is w, the feature map F output by each encoding module is i The size of can be expressed as:
[0102]
[0103] Where i = (0, 1, 2, 3, 4), F i Represents the feature map output by the nth encoding module, n=i+1.
[0104] According to the above formula (1), the size of the feature map F0 output by the first encoding module B1 is Compared with the formula recognition sample, the height and width are both halved; the size of the feature map F1 output by the second encoding module B2 is Compared with the feature map F0, the height and width are both halved; and so on, the height and width of each feature map are halved compared with the previous feature map.
[0105] The more times the convolution is performed, the deeper the level of feature information is. Therefore, in feature maps F0 to F4, feature maps F0 and F1 reflect the underlying feature semantics of the image color, texture, shape, etc. of the formula recognition sample, which can be used to determine the contour, position relationship, etc. of each character in the formula sample; feature maps F3 and F4 reflect the high-level semantic features of the formula recognition sample, which can be used to determine the mathematical concept corresponding to the formula in the formula recognition sample.
[0106] like Figure 3 As shown, in an embodiment of the present invention, a decoding input layer can be added between the encoder and the decoder to determine M target coding features according to the formula for identifying N coding features corresponding to the sample. Wherein, M is a positive integer less than or equal to N. Specifically, M=N can be directly taken, that is, the N coding features are not processed; or, M coding features can be selected from the N coding features as target coding features according to preset rules; or, at least two coding features can be feature fused to obtain fused coding features, and the fused coding features and other coding features can be used as target coding features, and so on.
[0107] Take M=N as an example, Figure 3 As shown, the M target coding features are respectively input into the decoder of the formula recognition model for decoding processing, that is, the feature graphs F0 to F4 are decoded at the same time to obtain the formula recognition results corresponding to each feature graph. The feature graph corresponding to each formula recognition result is different, and the proportion of feature information affecting the recognition result is also different. The joint loss value of the formula recognition model is calculated based on the M formula recognition results and the target recognition result. The feature information of each level corresponding to the formula recognition sample will have an impact on the joint loss value. The formula recognition model is trained based on the joint loss value. The feature extraction function of each encoding module can be fully considered in the training process of the formula recognition model. The feature information of each level of the formula recognition sample is introduced to train the formula recognition model. The trained formula recognition model has higher recognition accuracy and more accurate recognition results.
[0108] In an optional embodiment of the present invention, the step S13 of calculating the joint loss value of the formula recognition model according to the M formula recognition results corresponding to the formula recognition sample and the target recognition result includes:
[0109] Sub-step S131, according to each formula recognition result of the M formula recognition results corresponding to the formula recognition sample and the target recognition result, respectively calculating the loss value corresponding to each formula recognition result;
[0110] Sub-step S132: Calculate the joint loss value of the formula recognition model according to the loss value corresponding to each formula recognition result.
[0111] When calculating the joint loss value of the formula recognition model, the loss value corresponding to each of the M formula recognition results corresponding to the formula recognition sample can be calculated respectively. Specifically, the loss value corresponding to each formula recognition result can be calculated by calculating the cross entropy between the formula recognition result and the target calculation result.
[0112] Then, according to the loss value corresponding to each formula recognition result, the joint loss value of the formula recognition model is calculated. For example, the loss values corresponding to M formula recognition results are weighted and summed to obtain the joint loss value of the formula recognition model. The weight of the loss value corresponding to each formula recognition result can be set according to actual needs. For example, if you want to increase the influence of the underlying feature information on the formula recognition model, you can appropriately increase the weight of the loss value corresponding to the formula recognition results output by the first few encoding modules to increase the weight of the loss value corresponding to the formula recognition results output by the first few encoding modules. Figure 3 Taking the formula recognition model shown as an example, the weight of the loss value corresponding to the formula recognition result output by the first encoding module can be increased, and so on.
[0113] In an optional embodiment of the present invention, the step S12 of determining M target coding features according to the formula identifying N coding features corresponding to the sample includes:
[0114] Sub-step S121: fusing the coding features output by the Nth coding module of the encoder and the coding features output by the N-1th coding module to obtain a fused coding feature;
[0115] Sub-step S122: taking the fused coding feature and the coding features output by the first N-2 coding modules of the encoder as the target coding feature.
[0116] In the embodiment of the present invention, when the encoder of the formula recognition model includes multiple encoding modules, multiple encoding features will be obtained for a formula recognition sample, and the decoder of the formula recognition model needs to decode the multiple encoding features. In order to reduce the amount of data processed by the decoder and increase the training data of the formula recognition model, the encoding features output by the Nth encoding module of the encoder and the encoding features output by the N-1th encoding module can be fused. Figure 3 Taking the formula recognition model shown in the figure as an example, for a formula recognition sample, there are 5 coding features: feature map F0 to feature map F4. Feature map F3 and feature map F4 are fused to obtain the fused coding feature, which is recorded as feature map F d .
[0117] Then, the fused coding features and the coding features output by the first N-2 coding modules of the encoder are used as target coding features and are input into the decoder for decoding. d The formula is input into the decoder for decoding and 4 formula recognition results are obtained.
[0118] When calculating the loss value of the formula recognition model, the joint loss value is still calculated according to each formula recognition result and the target recognition result. The specific calculation process can refer to the aforementioned sub-steps S131 to S132, and the embodiment of the present invention will not be further described here.
[0119] It should be noted that, without affecting the recognition precision and accuracy of the formula recognition model, the coding features output by other coding modules may also be fused, and the embodiment of the present invention does not specifically limit this.
[0120] In an optional embodiment of the present invention, the sub-step S121 of fusing the coding features output by the Nth coding module of the encoder and the coding features output by the N-1th coding module to obtain the fused coding features includes:
[0121] Step A11: up-sample the coding feature output by the Nth coding module of the encoder to obtain a first coding feature, where the dimension of the first coding feature is the same as the dimension of the coding feature output by the N-1th coding module of the encoder;
[0122] Step A12: Fuse the first coding feature with the coding feature output by the N-1th coding module to obtain a fused coding feature.
[0123] The encoding module encodes the input data, which is essentially to extract features from the input data. Taking the encoding module using the CNN network as an example, the encoding module extracts multiple features in the input data into one feature by performing convolution processing on the input data. Compared with the input data, the dimension of the output data of the encoding module is reduced.
[0124] by Figure 3 Taking the formula recognition model shown in the figure as an example, the size of the feature map F3 output by the fourth encoding module is The size of the feature map F4 output by the fifth encoding module is Since the sizes of the feature maps are different, it is impossible to directly fuse the two feature maps. Therefore, it is necessary to sample the feature maps to make the sizes of the two feature maps consistent before performing feature fusion.
[0125] The coding features output by the Nth coding module are obtained by extracting features based on the coding features output by the N-1th coding module. The coding features output by the Nth coding module have a deeper level and a lower dimension. Therefore, it is necessary to obtain the first coding features by upsampling the coding features output by the Nth coding module, so that the dimension of the first coding features is the same as the dimension of the coding features output by the N-1th coding module of the encoder. Then, the first coding features and the coding features output by the N-1th coding module are fused to obtain fused coding features.
[0126] In an optional embodiment of the present invention, the sub-step S121 of fusing the coding features output by the Nth coding module of the encoder and the coding features output by the N-1th coding module to obtain the fused coding features includes:
[0127] Step A21: down-sample the coding feature output by the N-1th coding module of the encoder to obtain a second coding feature, where the dimension of the second coding feature is the same as the dimension of the coding feature output by the Nth coding module of the encoder;
[0128] Step A22: Fuse the second coding feature with the coding feature output by the Nth coding module to obtain a fused coding feature.
[0129] In the embodiment of the present invention, compared with the coding features output by the Nth coding module, the coding features output by the N-1th coding module have a lower level and a higher dimension. Therefore, in addition to upsampling the coding features output by the Nth coding module and performing feature fusion based on the first coding features after upsampling, the coding features output by the N-1th coding module of the encoder can also be downsampled to obtain a second coding feature, so that the dimension of the second coding feature is the same as the dimension of the coding feature output by the Nth coding module of the encoder. Then, the first coding feature and the coding feature output by the Nth coding module are fused to obtain a fused coding feature.
[0130] In an optional embodiment of the present invention, the method further includes:
[0131] Step S21, obtaining a target image, wherein the target image contains a formula to be recognized;
[0132] Step S21: input the target image into the trained formula recognition model to perform formula recognition processing to obtain a formula recognition result corresponding to the target image.
[0133] In the embodiment of the present invention, after obtaining the trained formula recognition model, the trained formula recognition model can be used to perform formula recognition processing on the target image to obtain the formula recognition result corresponding to the target image. Compared with the existing formula recognition model, the formula recognition result obtained by the trained formula recognition model in the embodiment of the present invention has higher accuracy.
[0134] In summary, the embodiment of the present invention extracts the coding features output by each coding module, decodes the coding features output by each coding module, obtains the formula recognition result corresponding to each coding module, and calculates the joint loss value of the formula recognition model according to the formula recognition result corresponding to each coding module. Different coding modules output different coding features, and the corresponding formula recognition results are also different. Therefore, the coding features extracted by each coding module affect the joint loss value of the formula recognition model. The formula recognition model is trained based on the joint loss value. It is possible to fully consider the feature extraction function of each coding module in the training process of the formula recognition model, introduce feature information at all levels of formula recognition samples to train the formula recognition model, and the trained formula recognition model has higher recognition accuracy and more accurate recognition results.
[0135] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.
[0136] Device Embodiment
[0137] Reference Figure 4 , shows a structural block diagram of an embodiment of a training device for a formula recognition model of the present invention, the device may include:
[0138] The training data acquisition module 401 is used to acquire formula recognition samples and target recognition results corresponding to the formula recognition samples;
[0139] An iterative training module 402, configured to iteratively train a formula recognition model based on the formula recognition samples, wherein the formula recognition model includes an encoder and a decoder, and the encoder includes at least one encoding module;
[0140] A joint loss value calculation module 403 is used to calculate the joint loss value of the formula recognition model according to the formula recognition result corresponding to each encoding module output by the decoder in each round of training, and adjust the model parameters of the formula recognition model according to the joint loss value;
[0141] The training completion condition determination module 404 is used to obtain a trained formula recognition model when the joint loss value meets the convergence condition.
[0142] Optionally, the formula recognition model includes N serially connected encoding modules, where N is a positive integer greater than 1, and the joint loss value calculation module includes:
[0143] The encoding processing submodule is used to input the formula recognition sample into the encoder of the formula recognition model in each round of training, and sequentially encode the formula recognition sample through the N encoding modules of the encoder to obtain N encoding features corresponding to the formula recognition sample;
[0144] A target coding feature determination submodule is used to determine M target coding features by identifying N coding features corresponding to the sample according to the formula; wherein M is a positive integer less than or equal to N;
[0145] A decoding processing submodule, used for inputting the M target coding features corresponding to the formula recognition sample into the decoder of the formula recognition model respectively, and performing decoding processing on the M target coding features by the decoder to obtain M formula recognition results corresponding to the formula recognition sample;
[0146] The joint loss value calculation submodule is used to calculate the joint loss value of the formula recognition model according to the M formula recognition results corresponding to the formula recognition sample and the target recognition result.
[0147] Optionally, the joint loss value calculation submodule includes:
[0148] A first loss value calculation unit, configured to calculate a loss value corresponding to each formula recognition result according to each formula recognition result of the M formula recognition results corresponding to the formula recognition sample and the target recognition result;
[0149] The second loss value calculation unit is used to calculate the joint loss value of the formula recognition model according to the loss value corresponding to each formula recognition result.
[0150] Optionally, the target coding feature determination submodule includes:
[0151] A coding feature fusion unit, used to fuse the coding feature output by the Nth coding module of the encoder and the coding feature output by the N-1th coding module to obtain a fused coding feature;
[0152] A target coding feature determination unit is used to use the fused coding feature and the coding features output by the first N-2 coding modules of the encoder as the target coding feature.
[0153] Optionally, the coding feature fusion unit includes:
[0154] an upsampling processing subunit, configured to perform upsampling processing on the coding feature output by the Nth coding module of the encoder to obtain a first coding feature, wherein the dimension of the first coding feature is the same as the dimension of the coding feature output by the N-1th coding module of the encoder;
[0155] The first coding feature fusion subunit is used to fuse the first coding feature with the coding feature output by the N-1th coding module to obtain a fused coding feature.
[0156] Optionally, the coding feature fusion unit includes:
[0157] a downsampling processing subunit, configured to perform downsampling processing on the coding feature output by the N-1th coding module of the encoder to obtain a second coding feature, wherein the dimension of the second coding feature is the same as the dimension of the coding feature output by the Nth coding module of the encoder;
[0158] The second coding feature fusion subunit is used to fuse the second coding feature with the coding feature output by the Nth coding module to obtain a fused coding feature.
[0159] Optionally, the device further comprises:
[0160] A target image acquisition module, used to acquire a target image, wherein the target image contains a formula to be identified;
[0161] The formula recognition processing module is used to input the target image into the trained formula recognition model for formula recognition processing to obtain the formula recognition result corresponding to the target image.
[0162] In summary, the embodiment of the present invention extracts the coding features output by each coding module, decodes the coding features output by each coding module, obtains the formula recognition result corresponding to each coding module, and calculates the joint loss value of the formula recognition model according to the formula recognition result corresponding to each coding module. Different coding modules output different coding features, and the corresponding formula recognition results are also different. Therefore, the coding features extracted by each coding module affect the joint loss value of the formula recognition model. The formula recognition model is trained based on the joint loss value. It is possible to fully consider the feature extraction function of each coding module in the training process of the formula recognition model, introduce feature information at all levels of formula recognition samples to train the formula recognition model, and the trained formula recognition model has higher recognition accuracy and more accurate recognition results.
[0163] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0164] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0165] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0166] An embodiment of the present invention provides a device for model training, the device comprising a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors. The one or more programs include instructions for performing the following operations:
[0167] P11. Obtain a formula recognition sample and a target recognition result corresponding to the formula recognition sample;
[0168] P12. Based on the formula recognition sample, iteratively train a formula recognition model, wherein the formula recognition model includes an encoder and a decoder, and the encoder includes at least one encoding module;
[0169] P13. In each round of training, according to the formula recognition results corresponding to each encoding module output by the decoder, a joint loss value of the formula recognition model is calculated, and model parameters of the formula recognition model are adjusted according to the joint loss value;
[0170] P14. When the joint loss value meets the convergence condition, a trained formula recognition model is obtained.
[0171] Optionally, the formula recognition model includes N serially connected encoding modules, where N is a positive integer greater than 1, and in each round of training, according to the formula recognition result corresponding to each encoding module output by the decoder, calculating the joint loss value of the formula recognition model includes:
[0172] In each round of training, the formula recognition sample is input into the encoder of the formula recognition model, and the formula recognition sample is encoded in turn by the N encoding modules of the encoder to obtain N encoding features corresponding to the formula recognition sample;
[0173] According to the formula, N coding features corresponding to the sample are identified to determine M target coding features; wherein M is a positive integer less than or equal to N;
[0174] Inputting the M target coding features corresponding to the formula recognition sample into the decoder of the formula recognition model respectively, and decoding the M target coding features by the decoder to obtain M formula recognition results corresponding to the formula recognition sample;
[0175] The joint loss value of the formula recognition model is calculated according to the M formula recognition results corresponding to the formula recognition sample and the target recognition result.
[0176] Optionally, calculating the joint loss value of the formula recognition model according to the M formula recognition results corresponding to the formula recognition sample and the target recognition result includes:
[0177] According to each formula recognition result of the M formula recognition results corresponding to the formula recognition sample and the target recognition result, respectively calculating the loss value corresponding to each formula recognition result;
[0178] According to the loss value corresponding to each formula recognition result, the joint loss value of the formula recognition model is calculated.
[0179] Optionally, determining M target coding features by identifying N coding features corresponding to the sample according to the formula includes:
[0180] Fusing the coding feature output by the Nth coding module of the encoder and the coding feature output by the N-1th coding module to obtain a fused coding feature;
[0181] The fused coding features and the coding features output by the first N-2 coding modules of the encoder are used as target coding features.
[0182] Optionally, fusing the coding feature output by the Nth coding module of the encoder and the coding feature output by the N-1th coding module to obtain the fused coding feature includes:
[0183] Performing upsampling processing on the coding feature output by the Nth coding module of the encoder to obtain a first coding feature, where the dimension of the first coding feature is the same as the dimension of the coding feature output by the N-1th coding module of the encoder;
[0184] The first coding feature and the coding feature output by the N-1th coding module are fused to obtain a fused coding feature.
[0185] Optionally, fusing the coding feature output by the Nth coding module of the encoder and the coding feature output by the N-1th coding module to obtain the fused coding feature includes:
[0186] Downsampling the encoding feature output by the N-1th encoding module of the encoder to obtain a second encoding feature, where the dimension of the second encoding feature is the same as the dimension of the encoding feature output by the Nth encoding module of the encoder;
[0187] The second coding feature and the coding feature output by the Nth coding module are fused to obtain a fused coding feature.
[0188] Optionally, the device is further configured to execute, by one or more processors, the one or more programs including instructions for performing the following operations:
[0189] Acquire a target image, wherein the target image contains a formula to be recognized;
[0190] The target image is input into the trained formula recognition model for formula recognition processing to obtain a formula recognition result corresponding to the target image.
[0191] Figure 5 8 is a block diagram of an apparatus 800 for model training according to an exemplary embodiment. For example, the apparatus 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.
[0192] Reference Figure 5 , the device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .
[0193] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.
[0194] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0195] The power supply component 806 provides power to the various components of the device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 800.
[0196] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
[0197] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the device 800 is in an operating mode, such as a call mode, a recording mode, and a voice information processing mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.
[0198] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.
[0199] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the device 800, the sensor assembly 814 can also detect the position change of the voice processing device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
[0200] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency information processing (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
[0201] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.
[0202] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by the processor 820 of the device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0203] Figure 61 is a schematic diagram of the structure of the server in some embodiments of the present invention. The server 1900 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1922 (for example, one or more processors) and memory 1932, and one or more storage media 1930 (for example, one or more mass storage devices) storing application programs 1942 or data 1944. Among them, the memory 1932 and the storage medium 1930 can be short-term storage or permanent storage. The program stored in the storage medium 1930 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the server. Furthermore, the central processing unit 1922 can be configured to communicate with the storage medium 1930 and execute a series of instruction operations in the storage medium 1930 on the server 1900.
[0204] The server 1900 may also include one or more power supplies 1926, one or more wired or wireless network interfaces 1950, one or more input and output interfaces 1958, one or more keyboards 1956, and / or, one or more operating systems 1941, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.
[0205] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a device (server or terminal), enables the device to perform Figure 2 The training method of the formula recognition model is shown.
[0206] A non-temporary computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a device (server or terminal), enables the device to execute a training method for a formula recognition model, the method comprising: obtaining a formula recognition sample and a target recognition result corresponding to the formula recognition sample; based on the formula recognition sample, iteratively training the formula recognition model, the formula recognition model comprising an encoder and a decoder, the encoder comprising at least one encoding module; in each round of training, according to the formula recognition result corresponding to each encoding module output by the decoder, calculating the joint loss value of the formula recognition model, and adjusting the model parameters of the formula recognition model according to the joint loss value; when the joint loss value meets the convergence condition, a trained formula recognition model is obtained.
[0207] Those skilled in the art will readily appreciate other embodiments of the invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in this disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.
[0208] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.
[0209] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
[0210] The above is a detailed introduction to a training method for a formula recognition model, a training device for a formula recognition model and a device for model training provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A training method for a formula recognition model, It is characterized in that The method comprises: Obtaining a formula recognition sample and a target recognition result corresponding to the formula recognition sample; Based on the formula recognition samples, iteratively training a formula recognition model, wherein the formula recognition model includes an encoder and a decoder, and the encoder includes at least one encoding module; In each round of training, according to the formula recognition results corresponding to each encoding module output by the decoder, a joint loss value of the formula recognition model is calculated, and model parameters of the formula recognition model are adjusted according to the joint loss value; When the joint loss value satisfies the convergence condition, a trained formula recognition model is obtained; The formula recognition model includes N serially connected encoding modules, where N is a positive integer greater than 1. In each round of training, the joint loss value of the formula recognition model is calculated according to the formula recognition result corresponding to each encoding module output by the decoder, including: In each round of training, the formula recognition sample is input into the encoder of the formula recognition model, and the formula recognition sample is encoded in turn by the N encoding modules of the encoder to obtain N encoding features corresponding to the formula recognition sample; According to the formula, N coding features corresponding to the sample are identified to determine M target coding features; wherein M is a positive integer less than or equal to N; Inputting the M target coding features corresponding to the formula recognition sample into the decoder of the formula recognition model respectively, and decoding the M target coding features by the decoder to obtain M formula recognition results corresponding to the formula recognition sample; The joint loss value of the formula recognition model is calculated according to the M formula recognition results corresponding to the formula recognition sample and the target recognition result.
2. The method according to claim 1, It is characterized in that The calculating the joint loss value of the formula recognition model according to the M formula recognition results corresponding to the formula recognition sample and the target recognition result includes: According to each formula recognition result of the M formula recognition results corresponding to the formula recognition sample and the target recognition result, respectively calculating the loss value corresponding to each formula recognition result; According to the loss value corresponding to each formula recognition result, the joint loss value of the formula recognition model is calculated.
3. The method according to claim 1, It is characterized in that The determining of M target coding features by identifying N coding features corresponding to the sample according to the formula includes: Fusing the coding feature output by the Nth coding module of the encoder and the coding feature output by the N-1th coding module to obtain a fused coding feature; The fused coding features and the coding features output by the first N-2 coding modules of the encoder are used as target coding features.
4. The method according to claim 3, It is characterized in that The step of fusing the coding features output by the Nth coding module of the encoder and the coding features output by the N-1th coding module to obtain the fused coding features includes: Performing upsampling processing on the coding feature output by the Nth coding module of the encoder to obtain a first coding feature, where the dimension of the first coding feature is the same as the dimension of the coding feature output by the N-1th coding module of the encoder; The first coding feature and the coding feature output by the N-1th coding module are fused to obtain a fused coding feature.
5. The method according to claim 3, It is characterized in that The step of fusing the coding features output by the Nth coding module of the encoder and the coding features output by the N-1th coding module to obtain the fused coding features includes: Downsampling the encoding feature output by the N-1th encoding module of the encoder to obtain a second encoding feature, where the dimension of the second encoding feature is the same as the dimension of the encoding feature output by the Nth encoding module of the encoder; The second coding feature and the coding feature output by the Nth coding module are fused to obtain a fused coding feature.
6. The method according to any one of claims 1 to 5, It is characterized in that The method further comprises: Acquire a target image, wherein the target image contains a formula to be recognized; The target image is input into the trained formula recognition model for formula recognition processing to obtain a formula recognition result corresponding to the target image.
7. A training device for a formula recognition model, It is characterized in that The device comprises: A training data acquisition module, used to acquire formula recognition samples and target recognition results corresponding to the formula recognition samples; An iterative training module, used for iteratively training a formula recognition model based on the formula recognition samples, wherein the formula recognition model includes an encoder and a decoder, and the encoder includes at least one encoding module; A joint loss value calculation module, used to calculate the joint loss value of the formula recognition model according to the formula recognition result corresponding to each encoding module output by the decoder in each round of training, and adjust the model parameters of the formula recognition model according to the joint loss value; A training completion condition determination module, used to obtain a training completed formula recognition model when the joint loss value meets the convergence condition; The formula recognition model includes N serially connected encoding modules, where N is a positive integer greater than 1, and the joint loss value calculation module includes: The encoding processing submodule is used to input the formula recognition sample into the encoder of the formula recognition model in each round of training, and sequentially encode the formula recognition sample through the N encoding modules of the encoder to obtain N encoding features corresponding to the formula recognition sample; A target coding feature determination submodule is used to determine M target coding features by identifying N coding features corresponding to the sample according to the formula; wherein M is a positive integer less than or equal to N; A decoding processing submodule, used for inputting the M target coding features corresponding to the formula recognition sample into the decoder of the formula recognition model respectively, and performing decoding processing on the M target coding features by the decoder to obtain M formula recognition results corresponding to the formula recognition sample; The joint loss value calculation submodule is used to calculate the joint loss value of the formula recognition model according to the M formula recognition results corresponding to the formula recognition sample and the target recognition result.
8. The device according to claim 7, It is characterized in that The joint loss value calculation submodule includes: A first loss value calculation unit, configured to calculate a loss value corresponding to each formula recognition result according to each formula recognition result of the M formula recognition results corresponding to the formula recognition sample and the target recognition result; The second loss value calculation unit is used to calculate the joint loss value of the formula recognition model according to the loss value corresponding to each formula recognition result.
9. The device according to claim 7, It is characterized in that The target coding feature determination submodule includes: A coding feature fusion unit, used to fuse the coding feature output by the Nth coding module of the encoder and the coding feature output by the N-1th coding module to obtain a fused coding feature; A target coding feature determination unit is used to use the fused coding feature and the coding features output by the first N-2 coding modules of the encoder as the target coding feature.
10. The device according to claim 9, It is characterized in that The coding feature fusion unit comprises: an upsampling processing subunit, configured to perform upsampling processing on the coding feature output by the Nth coding module of the encoder to obtain a first coding feature, wherein the dimension of the first coding feature is the same as the dimension of the coding feature output by the N-1th coding module of the encoder; The first coding feature fusion subunit is used to fuse the first coding feature with the coding feature output by the N-1th coding module to obtain a fused coding feature.
11. The device according to claim 9, It is characterized in that The coding feature fusion unit comprises: a downsampling processing subunit, configured to perform downsampling processing on the coding feature output by the N-1th coding module of the encoder to obtain a second coding feature, wherein the dimension of the second coding feature is the same as the dimension of the coding feature output by the Nth coding module of the encoder; The second coding feature fusion subunit is used to fuse the second coding feature with the coding feature output by the Nth coding module to obtain a fused coding feature.
12. A device for model training, It is characterized in that The device includes a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors. The one or more programs include instructions for executing the training method of the formula recognition model as described in any one of claims 1 to 6.
13. A machine-readable medium having instructions stored thereon, which, when executed by one or more processors, causes a device to execute the method for training a formula recognition model as described in any one of claims 1 to 6.
Citation Information
Patent Citations
Handwritten formula recognition method based on end-to-end network model
CN111738169A
Dialogue intention recognition model training method and device, computer equipment and medium
CN112766319A
Training method and device for nested named entity recognition model
CN112800768A
Text recognition method and device, equipment and medium
CN113283427A
Formula identification method and device and device for formula identification
CN113408417A
