Formula identification method and device, model training method and device, medium and equipment

Abstract

The invention relates to a formula identification method and device, a model training method and device, a medium and equipment, and aims to improve the accuracy of a formula identification model obtained by training for mixed formula identification. The model training method comprises the steps of obtaining a to-be-recognized formula image sample and a recognition result label corresponding to the to-be-recognized formula image sample; performing feature extraction on the to-be-identified formula image sample to obtain a cell mapping feature sample corresponding to the to-be-identified formula image sample; inputting the cell mapping feature sample into a preset visual language model to obtain a corresponding formula recognition result sample; and according to a loss value between the formula identification result sample and the identification result label, adjusting model parameters of a preset visual language model to obtain a trained formula identification model.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a formula recognition method, a model training method, an apparatus, a medium, and equipment. Background Art

[0002] In intelligent education, for high-grade comprehensive science subjects such as mathematics, physics, and chemistry, the use of test paper OCR (Optical Character Recognition) recognition functions is often involved in the process of teachers assisting in marking and compiling test papers.

[0003] However, in some exam recognition scenarios, it can be difficult to distinguish whether formulas used in exams belong to mathematics or physics. For example, students may need to understand, apply, and solve complex problems that include physics, mathematics, and engineering formulas. Therefore, to facilitate business integration, reduce communication costs, and reduce redundancy in open platforms, it is necessary to develop a unified version of the formula recognition algorithm. Summary of the Invention

[0004] The purpose of the present disclosure is to provide a formula recognition method, a model training method, an apparatus, a medium and a device to improve the accuracy of the trained formula recognition model for hybrid formula recognition.

[0005] In order to achieve the above objectives, according to a first aspect of an embodiment of the present disclosure, a model training method is provided, the method comprising: Obtaining a formula image sample to be identified and a recognition result label corresponding to the formula image sample to be identified; Performing feature extraction on the formula image sample to be identified to obtain a cell mapping feature sample corresponding to the formula image sample to be identified; Inputting the cell mapping feature sample into a preset visual language model to obtain a corresponding formula recognition result sample; According to the loss value between the formula recognition result sample and the recognition result label, the model parameters of the preset visual language model are adjusted to obtain a trained formula recognition model.

[0006] Optionally, the extracting features of the formula image sample to be identified to obtain a cell mapping feature sample corresponding to the formula image sample to be identified includes: Determining a point mapping dynamic system of the formula image sample to be identified; wherein the point mapping dynamic system is used to represent a discretized cell state space; Determine a cell mapping feature sample of the formula image sample to be identified in the cell state space.

[0007] Optionally, the preset visual language model includes a multi-layer network structure; a prompt embedding vector is set in at least one layer of the network structure of the preset visual language model; wherein the prompt embedding vector serves as a model parameter of the preset visual language model.

[0008] Optionally, adjusting the model parameters of the preset visual language model according to the loss value between the formula recognition result sample and the recognition result label to obtain a trained formula recognition model includes: Determine a loss value between the formula recognition result sample and the recognition result label based on a relative entropy loss function; The model parameters of the preset visual language model are adjusted according to the loss value to obtain a trained formula recognition model.

[0009] Optionally, the cell mapping feature sample includes at least one of a formula shape feature, a color feature, a line feature, and a symbol representation feature.

[0010] According to a second aspect of an embodiment of the present disclosure, a formula recognition method is provided, the method comprising: Obtaining the formula image to be recognized; Inputting the formula image to be recognized into a formula recognition model to obtain a formula recognition result output by the formula recognition model; The formula recognition model is obtained by training using the model training method described in any one of the first aspects.

[0011] According to a third aspect of an embodiment of the present disclosure, a model training device is provided, comprising: A first acquisition module is used to acquire a formula image sample to be identified and a recognition result label corresponding to the formula image sample to be identified; An extraction module, configured to extract features from the formula image sample to be identified, and obtain a cell mapping feature sample corresponding to the formula image sample to be identified; A first input module is used to input the cell mapping feature sample into a preset visual language model to obtain a corresponding formula recognition result sample; An adjustment module is used to adjust the model parameters of the preset visual language model according to the loss value between the formula recognition result sample and the recognition result label to obtain a trained formula recognition model.

[0012] According to a fourth aspect of an embodiment of the present disclosure, a formula recognition device is provided, the formula recognition device comprising: A second acquisition module is used to acquire the formula image to be recognized; A second input module is used to input the formula image to be recognized into a formula recognition model to obtain a formula recognition result output by the formula recognition model; The formula recognition model is obtained by training using the model training method described in any one of the first aspects.

[0013] According to a fifth aspect of an embodiment of the present disclosure, a non-temporary computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the method described in any one of the first aspect or the second aspect.

[0014] According to a sixth aspect of an embodiment of the present disclosure, there is provided an electronic device, including: a memory having a computer program stored thereon; A processor is used to execute the computer program in the memory to implement the steps of the method described in any one of the first aspect or the second aspect above.

[0015] Through the above technical solution, numerical generalized cell mapping is used for feature extraction during the model training process, so as to adapt to feature extraction in complex situations and ensure the accuracy of feature extraction of mixed formulas. In addition, the model training is combined with a preset large-scale visual language model to improve the robustness of model training, so that the trained model can have good detection and analysis effects when facing unclear, ambiguous or erroneous formulas, thereby improving the accuracy of the trained formula recognition model for mixed formula recognition.

[0016] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following detailed description, they are used to explain the present disclosure but do not constitute a limitation of the present disclosure. In the accompanying drawings: Figure 1 It is a flowchart of a model training method shown in an exemplary embodiment of the present disclosure.

[0018] Figure 2 The figure is a flow chart of a formula recognition method shown in an exemplary embodiment of the present disclosure.

[0019] Figure 3 FIG. 4 is a flowchart of a formula recognition method shown in another exemplary embodiment of the present disclosure.

[0020] Figure 4 It is a block diagram of a model training device shown in an exemplary embodiment of the present disclosure.

[0021] Figure 5It is a block diagram of a formula recognition device shown in an exemplary embodiment of the present disclosure.

[0022] Figure 6 It is a block diagram of an electronic device shown in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] The following describes the specific embodiments of the present disclosure in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present disclosure and are not intended to limit the present disclosure.

[0024] As mentioned in the background, intelligent education often involves the identification of multidisciplinary formulas, that is, the identification of mixed formulas. To this end, related technologies provide detection methods for single models and general models.

[0025] Single-model detection methods are suitable for scenarios where different disciplines must be identified first, and then formula detection and recognition must rely solely on specific models. However, this approach is inefficient. Current general-purpose model detection methods often lack robustness, resulting in poor recognition of complex formulas and an inability to guarantee accuracy. Therefore, accurately and quickly detecting mixed formulas across multiple disciplines in intelligent education scenarios has become a pressing issue.

[0026] In view of this, the embodiments of the present disclosure provide a formula recognition method, a model training method, an apparatus, a medium and a device, which use numerical generalized cell mapping to perform feature extraction during the model training process, thereby adapting to feature extraction in complex situations and ensuring the accuracy of feature extraction of mixed formulas. The model training is carried out in combination with a preset large-scale visual language model, thereby improving the robustness of the model training, so that the trained model can have good detection and analysis effects when facing unclear, ambiguous or erroneous formulas, thereby improving the accuracy of the trained formula recognition model for mixed formula recognition.

[0027] Figure 1 This is a flow chart of a model training method shown in an exemplary embodiment of the present disclosure. Figure 1 As shown, the model training method may include steps S101 to S104.

[0028] In step S101 , a formula image sample to be identified and a recognition result label corresponding to the formula image sample to be identified are obtained.

[0029] It should be understood that the formula image sample to be identified includes a formula, and the formula sample can be identified from the formula image sample to be identified using formula detection and recognition technology. The recognition result label corresponding to the formula image sample to be identified includes a formula label and an identification frame label for identifying the formula label.

[0030] In step S102, feature extraction is performed on the formula image sample to be identified to obtain a cell mapping feature sample corresponding to the formula image sample to be identified.

[0031] There are many ways to extract features from image samples of the recognition formula, for example, features can be extracted from images through densely connected convolutional networks (DenseNet) or residual neural networks (ResNet).

[0032] In the disclosed embodiments, numerical generalized cell mapping is used to describe the formula image samples to be identified, thereby extracting corresponding cell mapping feature samples for the formula image samples to be identified. This allows the cell mapping method to accurately describe formula feature extraction in complex situations, ensuring the accuracy of feature extraction for mixed formulas.

[0033] In step S103, the cell mapping feature sample is input into a preset visual language model to obtain a corresponding formula recognition result sample.

[0034] Among them, the preset visual language model can refer to a large-scale visual language model (Qwen-VL-Chat). By combining the preset large-scale visual language model for model training, the robustness of model training can be improved, so that the trained model can have good detection and analysis effects when facing unclear, ambiguous or erroneous formulas, thereby improving the accuracy of the trained formula recognition model for mixed formula recognition.

[0035] In step S104, the model parameters of the preset visual language model are adjusted according to the loss value between the formula recognition result sample and the recognition result label to obtain a trained formula recognition model.

[0036] The formula recognition result sample may include the recognized formula sample and the identification box sample used to identify the formula sample. Based on this, a loss function can be used to determine the loss value between the formula recognition result sample and the recognition result label, and the model parameters of the preset visual language model can be adjusted based on the loss value. Thus, after the trained preset visual language model reaches the preset training stop condition, a trained formula recognition model can be obtained.

[0037] Among them, the preset training stop condition may include the number of training times being equal to the preset number of training times and / or the loss value being equal to the preset loss value. The preset number of training times and the preset loss value may be determined according to actual conditions, and this disclosure does not specifically limit this. In one possible implementation, the loss value between the formula recognition result sample and the recognition result label may be determined based on the relative entropy loss function (KL, Kullback-Leibler Divergence), and the model parameters of the preset visual language model may be adjusted based on the loss value to obtain a trained formula recognition model.

[0038] Optionally, step S102 may include: Determine a point mapping dynamic system of a formula image sample to be identified; wherein the point mapping dynamic system is used to represent a discretized cell state space; Determine the cell mapping feature samples of the formula image samples to be identified in the cell state space.

[0039] The cell mapping feature sample may include at least one of a formula shape feature, a color feature, a line feature, and a symbol representation feature.

[0040] It should be understood that each cell of a numerical generalized cell mapping can have multiple image cells. When a dynamical system is described using a generalized cell mapping (GCM), it can correspond to a time-homogeneous Markov chain. Therefore, in one possible implementation, the formula image samples to be identified can be discretized into a point mapping dynamical system using a Poincare section, and a cell mapping can be constructed based on this point mapping dynamical system.

[0041] For example, referring to the following formula 1, the original system where the formula image sample to be identified is located is discretized into a point mapping dynamic system through the Poincare section.

[0042] (Formula 1) in, is the state vector, which represents the state of the system when it crosses the Poincare section for the nth time. is the state vector, which represents the state of the system when it crosses the Poincare section for the n+1th time. represents the Poincare mapping function, which is used to describe the evolution of the system state from one section to the next. Represents system parameters such as the frequency and amplitude of external driving forces, internal damping coefficients, and other parameters that affect system behavior. represents an n-dimensional real vector space, represents an m-dimensional real vector space.

[0043] On this basis, in order to construct cell mapping from point mapping dynamic system, first Select the preset research range from the above, and discretize the selected preset research range into a cell state space, using one cell to replace all points in the cell. The direction has a magnitude of The edges (i = 1, 2, ..., N), so the interval along each edge can be defined as an integer , which satisfies the following formula 2.

[0044] (Formula 2) In this way, the generalized cell map state space can be described by the following formula 3.

[0045] (Formula 3) in, is the state vector, which represents the state of the system at the nth iteration and contains all the information of the system at that moment. is the state vector, which represents the state of the system at the n+1th iteration. Represents the mapping matrix or transfer matrix, which defines how the system state transfers from one step to the next, that is, the matrix includes the transition probability or transition rule from the current cell to the next cell.

[0046] Through the above transformation, the features of the formula image samples to be identified as the original data can be carefully extracted. For formulas of different disciplines, the corresponding characteristics can be extracted, that is, the corresponding cell mapping feature samples. The extracted cell mapping feature samples can then be used as input for subsequent network classification.

[0047] It should be understood that for image feature extraction, a densely connected convolutional network or a residual network can be used as an encoder. Compared to a residual network, a densely connected convolutional network's dense connections across feature maps of different scales can better reflect the scale characteristics of characters of different sizes, facilitating the subsequent decoding of the meaning of characters of different sizes at different positions.

[0048] Optionally, the preset visual language model may include a multi-layer network structure. On this basis, the hint embedding vector may be inserted into at least one layer of the network structure of the preset visual language model, that is, the hint embedding vector is set in at least one layer of the network structure of the preset visual language model.

[0049] The cue embedding vector serves as a model parameter for the pre-set visual language model. This allows adjustments to the cue embedding vector during model training, fine-tuning the pre-set visual language model and making the trained model more adaptable to specific tasks. Compared to training a new model from scratch, fine-tuning significantly saves computing resources and also yields better performance.

[0050] It should be noted that the preset visual language model in the embodiments of the present disclosure may refer to the Large-Scale Visual Language Model (Qwen-VL-Chat), a variant of the Transformer model that uses an attention mechanism to enable the decoder to focus on its region of interest, thereby better aligning feature information and guiding the network to assign a higher probability of attention to unattended areas. The network structure of the preset visual language model may include a normalization layer, a self-attention layer, and an activation function.

[0051] Among them, the normalization layer normalizes each cell mapping feature sample in the feature dimension so that the output has a mean of 0 and a variance of 1 in each feature dimension. This method can accelerate the convergence of the model and help solve the problems of gradient disappearance and gradient explosion.

[0052] The self-attention layer uses the self-attention mechanism, which is a core component of the Transformer model. In specific implementations, given an input sequence, the self-attention mechanism can calculate a weight vector based on the relationship between each element in the sequence and other elements, and use this weight vector to perform a weighted average on the input sequence. In this way, the model can better capture long-distance dependencies in the sequence. In the embodiment of the present disclosure, the self-attention layer can be followed by a residual connection. The residual connection allows information to be passed directly from the front layer to the back layer, which helps solve the gradient disappearance problem in deep networks.

[0053] For example, the activation function can use a gated linear unit (GLU). As a nonlinear activation function, the GLU consists of two main components: a linear transformation and a gating mechanism. The linear transformation is responsible for extracting input features, while the gating mechanism is responsible for controlling the flow of information. In this way, the GLU is better able to handle complex tasks. In the disclosed embodiment, a residual connection can also be connected after the GLU.

[0054] In addition, a normalization layer can be added between the self-attention layer and the activation function for normalization. The normalization layer between the self-attention layer and the activation function can be used to stabilize the input distribution of the intermediate layer and reduce the internal covariance shift, thereby facilitating gradient flow and improving the stability and generalization ability of the model.

[0055] Therefore, in one embodiment, the hint embedding vector can be inserted into the input layer or intermediate layer of the model. In another embodiment, continuous and independent hint embedding vectors can be inserted into all layers of the model network structure. In this way, the hint embedding vector can be trained to adjust the model's behavior and adapt it to specific tasks.

[0056] In this way, by combining the visual capabilities of the preset visual language model and the fine-tuning scheme to detect and identify formulas, the robustness of model training can be effectively improved, so that the trained model can also have good detection and analysis effects when facing unclear, ambiguous or erroneous formulas, thereby improving the accuracy of the trained formula recognition model for mixed formula recognition.

[0057] Moreover, the trained formula recognition model can be used as a unified model to recognize mixed formulas, thereby avoiding the efficiency problems caused by using a single model to detect a specific subject.

[0058] Figure 2 FIG. 1 is a flow chart of a formula recognition method according to an exemplary embodiment of the present disclosure. Figure 2 As shown, the formula recognition method may include steps S201 to S202.

[0059] In step S201, a formula image to be recognized is obtained.

[0060] In step S202, the formula image to be recognized is input into the formula recognition model to obtain the formula recognition result output by the formula recognition model.

[0061] The formula recognition model is obtained by training through any of the above-mentioned model training methods.

[0062] It is not difficult to understand that since numerical generalized cell mapping is used for feature extraction during the model training process, it adapts to feature extraction in complex situations and ensures the accuracy of feature extraction of mixed formulas. In addition, the preset large-scale visual language model is combined for model training, which improves the robustness of model training. As a result, the trained model can have good detection and analysis effects when facing unclear, ambiguous or erroneous formulas, thereby improving the accuracy of the trained formula recognition model for mixed formula recognition.

[0063] Figure 3FIG. 1 is a flow chart of a formula recognition method shown in another exemplary embodiment of the present disclosure. Figure 3 As shown, the formula recognition method may include steps S301 to S304.

[0064] In step S301, an OCR image of an educational scene is obtained, and the OCR image is preprocessed to obtain a formula image to be recognized.

[0065] In step S302, a cell mapping method is used to perform feature processing on the formula image to be identified to obtain cell mapping features; the cell mapping features include at least one of formula shape features, color features, line features, and symbol representation features.

[0066] In step S303, the cell mapping features are input into the network backbone for detection, recognition and classification.

[0067] The network backbone includes a multidisciplinary formula encoder, an input mapper, a trained formula recognition model, an output mapper, and a multidisciplinary formula decoder.

[0068] In step S304, the formula recognition result is output.

[0069] By using the formula recognition model trained by the above-mentioned model training method to perform formula recognition, it is possible to cope with mixed formulas of different types and complexities as well as the ever-changing educational text content, and accurately classify and parse the identified mixed formulas to ensure that they are accurately understood and used, thereby conveniently supporting calls from different business units and reducing communication costs.

[0070] Based on the same inventive concept, the present disclosure also provides a model training device, see Figure 4 , Figure 4 FIG. 1 is a block diagram of a model training device according to an exemplary embodiment of the present disclosure. Figure 4 As shown, the model training device 100 is used to train a formula recognition model. The model training device 100 includes a first acquisition module 101 , an extraction module 102 , a first input module 103 , and an adjustment module 104 .

[0071] The first acquisition module 101 is used to acquire a formula image sample to be identified and a recognition result label corresponding to the formula image sample to be identified; An extraction module 102 is configured to perform feature extraction on the formula image sample to be identified, and obtain a cell mapping feature sample corresponding to the formula image sample to be identified; A first input module 103 is configured to input the cell mapping feature sample into a preset visual language model to obtain a corresponding formula recognition result sample; The adjustment module 104 is used to adjust the model parameters of the preset visual language model according to the loss value between the formula recognition result sample and the recognition result label to obtain a trained formula recognition model.

[0072] Through the above technical solution, numerical generalized cell mapping is used for feature extraction during the model training process, so as to adapt to feature extraction in complex situations and ensure the accuracy of feature extraction of mixed formulas. In addition, the model training is combined with a preset large-scale visual language model to improve the robustness of model training, so that the trained model can have good detection and analysis effects when facing unclear, ambiguous or erroneous formulas, thereby improving the accuracy of the trained formula recognition model for mixed formula recognition.

[0073] In one embodiment, the extraction module 102 is configured to: Determining a point mapping dynamic system of the formula image sample to be identified; wherein the point mapping dynamic system is used to represent a discretized cell state space; Determine a cell mapping feature sample of the formula image sample to be identified in the cell state space.

[0074] In one embodiment, the preset visual language model includes a multi-layer network structure; a prompt embedding vector is set in at least one layer of the network structure of the preset visual language model; wherein the prompt embedding vector serves as a model parameter of the preset visual language model.

[0075] In one embodiment, the adjustment module 104 is configured to: Determine a loss value between the formula recognition result sample and the recognition result label based on a relative entropy loss function; The model parameters of the preset visual language model are adjusted according to the loss value to obtain a trained formula recognition model.

[0076] In one embodiment, the cell mapping feature sample includes at least one of a formula shape feature, a color feature, a line feature, and a symbol representation feature.

[0077] Regarding the model training device 100 in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method and will not be elaborated here.

[0078] Based on the same inventive concept, the present disclosure also provides a formula recognition device, see Figure 5 , Figure 5 FIG. 1 is a block diagram of a formula recognition device according to an exemplary embodiment of the present disclosure. Figure 5 As shown, the formula recognition device 200 includes: The second acquisition module 201 is used to acquire the formula image to be recognized; The second input module 202 is used to input the formula image to be recognized into the formula recognition model to obtain the formula recognition result output by the formula recognition model; wherein, the formula recognition model is trained by any of the above-mentioned model training methods.

[0079] Regarding the formula recognition device 200 in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated on here.

[0080] Figure 6 FIG. 6 is a block diagram of an electronic device 600 according to an exemplary embodiment. Figure 6 As shown, the electronic device 600 may include: a processor 601 , a memory 602 , and may further include one or more of a multimedia component 603 , an input / output (I / O) interface 604 , and a communication component 605 .

[0081] The processor 601 is used to control the overall operation of the electronic device 600 to complete all or part of the steps in the above-mentioned model training method or formula recognition method. The memory 602 is used to store various types of data to support the operation of the electronic device 600. This data may include, for example, instructions for any application or method operating on the electronic device 600, as well as application-related data such as contact information, sent and received messages, images, audio, video, etc. The memory 602 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The multimedia component 603 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in the memory 602 or sent through the communication component 605. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 604 provides an interface between the processor 601 and other interface modules. The above-mentioned other interface modules may be a keyboard, a mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 605 is used for wired or wireless communication between the electronic device 600 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G or 4G, or a combination of one or more of them, so the corresponding communication component 605 may include: a Wi-Fi module, a Bluetooth module, an NFC module.

[0082] In an exemplary embodiment, the electronic device 600 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to execute the above-mentioned model training method or formula recognition method.

[0083] In another exemplary embodiment, a computer-readable storage medium including program instructions is further provided. When the program instructions are executed by a processor, the steps of the above-described model training method or formula recognition method are implemented. For example, the computer-readable storage medium may be the aforementioned memory 602 including the program instructions. The program instructions may be executed by the processor 601 of the electronic device 600 to implement the above-described model training method or formula recognition method.

[0084] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program that can be executed by a processor. When the computer program is executed by the processor, the steps of the above-mentioned model training method or formula recognition method are implemented.

[0085] The preferred embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings. However, the present disclosure is not limited to the specific details of the above embodiments. Within the technical concept of the present disclosure, various simple modifications can be made to the technical solutions of the present disclosure, and these simple modifications all fall within the scope of protection of the present disclosure.

[0086] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any appropriate manner without contradiction. In order to avoid unnecessary repetition, the present disclosure will not further describe various possible combinations.

[0087] In addition, the various embodiments of the present disclosure may be arbitrarily combined, and as long as they do not violate the concept of the present disclosure, they should also be regarded as the contents disclosed by the present disclosure.

Claims

1. A model training method, characterized in that: The method comprises: Obtaining a formula image sample to be identified and a recognition result label corresponding to the formula image sample to be identified; Performing feature extraction on the formula image sample to be identified to obtain a cell mapping feature sample corresponding to the formula image sample to be identified; Inputting the cell mapping feature sample into a preset visual language model to obtain a corresponding formula recognition result sample; According to the loss value between the formula recognition result sample and the recognition result label, the model parameters of the preset visual language model are adjusted to obtain a trained formula recognition model.

2. The method according to claim 1, characterized in that The extracting features of the formula image sample to be identified to obtain a cell mapping feature sample corresponding to the formula image sample to be identified includes: Determining a point mapping dynamic system of the formula image sample to be identified; wherein the point mapping dynamic system is used to represent a discretized cell state space; Determine a cell mapping feature sample of the formula image sample to be identified in the cell state space.

3. The method according to claim 1, characterized in that The preset visual language model includes a multi-layer network structure; a prompt embedding vector is set in at least one layer of the network structure of the preset visual language model; wherein the prompt embedding vector serves as a model parameter of the preset visual language model.

4. The method according to claim 1, wherein The step of adjusting the model parameters of the preset visual language model according to the loss value between the formula recognition result sample and the recognition result label to obtain a trained formula recognition model includes: Determine a loss value between the formula recognition result sample and the recognition result label based on a relative entropy loss function; The model parameters of the preset visual language model are adjusted according to the loss value to obtain a trained formula recognition model.

5. The method according to any one of claims 1 to 4, characterized in that The cell mapping feature sample includes at least one of a formula shape feature, a color feature, a line feature, and a symbol representation feature.

6. A formula recognition method, characterized in that: The method comprises: Obtaining the formula image to be recognized; Inputting the formula image to be recognized into a formula recognition model to obtain a formula recognition result output by the formula recognition model; Wherein, the formula recognition model is obtained by training through the model training method described in any one of claims 1-5.

7. A model training device, characterized in that: The model training device comprises: A first acquisition module is used to acquire a formula image sample to be identified and a recognition result label corresponding to the formula image sample to be identified; An extraction module, configured to extract features from the formula image sample to be identified, and obtain a cell mapping feature sample corresponding to the formula image sample to be identified; A first input module is used to input the cell mapping feature sample into a preset visual language model to obtain a corresponding formula recognition result sample; An adjustment module is used to adjust the model parameters of the preset visual language model according to the loss value between the formula recognition result sample and the recognition result label to obtain a trained formula recognition model.

8. A formula recognition device, characterized in that: The formula recognition device comprises: A second acquisition module is used to acquire the formula image to be recognized; A second input module is used to input the formula image to be recognized into a formula recognition model to obtain a formula recognition result output by the formula recognition model; Wherein, the formula recognition model is obtained by training through the model training method described in any one of claims 1-5.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 5 or claim 6 are implemented.

10. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 5 or claim 6.

Images