Formula identification method and device, equipment and medium

Through the formula recognition model of multi-task joint training, combined with visual and language decoders, the end-to-end recognition is used to use formula context information, which solves the problem of low accuracy in formula recognition in the prior art, and achieves higher recognition accuracy and Model performance.

CN119049068APending Publication Date: 2024-11-29DOUYIN VISION CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411017645.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

In the existing formula recognition technology, the two-stage recognition method that first performs image area detection and then performs formula recognition has low recognition accuracy and fails to effectively utilize the context information of the formula, resulting in low error accumulation and accuracy.

Method used

The formula recognition model based on multi-task joint training is adopted, and the target image and formula recognition prompt words are input, combined with the visual encoder and the language decoder for end-to-end formula recognition, considering the formula context information, and improving the model's performance and generalization capabilities.

Benefits of technology

It effectively improves the accuracy of formula recognition, avoids error accumulation, improves the global perception and understanding ability of the model, realizes formula recognition in a paragraph, and enhances the ability to identify formula content and location.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119049068A_ABST
    Figure CN119049068A_ABST
Patent Text Reader

Abstract

The embodiment of the invention relates to a formula recognition method and device, equipment and a medium. The method comprises the steps that a target image including a target formula is acquired; inputting the target image and the formula identification cue word into a formula identification model, and outputting to obtain the formula content and the formula position of the target formula; wherein the formula recognition model is obtained based on a plurality of cue words through multi-task joint training, and the plurality of cue words comprise formula recognition cue words and formula context cue words. By adopting the technical scheme, joint learning and knowledge sharing can be performed from multiple tasks through the multi-task joint training formula identification model, the performance and generalization ability of the model can be improved, the formula context is considered during identification, the accuracy of formula identification of the model is effectively improved, and the user experience is improved. When the model is applied, one-section type formula identification is realized in an end-to-end mode, error accumulation in two-section type identification is avoided, and the accuracy of formula identification is further improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a formula recognition method, device, equipment and medium. Background Art

[0002] In the related art, when performing formula recognition, the formula area of ​​the image is first detected, and then the formula area is cut out and the formula recognition is performed independently on the formula area. However, this two-step recognition method of first performing area detection and then performing formula recognition has low recognition accuracy. Summary of the invention

[0003] In order to solve the above technical problems, the present disclosure provides a formula recognition method, device, equipment and medium.

[0004] The present disclosure provides a formula recognition method, the method comprising:

[0005] obtaining a target image including a target formula;

[0006] Input the target image and the formula recognition prompt word into the formula recognition model, and output the formula content and formula position of the target formula;

[0007] The formula recognition model is obtained through multi-task joint training based on multiple prompt words, and the multiple prompt words include the formula recognition prompt words and formula context prompt words.

[0008] The present disclosure also provides a formula recognition device, the device comprising:

[0009] An acquisition module, used for acquiring a target image including a target formula;

[0010] A processing module is used to input the target image and formula recognition prompt words into a formula recognition model, and output the formula content and formula position of the target formula; wherein the formula recognition model is obtained through multi-task joint training based on multiple prompt words, and the multiple prompt words include the formula recognition prompt words and formula context prompt words.

[0011] An embodiment of the present disclosure also provides an electronic device, which includes: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement the formula recognition method provided by the embodiment of the present disclosure.

[0012] The embodiment of the present disclosure further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the formula recognition method provided by the embodiment of the present disclosure.

[0013] The technical solution provided by the embodiment of the present disclosure has the following advantages over the prior art: the formula recognition solution provided by the embodiment of the present disclosure obtains a target image including a target formula; inputs the target image and the formula recognition prompt word into the formula recognition model, and outputs the formula content and formula position of the target formula; wherein the formula recognition model is obtained through multi-task joint training based on multiple prompt words, and the multiple prompt words include formula recognition prompt words and formula context prompt words. By adopting the above technical solution, multi-task joint training is performed in multiple dimensions such as formula recognition and formula context recognition to obtain a formula recognition model, and the image and the formula recognition prompt word are input into the formula recognition model to obtain the formula content and formula position of the formula. Through multi-task joint training, the formula recognition model can learn and share knowledge from multiple tasks, which can improve the performance and generalization ability of the model, and consider the formula context during recognition, effectively improving the accuracy of the formula recognition of the model, and realizing one-segment formula recognition in an end-to-end manner when the model is applied, avoiding the error accumulation in two-segment recognition, and further improving the accuracy of formula recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0015] Figure 1 A flowchart of a formula recognition method provided in some embodiments of the present disclosure;

[0016] Figure 2 A flowchart of another formula recognition method provided in some embodiments of the present disclosure;

[0017] Figure 3 A schematic diagram of formula recognition provided for some embodiments of the present disclosure;

[0018] Figure 4 A flowchart of another formula recognition method provided in some embodiments of the present disclosure;

[0019] Figure 5 A schematic diagram of the structure of a formula recognition device provided in some embodiments of the present disclosure;

[0020] Figure 6 A schematic diagram of the structure of an electronic device provided for some embodiments of the present disclosure. DETAILED DESCRIPTION

[0021] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0022] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0023] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0024] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0025] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0026] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0027] When performing formula recognition, the formula area of ​​the image is first detected, and then the formula area is cut out and the formula recognition is performed independently on the formula area. However, this two-step recognition method of first detecting and then recognizing has the following two defects.

[0028] First, the recognition of the formula involves two models, the region determination model and the region recognition model. The two models cannot be jointly optimized, and the recognition of the formula is restricted by the model architecture. If the determined formula area is inaccurate, the region recognition model cannot recognize the formula in the inaccurate area, resulting in the problem of error accumulation.

[0029] Second, the context of the formula is not utilized. Generally, the formulas in articles do not appear independently. Usually, the formulas have rich contexts, which are helpful for the recognition and understanding of the formulas. The method of cropping the formula area and then recognizing it loses this information, resulting in a lower accuracy rate.

[0030] In order to solve the above problems, the embodiments of the present disclosure provide a formula recognition method, which is introduced below in conjunction with specific embodiments.

[0031] Figure 1 The flowchart of the formula recognition method provided by some embodiments of the present disclosure is shown in FIG. 1 , which can be executed by a formula recognition device, wherein the device can be implemented by software and / or hardware and can generally be integrated in an electronic device. Figure 1 As shown, the method includes:

[0032] Step 101: Acquire a target image including a target formula.

[0033] The target formula may be a formula to be identified in the target image, and the target formula may be a formula recorded in an article. This embodiment does not limit the type of the target formula, for example, the target formula may include a mathematical formula or a physical formula. This embodiment does not limit the number of the target formulas, and the number of the target formulas may be one or more.

[0034] The target image may be an image obtained by capturing an image of the article where the target formula is located, and the target image may also record the formula context of the target formula. The formula context may be the text located around the target formula in the article. For example, the formula context may include the text located before the target formula in the article, and / or the text located after the target formula in the article. It is understandable that the formula context may include text that records the symbolic meaning, formula meaning, application scope, etc. of the target formula to explain the target formula.

[0035] In the disclosed embodiment, the user can take a photo or screenshot of the part of the article including the target formula to obtain a target image including the target formula and its corresponding formula context, and then input the target image into the formula recognition device, which receives the target image. Alternatively, the formula recognition device can automatically take a photo or screenshot of the article to obtain a target image including the target formula and its corresponding formula context.

[0036] Step 102: input the target image and formula recognition prompt words into the formula recognition model, and output the formula content and formula position of the target formula; wherein the formula recognition model is obtained through multi-task joint training based on multiple prompt words, and the multiple prompt words include formula recognition prompt words and formula context prompt words.

[0037] Among them, the formula recognition model can be a multimodal model, and the formula recognition model can realize formula recognition in an end-to-end form. This embodiment does not limit the type of the formula recognition model. For example, the formula recognition model can be a visual language model. Prompt refers to the input information provided to the model, which is used to guide the model to generate corresponding outputs. The prompt word can enable the model to better understand the needs and intentions of the user. The formula recognition prompt word can be a text sentence that guides the model to perform formula recognition. The formula context prompt word can be a text sentence used to guide the model to perform formula context recognition. Multi-task joint training can be a training method that allows the model to consider the goals and constraints of multiple tasks simultaneously during the training process. In this embodiment, the multi-task joint training can enable the model to learn formula recognition and formula context recognition at the same time, thereby utilizing the correlation and complementarity between formula recognition and formula context recognition, and improving the performance of each individual function. The multi-task joint training is also called multi-task learning.

[0038] The formula content may be a target formula recorded in a text format, and this embodiment does not limit the text format. The formula position may be the position of the target formula in the target image, and the formula position may be recorded by at least one coordinate, for example, the formula position may be recorded by two coordinates (x1, y1) and (x2, y2).

[0039] In this embodiment, multi-task joint training is performed in advance based on the formula recognition prompt words and formula context prompt words as training data to obtain a formula recognition model. The formula recognition device inputs the formula recognition prompt words and the target image recording the target formula and the formula context into the formula recognition model, and the formula recognition model performs end-to-end recognition on the target formula in the target image to identify the formula content and the formula location.

[0040] In some embodiments of the present disclosure, the formula recognition model includes a visual encoder and a language decoder. The visual encoder can be an encoder for extracting features from an image, and the language decoder can be a decoder for decoding the formula content and formula position in a text format, and the language decoder is also called a language model. Figure 2 A flow chart of another formula recognition method provided in some embodiments of the present disclosure is shown as follows: Figure 2 As shown, in this embodiment, the target image and the formula recognition prompt word are input into the formula recognition model, and the formula content and formula position of the target formula are output, including:

[0041] Step 201: Input the target image into the visual encoder of the formula recognition model to extract features and obtain visual features.

[0042] The visual feature may be a feature extracted from the target image from the visual dimension, representing the features that can be seen from the user's perspective, and the visual feature may include features of various elements such as text, formulas, and tables included in the target image. The visual feature may include multiple unit features (Token), and a unit feature refers to a one-dimensional feature of an image block obtained by segmenting the target image in the visual dimension.

[0043] In this embodiment, the formula recognition device can input the target image into the visual encoder of the formula recognition model, and the visual encoder can perform feature extraction on the target image to obtain visual features. The specific extraction method can be, for example: the visual encoder can divide the target image into multiple image blocks, and extract the block features of each image block to obtain multiple block features, and transform the multiple block features from two-dimensional data into one-dimensional data through stretching, and convert each block feature into a unit feature to obtain multiple unit features, and determine the multiple unit features as visual features.

[0044] Step 202: Input the visual features and the formula recognition prompt words into the language decoder of the formula recognition model for autoregressive decoding to obtain the formula content and formula position of the output target formula.

[0045] In this embodiment, the formula recognition device can input the visual features and formula recognition prompt words output by the visual encoder into the language decoder of the formula recognition model. The language decoder performs implicit formula context extraction on the visual features and formula recognition prompt words, implements autoregressive decoding, and then outputs the formula content and formula position of the target formula.

[0046] Figure 3 A schematic diagram of formula recognition provided for some embodiments of the present disclosure, such as Figure 3 As shown, the user can input the target image containing the target formula and its corresponding formula recognition prompt word into the formula recognition model. The formula recognition prompt word can be: recognize all formula contents and formula positions in the image. The visual decoder in the formula recognition model takes the target image as input data, extracts the image features of the target image, and sends the image features to the language decoder. The language decoder takes the image features and the formula recognition prompt word as input data, and sequentially decodes the formula position 1 and formula content 1 of formula 1 in the target image, and the formula position 2 and formula content 2 of formula 2.

[0047] In the above scheme, the feature extraction of the image is performed based on the visual encoder, and the visual features and prompt words are decoded by the language decoder to generate the formula recognition result, thereby realizing the formula recognition based on multimodal data.

[0048] In some embodiments of the present disclosure, during the decoding process, the language decoder determines the formula context of the target formula based on visual features and the identified formula position of the target formula, and determines the formula content of the target formula based on the formula context of the target formula and the visual features.

[0049] In this embodiment, the language decoder can decode the visual features to obtain the formula position of the target formula. The formula context of the target formula is determined by neural network calculation based on the visual features and the formula position. Furthermore, the formula content of the target formula is determined by neural network calculation based on the visual features and the context of the target formula.

[0050] In the above scheme, during the decoding process of the language decoder, the extraction of the formula context is implicitly realized. Since the formula context helps the model understand and recognize the formula, the formula content is subsequently extracted based on the formula context, making the final recognized formula content more accurate.

[0051] The formula recognition scheme provided by the embodiment of the present disclosure obtains a target image including a target formula; inputs the target image and formula recognition prompt words into a formula recognition model, and outputs the formula content and formula position of the target formula; wherein the formula recognition model is obtained through multi-task joint training based on multiple prompt words, and the multiple prompt words include formula recognition prompt words and formula context prompt words. By adopting the above technical scheme, multi-task joint training is performed in multiple dimensions such as formula recognition and formula context recognition to obtain a formula recognition model, and the image and formula recognition prompt words are input into the formula recognition model to obtain the formula content and formula position of the formula. Through multi-task joint training, the formula recognition model can learn and share knowledge from multiple tasks, which can improve the performance and generalization ability of the model, and consider the formula context during recognition, effectively improving the accuracy of the model's formula recognition. When the model is applied, one-stage formula recognition is realized in an end-to-end manner, avoiding the error accumulation in two-stage recognition, and further improving the accuracy of formula recognition.

[0052] In addition, the formula context promotes the understanding and recognition of the formula. Multi-task joint training can enable the formula recognition model to have the advantages of being trained by different tasks. The formula recognition model has more comprehensive formula perception and understanding capabilities and better formula recognition accuracy.

[0053] Figure 4 A flowchart of another formula recognition method provided in some embodiments of the present disclosure is shown as follows: Figure 4 As shown, in some embodiments of the present disclosure, the formula recognition method further includes:

[0054] Step 401: construct multiple sample data based on multiple prompt words.

[0055] Among them, the multiple prompt words may include prompt words for guiding the model to realize different functions, and the multiple prompt words may include at least two of formula positioning prompt words, formula content recognition prompt words, formula extraction prompt words, and formula context prompt words. The sample data may include data for training the basic model so that the basic model has the formula recognition function. In some embodiments of the present disclosure, each type of sample data corresponds to a training task. A training task can be a task that enables the basic model to have a specific function through training.

[0056] In this embodiment, for each prompt word, one or more data types corresponding to the prompt word are determined, and the prompt word is combined with the corresponding type of data to obtain one or more sample data corresponding to the prompt word.

[0057] In some embodiments of the present disclosure, a variety of sample data are constructed based on a plurality of prompt words, including at least two of the following:

[0058] Optionally, multiple sample data are constructed based on multiple prompt words, including: constructing corresponding sample data based on formula positioning prompt words, sample images and formula position labels in formula recognition prompt words, or based on formula positioning prompt words, sample images, formula content and formula position labels.

[0059] The formula location prompt word may be a prompt word that guides the model to determine the location of the formula. The sample image may be an image containing a formula in the sample data. The formula location label may be a label (Label) of the location of the formula in the sample image, and the label may be known and expected output data associated with the input data for training the basic model.

[0060] In this embodiment, in the sample data set for the formula location task in the training task, the input data of the basic model may include formula location prompt words and sample images, and the output data of the basic model may include formula location labels of all sample formulas in the sample images. Through this formula location task, the model can be trained to have the ability to identify all formula locations in an image.

[0061] In the sample data set for the formula location retrieval task in the training task, the input data of the basic model may include formula location prompt words, sample images, and part or all of the formula content, and the output data of the basic model may include the formula location label of the formula corresponding to the formula content in the sample image. Through this formula location retrieval task, the model can be trained to recognize the formula location of a specified formula in an image.

[0062] Optionally, multiple sample data are constructed based on multiple prompt words, including: constructing corresponding sample data according to formula content recognition prompt words, sample images, formula content labels, and formula positions or formula contexts in formula recognition prompt words.

[0063] The formula content recognition prompt word may be a prompt word that guides the model to determine the formula content. The formula content label may be a label that records the text content corresponding to the formula in the sample image.

[0064] In this embodiment, in the sample data set for the content recognition task based on the formula position in the training task, the input data of the basic model may include the formula content recognition prompt word, the sample image, and the formula position, and the output data of the basic model may include the formula content label of the formula corresponding to the formula position in the sample image. Through this content recognition task based on the formula position, the model can be trained to recognize the ability of the formula content at a specified position in an image.

[0065] In the sample data set for the content recognition task based on formula context in the training task, the input data of the basic model may include formula content recognition prompt words, sample images, and formula contexts, and the output data of the basic model may include the formula content label of the sample formula corresponding to the formula context in the sample image. Through this content recognition task based on formula context, the model can be trained to recognize the formula content of the formula surrounded by the specified formula context in the image.

[0066] Optionally, constructing a variety of sample data based on a plurality of prompt words includes: extracting prompt words, sample texts and formula content labels according to formulas to construct corresponding sample data.

[0067] The formula extraction prompt word may be a prompt word that guides the model to extract the formula content contained in the text. The sample text may be a text containing the formula content, the sample text may include the formula content, the formula context and other texts, and the sample text may be an article in text format, in which the formula exists.

[0068] In this embodiment, in the sample data set for the text formula extraction task in the training task, the input data of the basic model may include formula extraction prompt words and sample text, and the output data of the basic model may include formula content labels. Through this text formula extraction task, the model's ability to extract formula content from a piece of text can be trained.

[0069] Optionally, a variety of sample data are constructed based on a plurality of prompt words, including: constructing corresponding sample data according to formula context prompt words, sample images, formula positions and formula context labels.

[0070] The formula context label may be a label recording the formula context corresponding to the formula in the sample image.

[0071] In this embodiment, in the sample data set for the formula context extraction task in the training task, the input data of the basic model may include formula context prompt words, sample images, and formula positions, and the output data of the basic model may include the formula context label of the formula corresponding to the formula position in the sample image. Through this formula context extraction task, the model can be trained to extract the formula context corresponding to the formula at the specified position in the image.

[0072] Step 402: Perform multi-task joint training on the basic model based on various sample data to obtain a formula recognition model.

[0073] Among them, the basic model can be a visual language model that has not completed multi-task joint training.

[0074] In this embodiment, after determining multiple types of sample data, for each sample data, the data in the sample data that is not a label type is used as the input data of the basic model, and the data in the sample data that is a label type is used as the output data of the basic model. The basic model is trained jointly for multiple tasks so that the basic model has at least two functions of formula locating function, formula location retrieval function, content recognition function based on formula location, content recognition function based on formula context, text formula extraction function, and formula context extraction function. The training is completed to obtain the formula recognition model.

[0075] In some embodiments of the present disclosure, a basic model is trained with multiple tasks based on various sample data to obtain a formula recognition model, including:

[0076] Obtain the allocation ratio of multiple training tasks corresponding to multiple sample data; train the basic model using the corresponding sample data in the proportion of each training task according to the allocation ratio until all the training tasks are completed; and determine the successfully trained basic model as the formula recognition model.

[0077] The allocation ratio may be the quantitative ratio of sample data corresponding to different training tasks. The allocation ratio may be set according to user needs, etc., and this embodiment does not limit the allocation ratio. For example, the allocation ratio may be the same for multiple training tasks.

[0078] In this embodiment, corresponding allocation ratios are pre-set for multiple training tasks. The formula recognition device can select multiple target sample data that meet the proportions of each training task from the multiple sample data corresponding to the multiple training tasks, and perform multi-task joint training on the basic model based on the multiple target sample data, so as to execute the corresponding training tasks according to the allocation ratio. After completing the training of the basic model, the formula recognition model is obtained. Therefore, by adjusting the allocation ratio, the proportion of different training tasks in the process of basic model training can be adjusted, making the training of the basic model more flexible.

[0079] In the above scheme, by performing multi-task joint training on the basic model, the trained formula recognition model can have the advantages of different training tasks and make different training tasks promote each other, thereby improving the model's formula recognition ability and enabling the model to have more comprehensive formula perception and formula understanding capabilities.

[0080] Next, the formula recognition method in the embodiment of the present disclosure is further explained through a specific example.

[0081] In this embodiment, a formula recognition model of the visual language model type is used to perform end-to-end modeling on images containing formulas. In the training phase of the formula recognition model, different prompt words are used to perform multi-modal multi-task joint training on the basic model. In the application phase of the formula recognition model, the formula content and formula position of the target formula are extracted from the target image containing the target formula through the formula recognition model. The specific steps are as follows:

[0082] Step a1, construct a basic model. The basic model includes a visual encoder and a language decoder. The visual encoder is used to extract block features, and transform the block features into unit features, and determine the visual features based on the unit features. The language decoder is used to autoregressively decode the input formula recognition prompt words and visual features, and output the corresponding formula position and formula content.

[0083] Step a2: Perform multi-modal and multi-task joint training on the basic model to obtain a formula recognition model.

[0084] The sample data corresponding to the training task is set, including at least the following two types:

[0085] For the formula localization task, the input data includes sample images and formula localization prompt words, and the output data includes the formula location labels of all formulas in the sample images.

[0086] The formula location retrieval task, the input data includes sample images, formula location prompt words, and formula content, and the output data includes the formula location label of the formula corresponding to the formula content.

[0087] The formula context extraction task, the input data includes sample images, formula context prompt words, formula positions, and the output data includes formula context labels.

[0088] The content recognition task based on the formula location includes the input data of the sample image, the formula content recognition prompt word, and the formula location, and the output data includes the formula content label.

[0089] The content recognition task based on formula context includes sample images, formula content recognition prompt words, and formula context, and the output data includes formula content labels.

[0090] For the text formula extraction task, the input data includes sample text containing formula content and formula extraction prompt words, and the output data includes formula content labels.

[0091] The above sample data are selected according to the distribution ratio, and the basic model is trained with multi-tasks according to the selected sample data to improve the comprehensive formula perception and comprehension ability of the basic model, and make full use of the advantages of different training tasks to promote each other and improve the formula recognition performance, so as to obtain a formula recognition model that has completed the training.

[0092] Step a3: Perform end-to-end formula recognition based on the formula recognition model. Specifically, a target image containing a target formula and a formula recognition prompt word are input into the formula recognition model. The target image is input into the visual encoder in the formula recognition model to extract visual features, and then the visual features and the formula recognition prompt word are input into the language decoder in the formula recognition model to decode the formula content recorded in the target image.

[0093] In the above scheme, based on the formula recognition model, the target formula in the target image is recognized end-to-end, without relying on the two-stage recognition method of first detection and then recognition in the related technology. The formula context of end-to-end formula recognition is fully acquired and utilized, giving full play to the advantages of end-to-end recognition.

[0094] Figure 5 This is a schematic diagram of the structure of a formula recognition device provided by some embodiments of the present disclosure. The device can be implemented by software and / or hardware and can generally be integrated into an electronic device. Figure 5 As shown, the device comprises:

[0095] An acquisition module 501 is used to acquire a target image including a target formula;

[0096] Processing module 502 is used to input the target image and formula recognition prompt words into a formula recognition model, and output the formula content and formula position of the target formula; wherein the formula recognition model is obtained through multi-task joint training based on multiple prompt words, and the multiple prompt words include the formula recognition prompt words and formula context prompt words.

[0097] In some embodiments of the present disclosure, the formula recognition model includes a visual encoder and a language decoder, and the processing module 502 includes:

[0098] Inputting the target image into the visual encoder of the formula recognition model to extract features and obtain visual features;

[0099] The visual features and the formula recognition prompt words are input into the language decoder of the formula recognition model for autoregressive decoding, so as to obtain the formula content and formula position of the target formula as output.

[0100] In some embodiments of the present disclosure, the language decoder determines the formula context of the target formula based on the visual features and the identified formula position of the target formula during the decoding process, and determines the formula content of the target formula based on the formula context of the target formula and the visual features.

[0101] In some embodiments of the present disclosure, the device further includes:

[0102] A construction module, used for constructing a variety of sample data based on a plurality of prompt words;

[0103] The training module is used to perform multi-task joint training on the basic model based on the multiple sample data to obtain the formula recognition model.

[0104] In some embodiments of the present disclosure, the building blocks include at least two of the following:

[0105] A first construction submodule is used to construct corresponding sample data according to the formula location prompt word, the sample image and the formula location label in the formula recognition prompt word, or according to the formula location prompt word, the sample image, the formula content and the formula location label;

[0106] A second construction submodule is used to construct corresponding sample data according to the formula content recognition prompt word, the sample image, the formula content label, and the formula position or the formula context in the formula recognition prompt word;

[0107] The third construction submodule is used to extract prompt words, sample text and formula content labels according to the formula to construct corresponding sample data;

[0108] The fourth construction submodule is used to construct corresponding sample data according to the formula context prompt words, sample images, formula positions and formula context labels.

[0109] In some embodiments of the present disclosure, each type of sample data corresponds to a training task.

[0110] In some embodiments of the present disclosure, the training module is used to:

[0111] Obtaining allocation ratios of multiple training tasks corresponding to the multiple types of sample data;

[0112] Training the basic model using the corresponding sample data in the proportion of each training task according to the allocation ratio until all the training tasks are completed;

[0113] The successfully trained basic model is determined as the formula recognition model.

[0114] The formula recognition device provided in the embodiments of the present disclosure can execute the formula recognition method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0115] The embodiment of the present disclosure provides a computer program product, including a computer program / instruction, which implements the steps of the above formula recognition method when executed by a processor.

[0116] Figure 6 A schematic diagram of the structure of an electronic device provided for some embodiments of the present disclosure.

[0117] The following specific reference Figure 6 , which shows a schematic diagram of the structure of an electronic device 600 suitable for implementing the embodiment of the present disclosure. The electronic device 600 in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0118] like Figure 6As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0119] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0120] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the formula recognition method of the embodiment of the present disclosure are executed.

[0121] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0122] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0123] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0124] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: obtains a target image including a target formula; inputs the target image and formula recognition prompt words into a formula recognition model, and outputs the formula content and formula position of the target formula; wherein the formula recognition model is obtained through multi-task joint training based on multiple prompt words, and the multiple prompt words include formula recognition prompt words and formula context prompt words.

[0125] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0126] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0127] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0128] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0129] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0130] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the type, scope of use, usage scenarios, etc. of the information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0131] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0132] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0133] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A formula recognition method, characterized in that: include: obtaining a target image including a target formula; Input the target image and the formula recognition prompt word into the formula recognition model, and output the formula content and formula position of the target formula; The formula recognition model is obtained through multi-task joint training based on multiple prompt words, and the multiple prompt words include the formula recognition prompt words and formula context prompt words.

2. The method according to claim 1, characterized in that The formula recognition model includes a visual encoder and a language decoder. The target image and the formula recognition prompt word are input into the formula recognition model, and the formula content and formula position of the target formula are output, including: Inputting the target image into the visual encoder of the formula recognition model to extract features and obtain visual features; The visual features and the formula recognition prompt words are input into the language decoder of the formula recognition model for autoregressive decoding, so as to obtain the formula content and formula position of the target formula as output.

3. The method according to claim 2, characterized in that During the decoding process, the language decoder determines the formula context of the target formula based on the visual features and the identified formula position of the target formula, and determines the formula content of the target formula based on the formula context of the target formula and the visual features.

4. The method according to claim 1, characterized in that: The method further comprises: Construct a variety of sample data based on multiple prompt words; Based on the multiple sample data, multi-task joint training is performed on the basic model to obtain the formula recognition model.

5. The method according to claim 4, characterized in that Construct a variety of sample data based on multiple prompt words, including at least two of the following: Constructing corresponding sample data according to the formula positioning prompt word, the sample image and the formula position label in the formula recognition prompt word, or according to the formula positioning prompt word, the sample image, the formula content and the formula position label; Constructing corresponding sample data according to the formula content recognition prompt word, the sample image, the formula content label, and the formula position or the formula context in the formula recognition prompt word; Extract prompt words, sample text and formula content labels according to the formula, and construct corresponding sample data; Corresponding sample data is constructed according to the formula context prompt words, sample images, formula positions and formula context labels.

6. The method according to claim 4, characterized in that Each type of sample data corresponds to a training task.

7. The method according to claim 4, characterized in that Performing multi-task joint training on the basic model based on the multiple sample data to obtain the formula recognition model includes: Obtaining allocation ratios of multiple training tasks corresponding to the multiple types of sample data; Training the basic model using the corresponding sample data in the proportion of each training task according to the allocation ratio until all the training tasks are completed; The successfully trained basic model is determined as the formula recognition model.

8. A formula recognition device, characterized in that: include: An acquisition module, used for acquiring a target image including a target formula; A processing module is used to input the target image and formula recognition prompt words into a formula recognition model, and output the formula content and formula position of the target formula; wherein the formula recognition model is obtained through multi-task joint training based on multiple prompt words, and the multiple prompt words include the formula recognition prompt words and formula context prompt words.

9. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the formula recognition method described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the formula recognition method described in any one of claims 1 to 7.

Citation Information

Patent Citations

  • Prompt information generation method and device, object recognition method and device and unmanned vehicle

    CN117612120A

  • Method and system for identifying handwritten answer mathematical formula with question prompt

    CN117612184A

  • Multimedia content classification method and device, electronic equipment and storage medium

    CN117648606A

  • Method and device for realizing Internet harmful image detection processing based on large language model, processor and computer readable storage medium

    CN117876911A

  • Target detection method and device, equipment and medium

    CN118196398A