A training method for an image processing model and related products

Abstract

The embodiments of the present invention provide a method for training an image processing model and related products thereof. The image processing model includes a recognition model, and the training method includes: obtaining training pictures, wherein the training pictures include labeled pure text pictures, pure formula pictures, and text-formula mixed pictures, and the annotations include word segmentation according to word segmentation rules applicable to formulas and texts; inputting the training pictures into the recognition model for training to obtain output results; calculating the loss function based on the annotations in the training pictures and the output results to update the parameters of the recognition model in back propagation. The embodiments of the present invention use only one model, which reduces resource consumption and improves recognition quality. In addition, a method for image processing using an image processing model, as well as a computing device and a computer-readable storage medium are also provided.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of image processing technology. More specifically, embodiments of the present invention relate to a method for training an image processing model, a method for performing image processing using an image processing model, a computing device, and a computer-readable storage medium. Background Art

[0002] This section is intended to provide background or context for embodiments of the present invention as recited in the claims. The description herein may include concepts that could be explored, but not necessarily concepts that have been previously conceived or explored. Therefore, unless otherwise indicated herein, the material described in this section is not prior art with respect to the specification and claims of this application and is not admitted to be prior art by inclusion in this section.

[0003] With the rapid development of artificial intelligence (AI) and intelligent hardware, AI is being applied to an increasingly diverse range of business scenarios. This is particularly true in educational settings, such as automated grading, error entry, automated assessment, and multilingual translation. These scenarios require the use of optical character recognition (OCR) to automatically detect text, eliminating manual text entry and improving efficiency.

[0004] In actual scenarios, documents may contain not only text but also formulas, such as mixed text and formulas (including both text and formula areas in one line). This type of text is a challenge for the recognition model if it is not separated. In order to solve the problem of detecting text with formulas, the existing technology divides the text into multiple areas and uses different detection models to identify text and formulas respectively. For example, a text detection model is used to identify the text area, and a formula detection model is used to identify the formula area. The disadvantages of this solution are:

[0005] First, this solution requires two detection models, each for identifying a specific region. Therefore, the solution's recognition performance depends heavily on how the regions are divided. If the regions are improperly divided or the positions of the region boundaries are misaligned, the final recognition quality will be poor. In practical applications, region boundaries are often difficult to determine, resulting in the solution's poor robustness.

[0006] Second, two detection models require twice as many resources, and each model requires training and engineering, resulting in higher costs.

[0007] Third, detection models typically use recurrent neural networks or convolutional neural networks. For example, for an image, a convolutional neural network extracts features, encodes them using a recurrent neural network, and finally decodes them using a decoder to obtain the recognition result. Because the reasoning structure of recurrent and convolutional neural networks is linear and step-by-step, the input of subsequent text depends on information from the previous text, which results in slower model reasoning speed.

[0008] Fourth, the use of this solution is greatly limited in offline scenarios. Since the server's computing resources cannot be utilized in offline scenarios, and the solution itself relies on a large amount of computing resources, the implementation effect of this solution in offline scenarios is further affected. Summary of the Invention

[0009] The existing technology suffers from the aforementioned drawbacks when processing documents containing formulas. Therefore, there is a strong need for an improved image processing model training method to obtain a new image processing model, and to utilize the image processing model to perform an image processing method that addresses at least one of the aforementioned drawbacks. The present invention also provides a computing device and a computer-readable storage medium.

[0010] In this context, embodiments of the present invention are intended to provide a method for training an image processing model and related products.

[0011] In a first aspect of an embodiment of the present invention, a training method for an image processing model is provided, wherein the image processing model includes a recognition model, and the training method includes: obtaining training pictures, wherein the training pictures include labeled pure text pictures, pure formula pictures, and text-formula mixed pictures, and the annotations include word segmentation according to word segmentation rules applicable to formulas and texts; inputting the training pictures into the recognition model for training to obtain output results; calculating a loss function based on the annotations in the training pictures and the output results to update the parameters of the recognition model in back propagation.

[0012] In one embodiment, the word segmentation rules include: using formula syntax to mark symbols that appear in both text and formula; using formula syntax to mark symbols that are exclusive to formulas; and using general text to mark symbols that are exclusive to texts.

[0013] In another embodiment, the recognition model includes a feature extractor, an encoder and a decoder, and the training picture is input into the recognition model for training to obtain an output result, which includes: using the feature extractor to extract a first feature of the training picture containing visual information; using the encoder to encode the first feature to obtain a second feature containing semantic information; and using the decoder to map the second feature into text or a formula to obtain the output result.

[0014] In another embodiment, the recognition model also includes a blocking module and a merging module; the training method also includes: using the blocking module to block the training image to obtain at least two sub-images with overlapping parts, and using the feature extractor and encoder to process each sub-image separately; and using the merging module to merge the features obtained by processing each sub-image to obtain the second feature.

[0015] In another embodiment, the training image is divided into blocks, including: dividing the target image into blocks according to a predetermined size to obtain sub-images of the same size, wherein adjacent sub-images have overlapping parts, and using a specified value to fill in the parts of the sub-images where the image content size is smaller than the predetermined size.

[0016] In one embodiment, obtaining the training image includes performing one or more of the following processing modes on the training image to perform data augmentation: rotation transformation, perspective transformation, contrast adjustment, blurring, and noise addition.

[0017] In another embodiment, the feature extractor adopts a convolutional neural network structure or a model based on an attention mechanism; or, the encoder adopts the encoder part of a model based on an attention mechanism; or, the decoder adopts a fully connected structure.

[0018] In another embodiment, the image processing model also includes a classification model, and the training method also includes training the classification model as follows: freezing the parameters of the recognition model; and training the classification model using the recognition result of the recognition model and the second feature extracted from the recognition model to classify the recognition result to obtain a formula area and / or a text area.

[0019] In another embodiment, the classification model includes an embedding layer, an encoding layer and a classification layer, and training the classification model includes: using the embedding layer to convert the recognition result into a word vector; using the encoding layer to encode the word vector and the second feature to obtain an encoding result; using the classification layer to classify the encoding result to obtain the formula area and / or text area.

[0020] In one embodiment, the formula area includes: a formula starting position and / or a formula ending position.

[0021] In another embodiment, the encoding layer adopts the encoder part, convolutional neural network structure or recurrent neural network structure in the attention mechanism model; or, the classification layer adopts the convolutional neural network structure or the fully connected structure.

[0022] In a second aspect of an embodiment of the present invention, a method for image processing using an image processing model is provided, wherein the image processing model has been trained according to the training method described in any one of the first aspects of the embodiment of the present invention; the method comprises: obtaining a target image; processing the target image according to the image processing model to recognize text and / or formulas in the target image.

[0023] In one embodiment, acquiring the target image includes: acquiring a complete image including the identified target; or acquiring multiple consecutive video frames, and fusing the multiple consecutive video frames to obtain a complete image including the identified target.

[0024] In a third aspect of an embodiment of the present invention, a computing device is provided, comprising: a processor configured to execute program instructions; and a memory configured to store the program instructions, wherein when the program instructions are loaded and executed by the processor, the processor executes the training method of the image processing model according to any one of the first aspect of the embodiment of the present invention or executes the image processing method according to any one of the second aspect of the embodiment of the present invention.

[0025] In a fourth aspect of the embodiments of the present invention, a computer-readable storage medium is provided, in which program instructions are stored. When the program instructions are loaded and executed by a processor, the processor executes the training method of the image processing model according to any one of the first aspects of the embodiments of the present invention or executes the image processing method according to any one of the second aspects of the embodiments of the present invention.

[0026] The image processing model training method according to an embodiment of the present invention uses a single recognition model to simultaneously recognize formulas and text, replacing the dual-model approach of the prior art. Consequently, when loading the model, the memory occupied is reduced, which is crucial for implementing the image processing method on intelligent hardware. Furthermore, since the image processing method does not require region segmentation, it is more robust.

[0027] In some other embodiments of the present invention, the encoder in the recognition model can adopt an encoder part based on the attention mechanism model (such as a transformer encoder), abandoning the traditional recurrent neural network or convolutional neural network. Since the encoder part based on the attention mechanism model is a highly parallel network, it can accelerate the reasoning speed of the recognition model without reducing the recognition ability. In addition, based on the current architecture and development direction of processor chips, the parallel processing capability of processor chips is constantly increasing, so the embodiments of the present invention are also more suitable for current and future processor chip architectures.

[0028] In other embodiments of the present invention, a partitioning module is incorporated into the recognition model. This partitioning module can divide an image into multiple overlapping sub-images, allowing for parallel processing of these sub-images, significantly improving inference speed. Furthermore, processing each sub-image separately reduces interference from other sub-images, thereby ensuring inference accuracy.

[0029] In some further embodiments of the present invention, the image processing model may further include a classification model, through which the formula area and / or text area in the recognition result may be indicated to meet the needs of the user.

[0030] In general, based on the above technical effects, the implementation methods of the present invention are very friendly to offline scenarios, so that the present invention can be well applied to various offline scenarios. For example, it can be used in smart hardware in the field of auxiliary learning to quickly and accurately realize automatic correction, wrong question entry and automatic judgment of mathematics, physics and chemistry subjects. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily apparent by reading the following detailed description with reference to the accompanying drawings, in which several embodiments of the present invention are shown by way of example and not limitation, in which:

[0032] Figure 1 A block diagram schematically illustrates an exemplary computing system 100 suitable for implementing embodiments of the present invention;

[0033] Figure 2 Schematically shows a flow chart of a method 200 for training an image processing model according to one embodiment of the present invention;

[0034] Figure 3 The following schematically shows a training framework diagram of a recognition model according to an embodiment of the present invention;

[0035] Figure 4 The following schematically shows a training framework diagram of a classification model according to an embodiment of the present invention;

[0036] Figure 5 Schematically shows a training framework diagram of a classification model according to another embodiment of the present invention;

[0037] Figure 6 Schematically shows a flow chart of an image processing method 600 according to an embodiment of the present invention;

[0038] Figure 7 Schematically shows a flow chart of an image processing method 700 according to another embodiment of the present invention; and

[0039] Figure 8 A schematic block diagram of a computing device 1000 according to an embodiment of the present invention is schematically shown.

[0040] In the drawings, the same or corresponding reference numerals denote the same or corresponding parts. DETAILED DESCRIPTION

[0041] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided solely to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0042] Figure 1 1 shows a block diagram of an exemplary computing system 100 suitable for implementing embodiments of the present invention. Figure 1 As shown, the computing system 100 may include a central processing unit (CPU) 101, a random access memory (RAM) 102, a read-only memory (ROM) 103, a system bus 104, a hard disk controller 105, a keyboard controller 106, a serial interface controller 107, a parallel interface controller 108, a display controller 109, a hard disk 110, a keyboard 111, a serial peripheral device 112, a parallel peripheral device 113, and a display 114. Of these devices, the CPU 101, RAM 102, ROM 103, hard disk controller 105, keyboard controller 106, serial controller 107, parallel controller 108, and display controller 109 are coupled to the system bus 104. The hard disk 110 is coupled to the hard disk controller 105, the keyboard 111 is coupled to the keyboard controller 106, the serial peripheral device 112 is coupled to the serial interface controller 107, the parallel peripheral device 113 is coupled to the parallel interface controller 108, and the display 114 is coupled to the display controller 109. It should be understood that Figure 1 The structured block diagram is only for the purpose of illustration, rather than for limiting the scope of the present invention. In some cases, some devices may be added or reduced according to specific circumstances.

[0043] Those skilled in the art will appreciate that embodiments of the present invention may be implemented as a system, method, or computer program product. Therefore, the present disclosure may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of hardware and software, generally referred to herein as a "circuit," "module," "unit," or "system." Furthermore, in some embodiments, the present invention may also be implemented in the form of a computer program product in one or more computer-readable media containing computer-readable program code.

[0044] Any combination of one or more computer-readable media can be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (non-exhaustive examples) of computer-readable storage media can include, for example: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0045] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0046] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0047] The computer program code for performing the operations of the present invention can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0048] The following will describe the embodiments of the present invention with reference to the flowcharts of the methods and block diagrams of the devices (or systems) according to the embodiments of the present invention. It should be understood that each block in the flowcharts and / or block diagrams, as well as the combination of blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, and these computer program instructions are executed by the computer or other programmable data processing device to produce a device that implements the functions / operations specified in the blocks in the flowcharts and / or block diagrams.

[0049] These computer program instructions can also be stored in a computer-readable medium that enables a computer or other programmable data processing device to operate in a specific manner. In this way, the instructions stored in the computer-readable medium produce a product that includes an instruction device that implements the functions / operations specified in the blocks in the flowchart and / or block diagram.

[0050] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, such that the instructions executed on the computer or other programmable apparatus provide a process that implements the functions / operations specified in the blocks in the flowchart and / or block diagram.

[0051] According to an embodiment of the present invention, a training method for an image processing model and related products thereof are proposed. In addition, the number of any element in the accompanying drawings is for illustration only and not for limitation, and any naming is only for distinction and does not have any limiting meaning.

[0052] The principles and spirit of the present invention are explained in detail below with reference to several representative embodiments of the present invention. SUMMARY OF THE INVENTION

[0054] The inventors discovered that formulas are fundamentally different from traditional text and contain structural information, which poses a challenge to deep learning-based recognition technologies. Therefore, existing methods use text detection to separate text and formulas, then recognize them separately. However, existing solutions require two detection models to identify corresponding regions, leading to various drawbacks such as poor recognition quality, poor robustness, high costs, and slow processing speeds.

[0055] Based on this, the inventors discovered that a single image processing model can be used to identify pure text, pure formulas, or text with formulas. This eliminates the need for segmenting text regions and requiring two detection models, thereby improving recognition quality and robustness, reducing resource consumption, and also lowering training and engineering costs. Accordingly, to train the image processing model, it is necessary to annotate images of pure text, pure formulas, and mixed text and formulas.

[0056] In addition, the inventors discovered that the recurrent neural network in the image processing model can be replaced with a model based on the attention mechanism, which can accelerate the model's reasoning speed without reducing recognition capabilities. For example, the encoder part of the model based on the attention mechanism can be used to replace the recurrent neural network.

[0057] Furthermore, the inventors have also discovered that a block-based approach can be used to accelerate reasoning speed.

[0058] After introducing the basic principles of the present invention, various non-limiting embodiments of the present invention are described in detail below.

[0059] Exemplary Methods

[0060] Reference below Figure 2 This section describes a method for training an image processing model according to an exemplary embodiment of the present invention. It should be noted that the embodiments of the present invention can be applied to any applicable scenario. The purpose of the training method is to obtain an image processing model that can be used to recognize text containing formulas.

[0061] Figure 2 The flowchart of the training method 200 of the image processing model according to one embodiment of the present invention is schematically shown.

[0062] A method for training an image processing model includes steps S201 to S203, wherein the image processing model includes a recognition model. Depending on the actual scenario, the image processing model may also include other models besides the recognition model. These other models can provide more information based on image recognition. For example, the other models may be classification models that further provide information about the location of formulas. This embodiment primarily describes the training process of the recognition model; other models will be described in detail in subsequent embodiments.

[0063] like Figure 2 As shown, in step S201, training images are obtained, wherein the training images include annotated pure text images, annotated pure formula images, and annotated mixed text and formula images, wherein the annotations include word segmentation according to word segmentation rules applicable to formulas and texts.

[0064] In step S202, the training picture is input into the recognition model for training to obtain an output result.

[0065] In step S203, a loss function is calculated based on the annotations in the training image and the output results to update the parameters of the recognition model in back propagation.

[0066] The training images include three types: labeled text-only images, labeled formula-only images, and labeled images mixed with text and formulas. The coexistence of these three types of images empowers the trained recognition model to identify each type of image. Furthermore, since the three types of images are somewhat correlated—for example, labeled text-only images and labeled formula-only images both correlate with labeled text-mixed images—the trained recognition model is sensitive to the differences between text and formulas, demonstrating improved reasoning capabilities.

[0067] Specifically, in order to obtain annotated images in step S201, it is necessary to first obtain a large number of original images, such as original text-only images, original formula-only images, and original text-formula mixed images. It should be noted that in the original text-formula mixed images, the ratio of formulas to text is random, and the location of formulas is also random, thereby improving data quality.

[0068] Next, the original image needs to be annotated. For example, manual annotation can be performed after pre-annotation (automatic annotation). Of course, it is also possible to perform only automatic annotation or manual annotation. When performing automatic or manual annotation, you can first formulate word segmentation rules to apply to formulas and text.

[0069] Specifically, the word segmentation rules include: using formula syntax to mark symbols that appear in both text and formulas; using formula syntax to mark symbols that appear exclusively in formulas; and using general text to mark symbols that appear exclusively in text. The formula syntax can use LaTeX syntax. LaTeX is a TEX-based typesetting system that excels at generating complex tables and mathematical formulas.

[0070] For example, for symbols that appear in both text and formulas, the latex syntax is used, such as the circle symbol "①", which is marked as "\textcircle{1}", and the division sign " / " is marked as "\div". For symbols that are exclusive to formulas, the latex syntax is used, such as the fraction "1 / 2" is marked as "\frac{1}{2}", the cube of 2 is marked as "2 3 ” is marked as 2^{3}. For symbols exclusive to the text, universal text annotation is adopted, including but not limited to Chinese, English, Japanese and Korean. It should be noted here that for the sake of universality, quasi-structured text will be divided into multiple parts when annotated. The usual practice is to divide “\frac{1}{2}” into “\frac”, “{”, “1”, “}”, “{”, “2”, “}”. In order to adapt to the recognition model, the mergeable parts are merged in this embodiment. For example, “\frac{1}{2}” is marked as “\frac{”, “1”, “}{”, “2”, “}”.

[0071] Next, the segmented image is converted into a one-hot encoding. One-hot encoding, also known as single-bit encoding, primarily uses an N-bit state register to encode N states. Each state has its own register bit, and only one bit is active at any given time. One-hot encoding represents categorical variables as binary vectors. Generally speaking, one-hot encoding is the process of converting categorical variables into a form that is easily usable by machine learning algorithms. Converting images into one-hot encodings is a common technique in machine learning and will not be discussed further here.

[0072] Finally, pure text images, pure formula images, and mixed text and formula images are proportionately formed into samples and input into the recognition model for training. For example, pure text images account for 30% of the total samples, pure formula images account for 30% of the total samples, and mixed text and formula images account for 40% of the total samples. Of course, in other embodiments, the proportions of various types of images can be adjusted as needed.

[0073] Step S202 and step S203 are common technical means for model training in the field of machine learning. That is, first, the training image is input into the recognition model for training to obtain an output result, and then the loss function is calculated based on the annotation in the training image and the output result, and then the back propagation algorithm is executed according to the loss function, and the parameters of the recognition model are updated according to the learning rate; this is iterated continuously until the conditions are met (for example, the loss function is small enough); and finally a trained recognition model is obtained. As an example, the loss function can use CTC (Connectionist Temporal Classification); in other embodiments, other types of loss functions in the prior art can also be selected.

[0074] The above describes the process of the training method of the recognition model. The following will further explain the training method in detail in combination with the structure of the recognition model.

[0075] Figure 3 The training framework diagram of a recognition model according to one embodiment of the present invention is schematically shown. The recognition model includes a feature extractor 303, an encoder 304, and a decoder 306. Inputting a training image into the recognition model for training to obtain an output result includes: extracting a first feature of the training image containing visual information using the feature extractor 303; encoding the first feature using the encoder 304 to obtain a second feature containing semantic information; and mapping the second feature into text or a formula using the decoder 306 to obtain the output result.

[0076] In the recognition model, the feature extractor 303 is responsible for extracting low-dimensional features containing visual information from the input image (the low-dimensional features are referred to as first features in the present invention), thereby converting the image into a feature map. In the fields of image processing, machine learning, and pattern recognition, feature extraction is a primary operation process. For example, the feature extractor 303 can check each pixel in an image or image area to determine whether the pixel represents a feature; the input image is generally smoothed in the scale space, and then one or more features of the image are obtained by derivative operations. In one embodiment, the structure of the feature extractor 303 can use structures commonly used in the prior art such as convolutional neural networks or models based on attention mechanisms.

[0077] The encoder 304 is responsible for encoding the low-dimensional visual features to obtain high-dimensional features containing semantic information (the high-dimensional features are referred to as second features in the present invention); in the embodiment of the present invention, low-dimensional features and high-dimensional features are relative, and the features extracted by the feature extractor 303 are primary features, so the embodiment of the present invention calls them low-dimensional features; the encoder 304 encodes the low-dimensional features to obtain deeper features, so the embodiment of the present invention calls them high-dimensional features.

[0078] In one embodiment, the encoder 304 can adopt a convolutional neural network, a recurrent neural network, and a model based on an attention mechanism. Preferably, the encoder 304 uses the encoder part of the model based on the attention mechanism, such as the encoder part of the transformer, wherein the transformer is a new type of neural network structure, and its encoder part (encoder) is very suitable as the encoder 304 in the recognition model of the embodiment of the present invention. That is to say, the encoder part of the new neural network structure transformer is adopted, and the traditional recurrent neural network or convolutional neural network is discarded; because the neural network structure transformer is a highly parallelized network, it can accelerate the reasoning speed of the recognition model without reducing the recognition ability. In addition, based on the current processor chip architecture and development direction, the parallel processing capability of the processor chip is constantly increasing, so the embodiment of the present invention is also more suitable for current and future processor chip architectures.

[0079] Decoder 306 is responsible for mapping the extracted high-dimensional features to text categories. Preferably, decoder 306 adopts a fully connected structure. With a fully connected structure, the algorithm is easily deployed on the offline side, making the present invention suitable for use in offline scenarios.

[0080] like Figure 3 As shown, in order to improve the training effect, in addition to the above-mentioned feature extractor 303, encoder 304 and decoder 306, the recognition model of the present invention further adds a data augmentation module 301, a blocking module 302 and a merging module 305.

[0081] The data augmentation module 301 generates a large number of new samples based on the limited original samples by performing processing such as flipping, cropping, transforming, and adding noise to the original samples, thereby enhancing the generalization ability of the recognition model. In addition, the data augmentation module 301 can also simulate the rotation, perspective transformation, noise, contrast, and blur that exist in real images.

[0082] The block module 302 is used to divide the training image into multiple sub-images with overlapping parts, and then process each sub-image through the encoder 304. The significance of using the block module 302 is that, on the one hand, multiple sub-images can be processed in parallel through the block module 302; on the other hand, text that is farther away usually plays a smaller role in the recognition of the current text than text that is closer. For example, the previous paragraph and the next paragraph of the current text are more closely related to the current text, while the text that is more than ten words before and after the current word is more distant from the current text. In other words, processing each sub-image separately can reduce the interference of redundant information (information of other sub-images) that is farther away, thereby ensuring the accuracy of reasoning.

[0083] The merging module 305 is a module corresponding to the blocking module 302 and performs an operation completely opposite to that of the blocking module 302 , that is, the merging module 305 merges multiple sub-images into one image.

[0084] The following combination Figure 3 The training process of the recognition model is described in detail.

[0085] First, data augmentation is performed on the training image using one or more of the following processing methods: rotation, perspective transformation, contrast adjustment, blurring, and noise addition. Data augmentation, also known as data enhancement or data expansion, can generate more variations from limited data without substantially increasing the data volume, allowing the limited data to generate the equivalent value of more data and saving significant manual labeling costs. In practical applications, an original training image can be subjected to one or more processing methods; for example, the original training image can be rotated and contrast adjusted to generate a new training image. The specific processing method chosen should be determined based on training requirements and costs.

[0086] For example, after data augmentation, a training image P is generated. For the convenience of the following description, the training image P can be represented as three parts: part A, part B, and part C. For example, parts A and B are of equal size, and the remaining part C is smaller than parts A and B.

[0087] Then, the training image is segmented using the segmentation module 302 to obtain at least two sub-images with overlapping portions. For example, for training image P, the segmentation module 302 segments it into three sub-images PA, PB, and PC with overlapping portions. Sub-image PA comprises portion A, which also includes a blank portion and portion B1 that overlaps with sub-image PB; sub-image PB comprises portion B, which also includes portion A1 that overlaps with sub-image PA and portion C1 that overlaps with sub-image PC; and sub-image PC comprises portion C, which also includes portion B2 that overlaps with sub-image PB and a blank portion.

[0088] Specifically, the training image can be divided into blocks according to a predetermined size to obtain sub-images of the same size, wherein adjacent sub-images have overlapping parts, and the parts of the sub-images where the image content size is smaller than the predetermined size are filled with a specified value. Among them, sub-images PA, PB and PC are adjacent sub-images in sequence. By filling with the specified value, the sizes of the three sub-images containing part A, part B and part C, which originally have different sizes, are made the same. The blank part is filled with the specified value to facilitate processing by the feature extractor 303. For example, the blank part can be filled with the specified value "0". In other embodiments, the blank part can also be filled with other numerical values instead of the specified value "0", such as the specified value "255". In addition, the sizes of the sub-images can also be unequal. Of course, in this case, the structure of the feature extractor 303 needs to be adjusted to adapt.

[0089] Next, each sub-image is processed separately using the feature extractor 303 and encoder 304. Specifically, the sub-images processed by the blocking module 302 are input into the feature extractor 303 for processing, and corresponding low-dimensional features are obtained for each sub-image. The low-dimensional features of each sub-image are then input into the encoder 304 for processing, and the corresponding high-dimensional features are obtained for each sub-image. The feature extractor 303 and encoder 304 have been described in detail above and will not be repeated here.

[0090] As an example, the blocking module 302 processes the sub-images PA, PB and PC, which are processed by the feature extractor 303 and the encoder 304 respectively to obtain the high-dimensional features of the main part of each sub-image. Among them, the overlapping part is redundant, but the overlapping part has certain significance for the feature extraction and encoding of the main part. Therefore, the overlapping part can be retained during feature extraction and encoding, and is no longer needed after feature extraction and encoding. For example, for the sub-image PA, the overlapping part B1 with the sub-image PB has a certain positive effect on the feature extraction of its main part A. Similarly, for the sub-image PB, the overlapping part A1 with the sub-image PA and the overlapping part C1 with the sub-image PC have a certain positive effect on the feature extraction of its main part B; for the sub-image PC, the overlapping part B2 with the sub-image PB has a certain positive effect on the feature extraction of its main part C. Finally, in the output of the encoder 304, the overlapping part is replaced by Figure 3 For example, the overlapping portion B1 of the sub-image PA, the overlapping portions A1 and C1 of the sub-image PB, and the overlapping portion B2 of the sub-image are all represented as blanks, which means that the data in these portions are no longer needed and can be discarded in subsequent processes.

[0091] Next, the merging module 305 is used to merge the features obtained by processing each sub-image to obtain the second feature. For example, the merging module 305 concatenates the high-dimensional features corresponding to the sub-images PA, PB, and PC, removes the redundant parts, and removes the Figure 3 The blank part in , we can get the complete high-dimensional features corresponding to the training image P. It should be noted that Figure 3 In the example, after being processed by encoder 304, the image has been converted into corresponding data information. Therefore, A, B, and C after being processed by encoder 304 represent the data corresponding to the A, B, and C parts of the training image, rather than the training image itself. Similarly, the output of merging module 305 uses the same format as the training image P, but its meaning refers to the data corresponding to the training image P, rather than the training image P itself.

[0092] Finally, the complete high-dimensional features are input into the decoder 306 to obtain the output result; based on the output result and the annotation result, the loss function is calculated, the gradient of the parameters of each layer in the recognition model is calculated by the back propagation algorithm, and the parameters are updated according to the learning rate. After several iterations, the trained recognition model and the corresponding merging module 307 are finally obtained.

[0093] In a practical application scenario, users not only want to know the merge module 307, i.e., the recognition of text and formulas, but also want to know the location of the formula or text. Therefore, in some embodiments of the present invention, the image processing model includes not only a recognition model but also a classification model, which serves this purpose. This is described below with reference to the accompanying drawings.

[0094] Figure 4 The training framework diagram of the classification model according to one embodiment of the present invention is schematically shown. It should be noted that the classification model is plug-and-play, that is, the classification model is optional and the training process of the classification model can be added to the training process of the recognition model.

[0095] Training the classification model includes: freezing the parameters of the recognition model; and training the classification model using the merging module 307 of the recognition model to classify the merging module 307 to obtain a formula region and / or a text region. Freezing the parameters of the recognition model means that the recognition model is fixed. In other words, while training the classification model, the training of the recognition model is suspended, or the training of the classification model is performed based on the trained recognition model.

[0096] The classification model can adopt various types of structures. The classification model adopted in the embodiment of the present invention is as follows: Figure 4 As shown, the classification model includes an embedding layer 401, an encoding layer 402, and a classification layer 403. Training the classification model includes:

[0097] First, the merging module 307 is converted into a word vector using the embedding layer 401; a word vector refers to converting a word (e.g., a word in one-hot encoding) into a distributed representation, that is, representing the word as a continuous dense vector of a fixed length. Because word vectors can express similar relationships between natural languages and can contain more information, word vectors are widely used in the fields of machine learning and pattern recognition. Generally speaking, methods for generating word vectors include statistical-based methods and language model-based methods.

[0098] Then, the word vector is encoded using the encoding layer 402 to obtain an encoding result. In one embodiment, the encoding layer 402 can adopt the same structure as the encoder 304 in the recognition model; for example, the encoding layer 402 preferably adopts the encoder part of the transformer, or can also adopt a convolutional neural network structure or a recurrent neural network structure.

[0099] Finally, the classification layer 403 is used to classify the encoded results to obtain the formula area and / or text area. Based on the output formula area and / or text area and the annotation results, the loss function of the classification model is calculated, for example, a cross-entropy loss function can be calculated. Then, the gradients of the parameters of each layer are calculated using the backpropagation algorithm, and the parameters are updated according to the learning rate, thereby completing the training of the classification model. The classification layer 403 can adopt a fully connected structure.

[0100] For example, since there are more characters and fewer formulas in general texts, and formulas are more eye-catching, the embodiment of the present invention preferably obtains the formula area through the classification layer 403. The formula area is the location of the formula, which can be expressed as the starting position of the formula, the ending position of the formula, or the center position of the formula, or as a range from the starting position to the ending position. Specifically, the formula area can be represented by pixel coordinates or proportional values; for example, the formula area is represented as (30,75)-(30,85), that is, the formula area starts from the 30th row and 75th column to the 30th row and 85th column. For another example, the formula area is represented as (50%, 60%), that is, the starting position of the formula is at 50% horizontally and 60% vertically of the page.

[0101] Figure 5 The following schematically shows a training framework diagram of a classification model according to another embodiment of the present invention. Figure 5 As shown, Figure 4 Compared to the embodiment shown, Figure 5 The difference between the shown embodiments is that Figure 5 The classification model shown also leverages the high-dimensional features extracted in the recognition model.

[0102] Specifically, the classification model includes an embedding layer 401, an encoding layer 402 and a classification layer 403. Training the classification model includes: using the embedding layer 401 to convert the merging module 307 into a word vector; using the encoding layer 402 to encode the word vector and the high-dimensional features of the recognition model shown to obtain an encoding result; using the classification layer 403 to classify the encoding result to obtain the formula area and / or text area.

[0103] Among them, combined Figure 3 As shown, the high-dimensional features of the recognition model refer to the complete high-dimensional features obtained after merging by the merging module 305. The high-dimensional features and the word vectors generated by the embedding layer 401 are sent to the encoding layer 402 for encoding. Since the information contained in the high-dimensional features is not completely consistent with the recognition results 307, the trained classification model is more robust.

[0104] The above describes the training method of the image processing model, and the trained image processing model can be used for the image processing process in actual application scenarios. As an example, one application scenario may be: a dictionary pen. As a learning tool, the dictionary pen can realize instant scanning and searching to facilitate the students' learning process. In addition, the dictionary pen can also be used as a teaching aid, that is, to correct essays instead of teachers, to help students quickly improve their writing skills. By deploying the image processing model trained by the embodiment of the present invention in the dictionary pen, the application scenarios of the dictionary pen can be broadened. For example, mathematics, physics and chemistry homework containing formulas can be identified and corrected, further improving students' learning efficiency and providing students with more comprehensive learning assistance. The image processing method is described in detail below with reference to the accompanying drawings.

[0105] Figure 6 The flowchart of the image processing method 600 according to one embodiment of the present invention is schematically shown, which includes step S601 and step S602.

[0106] In step S601, a target image is obtained; wherein the target image is an image containing a recognition target, and the recognition target may be text, a formula, or a mixture of text and formula.

[0107] Specifically, obtaining the target image includes: obtaining a complete image including the identified target, or obtaining multiple consecutive video frames, and fusing the multiple consecutive video frames to obtain a complete image including the identified target.

[0108] A wide-angle camera can be used to capture a complete image of the target. Alternatively, a high-frame-rate camera can be used to capture a series of multiple consecutive video frames, which are then fused to obtain a complete image of the target. Specifically, when fusion is performed, two consecutive video frames need to be matched, the relative displacement between the two frames is calculated, and the frames are fused based on the relative displacement to obtain a complete image.

[0109] In step S602, the target image is processed according to the image processing model to obtain a recognition result, i.e., a recognized target. In one embodiment, if the recognition target is known to be a mixed text and formula pattern, a detection algorithm can be used to first obtain the mixed text and formula pattern area, and information in other areas can be removed. Then, recognition can be performed according to the recognition model to obtain a recognition result.

[0110] Figure 7 The flowchart of an image processing method 700 according to another embodiment of the present invention is schematically shown, which includes steps S701, S702 and S703. Step S703 is used to further obtain a formula area and / or a text area.

[0111] Step S701, step S702 and Figure 6 Step S601 and step S602 of the illustrated embodiment are the same and will not be described in detail here.

[0112] In step S703, the recognition result obtained in step S702 is input into the classification model trained in the embodiment of the present invention (see Figure 4 and Figure 5 Corresponding embodiment), obtain the formula area and / or text area.

[0113] Since the process of reasoning using the recognition model and the classification model is consistent with the forward process of the training process, the specific reasoning process will not be repeated here.

[0114] In summary, the image processing method of the present invention can capture images using a wide-angle camera or a high-frame-rate camera, then recognize the text and formulas contained therein, quickly extracting useful information. This facilitates downstream tasks such as grading exercises and collecting incorrect answers. Furthermore, based on the training method, the image processing method of the present invention is suitable for offline business scenarios and can achieve faster inference speeds and higher accuracy while using fewer resources, thereby enhancing the student experience.

[0115] Exemplary devices

[0116] After introducing the method of the exemplary embodiment of the present invention, next, reference is made to Figure 8 The following describes products related to the method for training an image processing model according to an exemplary embodiment of the present invention.

[0117] Figure 8 Schematically shows a schematic block diagram of a computing device 1000 according to an embodiment of the present invention. Figure 8 As shown, the computing device 1000 may include a processor 1001 and a memory 1002. The memory 1002 stores computer instructions for executing the training method of the image processing model described in the embodiment of the present invention, or computer instructions for the image processing method described in the embodiment of the present invention. When the computer instructions are executed by the processor 1001, the device 1000 executes the above Figure 2 、 Figure 6 or Figure 7For example, in some embodiments, the computing device 1000 may execute a training method to train an image processing model to obtain a trained image processing model. In another example, the computing device 1000 may process an input target image based on the trained image processing model to recognize text and formulas therein. Based on this, the computing device 1000 may enable the image processing method to obtain more accurate inference results, faster inference speed, and greater robustness.

[0118] In some implementation scenarios, the image processing model also includes a classification model for providing text or formula areas, so the memory 1002 of the computing device 1000 can also store computer instructions for training the classification model, or computer instructions for further processing based on the classification model.

[0119] In addition, in actual applications, the computing device 1000 may also include a wide-angle camera or a high-frame-rate camera to capture the actual scene and obtain an image containing the recognized target. To display the recognition results, the computing device 1000 may also include a display or other device or module for interacting with the user. It should be noted that the embodiments of the present invention do not limit the structure of the computing device 1000.

[0120] It should be noted that although several modules of the image processing model are mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more modules described above can be embodied in a single module. Conversely, the features and functions of a single module described above can be further divided and embodied by multiple modules.

[0121] Furthermore, although the operations of the present method are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in that particular order, or that all of the operations shown must be performed to achieve the desired results. Rather, the steps depicted in the flowcharts may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into a single step, and / or a single step may be broken down into multiple steps.

[0122] The use of the verbs "comprise", "include" and their conjugations in the application documents does not exclude the presence of elements or steps other than those stated in the application documents. The article "a" or "an" preceding an element does not exclude the presence of a plurality of such elements.

[0123] Although the spirit and principles of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the specific embodiments disclosed, and the division into various aspects does not mean that the features of these aspects cannot be combined to benefit. Such division is merely for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. The scope of the appended claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. A training method for an image processing model, characterized in that: The image processing model includes a recognition model, and the training method includes: Acquire training images, wherein the training images include annotated pure text images, pure formula images, and mixed text and formula images, and the annotations include word segmentation according to word segmentation rules applicable to formulas and texts; Inputting the training image into the recognition model for training to obtain an output result, wherein the recognition model includes a feature extractor, an encoder and a decoder, the encoder uses the encoder part of the model based on the attention mechanism, and the decoder adopts a fully connected structure; Calculating a loss function based on the annotations in the training image and the output results to update the parameters of the recognition model in back propagation; The recognition model further includes a block module and a merging module; the training method further includes: Using the blocking module to block the training image to obtain at least two sub-images with overlapping parts, and using the feature extractor and encoder to process each sub-image separately; and The features obtained by processing each sub-image are merged using the merging module to obtain a second feature.

2. The training method according to claim 1, characterized in that The word segmentation rules include: For symbols that appear in both text and formulas, use formula syntax to mark them; For symbols that are exclusive to formulas, use formula syntax to mark them; For symbols that are exclusively used by text, general text annotation is used.

3. The training method according to claim 1, characterized in that Inputting the training image into the recognition model for training to obtain output results includes: Extracting a first feature containing visual information from the training image using the feature extractor; Encoding the first feature using the encoder to obtain a second feature containing semantic information; The second feature is mapped into text or a formula using the decoder to obtain the output result.

4. The training method according to claim 1, characterized in that Obtaining training images includes: Perform one or more of the following processing methods on the training images to perform data augmentation: rotation transformation, perspective transformation, contrast adjustment, blurring, and adding noise.

5. The training method according to any one of claims 1 to 4, characterized in that: The image processing model further includes a classification model, and the training method further includes training the classification model as follows: freezing parameters of the recognition model; The classification model is trained using the recognition result of the recognition model and the second feature extracted from the recognition model to classify the recognition result to obtain a formula area and / or a text area.

6. The training method according to claim 5, characterized in that The classification model includes an embedding layer, an encoding layer, and a classification layer. Training the classification model includes: Converting the recognition result into a word vector using the embedding layer; Encoding the word vector and the second feature using the encoding layer to obtain an encoding result; The encoding result is classified using the classification layer to obtain the formula area and / or text area.

7. A method for image processing using an image processing model, wherein the image processing model has been trained according to the training method according to any one of claims 1 to 6; characterized in that: The method comprises: Get the target image; The target image is processed according to the image processing model to recognize text and / or formulas in the target image.

8. A computing device, characterized in that: include: a processor configured to execute program instructions; as well as A memory configured to store the program instructions, which, when loaded and executed by the processor, enables the processor to execute the training method of the image processing model according to any one of claims 1 to 6 or the image processing method according to claim 7.

9. A computer-readable storage medium, characterized in that Program instructions are stored therein, and when the program instructions are loaded and executed by a processor, the processor executes the training method of the image processing model according to any one of claims 1 to 6 or the image processing method according to claim 7.

Images