Image processing method, apparatus, device, and medium

Abstract

The present disclosure relates to an image processing method, device, equipment and medium, the method comprising: acquiring a target image containing a target formula; inputting the target image into a pre-trained formula recognition model to perform formula recognition, obtaining a first character prediction result and a first relationship prediction result corresponding to each character in the target formula; wherein the formula recognition model is trained based on a formula sample image with labeled information, and the labeled information includes: a character sequence corresponding to the formula sample, a semantic feature vector and a character detection box; and obtaining a recognition result of the target formula in the target image according to the first character prediction result and the first relationship prediction result corresponding to each character. The present disclosure can improve the formula recognition accuracy.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image processing method, apparatus, device and medium. Background Technology

[0002] Natural scene text recognition is the process of identifying character sequences from images containing text content. The text content in images is very diverse, with complex formula arrangements, such as alternating single and multiple lines, including superscripts and subscripts, and varying font sizes. Compared to conventional text recognition, the complexity of formula arrangements makes formula recognition much more difficult and yields poor results when using conventional text recognition methods. Summary of the Invention

[0003] In order to solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, this disclosure provides an image processing method, apparatus, device and medium.

[0004] According to one aspect of this disclosure, an image processing method is provided, comprising:

[0005] Obtain the target image containing the target formula;

[0006] The target image is input into a pre-trained formula recognition model for formula recognition, and the first character prediction result and the first relation prediction result corresponding to each character in the target formula are obtained; wherein, the formula recognition model is trained based on formula sample images with annotation information, the annotation information including: the character sequence, semantic feature vector and character detection box corresponding to the formula sample;

[0007] Based on the prediction result of the first character and the prediction result of the first relationship corresponding to each character, the recognition result of the target formula in the target image is obtained.

[0008] According to another aspect of this disclosure, an image processing apparatus is provided, comprising:

[0009] The image acquisition module is used to acquire the target image containing the target formula;

[0010] The formula recognition module is used to input the target image into a pre-trained formula recognition model to perform formula recognition, and obtain the first character prediction result and the first relation prediction result corresponding to each character in the target formula; wherein, the formula recognition model is trained based on formula sample images with annotation information, the annotation information including: the character sequence, semantic feature vector and character detection box corresponding to the formula sample;

[0011] The result acquisition module is used to obtain the recognition result of the target formula in the target image based on the first character prediction result and the first relationship prediction result corresponding to each character.

[0012] According to another aspect of this disclosure, an electronic device is provided, the electronic device comprising: a processor; and a memory storing a program, wherein the program includes instructions that, when executed by the processor, cause the processor to perform the image processing method described above.

[0013] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing the computer to perform an image processing method.

[0014] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0015] The image processing method, apparatus, device, and medium provided in this disclosure acquire a target image containing a target formula; input the target image into a pre-trained formula recognition model for formula recognition, obtaining a first character prediction result and a first relation prediction result corresponding to each character in the target formula; and obtain the recognition result of the target formula in the target image based on the first character prediction result and the first relation prediction result corresponding to each character. The above technical method can improve the accuracy of formula recognition. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0017] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of an image processing method provided in an embodiment of the present disclosure;

[0019] Figure 2 A flowchart of the model training method provided in this embodiment of the disclosure;

[0020] Figure 3 This is a schematic diagram of the structure of the image processing apparatus provided in the embodiments of this disclosure;

[0021] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0022] To better understand the above-described objects, features, and advantages of this disclosure, embodiments of the disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of the disclosure are shown in the drawings, it should be understood that the disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0023] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0024] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc., used in this disclosure are only used to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0025] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0026] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0027] Text recognition in natural scenes is an extremely challenging task. Besides the complexity of image backgrounds and varying lighting, the complexity of the output space is also a major difficulty. Since text is composed of a variable number of letters, text recognition in natural scenes requires identifying sequences of varying lengths from images. Currently, two main recognition methods are commonly used: one is a bottom-up strategy that breaks down the problem into character detection, character recognition, and character combination, solving them one by one; the other is a holistic analysis strategy, i.e., a sequence-to-sequence approach, where the image is first encoded, and then the sequence is decoded to directly obtain the entire string. However, the first method requires character-level annotation, meaning the position and information of each character in the input image need to be labeled, consuming significant human resources. While the second method is simpler to annotate, requiring only string transcription, its recognition results may include over- or under-identified characters.

[0028] The text content in text images is very diverse, including Chinese, English, numbers, or combinations thereof, as well as formulas. Chinese and English text or numbers generally have a simple, WYSIWYG (What You See Is What You Get) structure, with similar font sizes and a linear overall structure. Formulas, however, have a more complex structure, involving alternating single and multiple lines, superscripts and subscripts, and varying font sizes, resulting in a non-linear structure. Therefore, it's understandable that recognizing formulas with their complex structures is more challenging than conventional text recognition.

[0029] Currently, conventional text recognition methods can be used to recognize formulas, but the accuracy and performance are low. For example, character-based methods are costly, and the character detection part is difficult to detect accurately due to variations in character size and position; sequence-to-sequence methods cannot achieve good alignment and often result in missed or over-identified characters.

[0030] Among them, the CRNN (Convolutional Recurrent Neural Network) model is a sequence-to-sequence text recognition method that achieves a good balance between accuracy and speed, but it still suffers from problems such as excessively large number of parameters and long training cycles. The Transformer, as a highly effective encoder-decoder network structure, has achieved excellent results in tasks such as natural language processing and computer vision. Decoding can be performed using either CTC (Connectionist Temporal Classification) or Attention-based methods. For the recognition of complex formulas, neither CTC nor Attention-based methods can effectively align predicted characters with their corresponding feature regions, resulting in poor performance for both. However, the Attention method is more accurate than the CTC method. The inventors have found that this is because: firstly, the alignment method of the Attention method is more flexible and can utilize global information; secondly, the Attention method can learn an implicit language model during the decoding process. Furthermore, considering that both CTC-based and Attention-based decoding structures are linear, decoding from left to right with only one character per vertical position, while formulas are non-linear structures where each position may contain one or more characters, a linear decoding approach would require forcibly learning an irregular or even highly complex alignment pattern. For the network, the more complex the pattern, the greater the learning difficulty and the harder it is to guarantee results.

[0031] For formula diagrams, the following key information can be considered: the number of characters in the entire formula, the position information of each character, the structural information between characters, and the semantic information of the overall formula.

[0032] Based on the above, in order to improve the recognition accuracy of complex formulas, this disclosure provides an image processing method, apparatus, device, and medium. This embodiment constructs a new network structure formula recognition model based on existing network models, enabling the model to comprehensively utilize character quantity, character position, and semantic information to perform formula recognition on images, thereby improving the accuracy of formula recognition. For ease of understanding, the embodiments of this disclosure are described below.

[0033] Figure 1 This disclosure provides a flowchart of an image processing method according to an embodiment of the present disclosure. This method can be executed by an image processing device configured in a terminal or server, which can be implemented in software and / or hardware. (Refer to...) Figure 1The method includes the following steps:

[0034] Step S102: Obtain the target image containing the target formula.

[0035] This embodiment allows the acquisition of a target image on a terminal via image selection, image capture, or image upload. The target formula contained in the target image can be an expression in natural sciences such as mathematics, physics, and chemistry, using mathematical symbols to represent the relationship between several quantities. The characters in the target formula can include Chinese characters, English characters, numbers, and / or operators, and each position in the target formula includes at least one character.

[0036] Step S104: Input the target image into the pre-trained formula recognition model to perform formula recognition, and obtain the first character prediction result and the first relation prediction result corresponding to each character in the target formula; wherein, the formula recognition model is trained based on the formula sample image with annotation information, the annotation information includes: the character sequence, semantic feature vector and character detection box corresponding to the formula sample.

[0037] In this embodiment, the annotation information of the formula sample image includes: the character sequence, semantic feature vector, and character detection box corresponding to the formula sample; wherein, the character sequence can represent the characters and the number of characters in the formula sample, the semantic feature vector can represent the logical semantic information of the proposition defined or expressed by the formula sample, and the character detection box can represent the position of each character in the formula sample. The formula recognition model trained using the formula sample image with the above annotation information can learn the number of characters, character positions, and semantic information of the formula sample. Based on this, this embodiment inputs the target image containing the target formula into the trained formula recognition model. When performing formula recognition on the target image, the formula recognition model can fully utilize multi-dimensional information such as the number of characters, character positions, and semantic information of the target formula, thereby improving the accuracy of formula recognition.

[0038] The formula recognition model may include: a feature mapping module, a self-attention module, and a semantic extraction module connected in sequence, and a character prediction module connected to the self-attention module and the semantic extraction module respectively. The feature mapping module takes the target image as input and outputs a first mapping feature corresponding to the target image; the self-attention module takes the first mapping feature as input and outputs a second mapping feature; the semantic extraction module takes the second mapping feature as input and outputs a first global semantic vector; the character prediction module takes the second mapping feature and the first global semantic feature as input and outputs a first character prediction result and a first relation prediction result for each character in the target formula. The first relation prediction result is used to indicate the character c at any position i. i The character c at the adjacent position i-1 i-1Relationships, specifically such as square, cube, superscript, subscript, etc.

[0039] Step S106: Based on the prediction result of the first character and the prediction result of the first relationship corresponding to each character, the recognition result of the target formula in the target image is obtained.

[0040] It is understandable that after obtaining the first character prediction result and the first relation prediction result for each character, the first character prediction results of each character can be arranged according to the first relation prediction result to obtain the recognition result of the target formula in the target image.

[0041] This disclosure provides an image processing method that involves acquiring a target image containing a target formula; inputting the target image into a pre-trained formula recognition model for formula recognition, obtaining a first character prediction result and a first relation prediction result for each character in the target formula; and obtaining the recognition result of the target formula in the target image based on the first character prediction result and the first relation prediction result for each character. This image processing method allows the formula recognition model to fully utilize the number of characters, character positions, and semantic information, thereby performing formula recognition from multiple dimensions when processing the target image, effectively improving the accuracy of formula recognition.

[0042] To enable the formula recognition model to be directly applied to formula recognition in images, it needs to be trained. The parameters of the formula recognition model need to be obtained through training. The purpose of training the model is to ultimately determine the parameters that meet the requirements. Using the trained parameters, the formula recognition model can achieve the expected formula recognition results. Figure 2 As shown in the figure, this embodiment provides a training method for a formula recognition model, which is illustrated in the following steps:

[0043] Step S202: Obtain the first training set and the second training set; wherein, the formula sample images in the first training set are labeled with the character sequence, semantic feature vector and character detection box corresponding to the formula sample, and the formula sample images in the second training set are labeled with the character sequence and semantic feature vector corresponding to the formula sample.

[0044] In one embodiment, multiple original formula sample images are acquired. Each original formula sample image includes a single line of formula text content, i.e., a formula sample. To increase the richness and diversity of the training samples, the formula samples in this embodiment can be in styles such as straight text, slanted text, and / or curved text, and the original formula sample images can be images with effects such as blurred images, photocopied images, or high-resolution images. Text character information is annotated on the original formula sample images to obtain the character sequence corresponding to the formula sample; specifically, for example, the LaTeX character sequence corresponding to the formula sample can be annotated manually.

[0045] This embodiment can also establish a dictionary based on the character sequences annotated in multiple original formula sample images. The dictionary includes characters from multiple original formula sample images. Specifically, the dictionary is established by performing single-character set operations on the annotated character sequences, ensuring that each character in the dictionary is independent and non-repeating. This dictionary can be used as a character database for subsequent character recognition based on the formula recognition model, determining the specific character in the character recognition database according to the recognition probability.

[0046] Character detection boxes are labeled on the original formula sample images. In one specific method, the acquired multiple original formula sample images can be divided into a first image set and a second image set according to a preset allocation ratio, for example, the number of images in the first image set is one-ninth of the number of images in the second image set. Then, character detection boxes are labeled on each character in the original formula sample images of the first image set, and the character detection boxes can be rectangles.

[0047] The original formula sample image is labeled with semantic feature vectors. An example of obtaining semantic feature vectors is provided below.

[0048] Obtain a second formula sample image labeled with character sequences and character detection boxes; or, in other words, the second formula sample image is any original formula sample image labeled with character sequences and character detection boxes.

[0049] Based on the labeled character sequence and character detection boxes, each character in the sample image of the second formula is encoded using triplet encoding to obtain a triplet structure. The triplet structure includes: the current character, the adjacent character at the previous position corresponding to the current character, and the relationship between the current character and the adjacent characters. In a specific example, based on the character sequence and character detection boxes, each pair of adjacent characters in the sample formula is determined. Each pair of adjacent characters includes the current character c corresponding to any position i. i The character c that is adjacent to the previous character at position i-1. i-1 In natural sciences such as mathematics and physics, there are finite relationships between adjacent characters in formulas, such as square, cube, superscript, and subscript. Therefore, based on the syntax rules of the sample formula, the relationship between each pair of adjacent characters is determined. Each pair of adjacent characters and the relationship between them are encoded into a triplet structure, which can be represented as (node1, node2, relation), where node1 and node2 correspond to the current character 'c'. i The character c adjacent to the previous position i-1 .

[0050] Next, word embedding vectors corresponding to the triplet structure are obtained through word embedding method; the word embedding vectors corresponding to each character are input into the pre-trained semantic information extraction model to obtain the semantic feature vectors corresponding to the character sequences of the sample image of the second formula.

[0051] The semantic information extraction model can be built based on the Transformer model. The main structure of the semantic information extraction model is basically the same as that of the Transformer, with the following two differences: First, in the position encoding part of the encoder, a grid is used to divide the characters in the sample image of the second formula to ensure that each character has a two-dimensional coordinate position; the input of the encoder is a one-dimensional sine and cosine vector, and the output is a two-dimensional vector. Second, the decoder part of the semantic information extraction model performs decoding only once; the input of the decoder is any character 'c' in the character sequence. i The word embedding vector, the output is the next character c i-1 The prediction results. In this embodiment, Transformer can not only greatly accelerate the training and inference time of the network, but also effectively improve the accuracy of various tasks. Therefore, by extracting semantic feature vectors through a semantic information extraction model built on Transformer, the extracted semantic feature vectors can achieve better results in terms of accuracy and precision.

[0052] The semantic information extraction model needs to be pre-trained. Possible training methods include: using a first image set labeled with character sequences and character detection boxes, and a second image set labeled with character sequences to train the model to be trained. When the training is completed, only the encoder is retained to obtain the trained semantic information extraction model.

[0053] The word embedding vectors corresponding to each character are input into the trained semantic information extraction model. The word embedding vectors are one-dimensional sine and cosine vectors. The semantic information extraction model encodes the word embedding vectors corresponding to each character into two-dimensional vectors, thus obtaining the semantic feature vectors corresponding to the character sequences of the second formula sample image. Then, the semantic feature vectors are labeled on the original formula sample image.

[0054] According to the above embodiment, multiple original formula sample images are divided into a first image set and a second image set. The images in the first image set are labeled with character sequences, semantic feature vectors and character detection boxes, and the images in the second image set are labeled with character sequences and semantic feature vectors. Therefore, the first image set with the above-mentioned information annotation is used as the first training set, and the second image set with the above-mentioned information annotation is used as the second training set.

[0055] Step S204: Obtain the first model to be trained; the first model includes: a feature mapping module, a self-attention module, a semantic extraction module, a character prediction module, a convolution module, and a quantity prediction module.

[0056] In one embodiment, the feature mapping module can use a ResNet network. ResNet effectively addresses the performance degradation issue that occurs as the number of network layers increases, allowing for more complex feature pattern extraction with increased layer depth. The core of ResNet is a structure called a residual block. The main characteristic of the residual structure is its cross-layer skip connections; a block contains multiple convolutional layers, and the output of the input after passing through the block is added to the input channel-by-channel, point-by-point. This is equivalent to the input having two branches: one passing through the block, and the other quickly bypassing the block, before the two branches are merged. ResNet achieves good results in natural scene image classification.

[0057] In one specific example, the feature mapping module can use a ResNet18 network, which includes N blocks. Each block consists of several convolutional operations, and the output of each block is the input of the next block. Here, the output of each block is obtained, resulting in N sets of mapped features. This embodiment considers that earlier blocks can obtain shallow detail features, while later blocks can obtain high-level semantic features. However, the features obtained by later blocks may lose a significant amount of detail information. Therefore, the N sets of mapped features can be scaled to the same size and concatenated to obtain a better representation through information fusion, ultimately yielding the third mapped feature corresponding to the target image. In one example, for cases where there may be multiple characters at the same vertical position in the formula, the number of blocks in the height direction is no less than four. The third mapped feature obtained after the feature mapping module is used as the input to the self-attention module.

[0058] The self-attention module can use two self-attention layers to further extract features from the third mapping features extracted by the feature mapping module, resulting in a new set of feature maps, which is the fourth mapping feature. The fourth mapping feature has the same dimension as the third mapping feature.

[0059] The semantic extraction module mainly consists of a convolutional layer and a max pooling layer. The size of the pooling window is consistent with the scale of the second mapping feature, thus obtaining the second global semantic vector.

[0060] The convolutional module runs in parallel with the semantic extraction module and can include three convolutional layers of equal width, outputting a one-channel mapping feature. The convolutional module is mainly used to predict the character center point of each character in the sample formula based on the fourth mapping feature.

[0061] The quantity prediction module runs in parallel with the convolution module and the semantic extraction module, and can include two convolutional layers and one fully connected layer. The two convolutional layers extract features from the fourth mapping feature, and the fully connected layer combines the features extracted by the first two convolutional layers to predict the number of characters in the sample formula.

[0062] The character prediction module includes a word embedding layer, an attention layer, and an LSTM (Long Short-Term Memory) unit, which acts as the decoder. The input to the character prediction module includes the fourth mapping feature and the second global semantic vector. Using the second global semantic vector as the initial hidden state and the fourth mapping feature as the key and value, the module calculates and outputs the prediction result for the current character and the relationship between the current character and the character at the previous adjacent position.

[0063] Step S206: Train the first model using the first training set. The specific training process includes the following:

[0064] The first formula sample image is input into the first model; wherein, the first formula sample image is the formula sample image in the first training set.

[0065] The feature mapping module outputs the third mapping feature corresponding to the first formula sample image; the self-attention module outputs the fourth mapping feature based on the third mapping feature; the semantic extraction module outputs the second global semantic vector based on the fourth mapping feature; the convolution module outputs the character center point of each character in the formula sample of the first formula sample image based on the fourth mapping feature; the quantity prediction module outputs the number of characters in the formula sample of the first formula sample image based on the fourth mapping feature; and the character prediction module outputs the second character prediction result and the second relation prediction result corresponding to each character in the formula sample of the first formula sample image based on the fourth mapping feature and the second global semantic vector.

[0066] The first model is trained based on the character sequence, semantic feature vector, and character detection box corresponding to the formula sample marked on the first formula sample image, the second character prediction result, the second relation prediction result, the second global semantic vector, the character center point and the number of characters, and the preset loss function.

[0067] This step specifically includes: calculating the first loss function value of the semantic extraction module based on the semantic feature vector labeled on the first formula sample image, the second global semantic vector, and the L1 loss function. The L1 loss function, also known as the minimum absolute value bias or absolute value loss function, is used to minimize the absolute difference between the target value (labeled semantic feature vector) and the estimated value (second global semantic vector).

[0068] Based on the character detection box, character center point, and classification loss function Focal Loss corresponding to the formula sample marked on the first formula sample image, the second loss function value of the convolution module is calculated.

[0069] Based on the character detection boxes, character counts, and multi-class cross-entropy loss function corresponding to the formula samples marked on the first formula sample image, the third loss function value of the quantity prediction module is calculated.

[0070] Based on the second character prediction result and the second relation prediction result, the predicted character sequence of the formula sample in the first formula sample image is obtained; based on the predicted character sequence, the character sequence corresponding to the formula sample marked on the first formula sample image, and the multi-class cross-entropy loss function, the fourth loss function value of the character prediction module is calculated.

[0071] The first model is trained based on the first, second, third, and fourth loss function values. For example, the first, second, third, and fourth loss function values ​​can be weighted according to preset weight coefficients to obtain a comprehensive loss function value, and the first model is trained based on the comprehensive loss function value.

[0072] Step S208: After the first model is trained, the parameters of the convolutional module are kept unchanged to obtain the second model.

[0073] Step S210: Train the second model using the second training set. The method of training the second model using the second training set is similar to step S206 above, except that the convolutional module no longer participates in the training process, while the other modules are trained using the second training set.

[0074] Step S212: When the second model training converges, the feature mapping module, self-attention module, semantic extraction module, and character prediction module are determined as the formula recognition model. When the second model training converges, the convolution module and quantity prediction module are removed, and the remaining feature mapping module, self-attention module, semantic extraction module, and character prediction module are determined as the formula recognition model and directly used for formula recognition.

[0075] The above embodiment divides the training process into two stages: a first stage using a first training set to train the second model, and a second stage using a second training set to train the formula recognition model. This effectively improves the model's learning rate and shortens the training cycle. The formula recognition model trained by the above method can recognize formulas from input target images, effectively improving the accuracy of formula recognition.

[0076] Thus, a trained formula recognition model can be obtained based on the above embodiments. Building upon this, this embodiment provides an image processing method based on the formula recognition model, specifically including the following:

[0077] Obtain a target image containing the target formula. Input the target image into a pre-trained formula recognition model for formula recognition; the formula recognition model includes: a feature mapping module, a self-attention module, a semantic extraction module, and a character prediction module; wherein:

[0078] The target image is input into the feature mapping module, which performs feature mapping on the target image to obtain the first mapping feature corresponding to the target image. The self-attention module extracts features from the first mapping feature to obtain the second mapping feature. The semantic extraction module extracts semantic features from the second mapping feature to obtain the first global semantic vector. The character prediction module, based on the second mapping feature and the first global semantic feature, obtains the first character prediction result and the first relation prediction result for each character in the target formula. The first relation prediction result is used to indicate the relationship between the character and the character at the previous adjacent position.

[0079] The character prediction module includes a word embedding layer, an attention layer, and a decoder. During the prediction process, the word embedding layer vectorizes the second mapping features to obtain their semantic features. The attention layer then generates an attention vector based on these semantic features. Finally, the decoder uses the attention vector and the first global semantic features to obtain the first character prediction result and the first relation prediction result for each character in the target formula. To improve recognition accuracy, the first character prediction result can be obtained from a pre-established dictionary containing multiple characters.

[0080] Next, based on the prediction results of the first character and the prediction results of the first relationship corresponding to each character, the recognition result of the target formula in the target image is obtained.

[0081] In summary, the image processing method provided in the above embodiments uses a trained formula recognition model to recognize formulas in target images. The formula recognition model fully utilizes multi-dimensional information such as the number of characters, character positions, and semantic information contained in the target image, which can effectively improve the accuracy and precision of formula recognition.

[0082] Figure 3 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this disclosure. The image processing apparatus provided in this embodiment can execute the processing flow provided in the image processing method embodiment, such as... Figure 3 As shown, the image processing apparatus 300 includes:

[0083] Image acquisition module 302 is used to acquire a target image containing the target formula;

[0084] Formula recognition module 304 is used to input the target image into a pre-trained formula recognition model to perform formula recognition, and obtain the first character prediction result and the first relation prediction result corresponding to each character in the target formula; wherein, the formula recognition model is trained based on formula sample images with annotation information, the annotation information including: the character sequence, semantic feature vector and character detection box corresponding to the formula sample;

[0085] The result acquisition module 306 is used to obtain the recognition result of the target formula in the target image based on the first character prediction result and the first relationship prediction result corresponding to each character.

[0086] In some embodiments, the formula recognition model includes: a feature mapping module, a self-attention module, a semantic extraction module, and a character prediction module; the formula recognition module 304 is further configured to:

[0087] The target image is input to the feature mapping module, and the feature mapping module performs feature mapping on the target image to obtain the first mapped feature corresponding to the target image;

[0088] The self-attention module extracts features from the first mapping features to obtain the second mapping features;

[0089] The semantic extraction module extracts semantic features from the second mapping features to obtain a first global semantic vector.

[0090] Based on the second mapping feature and the first global semantic feature, the character prediction module obtains the first character prediction result and the first relationship prediction result for each character in the target formula; wherein, the first relationship prediction result is used to indicate the relationship between the character and the character at the adjacent previous position.

[0091] In some embodiments, the character prediction module includes: a word embedding layer, an attention layer, and a decoder; the formula recognition module 304 is further configured to:

[0092] The second mapping feature is vectorized through the word embedding layer to obtain the semantic features of the second mapping feature;

[0093] An attention vector is obtained through the semantic features of the attention layer based on the second mapping feature;

[0094] The decoder obtains the first character prediction result and the first relation prediction result for each character in the target formula based on the attention vector and the first global semantic feature.

[0095] In some embodiments, the image processing apparatus 300 further includes a model training module, which is used for:

[0096] Obtain a first training set and a second training set; wherein, the formula sample images in the first training set are labeled with the character sequence, semantic feature vector and character detection box corresponding to the formula sample, and the formula sample images in the second training set are labeled with the character sequence and semantic feature vector corresponding to the formula sample;

[0097] Obtain the first model to be trained; the first model includes: a feature mapping module, a self-attention module, a semantic extraction module, a character prediction module, a convolution module, and a quantity prediction module;

[0098] The first model is trained based on the first training set;

[0099] After the first model is trained, the parameters of the convolution module are kept unchanged to obtain the second model;

[0100] The second model is trained based on the second training set;

[0101] When the training of the second model converges, the feature mapping module, the self-attention module, the semantic extraction module, and the character prediction module are identified as the formula recognition model.

[0102] In some embodiments, the model training module is further configured to:

[0103] The first formula sample image is input into the first model; wherein, the first formula sample image is the formula sample image in the first training set;

[0104] The feature mapping module outputs the third mapping feature corresponding to the first formula sample image;

[0105] The self-attention module outputs a fourth mapping feature based on the third mapping feature;

[0106] The semantic extraction module outputs a second global semantic vector based on the fourth mapping feature;

[0107] The convolution module outputs the character center point of each character in the formula sample of the first formula sample image based on the fourth mapping feature;

[0108] The quantity prediction module outputs the number of characters in the formula sample of the first formula sample image based on the fourth mapping feature;

[0109] Based on the fourth mapping feature and the second global semantic vector, the character prediction module outputs the second character prediction result and the second relation prediction result corresponding to each character in the formula sample of the first formula sample image.

[0110] The first model is trained based on the character sequence, semantic feature vector, and character detection box corresponding to the formula sample marked on the first formula sample image, the second character prediction result, the second relation prediction result, the second global semantic vector, the character center point, the number of characters, and a preset loss function.

[0111] In some embodiments, the model training module is further configured to:

[0112] Based on the semantic feature vector annotated on the first formula sample image, the second global semantic vector, and the L1 loss function, the first loss function value of the semantic extraction module is calculated;

[0113] Based on the character detection box corresponding to the formula sample marked on the first formula sample image, the character center point, and the classification loss function Focal Loss, calculate the second loss function value of the convolution module;

[0114] The third loss function value of the quantity prediction module is calculated based on the character detection box corresponding to the formula sample marked on the first formula sample image, the number of characters, and the multi-class cross-entropy loss function.

[0115] Based on the second character prediction result and the second relationship prediction result, the predicted character sequence of the formula sample in the first formula sample image is obtained;

[0116] The fourth loss function value of the character prediction module is calculated based on the predicted character sequence, the character sequence corresponding to the formula sample marked on the first formula sample image, and the multi-class cross-entropy loss function.

[0117] The first model is trained based on the first loss function value, the second loss function value, the third loss function value, and the fourth loss function value.

[0118] In some embodiments, the image processing apparatus 300 further includes a semantic feature vector acquisition module, which is used for:

[0119] Obtain a second formula sample image labeled with character sequences and character detection boxes;

[0120] Based on the labeled character sequence and character detection box, each character in the sample image of the second formula is encoded into a triplet to obtain a triplet structure. The triplet structure includes: the current character, the adjacent character at the previous position of the current character, and the relationship between the current character and the adjacent character.

[0121] The word embedding vector corresponding to the triplet structure is obtained by word embedding method;

[0122] The word embedding vector corresponding to each character is input into a pre-trained semantic information extraction model to obtain the semantic feature vector corresponding to the character sequence of the sample image of the second formula.

[0123] The device provided in this embodiment has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0124] Exemplary embodiments of this disclosure also provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to cause the electronic device to perform a method according to an embodiment of this disclosure.

[0125] Exemplary embodiments of this disclosure also provide a computer program product, including a computer program, wherein, when executed by a processor of a computer, the computer program is used to cause the computer to perform a method according to an embodiment of this disclosure.

[0126] refer to Figure 4 The present invention describes a structural block diagram of an electronic device 400 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0127] like Figure 4As shown, the electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. The RAM 403 may also store various programs and data required for the operation of the device 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0128] Multiple components in electronic device 400 are connected to I / O interface 405, including: input unit 406, output unit 407, storage unit 408, and communication unit 409. Input unit 406 can be any type of device capable of inputting information to electronic device 400. Input unit 406 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device. Output unit 407 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 404 may include, but is not limited to, disks and optical discs. Communication unit 409 allows electronic device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth™ devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0129] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above. For example, in some embodiments, the image processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 400 via ROM 402 and / or communication unit 409. In some embodiments, the computing unit 401 may be configured to perform the image processing method by any other suitable means (e.g., by means of firmware).

[0130] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0131] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0132] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0133] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0134] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0135] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0136] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image processing method, characterized in that, include: Obtain the target image containing the target formula; The target image is input into a pre-trained formula recognition model for formula recognition, obtaining a first character prediction result and a first relationship prediction result for each character in the target formula. The formula recognition model is trained based on formula sample images with annotation information, including: the character sequence, semantic feature vector, and character detection box corresponding to the formula sample. The first relationship prediction result indicates the relationship between a character and the character at the preceding adjacent position. A first model is trained using a first training set to obtain a second model, and a second model is trained using a second training set to obtain the formula recognition model. The formula sample images in the first training set are annotated with the character sequence, semantic feature vector, and character detection box corresponding to the formula sample, and the formula sample images in the second training set are annotated with the character sequence and semantic feature vector corresponding to the formula sample. Based on the prediction result of the first character and the prediction result of the first relationship corresponding to each character, the recognition result of the target formula in the target image is obtained; The method further includes: Obtain a second formula sample image labeled with character sequences and character detection boxes; Based on the labeled character sequence and character detection box, each character in the sample image of the second formula is encoded into a triplet to obtain a triplet structure. The triplet structure includes: the current character, the adjacent character at the previous position of the current character, and the relationship between the current character and the adjacent character. The word embedding vector corresponding to the triplet structure is obtained by word embedding method; The word embedding vector corresponding to each character is input into a pre-trained semantic information extraction model to obtain the semantic feature vector corresponding to the character sequence of the sample image of the second formula.

2. The method according to claim 1, characterized in that, The formula recognition model includes: a feature mapping module, a self-attention module, a semantic extraction module, and a character prediction module; The step of inputting the target image into a pre-trained formula recognition model for formula recognition, and obtaining the character prediction result and relationship prediction result corresponding to each character in the target formula, includes: The target image is input to the feature mapping module, and the feature mapping module performs feature mapping on the target image to obtain the first mapped feature corresponding to the target image; The self-attention module extracts features from the first mapping features to obtain the second mapping features; The semantic extraction module extracts semantic features from the second mapping features to obtain a first global semantic vector. The character prediction module obtains the first character prediction result and the first relation prediction result for each character in the target formula based on the second mapping feature and the first global semantic feature.

3. The method according to claim 2, characterized in that, The character prediction module includes: a word embedding layer, an attention layer, and a decoder; the step of obtaining the first character prediction result and the first relation prediction result corresponding to each character in the target formula based on the second mapping feature and the first global semantic feature by the character prediction module includes: The second mapping feature is vectorized through the word embedding layer to obtain the semantic features of the second mapping feature; An attention vector is obtained through the semantic features of the attention layer based on the second mapping feature; The decoder obtains the first character prediction result and the first relation prediction result for each character in the target formula based on the attention vector and the first global semantic feature.

4. The method according to claim 1, characterized in that, The training process of the formula recognition model includes: Obtain the first and second training sets; Obtain the first model to be trained; the first model includes: a feature mapping module, a self-attention module, a semantic extraction module, a character prediction module, a convolution module, and a quantity prediction module; The first model is trained based on the first training set; After the first model is trained, the parameters of the convolution module are kept unchanged to obtain the second model; The second model is trained based on the second training set; When the training of the second model converges, the feature mapping module, the self-attention module, the semantic extraction module, and the character prediction module are identified as the formula recognition model.

5. The method according to claim 4, characterized in that, The step of training the first model based on the first training set includes: The first formula sample image is input into the first model; wherein, the first formula sample image is the formula sample image in the first training set; The feature mapping module outputs the third mapping feature corresponding to the first formula sample image; The self-attention module outputs a fourth mapping feature based on the third mapping feature; The semantic extraction module outputs a second global semantic vector based on the fourth mapping feature; The convolution module outputs the character center point of each character in the formula sample of the first formula sample image based on the fourth mapping feature; The quantity prediction module outputs the number of characters in the formula sample of the first formula sample image based on the fourth mapping feature; Based on the fourth mapping feature and the second global semantic vector, the character prediction module outputs the second character prediction result and the second relation prediction result corresponding to each character in the formula sample of the first formula sample image. The first model is trained based on the character sequence, semantic feature vector, and character detection box corresponding to the formula sample marked on the first formula sample image, the second character prediction result, the second relation prediction result, the second global semantic vector, the character center point, the number of characters, and a preset loss function.

6. The method according to claim 5, characterized in that, The step of training the first model based on the character sequence, semantic feature vector, and character detection box corresponding to the formula sample marked on the first formula sample image, the second character prediction result, the second relation prediction result, the second global semantic vector, the character center point, and the number of characters, and a preset loss function, includes: Based on the semantic feature vector annotated on the first formula sample image, the second global semantic vector, and the L1 loss function, the first loss function value of the semantic extraction module is calculated; Based on the character detection box corresponding to the formula sample marked on the first formula sample image, the character center point, and the classification loss function Focal Loss, calculate the second loss function value of the convolution module; The third loss function value of the quantity prediction module is calculated based on the character detection box corresponding to the formula sample marked on the first formula sample image, the number of characters, and the multi-class cross-entropy loss function. Based on the second character prediction result and the second relationship prediction result, the predicted character sequence of the formula sample in the first formula sample image is obtained; The fourth loss function value of the character prediction module is calculated based on the predicted character sequence, the character sequence corresponding to the formula sample marked on the first formula sample image, and the multi-class cross-entropy loss function. The first model is trained based on the first loss function value, the second loss function value, the third loss function value, and the fourth loss function value.

7. An image processing apparatus, characterized in that, include: The image acquisition module is used to acquire the target image containing the target formula; A formula recognition module is used to input the target image into a pre-trained formula recognition model for formula recognition, obtaining a first character prediction result and a first relationship prediction result for each character in the target formula; wherein, the formula recognition model is trained based on formula sample images with annotation information, the annotation information including: the character sequence, semantic feature vector, and character detection box corresponding to the formula sample; the first relationship prediction result is used to indicate the relationship between the character and the character at the adjacent previous position; a first model is trained using a first training set to obtain a second model, and a second model is trained using a second training set to obtain a formula recognition model; the formula sample images in the first training set are annotated with the character sequence, semantic feature vector, and character detection box corresponding to the formula sample, and the formula sample images in the second training set are annotated with the character sequence and semantic feature vector corresponding to the formula sample; The result acquisition module is used to obtain the recognition result of the target formula in the target image based on the first character prediction result and the first relationship prediction result corresponding to each character; The device further includes: Obtain a second formula sample image labeled with character sequences and character detection boxes; Based on the labeled character sequence and character detection box, each character in the sample image of the second formula is encoded into a triplet to obtain a triplet structure. The triplet structure includes: the current character, the adjacent character at the previous position of the current character, and the relationship between the current character and the adjacent character. The word embedding vector corresponding to the triplet structure is obtained by word embedding method; The word embedding vector corresponding to each character is input into a pre-trained semantic information extraction model to obtain the semantic feature vector corresponding to the character sequence of the sample image of the second formula.

8. An electronic device, characterized in that, The electronic device includes: Processor; and Stored program memory, The program includes instructions that, when executed by the processor, cause the processor to perform the image processing method according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the image processing method according to any one of claims 1 to 6.

Images