Data processing method and apparatus

Abstract

The application provides a data processing method and device, the method comprising: obtaining an original image, performing padding processing on the original image according to a preconfigured image scale space to obtain a padding processing image, so as to normalize the scale of the padding processing image, performing interpolation down-sampling processing on the padding processing image to obtain an interpolation down-sampling processing image with details information gain, extracting features of the interpolation down-sampling processing image according to a fine-grained global feature extraction network model, performing encoding processing on the features by using an encoder, and performing decoding processing on the encoded features by using a decoder based on an attention mechanism to obtain a formula recognition result in the original image. Therefore, the application can more accurately recognize fine-grained information in a formula.

Description

Technical Field

[0001] This application relates to the field of digital document analysis technology, and in particular to a data processing method and apparatus. Background Technology

[0002] In recent years, document recognition technology has developed rapidly. Many documents contain various formulas, making formula recognition an important component of document recognition. Because formulas are far more complex than text, formula recognition has become a challenging problem in document recognition.

[0003] However, formulas often contain fine-grained information such as superscript and subscript relationships, and accurately identifying this fine-grained information is a technical challenge. Summary of the Invention

[0004] This application provides a data processing method and apparatus for accurately identifying fine-grained information in formulas.

[0005] Firstly, this application provides a data processing method, including:

[0006] Obtain the original image.

[0007] The original image is padded according to the pre-configured image scale space to obtain a padded image, so that the scale of the padded image is normalized.

[0008] The filled image is subjected to interpolation downsampling to obtain an interpolated downsampling image with improved detail information gain.

[0009] Based on a fine-grained global feature extraction network model, features are extracted from the interpolated downsampling processed image.

[0010] An encoder is used to encode the features, and an attention-based decoder is used to decode the encoded features to obtain the formula recognition result in the original image.

[0011] Optionally, the fine-grained global feature extraction network model includes at least convolutional layers, pooling layers, and batch normalization layers. The pooling layers use two asymmetric pooling layers, n×m and m×n, where n is an integer greater than or equal to 2 and m is an integer greater than or equal to 1.

[0012] Optionally, an attention-based decoder is used to decode the encoded features, including:

[0013] Based on the final hidden state of the encoding process, the global fusion hidden state is obtained; the global fusion hidden state is used as the initial hidden state of the decoder, and the decoder based on the attention mechanism is used to decode the encoded features; wherein, the global fusion hidden state includes: the mean of the global hidden state of the encoder, or the mean of the final hidden state of each row of the encoder.

[0014] Optionally, the pre-configured image scale space includes multiple image grouping sizes. Based on the pre-configured image scale space, the original image is padded to obtain a padded image, thereby normalizing the scale of the padded image. This includes:

[0015] Based on the size of the original image, determine the image group size that is closest to the size of the original image from the pre-configured image scale space; based on the determined image group size, perform padding on the original image to obtain a padded image, wherein the size of the padded image is equal to the determined image group size, so as to normalize the scale of the padded image.

[0016] Optionally, the original image is padded according to a pre-configured image scale space to obtain a padded image. Before the scale of the padded image is normalized, the following steps are also included:

[0017] Multiple original images are clustered according to various preset clustering dimensions to obtain multiple image group sizes; the multiple clustering dimensions include width dimension, height dimension, and aspect ratio dimension.

[0018] Optionally, multiple original images can be clustered according to various preset clustering dimensions to obtain multiple image grouping sizes, including:

[0019] Based on the width dimension, multiple original images are clustered to obtain M different width calibration values; based on the height dimension, multiple original images are clustered to obtain N different height calibration values; based on the aspect ratio dimension, multiple original images are clustered to obtain K different aspect ratio calibration values; the product of each of the N different height calibration values ​​and the K different aspect ratio calibration values ​​is obtained to obtain K width values ​​corresponding to each different height calibration value; based on the N different height calibration values ​​and the K width values ​​corresponding to each different height calibration value, multiple image group sizes are obtained; where M, N, and K are integers greater than or equal to 2.

[0020] Secondly, this application provides a data processing apparatus, comprising:

[0021] The acquisition module is used to acquire the original image.

[0022] The filling processing module is used to fill the original image according to the pre-configured image scale space to obtain a filled image, so as to normalize the scale of the filled image.

[0023] The interpolation processing module is used to perform interpolation downsampling on the filled image to obtain an interpolated downsampling image with improved detail information gain.

[0024] The extraction module is used to extract features from the interpolated and downsampled image based on the fine-grained global feature extraction network model.

[0025] The encoding module is used to encode features.

[0026] The decoding module is used to decode the encoded features based on the attention mechanism to obtain the formula recognition result in the original image.

[0027] Optionally, the fine-grained global feature extraction network model includes at least convolutional layers, pooling layers, and batch normalization layers. The pooling layers use two asymmetric pooling layers, n×m and m×n, where n is an integer greater than or equal to 2 and m is an integer greater than or equal to 1.

[0028] Optional, decoding module, specifically used for:

[0029] Based on the final hidden state of the encoding process, the global fusion hidden state is obtained; the global fusion hidden state is used as the initial hidden state of the decoder, and the decoder based on the attention mechanism is used to decode the encoded features; wherein, the global fusion hidden state includes: the mean of the global hidden state of the encoder, or the mean of the final hidden state of each row of the encoder.

[0030] Optionally, the pre-configured image scale space includes multiple image grouping sizes, and the padding processing module is specifically used for:

[0031] Based on the size of the original image, determine the image group size that is closest to the size of the original image from the pre-configured image scale space; based on the determined image group size, perform padding on the original image to obtain a padded image, wherein the size of the padded image is equal to the determined image group size, so as to normalize the scale of the padded image.

[0032] Optionally, a scale space configuration module may also be included.

[0033] The scale space configuration module is used to fill the original image according to the pre-configured image scale space in the filling processing module to obtain the filled image. Before the scale of the filled image is normalized, multiple different original images are clustered according to multiple preset clustering dimensions to obtain multiple image group sizes. The multiple clustering dimensions include width dimension, height dimension, and aspect ratio dimension.

[0034] Optional, scale space configuration module, specifically used for:

[0035] Based on the width dimension, multiple original images are clustered to obtain M different width calibration values; based on the height dimension, multiple original images are clustered to obtain N different height calibration values; based on the aspect ratio dimension, multiple original images are clustered to obtain K different aspect ratio calibration values; the product of each of the N different height calibration values ​​and the K different aspect ratio calibration values ​​is obtained to obtain K width values ​​corresponding to each different height calibration value; based on the N different height calibration values ​​and the K width values ​​corresponding to each different height calibration value, multiple image group sizes are obtained; where M, N, and K are integers greater than or equal to 2.

[0036] Thirdly, this application provides a data processing apparatus, including: a memory and a processor;

[0037] Memory is used to store program instructions;

[0038] The processor is used to invoke program instructions in memory to execute the data processing method as described in the first aspect of this application.

[0039] Fourthly, this application provides a computer-readable storage medium storing computer program instructions, which, when executed, implement the data processing method as described in the first aspect of this application.

[0040] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the data processing method as described in the first aspect of this application.

[0041] The data processing method and apparatus provided in this application can adaptively fill the original image according to a pre-configured image scale space including multiple image group sizes to normalize the scale of the original image, use interpolation downsampling to enhance the detail information of the image, extract image features according to a fine-grained global feature extraction network model, and use the global fusion hidden state as the initial hidden state of the decoder. Therefore, this application can more accurately identify fine-grained information in the formula, thereby improving the overall recognition accuracy of the formula. At the same time, it can greatly improve the efficiency of batch training of data and the efficiency of model training. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 A scenario diagram illustrating a data processing method provided in an embodiment of this application;

[0044] Figure 2 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0045] Figure 3 A schematic diagram showing the formula image provided in an embodiment of this application before and after interpolation downsampling processing;

[0046] Figure 4 A schematic diagram of the organizational structure of a pooling module in a backbone network according to an embodiment of this application;

[0047] Figure 5 A flowchart illustrating a data processing method provided in another embodiment of this application;

[0048] Figure 6 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application;

[0049] Figure 7 This is a schematic diagram of the structure of a data processing apparatus provided in another embodiment of this application;

[0050] Figure 8 This is a schematic diagram of the structure of a data processing apparatus provided in another embodiment of this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] In recent years, document recognition technology has developed rapidly. Many documents contain various formulas, making formula recognition a crucial component of document recognition. Because formulas are far more complex than text, formula recognition has become a challenging problem in document recognition. However, formulas often contain fine-grained information such as subscript and superscript relationships, and accurately identifying this fine-grained information is a significant technical challenge.

[0053] With the development of neural machine translation technology, deep learning-based formula recognition methods have also seen a surge in research, migrating a number of encoder-decoder framework-based algorithms from image description tasks to document formula recognition. However, in practice, these methods often exhibit misidentification, omission, and duplication of subscripts and superscripts in formulas, thus affecting the overall accuracy of the recognition results. To alleviate these problems, it is necessary to refine the granularity of image feature extraction and input more detailed information into the recognition model. Furthermore, since formulas are two-dimensional text objects, while extracting sufficient details, it is also necessary to ensure the extraction of coarse-grained visual features such as spatial relationships between characters, corresponding to the logical relationships to obtain the final recognition result.

[0054] Currently, some methods employ a CNN network (convolutional network) with convolutional and max-pooling layers to extract global features from images. These feature maps are then encoded line by line and output to a decoder. The decoder uses an attention mechanism to assign different weights to the encoded information and fuses historical decoded label information using a conditional language model to obtain the final formula structure text description. Existing deep learning-based formula recognition tasks mostly use the aforementioned encoder-decoder text generation framework, differing only in the attention mechanism or encoder / decoder network structure employed. However, they generally suffer from low accuracy in identifying superscripts and subscripts and lack improved mechanisms for distinguishing between them.

[0055] Therefore, this application provides a data processing method and apparatus that focuses on the backbone network to improve the recognition accuracy of subscripts and superscripts, thereby improving the overall recognition accuracy of the algorithm. Figure 1 A scenario diagram illustrating a data processing method provided in an embodiment of this application, such as... Figure 1 As shown, the formulas in the book are identified using data processing methods on a computer. Figure 1 The specific recognition process is as follows: The original image is acquired; according to a pre-configured image scale space, the original image is padded to obtain a padded image; the scale of the padded image is normalized; interpolation downsampling is performed on the padded image to obtain an interpolated downsampling image with increased detail information gain; features of the interpolated downsampling image are extracted using a fine-grained global feature extraction network model; the features are encoded using an encoder; and the encoded features are decoded using an attention-based decoder to obtain the formula recognition result in the original image. In extracting global features from the formula image, this application retains as much image detail information as possible while extracting global features from the fine-grained formula image. Therefore, this application can more accurately recognize the fine-grained information in the formula.

[0056] Figure 2This is a flowchart illustrating a data processing method provided in an embodiment of this application. The method of this embodiment can be applied to an electronic device, which may be a terminal device, such as a mobile phone, tablet computer, laptop computer, or desktop computer. Figure 2 As shown, the method in this embodiment includes:

[0057] S201. Obtain the original image.

[0058] In this embodiment, specifically, the original image is an image containing a formula, i.e., a formula image. This original image may be input by a user to the electronic device executing this method embodiment, or it may be sent by another device to the electronic device executing this method embodiment. Therefore, the input original image is obtained.

[0059] S202. Based on the pre-configured image scale space, perform padding processing on the original image to obtain a padded image, so as to normalize the scale of the padded image.

[0060] In this embodiment, an image scale space has been pre-configured. The original image has its own scale, namely the width and height of the image. Therefore, according to the pre-configured image scale space, the original image is filled to normalize the scale of the filled image and obtain the filled image.

[0061] S203. Perform interpolation downsampling on the filled image to obtain an interpolation downsampling image with improved detail information gain.

[0062] In this embodiment, a filled image has been obtained. According to a preset scaling ratio, the filled image is subjected to interpolation downsampling processing to reduce the dimension of the formula image while filling in the detailed information of the formula image. Therefore, an interpolation downsampling processed image with increased detail information is obtained. Figure 3 A schematic diagram showing the formula image provided in an embodiment of this application before and after interpolation and downsampling processing, as shown below. Figure 3 As shown, the preset scaling ratio is, for example, 2. Figure 3 Image (a) in the image is the formula image before interpolation and downsampling. Figure 3 Image (b) is the formula image after interpolation downsampling with a scaling factor of 2. Figure 3 The effect of simply magnifying the formula image in (b) to the size of the formula image before interpolation and downsampling is as follows: Figure 3 As shown in (c), the white pixels represent added details in the formula image. It is evident that interpolation downsampling can simultaneously achieve dimensionality reduction and enrichment of detail information in the formula image.

[0063] Compared to existing pooling downsampling methods, the main difference between interpolation downsampling and pooling downsampling lies in the degree of detail preservation. Taking max pooling as an example, an m1×m1 pane of the original image is encoded as a single pixel. The value of this pixel is the maximum value of all pixels within the window. Therefore, scaling the image using pooling does not yield more image information. Interpolation downsampling, on the other hand, performs interpolation during image scaling, inserting new elements between the original image pixels using a given interpolation algorithm. Thus, interpolation downsampling can compress image information while preserving richer detail.

[0064] S204. Extract features from the interpolated downsampling processed image based on the fine-grained global feature extraction network model.

[0065] In this embodiment, a fine-grained global feature extraction network model and an interpolated downsampled image have been determined. Therefore, features of the interpolated downsampled image are extracted to obtain the features of the interpolated downsampled image.

[0066] Optionally, the fine-grained global feature extraction network model includes at least convolutional layers, pooling layers, and batch normalization layers. The pooling layers use two asymmetric pooling layers, n×m and m×n, where n is an integer greater than or equal to 2 and m is an integer greater than or equal to 1.

[0067] In this embodiment, the fine-grained global feature extraction network model is the backbone network, which is a multi-layer convolutional neural network constructed by intersecting convolutional layers, pooling layers, and batch normalization layers. The pooling layers are replaced with two asymmetric pooling layers, n×m and m×n, instead of the commonly used n×n pooling, to reduce the rate of feature size change and thus retain more fine-grained structural information. Figure 4 This is a schematic diagram of the organizational structure of a pooling module in a backbone network according to an embodiment of this application, as shown below. Figure 4 As shown, with n=2 and m=1, replacing the 2×2 pooling with two asymmetric pooling layers, 1×2 and 2×1, achieves the same downsampling effect. Furthermore, other network layers can be added between the two asymmetric pooling layers, allowing these other network layers to ( Figure 4 The "multiple convolutional layers" in the image are applied to a larger feature map, thereby reducing the loss of image details.

[0068] S205. The encoder encodes the features and the decoder based on the attention mechanism decodes the encoded features to obtain the formula recognition result in the original image.

[0069] In this embodiment, the feature map corresponding to the features of the interpolated downsampling image has been obtained. Therefore, the features of the feature map are encoded line by line by an encoder to obtain the image visual encoding information. The encoder can be a recurrent neural network (RNN) or various variants of RNN, such as Long Short-Term Memory (LSTM), Bidirectional Long Short-Term Memory (BiLSTM), or Gated Recurrent Unit (GRU) networks.

[0070] Attention-based decoders fuse image visual encoding information with current decoded label information to obtain context information for the current step, and then decode the features of the encoded feature map. The decoder may employ an RNN or its variants, with a conditional language model as its mathematical model. The decoder fuses historical decoded label information and current step context information to obtain the next label with the highest probability, and this process is repeated cyclically to obtain the final label sequence, thus yielding the formula recognition result from the original image.

[0071] The data processing method provided in this application involves acquiring an original image, padding the original image according to a pre-configured image scale space to obtain a padded image, thereby normalizing the scale of the padded image, performing interpolation downsampling on the padded image to obtain an interpolated downsampling image with enhanced detail information, extracting features from the interpolated downsampling image using a fine-grained global feature extraction network model, encoding the features using an encoder, and decoding the encoded features using an attention-based decoder to obtain the formula recognition result in the original image. Because this application can adaptively pad the original image according to a pre-configured image scale space to normalize the scale, perform interpolation downsampling to enhance the detail information, and extract features from the image using a fine-grained global feature extraction network model, it can more accurately identify fine-grained information in formulas.

[0072] exist Figure 2 Based on the illustrated embodiments, in some embodiments, Figure 5 A flowchart of a data processing method provided in another embodiment of this application is shown below. Figure 5 As shown, the method in this embodiment may include:

[0073] S501, Obtain the original image.

[0074] In this embodiment, the specific implementation process of S501 can be found in [reference needed]. Figure 2 The relevant descriptions of the embodiments shown will not be repeated here.

[0075] S502. Based on multiple preset clustering dimensions, perform clustering processing on multiple different original images to obtain multiple image grouping sizes.

[0076] In this embodiment, multiple clustering dimensions have been preset. Therefore, multiple different original images are clustered to obtain multiple image grouping sizes.

[0077] Optionally, multiple clustering dimensions can be used, including width, height, and aspect ratio. Based on the width dimension, multiple original images are clustered to obtain M different width calibration values. Based on the height dimension, multiple original images are clustered to obtain N different height calibration values. Based on the aspect ratio dimension, multiple original images are clustered to obtain K different aspect ratio calibration values. The product of each of the N different height calibration values ​​and the K different aspect ratio calibration values ​​is obtained to obtain the K width values ​​corresponding to each different height calibration value. The N different height calibration values ​​and the K width values ​​corresponding to each different height calibration value are combined into K*N width-height combinations. From a set of K*N width-to-height combinations, identify two width-to-height combinations whose width differences are less than a preset value. Remove the width-to-height combination least close to its specified width value from these two combinations. Also remove width-to-height combinations from the K*N combinations whose width values ​​exceed the maximum width of the images in the formula image sample set, and remove width-to-height combinations from the K*N combinations whose height values ​​exceed the maximum height of the images in the formula image sample set. The resulting K*n width-to-height combinations are then used as multiple clustering group sizes. Here, M, N, and K are integers greater than or equal to 2.

[0078] In this embodiment, for example, K-means clustering is performed on the width, height, and aspect ratio of multiple different original images. The highest degree of clustering is achieved when width is clustered into 5 groups, height into 3 groups, and aspect ratios into 7 groups. The resulting calibration values ​​are set as follows:

[0079] Width: 100, 240, 500, 1000, 1200, minimum 80, maximum 1800

[0080] Height: 80, 150, 200, minimum 40, maximum 300

[0081] Aspect ratio: 1.5, 3, 4, 6, 10, 15, 20

[0082] Among the multiple different original images, the minimum width of the image is 80 and the maximum width is 1800, and the minimum height of the image is 40 and the maximum height is 300. Based on the prior knowledge that the formula is mostly elongated shape, the height and aspect ratio are multiplied by the height calibration value to obtain the preset group size as: [120, 80], [240, 80], ..., [800, 80], ..., [4000, 200]. The intervals between [240, 80] and [320, 80] are relatively small, and 240 is closer to one of the width calibration parameters (240), so the latter is deleted; the group sizes [2250, 150], [3000, 150], ... [4000, 200] exceed the maximum width of 1800, so they are all deleted; based on the maximum width and maximum height of the images in multiple different original images, the group size [1800, 300] is added, and the final group sizes of multiple images are: [[120, 80], [240, 80], ..., [800, 80], ..., [1200, 200], [1800, 300]].

[0083] S503. The pre-configured image scale space includes multiple image group sizes. Based on the size of the original image, the image group size that is closest to the size of the original image is determined from the pre-configured image scale space.

[0084] In this embodiment, an image scale space has been configured based on the obtained multiple image group sizes. That is, the image scale space includes multiple image group sizes, each including width and height. The original image has its own dimensions, namely its width and height. Therefore, based on the dimensions of the original image, the image group size closest to the original image size is determined from the configured image scale space. For example, the final multiple image group sizes are: [[120, 80], [240, 80], ..., [800, 80], ..., [1200, 200], [1800, 300]]. If the size of an input formula image is [766, 47], then the image group size closest to the size of that formula image is determined to be [800, 80].

[0085] S504. Based on the determined image grouping size, the original image is padded to obtain a padded image, wherein the size of the padded image is equal to the determined image grouping size, so as to normalize the scale of the padded image.

[0086] In this embodiment, the image grouping size corresponding to the original image has been determined. Therefore, the original image is padded to obtain a padded image. The size of the padded image is equal to the determined image grouping size to normalize the padded image scale. The adaptive padding operation specifically involves sequentially searching for the smallest grouping size larger than the input image in the grouping size array, and then padding the input image according to that grouping size. During subsequent model training, the data generator takes data from one of the groups in batches each time, improving the efficiency of batch training and accelerating model convergence under the same parameter magnitude. For example, if the final image group sizes are: [[120, 80], [240, 80], ..., [800, 80], ..., [1200, 200], [1800, 300]], and the size of an input formula image is [766, 47], and the image group size closest to this formula image size has been determined to be [800, 80], then an adaptive clustering grouping and filling operation is performed on this formula image. The formula image size is normalized to [800, 80], resulting in a filled image. This operation can be performed on all input formula images.

[0087] S505. Perform interpolation downsampling on the filled image to obtain an interpolated downsampling image with improved detail information gain.

[0088] S506. Extract features from the interpolated downsampling processed image based on the fine-grained global feature extraction network model.

[0089] In this embodiment, the specific implementation process of S505 and S506 can be found in [reference needed]. Figure 2 The relevant descriptions of the embodiments shown will not be repeated here.

[0090] S507. The features are encoded using an encoder, and the global fusion hidden state is obtained based on the final hidden state of the encoding process.

[0091] In this embodiment, the features of the interpolated downsampling image have been obtained. Therefore, an encoder is used to encode the features, and the global fusion hidden state is obtained based on the final hidden state of the encoding process.

[0092] Optionally, the globally fused hidden state includes: the mean of the encoder's global hidden states, or the mean of the final hidden states of each row of the encoder. Specifically, the globally fused hidden state may include: the arithmetic mean of the encoder's global hidden states, or the harmonic mean of the encoder's global hidden states, or the weighted mean of the encoder's global hidden states, or the arithmetic mean of the final hidden states of each row of the encoder, or the harmonic mean of the final hidden states of each row of the encoder, or the weighted mean of the final hidden states of each row of the encoder. This application does not impose any limitations here, as long as it achieves the purpose of introducing global information.

[0093] In this embodiment, for example: the global fusion hidden state is obtained based on the arithmetic mean of the final hidden states of each row of the encoder. The feature map of the interpolated downsampled image has already been obtained, with a size of H×W×C, where H represents height, W represents width, and C represents the number of channels. It is assumed that the encoder used is an RNN recurrent neural network. Formally, the encoder recursively maps the feature map grid and the hidden state of the previous time step to the current hidden state row by row. Here, the output of each time step of the RNN is taken as its corresponding hidden state. The final image visual encoding information can be represented as {G}. h,w}, where h = 1…H, w = 1…W, G h,w =RNN{G h,w-1 ,g h,w That is, the state of each row at the last time step of the encoder RNN recurrent neural network output is G. h,W Take their average value F g As a global fusion hidden state, i.e.

[0094] S508. The global fusion hidden state is used as the initial hidden state of the decoder. An attention-based decoder is used to decode the encoded features to obtain the formula recognition result in the original image.

[0095] In this embodiment, the global fusion hidden state has been obtained. For example, in the above embodiment, the global fusion hidden state F has been obtained. g , will F g As the initial hidden state S0 of the decoder, then It can be seen that S0 matches the hidden state output of each row of the encoder with the same degree of matching.

[0096] The global mean of the encoded information is introduced for decoder hidden state initialization. This involves averaging the encoder's final hidden state output row by row and outputting it to the decoder module as the initial hidden state, known as the "first sight." The final hidden state of the encoder neural network summarizes the global visual features of the image. As the name suggests, this initial hidden state guides the attention weights at the initial time steps. The decoder's "first sight" focuses on global image features with equal attention weight, introducing global information and fusing coarse and fine-grained feature information. This significantly accelerates model training convergence and also significantly improves recognition performance.

[0097] Assuming the decoder also uses an RNN (Recurrent Neural Network), at time step t, the hidden state S is decoded from the previous time step. t-1 With image visual coding features {G h,w The alignment probability α is calculated according to Formula 1. t .

[0098] α t =σ(S t-1 {G h,w Formula 1

[0099] Furthermore, based on the alignment probability distribution, the weighted sum of the image visual coding features is calculated to obtain the context vector C. t ,in,

[0100] Based on the known context vector C t and the previous time step hidden state S t-1 The current hidden state S is calculated by the decoder. t and the output O of the current step t The calculation method is similar to that of the encoder network, i.e., S t =RNN(S) t-1 C t ), O t =θ(S) t C t The decoder projection layer maps the current output to a conditional probability distribution p on the tokenized vocabulary, where p = p(y). t+1 |y1,…y t ,{G h,w})=γ(O t During prediction, the tag with the highest conditional probability is selected as the decoding tag (tok) for the current step. t This process of looping and decoding yields the final formula image text description {tok}. t}, which is the formula recognition result in the original image.

[0101] The data processing method provided in this application improves the recognition accuracy of fine-grained information such as superscripts and subscripts in formulas, while also enhancing model training efficiency. A qualitative comparison of the recognition performance of the benchmark model WYGIWYS (What You Get Is What You See, WYGIWYS) and the model in this application was conducted on the formula recognition benchmark dataset im2markup-100k. Some comparison results are shown in Table 1. As can be seen from Table 1, for each original image, the recognition result labeled 'a' is that of the model in this application, and the recognition result labeled 'b' is that of the benchmark model WYGIWYS. The comparison results demonstrate that the model in this application does indeed show a significant improvement in the recognition accuracy of fine-grained information such as superscripts and subscripts.

[0102] Table 1 shows some examples of comparison results.

[0103]

[0104] The data processing method provided in this application can adaptively fill the original image according to a pre-configured image scale space including multiple image group sizes to normalize the scale of the original image, use interpolation downsampling to enhance the detail information of the image, extract image features according to a fine-grained global feature extraction network model, and use the global fusion hidden state as the initial hidden state of the decoder. Therefore, this application can more accurately identify the fine-grained information in the formula, thereby improving the overall recognition accuracy of the formula. At the same time, it can greatly improve the efficiency of batch training of data and the efficiency of model training.

[0105] Figure 6 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application, as shown below. Figure 6 As shown, the data processing device 600 in this embodiment includes: an acquisition module 601, a filling processing module 602, an interpolation processing module 603, an extraction module 604, an encoding module 605, and a decoding module 606.

[0106] The acquisition module 601 is used to acquire the original image.

[0107] The filling processing module 602 is used to fill the original image according to the pre-configured image scale space to obtain a filled image, so as to normalize the scale of the filled image.

[0108] The interpolation processing module 603 is used to perform interpolation downsampling processing on the filled image to obtain an interpolated downsampling image with increased detail information gain.

[0109] The extraction module 604 is used to extract features from the interpolated downsampling processed image based on the fine-grained global feature extraction network model.

[0110] Encoding module 605 is used to encode features.

[0111] The decoding module 606 is used to decode the encoded features based on the attention mechanism to obtain the formula recognition result in the original image.

[0112] Based on any of the embodiments shown above, the fine-grained global feature extraction network model includes at least a convolutional layer, a pooling layer, and a batch normalization layer. The pooling layer uses two asymmetric pooling layers, n×m and m×n, where n is an integer greater than or equal to 2 and m is an integer greater than or equal to 1.

[0113] Based on any of the embodiments shown above, the encoding module 605 is specifically used for:

[0114] Based on the final hidden state of the encoding process, the global fusion hidden state is obtained;

[0115] Decoding module 606 is specifically used for:

[0116] The global fusion hidden state is used as the initial hidden state of the decoder. An attention-based decoder is used to decode the encoded features. The global fusion hidden state includes the mean of the global hidden state of the encoder, or the mean of the final hidden state of each row of the encoder.

[0117] Based on any of the embodiments shown above, the pre-configured image scale space includes multiple image grouping sizes, and the filling processing module 602 is specifically used for:

[0118] Based on the size of the original image, determine the image group size that is closest to the size of the original image from the pre-configured image scale space; based on the determined image group size, perform padding on the original image to obtain a padded image, wherein the size of the padded image is equal to the determined image group size, so as to normalize the scale of the padded image.

[0119] Based on any of the embodiments shown above, the data processing device 600 further includes: a scale space configuration module 607; the scale space configuration module 607 is used to perform fill processing on the original image according to the pre-configured image scale space in the fill processing module 602 to obtain a filled image, and to perform cluster processing on multiple different original images according to multiple preset clustering dimensions before the scale normalization of the filled image, to obtain multiple image group sizes; the multiple clustering dimensions include width dimension, height dimension, and aspect ratio dimension.

[0120] Based on any of the embodiments shown above, the scale space configuration module 607 is specifically used for:

[0121] Based on the width dimension, multiple original images are clustered to obtain M different width calibration values; based on the height dimension, multiple original images are clustered to obtain N different height calibration values; based on the aspect ratio dimension, multiple original images are clustered to obtain K different aspect ratio calibration values; the product of each of the N different height calibration values ​​and the K different aspect ratio calibration values ​​is obtained to obtain K width values ​​corresponding to each different height calibration value; based on the N different height calibration values ​​and the K width values ​​corresponding to each different height calibration value, multiple image group sizes are obtained; where M, N, and K are integers greater than or equal to 2.

[0122] The apparatus of this embodiment can be used to execute the technical solutions of any of the method embodiments shown above. Its implementation principle and technical effect are similar, and will not be repeated here.

[0123] Figure 7 This is a schematic diagram of the structure of a data processing apparatus provided in another embodiment of this application, as shown below. Figure 7 As shown, the data processing device 700 of this embodiment includes a memory 701 and a processor 702. The memory 701 and the processor 702 are connected via a bus.

[0124] Memory 701 is used to store program instructions.

[0125] Processor 702 is used to execute program instructions from memory:

[0126] The process involves: acquiring the original image; padding the original image according to a pre-configured image scale space to obtain a padded image, thus normalizing the scale of the padded image; performing interpolation downsampling on the padded image to obtain an interpolated downsampling image with improved detail information gain; extracting features from the interpolated downsampling image using a fine-grained global feature extraction network model; encoding the features using an encoder and decoding the encoded features using an attention-based decoder to obtain the formula recognition result from the original image.

[0127] Based on any of the embodiments shown above, the fine-grained global feature extraction network model includes at least a convolutional layer, a pooling layer, and a batch normalization layer. The pooling layer uses two asymmetric pooling layers, n×m and m×n, where n is an integer greater than or equal to 2 and m is an integer greater than or equal to 1.

[0128] Based on any of the embodiments shown above, the processor 702 is specifically used for:

[0129] Based on the final hidden state of the encoding process, the global fusion hidden state is obtained; the global fusion hidden state is used as the initial hidden state of the decoder, and the decoder based on the attention mechanism is used to decode the encoded features; wherein, the global fusion hidden state includes: the mean of the global hidden state of the encoder, or the mean of the final hidden state of each row of the encoder.

[0130] Based on any of the embodiments shown above, the pre-configured image scale space includes multiple image grouping sizes, and the processor 702 is specifically used for:

[0131] Based on the size of the original image, determine the image group size that is closest to the size of the original image from the pre-configured image scale space; based on the determined image group size, perform padding on the original image to obtain a padded image, wherein the size of the padded image is equal to the determined image group size, so as to normalize the scale of the padded image.

[0132] Based on any of the embodiments shown above, the processor 702 is further configured to perform filling processing on the original image according to a pre-configured image scale space to obtain a filled image, so that before the scale of the filled image is normalized, multiple different original images are clustered according to multiple preset clustering dimensions to obtain multiple image group sizes; the multiple clustering dimensions include width dimension, height dimension, and aspect ratio dimension.

[0133] Based on any of the embodiments shown above, the processor 702 is specifically used for:

[0134] Based on the width dimension, multiple original images are clustered to obtain M different width calibration values; based on the height dimension, multiple original images are clustered to obtain N different height calibration values; based on the aspect ratio dimension, multiple original images are clustered to obtain K different aspect ratio calibration values; the product of each of the N different height calibration values ​​and the K different aspect ratio calibration values ​​is obtained to obtain K width values ​​corresponding to each different height calibration value; based on the N different height calibration values ​​and the K width values ​​corresponding to each different height calibration value, multiple image group sizes are obtained; where M, N, and K are integers greater than or equal to 2.

[0135] The apparatus of this embodiment can be used to execute the technical solutions of any of the method embodiments shown above. Its implementation principle and technical effect are similar, and will not be repeated here.

[0136] Figure 8 This is a schematic diagram of the structure of a data processing apparatus provided in another embodiment of this application, as shown below. Figure 8 As shown, for example, the data processing device 800 can be provided as a computer. (See reference...) Figure 8 The data processing apparatus 800 includes a processing component 801, which further includes one or more processors, and memory resources represented by memory 802 for storing instructions executable by the processing component 801, such as application programs. The application programs stored in memory 802 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 801 is configured to execute instructions to perform any of the method embodiments described above.

[0137] The data processing device 800 may also include a power supply component 803 configured to perform power management of the data processing device 800, a wired or wireless network interface 804 configured to connect the data processing device 800 to a network, and an input / output (I / O) interface 805. The data processing device 800 can operate on an operating system stored in memory 802, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0138] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described data processing method.

[0139] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described data processing method.

[0140] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0141] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in a data processing device.

[0142] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0143] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized in that, include: Obtain the original image; The original image is filled according to the pre-configured image scale space to obtain a filled image, so that the scale of the filled image is normalized; The filled image is subjected to interpolation downsampling to obtain an interpolated downsampling image with improved detail information gain; the interpolation downsampling includes interpolation during downsampling of the filled image; The feature map mesh of the interpolated downsampling image is extracted based on a fine-grained global feature extraction network model that includes an asymmetric pooling layer. Based on the final hidden state of the encoding process, a global fusion hidden state is obtained; the global fusion hidden state is used as the initial hidden state of the decoder, and an attention-based decoder is used to decode the feature map grid after encoding to obtain the formula recognition result in the original image. The global fusion hidden state includes: the mean of the global hidden states of the encoder, or the mean of the final hidden states of each row of the encoder. The step of obtaining the global fusion hidden state based on the final hidden state of the encoding process includes: The encoder recursively maps the hidden states of the feature map grid line by line to obtain the state of each line at the last time step; and the mean of the global hidden states is obtained based on the state of each line at the last time step.

2. The method according to claim 1, characterized in that, The fine-grained global feature extraction network model includes at least convolutional layers, pooling layers, and batch normalization layers. The pooling layers employ two asymmetric pooling layers, n×m and m×n, where n is an integer greater than or equal to 2 and m is an integer greater than or equal to 1.

3. The method according to any one of claims 1-2, characterized in that, The pre-configured image scale space includes multiple image grouping sizes. The step of padding the original image according to the pre-configured image scale space to obtain a padded image, thereby normalizing the scale of the padded image, includes: Based on the size of the original image, determine the image group size that is closest to the size of the original image from a pre-configured image scale space; The original image is padded according to the determined image grouping size to obtain a padded image, wherein the size of the padded image is equal to the determined image grouping size so as to normalize the scale of the padded image.

4. The method according to claim 3, characterized in that, The step of filling the original image according to a pre-configured image scale space to obtain a filled image, before the scale of the filled image is normalized, further includes: Multiple original images are clustered according to various preset clustering dimensions to obtain the grouping size of the multiple images; The various clustering dimensions include width dimension, height dimension, and aspect ratio dimension.

5. The method according to claim 4, characterized in that, The step of clustering multiple different original images according to preset multiple clustering dimensions to obtain the multiple image grouping sizes includes: Based on the width dimension, multiple original images are clustered to obtain M different width calibration values ​​in the width dimension; Based on the height dimension, the multiple original images are clustered to obtain N different height calibration values ​​on the height dimension; Based on the aspect ratio dimension, the multiple original images are clustered to obtain K different aspect ratio calibration values ​​in the aspect ratio dimension; Obtain the product of each of the N different height calibration values ​​and the K different aspect ratio calibration values ​​to obtain the K width values ​​corresponding to each different height calibration value; The dimensions of the multiple image groups are obtained based on the N different height calibration values ​​and the K width values ​​corresponding to each different height calibration value; Where M, N, and K are integers greater than or equal to 2.

6. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire the original image; The filling processing module is used to fill the original image according to a pre-configured image scale space to obtain a filled image, so as to normalize the scale of the filled image; An interpolation processing module is used to perform interpolation downsampling processing on the filled image to obtain an interpolated downsampling image with improved detail information gain; the interpolation downsampling includes interpolation during downsampling of the filled image; The extraction module is used to extract the feature map mesh of the interpolated downsampling image based on a fine-grained global feature extraction network model including an asymmetric pooling layer. The encoding module is used to obtain the global fusion hidden state based on the final hidden state of the encoding process, including: recursively mapping the hidden state of the feature map grid row by row using the encoding module to obtain the state of each row at the last time step; and obtaining the mean of the global hidden state based on the state of each row at the last time step. A decoding module is used to take the global fusion hidden state as the initial hidden state of the decoding module, wherein the global fusion hidden state includes: the mean of the global hidden states of the encoder, or the mean of the final hidden states of each row of the encoder. The decoding module is also used to decode the encoded feature map grid based on an attention mechanism to obtain the formula recognition result in the original image.

7. A data processing apparatus, characterized in that, include: Memory and processor; The memory is used to store program instructions; The processor is used to invoke program instructions in the memory to execute the data processing method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed, implement the data processing method as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the data processing method according to any one of claims 1-5.

Images