Method, device and electronic device for page block sequencing

Abstract

The application discloses a layout block sorting method, belongs to the field of optical character recognition, and helps to improve the efficiency of workflow execution. The method comprises the following steps: cutting a target image to obtain spatial information of at least two layout blocks to be sorted, and generating a layout block spatial information sequence; performing feature extraction and coding processing on the layout block spatial information sequence through an encoding network module of a pre-trained neural network model, obtaining a feature coding sequence corresponding to the layout block spatial information sequence, and each layout block feature vector in the feature coding sequence corresponds to a layout block to be sorted; and performing sorting prediction on the feature coding sequence through a decoding network module of the neural network model, and obtaining a sorting result of each layout block to be sorted. The method learns the complex mapping relationship between the spatial information of the layout block and the layout block sorting position through the neural network model, and is applied to the layout block sorting, so that the accuracy of the layout block sorting is improved.

Description

Technical Field

[0001] This application relates to the field of optical character recognition, and in particular to methods and apparatus for sorting page blocks, as well as electronic devices and computer-readable storage media. Background Technology

[0002] In Optical Character Recognition (OCR) applications, layout analysis is a crucial step. Layout analysis primarily involves analyzing and understanding text, images, and tables within an image. Typically, layout analysis begins by segmenting the image into regions that belong to text, images, or tables; these regions are called layout blocks. The layout blocks segmented through layout analysis are isolated regions, lacking a proper logical structure. Therefore, it is necessary to sort these layout blocks before understanding the content. Clearly, the sorting result of the layout blocks significantly impacts subsequent layout content analysis.

[0003] Existing page block sorting methods mainly include the following two types: The first type considers the global minimum distance between all page blocks and iteratively merges them according to line-major order; the second type scans optical character recognition text blocks according to line-major order to obtain page blocks and selects the nearest page block of the current page block as the next linked page block. However, existing page block sorting methods are sensitive to the position coordinates of page blocks and are only suitable for images with relatively fixed page block layouts. For cases where related content is distributed in page blocks that are far apart, the sorting accuracy is low.

[0004] It is evident that the existing methods for sorting page blocks still need improvement. Summary of the Invention

[0005] This application provides a method for sorting page blocks, which helps to improve the accuracy of page block sorting.

[0006] In a first aspect, embodiments of this application provide a method for sorting page blocks, including:

[0007] A sequence of page block spatial information is generated based on the spatial information of at least two page blocks to be sorted, wherein the at least two page blocks to be sorted are obtained by cutting a target image into page blocks;

[0008] The page block spatial information sequence is processed by the encoding network module of the pre-trained neural network model to extract features and encode them, thereby obtaining the feature encoding sequence corresponding to the page block spatial information sequence. Each page block feature vector in the feature encoding sequence corresponds one-to-one with the page block to be sorted.

[0009] The feature encoding sequence is sorted and predicted by the decoding network module of the neural network model to obtain the sorting result of each of the unsorted blocks.

[0010] Secondly, embodiments of this application provide a device for sorting page blocks, comprising:

[0011] The page block spatial information sequence generation module is used to generate a page block spatial information sequence based on the spatial information of at least two page blocks to be sorted, wherein the at least two page blocks to be sorted are obtained by page block cutting of a target image;

[0012] The feature encoding sequence acquisition module is used to extract and encode features from the spatial information sequence of the page blocks through the encoding network module of the pre-trained neural network model, and obtain the feature encoding sequence corresponding to the spatial information sequence of the page blocks, wherein each feature vector of the page block in the feature encoding sequence corresponds one-to-one with the page block to be sorted.

[0013] The sorting module is used to predict the sorting of the feature encoding sequence through the decoding network module of the neural network model, so as to obtain the sorting result of each of the unsorted blocks.

[0014] Thirdly, embodiments of this application also disclose an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the page block sorting method described in embodiments of this application.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, contains the steps of the page block sorting method disclosed in embodiments of this application.

[0016] The page block sorting method disclosed in this application generates a page block spatial information sequence based on the spatial information of at least two page blocks to be sorted, wherein the at least two page blocks to be sorted are obtained by segmenting a target image into page blocks; the encoding network module of a pre-trained neural network model performs feature extraction and encoding processing on the page block spatial information sequence to obtain a feature encoding sequence corresponding to the page block spatial information sequence, wherein each page block feature vector in the feature encoding sequence corresponds one-to-one with the page block to be sorted; the decoding network module of the neural network model performs sorting prediction on the feature encoding sequence to obtain the sorting result of each page block to be sorted, which helps to improve the accuracy of page block sorting.

[0017] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] Figure 1 This is a schematic flowchart of the page block sorting method according to Embodiment 1 of this application;

[0020] Figure 2 This is a schematic diagram of the page layout obtained by image segmentation;

[0021] Figure 3 This is a schematic diagram of the neural network model structure described in Embodiment 1 of this application;

[0022] Figure 4 This is a schematic diagram of the decoding network module structure of the neural network model in Embodiment 1 of this application;

[0023] Figure 5 This is another flowchart illustrating the page block sorting method of Embodiment 1 of this application;

[0024] Figure 6 It uses the existing page block sorting method to... Figure 2 A schematic diagram showing the sorting results of the page blocks shown.

[0025] Figure 7 The method of page block sorting disclosed in the embodiments of this application is used to sort the pages. Figure 2 A schematic diagram showing the sorting results of the page blocks shown.

[0026] Figure 8 This is one of the schematic diagrams of the device structure for sorting page blocks according to Embodiment 2 of this application;

[0027] Figure 9 This is the second schematic diagram of the device structure for sorting page blocks according to Embodiment 2 of this application.

[0028] Figure 10 A block diagram schematically illustrates an electronic device for performing the method according to this application; and

[0029] Figure 11 A storage unit for holding or carrying program code implementing the method according to this application is illustrated schematically. Detailed Implementation

[0030] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0031] Example 1

[0032] This application discloses a method for sorting page blocks, such as... Figure 1 As shown, the method includes steps 110 to 130.

[0033] Step 110: Generate a spatial information sequence of page blocks based on the spatial information of at least two page blocks to be sorted, wherein the at least two page blocks to be sorted are obtained by cutting a target image into page blocks.

[0034] The page block sorting method described in this application embodiment is used to sort several page blocks obtained after scanning and page block cutting of an image in the field of optical character recognition, so as to facilitate subsequent tasks to perform image content recognition or other operations based on the sorted page blocks.

[0035] In this application, the specific implementation method for cutting out page blocks from a target image is not limited. After image scanning and page cutting processing, the spatial information of each page block included in the image can be obtained. In some embodiments of this application, the spatial information of the page block is used to identify the position and size of the page block in a rectangular area in the image. For example, the spatial information can be represented as the coordinates of the upper left and lower right corners of the page block; it can also be represented as the coordinates of the lower left and upper right corners of the page block; or it can be represented as the coordinates of any three vertices of the page block; or it can be represented as the center coordinates and size of the page block, etc.

[0036] The spatial information of the page blocks described in the embodiments of this application is normalized spatial information. For example, the position coordinates and / or size of the page blocks cut from a target image are first normalized to map the position coordinates and / or size to a specified size image range. Then, the normalized position coordinates and / or size are used as the spatial information of the page blocks.

[0037] by Figure 2Taking the target image shown as an example, after being cut into page blocks, the target image is divided into 9 page blocks, that is... Figure 2 Pages 210 to 290.

[0038] In the embodiments of this application, the spatial information of each page block obtained by cutting a page block from an image is used as the feature of the page block. Based on the spatial information of the page block, the page blocks are sorted by a pre-trained neural network model. Therefore, for multiple page blocks obtained by cutting a page block from an image (i.e., page blocks to be sorted), the spatial information of each page block is first arranged sequentially to generate a page block spatial information sequence. Taking the spatial information represented by the upper left corner coordinates (x1, y1) and lower right corner coordinates (x2, y2) of the page block as an example, the spatial information of a page block can be represented as "x1, y1, x2, y2", that is, the spatial information of each page block can be represented by a set of data of length 4. Then, after the spatial information of n page blocks is arranged sequentially, a page block spatial information sequence of length n×4 will be obtained. In the page block spatial information sequence, each set of spatial information corresponds to a page block to be sorted.

[0039] Step 120: Using the encoding network module of a pre-trained neural network model, feature extraction and encoding processing are performed on the spatial information sequence of the page blocks to obtain the feature encoding sequence corresponding to the spatial information sequence of the page blocks.

[0040] In this context, each feature vector of a page block in the feature encoding sequence corresponds one-to-one with the page block to be sorted.

[0041] Next, the acquired page block spatial information sequence (hereinafter referred to as "PointArray") is input into a pre-trained neural network model. The neural network model performs feature extraction, feature encoding and decoding on the page block spatial information sequence PointArray. Finally, the sorting result of the page blocks to be sorted corresponding to each page block spatial information in the page block spatial information sequence PointArray is output.

[0042] like Figure 3As shown, in some embodiments of this application, the neural network model includes an encoding network module 310 and a decoding network module 320. Specifically, the encoding network module 310 is used to extract features from the spatial information of each page block in the input page block spatial information sequence PointArray based on the feature mapping relationship learned during the training phase, and to encode the features, outputting an encoded vector of the page block spatial information sequence PointArray. The decoding network module 320 is used to decode the encoded vector output by the encoding network module 310 based on the feature mapping relationship learned during the training phase, and output the decoding result of the encoded vector, i.e., outputting the sorting result corresponding to the spatial information of each page block in the page block spatial information sequence PointArray. This sorting result is the sorting result of the unsorted page blocks corresponding to the spatial information of each page block in the page block spatial information sequence PointArray.

[0043] In some embodiments of this application, the coding network module 310 further includes a feature embedding submodule and a feature coding submodule.

[0044] The feature embedding submodule is used to extract features and transform the dimensions of the input page block spatial information sequence PointArray, converting it into a feature vector sequence. In some embodiments of this application, the feature embedding submodule can be implemented using a linear transformation layer. After performing a linear transformation on the input page block spatial information sequence PointArray, the feature embedding submodule obtains a high-dimensional vector corresponding to the page block spatial information sequence PointArray, for example, denoted as "PointArrayF1". For example, for the aforementioned page block spatial information sequence of length n×4, after the linear transformation by the feature embedding submodule, the feature embedding submodule will output a feature vector sequence of length n×K, where the value of K depends on the dimension of the linear transformation layer; for example, the value of K can be 256 or 512.

[0045] In some embodiments of this application, the feature encoding submodule is used to further extract features from the feature vector sequence PointArrayF1 output by the feature embedding submodule, in order to extract relevant features for sorting in the feature vector sequence and establish the sequential relationship of the sequence. In some embodiments of this application, the feature encoding submodule can use any of the following network structures, including but not limited to: Long Short-Term Memory (LSTM) network, Multilayer Perceptron (MLP) network, and Transformer network. After further feature extraction of the feature vector sequence PointArrayF1, the feature encoding submodule will output a feature encoding sequence corresponding to the feature vector sequence PointArrayF1, hereinafter referred to as "PointArrayF2". After further feature extraction of the feature vector output by the feature embedding submodule, for the aforementioned page block spatial information sequence of length n×4 (i.e., the page block spatial information sequence composed of the spatial information of n page blocks to be sorted), the feature encoding submodule will output a feature encoding sequence of length n×m, where the value of m depends on the structure of the feature encoding submodule. Where n represents the number of feature encoding groups, each group of feature encodings corresponds to a set of spatial information, and each group of feature encodings corresponds to a page block to be sorted; m represents the dimension of the feature encoding corresponding to each page block to be sorted. The feature encoding sequence PointArrayF2 can be further represented as: Z v1 Z v2 ...Z vn Among them, Z vn express.

[0046] by Figure 2 Taking the unsorted page blocks 210 to 290 as an example, if the spatial information of the unsorted page blocks 210, 220, 230, ..., 290 in the page block spatial information sequence is arranged sequentially from front to back, then the feature encoding sequence output by the corresponding encoding network module will have the feature codes of the unsorted page blocks 210, 220, 230, ..., 290 arranged sequentially from front to back.

[0047] Step 130: The feature encoding sequence is sorted and predicted by the decoding network module of the neural network model to obtain the sorting result of each of the unsorted blocks.

[0048] In some embodiments of this application, the decoding network module 320 performs sorting prediction based on the feature encoding of each unsorted block in the feature encoding sequence PointArrayF2 output by the encoding network module 310, and outputs the sorting result of each unsorted block respectively.

[0049] In some embodiments of this application, the decoding network module is a pointer network. For example, the decoding network module can adopt the structure of a traditional Sequence2Sequence network with an attention mechanism. In a traditional attention mechanism network, after calculating the weights, the encoder's output state is weighted to obtain a vector C. In a pointer network, after calculating the weights, the encoder state with the highest probability is selected as the output. The pointer network obtains the prediction result by outputting a probability distribution, i.e., a pointer. When the pointer network performs decoding prediction, it finds the element with the largest weight in the current input sequence at each time step. Since the output result comes entirely from the input sequence, the pointer network can adapt to changes in the length of the input sequence. Specifically, regarding page block sorting, the number of page blocks obtained from different images may vary. By using a pointer network to decode and predict the encoded spatial information sequence of page blocks, it can adapt to page block spatial information sequences of varying lengths, thereby allowing page blocks to be sorted based on the performance of each page block to be sorted.

[0050] In some embodiments of this application, the feature encoding sequence is ordered and predicted using the decoding network module of the neural network model to obtain the sorting results of each of the unsorted blocks. This includes: using the decoding network module of the neural network model to order the feature encoding sequence based on an attention mechanism to obtain the sorting results of each of the unsorted blocks. As mentioned above, the decoding network module can adopt a traditional Sequence2Sequence network structure with an attention mechanism, where, during the decoding process, the attention mechanism network is used to calculate the correlation between each node and the previous output, i.e., the attention weight.

[0051] In some embodiments of this application, the step of using the decoding network module of the neural network model to perform sorting prediction on the feature encoding sequence based on an attention mechanism to obtain the sorting results of each of the unsorted page blocks includes: performing average pooling on the page block feature vectors in the feature encoding sequence to obtain a global feature vector of the feature encoding sequence; using the global feature vector, a dynamically determined first vector, and the feature encoding sequence as inputs to the decoding network module, and performing N rounds of sorting prediction on the feature encoding sequence through the decoding network module of the neural network model to obtain the sorting result of one of the unsorted page blocks output in each round of sorting prediction; wherein, the value of N is equal to the number of unsorted page blocks minus one; When performing the first round of sorting prediction, the first vector is a zero vector; when performing the Mth round of sorting prediction, the first vector is obtained by weighted summation of the feature vectors of each page block with corresponding weights, wherein the weights corresponding to the feature vectors of each page block are: the attention coefficients that match the feature vectors of the page blocks calculated when performing the (M-1)th round of sorting prediction, where M is an integer greater than 1 and less than or equal to N; the sorting results of the page block to be sorted obtained in each round of sorting prediction are aggregated to obtain the sorting results of the at least two page blocks to be sorted, wherein the sorting result of the page block to be sorted obtained in each round of sorting prediction is used to: indicate the page block to be sorted at the sorting position of the page block matching the current round.

[0052] Below, in conjunction with Figure 4 The diagram shown illustrates the decoding principle of the decoding network module and provides an example of the decoding process (i.e., the sorting and prediction process).

[0053] As mentioned earlier, during the decoding prediction in the pointer network, the element with the largest weight in the current input sequence is found at each time step. Therefore, in this embodiment of the application, when performing sorting prediction through the pointer network, for the encoding result of the spatial information sequence consisting of the spatial information of n unsorted blocks, n-1 predictions (i.e., n-1 time steps) are required to obtain the sorting result of the n unsorted blocks.

[0054] Still represented by the feature encoding sequence PointArrayF2: Z v1 Z v2 ...Z vn Among them, Z vn For example, Z v1 Z v2 ...Z vn Z represents the page block feature vectors of the page blocks to be sorted, corresponding to the spatial information of the 1st to nth elements in the page block sequence. During decoding and prediction, the page block feature vector Z is first... v1 Zv2 ...Z vn Perform average pooling (e.g., on the feature vector Z of the page layout). v1 Z v2 ...Z vn By performing a weighted average, the global feature vector of the feature-encoded sequence is obtained (referred to as "Z" in this paper). G (represented by ""). Then, using the global feature vector, the dynamically determined first vector, and the feature encoding sequence as input to the decoding network module, the PointArrayF2 feature encoding sequence is subjected to n-1 rounds of sorting prediction through the Sequence2Sequence model combined with an attention mechanism. In each round of sorting prediction, the page block to be sorted with the largest weight is selected as the sorting result output for the corresponding round. In this way, after n-1 rounds of sorting prediction, n-1 page blocks to be sorted are selected in sequence, thus completing the sorting prediction of the n page blocks to be sorted.

[0055] In some embodiments of this application, the attention mechanism in the Sequence2Sequence model requires the output of the previous time step as the input of the current time step, and combines it with the aforementioned global feature vector and the feature encoding sequence to perform the current step ranking prediction. In some embodiments of this application, the dynamically determined first vector is calculated from the output of the previous step. For the first step prediction, in this embodiment, the first vector is set as the zero vector; for the second step prediction, in this embodiment, the first vector is calculated based on the hidden layer output of the first step of the neural network model; for the third step prediction, the first vector is calculated based on the hidden layer output of the second step of the neural network model; and so on, for the (n-1)th step prediction, the first vector is calculated based on the hidden layer output of the (n-2)th step of the neural network model.

[0056] In some embodiments of this application, the global feature vector, the dynamically determined first vector, and the feature encoding sequence are used as inputs to the decoding network module. The decoding network module of the neural network model performs a round of sorting prediction on the feature encoding sequence, including: concatenating the dynamically determined first vector and the global feature vector to obtain a second vector for the current round of sorting prediction; calculating the correlation score between the feature vector of each page block in the feature encoding sequence and the second vector through the attention mechanism network in the decoding network module of the neural network model; obtaining the attention coefficient that matches the feature vector of the corresponding page block based on the correlation score; determining the page block feature vector that matches the largest attention coefficient as the target page block feature vector; and determining the page block to be sorted corresponding to the target page block feature vector as the sorting result of the page block to be sorted output in the current round of sorting prediction.

[0057] like Figure 4 As shown, taking the current step prediction as the first round (i.e., the first time step) as an example, the first vector V is set. input Let V be the zero vector, and then the first vector V... input and global eigenvector Z G By concatenating the vectors, we obtain the second vector h used for ranking prediction in the current round (i.e., the first round). c .

[0058] Subsequently, the attention mechanism network in the decoding network module of the neural network model is used to calculate the feature vector (i.e., the aforementioned Z) of each block in the feature encoding sequence PointArrayF2. v1 Z v2 ...Z vn ) and the second vector h c The relevance score.

[0059] In some embodiments of this application, the step of calculating the relevance score between the feature vector of each page block in the feature encoding sequence and the second vector through the attention mechanism network in the decoding network module of the neural network model includes: calculating the second vector through the attention mechanism network in the decoding network module of the neural network model to obtain a single query for the current round of ranking prediction; and calculating the key value corresponding to each page block feature vector in the feature encoding sequence through the attention mechanism network to obtain a key value for each page block feature vector in the current round of ranking prediction; for each page block feature vector, the relevance score between the page block feature vector and the second vector is determined by the following method: in response to the sorting result of the page block to be sorted corresponding to the page block feature vector being output, the relevance score between the page block feature vector and the second vector is set to a negative value, so that the attention coefficient matched by the page block feature vector is 0; in response to the sorting result of the page block to be sorted corresponding to the page block feature vector being not output, the relevance score between the page block feature vector and the second vector is calculated based on the key value corresponding to the page block feature vector and the single query.

[0060] In some embodiments of this application, the attention mechanism network calculates the feature vector of each page block and the second vector h using the following method. c The relevance score.

[0061] First, learnable network parameters W in the attention mechanism network are utilized. Q and W K Calculate q for each query at each prediction step. c And, calculate the key value k of each node at each prediction step. i The calculation formulas are as follows:

[0062] q c =W Q ·h c ;k i =W k ·Z vi ;

[0063] Where i represents the index of the eigenvector in the eigenvector sequence, Z vi Let i represent the i-th eigenvector in the eigenvector sequence. Taking the aforementioned eigenvector sequence containing n eigenvectors as an example, the value of i can be in the range of (1, n).

[0064] The single query q is calculated. c After considering the key values ​​of each node, calculate the relationship between each unoutput node and the query q. cThe relevance score is used as the attention coefficient of the corresponding node in the current step prediction. In some embodiments of this application, the i-th node is related to a single query q. c correlation score a c，i The calculation formula is as follows:

[0065]

[0066] Where, d h Let T represent the dimension of the node feature vector, T be the matrix transpose, s represent the number of prediction steps (i.e., prediction rounds), and s' represent the number of nodes already output. As can be seen from the above formula, when calculating the relationship between each node and the query q... c When calculating the relevance score, if the node is an already output node, its relevance score is set to a negative value.

[0067] In this way, by performing a linear mapping on the relevance score (such as through a softmax layer), we obtain an attention coefficient corresponding to the relevance score of each node. For a relevance score with a value of -inf, the corresponding attention coefficient is 0, and the attention coefficient for other relevance score values ​​is a number greater than 0 and less than 1.

[0068] In the first round of prediction, when calculating the relevance score, since there are no output nodes yet, n relevance scores greater than 0 will be obtained. By linearly mapping the relevance scores, n attention coefficients with values ​​between 0 and 1 can be obtained, and the sum of these n attention coefficients equals 1. Next, the node corresponding to the attention coefficient with the largest value is selected as the output node for the current round; that is, the feature vector of the page block matched by the attention coefficient with the largest value is chosen as the sorting result for the current round. For the first round of prediction (i.e., the first time step), the feature vector of the page block corresponding to the output node will be the first page block in the page block sequence. For example, when the attention coefficient matched by the second node is the largest, the page block feature vector Z... v2 The page block to be sorted will be selected as the first page block in the generated page block sequence.

[0069] Next, the second round of prediction (i.e., the second time step) is performed. At this point, based on the attention coefficients of each node calculated during the first round of prediction, the feature vector of each block in the feature encoding sequence PointArrayF2 (i.e., the aforementioned Z) needs to be adjusted. v1 Z v2 ...Z vnThe values ​​are weighted and summed, and the resulting vector is used as the first vector for prediction in this round. In some embodiments of this application, nodes that have been output are no longer subject to attention coefficient calculation in subsequent prediction steps and are not output again. Then, the first vector V is... input and global eigenvector Z G By concatenating the vectors, we obtain the second vector h used for ranking prediction in the current round (i.e., the second round). c Subsequently, the attention mechanism network in the decoding network module of the neural network model is used to calculate the feature vector (i.e., the aforementioned Z) of each block in the feature encoding sequence PointArrayF2. v1 Z v2 ...Z vn ) and the second vector h c The relevance score.

[0070] During the second round of prediction, when calculating the relevance score, since a node (such as the aforementioned block feature vector Z) has already been output after the first round of prediction, v2 If the corresponding node is not calculated, then when calculating the relevance score, the relevance score of that node will no longer be calculated, but will be set to a negative value. In this way, the attention coefficient of that node will be 0, thus achieving the effect that the node will no longer be output in the subsequent prediction process.

[0071] According to the prediction method of each round mentioned above, the feature vectors of n page blocks are predicted in n-1 rounds to determine the first to n-1 page blocks in the series of page blocks to be generated from front to back. Thus, the sorting prediction of the n page blocks to be sorted is completed.

[0072] In some embodiments of this application, the neural network model is pre-trained. For example... Figure 5 As shown, before the encoding network module of the pre-trained neural network model performs feature extraction and encoding processing on the spatial information sequence of the page block to obtain the feature encoding sequence corresponding to the spatial information sequence of the page block, the method further includes: step 100.

[0073] Step 100: Train the neural network model based on several training samples.

[0074] To help readers understand the solution in this application, the following examples illustrate specific implementation methods for training neural network models based on several training samples.

[0075] In some embodiments of this application, the sample data of each training sample is a page block spatial information sequence, which consists of the spatial information of at least two page blocks to be sorted. The two page blocks to be sorted are obtained by cutting a sample image into page blocks. The sample label of each training sample is a sequence of true values ​​of page block sorting positions corresponding to the corresponding page block spatial information sequence. The information in the true value sequence of page block sorting positions and the information at the corresponding positions in the corresponding page block spatial information sequence belong to the same page block to be sorted.

[0076] The method for generating the spatial information sequence of page blocks is described above and will not be repeated here.

[0077] In some embodiments of this application, the sample labels can be represented by a numerical sequence. For example, a sample image can be divided into L blocks. These L blocks can be manually sorted, and each block can be manually labeled with a number, for example, the first to the Lth blocks. Then, according to the labeled numbers from 1 to L, the spatial information of the blocks with corresponding numbers is arranged from front to back to obtain the spatial information sequence of the blocks corresponding to the image, which serves as the sample data. Then, the numbers 1 to L are arranged in ascending order to obtain a sequence number, and this sequence number is used as the sample label corresponding to the spatial information sequence of the blocks.

[0078] In some embodiments of this application, training the neural network model includes: for each training sample, performing the following sorting prediction operation to obtain the prediction loss of the training sample: using the encoding network module of the neural network model to extract and encode features from the spatial information sequence of the page blocks of the training sample, obtaining a feature encoding sequence corresponding to the spatial information sequence of the page blocks, wherein each page block feature vector in the feature encoding sequence corresponds one-to-one with the page blocks to be sorted; using the decoding network module of the neural network model to perform sorting prediction on the feature encoding sequence, obtaining the predicted sorting result value of each page block to be sorted; calculating the prediction loss of the training sample based on the predicted sorting result value of each page block to be sorted in the training sample and the sample label of the training sample; calculating the model loss of the neural network model based on the prediction loss of all the training samples, and iteratively training the neural network model with the goal of optimizing the model loss.

[0079] When the encoding network module of the neural network model performs feature extraction and encoding processing on the spatial information sequence of the page blocks of the training samples, the feature encoding sequence corresponding to the spatial information sequence of the page blocks of the training samples is obtained. For a detailed implementation of the feature extraction and encoding processing of the spatial information sequence of the page blocks of the training samples using the encoding network module of the neural network model, please refer to the description in step 120 above regarding the feature extraction and encoding processing of the spatial information sequence of the page blocks of the target image using the encoding network module of the neural network model; it will not be repeated here.

[0080] The specific implementation method for predicting the sorting result of each unsorted block in the corresponding training sample by performing sorting prediction on the feature encoding sequence of the training sample through the decoding network module of the neural network model is described in step 130 above, and will not be repeated here.

[0081] During the training phase, for each training sample's block spatial information sequence input to the neural network model, the decoding network module of the neural network model outputs a corresponding sorting result. This output sorting result is the predicted sorting result for each block to be sorted in the corresponding training sample. Further, based on the error between the predicted sorting result and the actual sorting result (i.e., the labeled sorting position of each block in the sample label of the corresponding training sample) between each training sample's predicted sorting result and the actual sorting result, the neural network model's prediction loss for the corresponding training sample is calculated (for example, the error can be represented by the cross-entropy between the predicted and actual sorting results). Finally, based on the neural network model's prediction loss for all training samples, the model loss of the neural network model is calculated. With the goal of minimizing the model loss, the parameters of the neural network model are optimized using methods such as error forward propagation, and the neural network model is iteratively trained until the model loss converges.

[0082] As can be seen from the training process of the neural network model, the neural network model learns the mapping relationship between the spatial information of page blocks and the sorting position of page blocks through the training process. Therefore, in the application stage, for the input sequence of page block spatial information, the neural network model can predict the sorting position of the page blocks corresponding to the sequence of page block spatial information.

[0083] The page block sorting method disclosed in this application generates a page block spatial information sequence based on the spatial information of at least two page blocks to be sorted, wherein the at least two page blocks to be sorted are obtained by segmenting a target image into page blocks; the encoding network module of a pre-trained neural network model performs feature extraction and encoding processing on the page block spatial information sequence to obtain a feature encoding sequence corresponding to the page block spatial information sequence, wherein each page block feature vector in the feature encoding sequence corresponds one-to-one with the page block to be sorted; the decoding network module of the neural network model performs sorting prediction on the feature encoding sequence to obtain the sorting result of each page block to be sorted, which helps to improve the accuracy of page block sorting.

[0084] The page block sorting method disclosed in this application learns the complex mapping relationship between the spatial information of page blocks and the sorting position of page blocks through a neural network model and applies it to page block sorting. It can adapt to more page block distribution situations and greatly improves the accuracy of page block sorting for complex pages compared to sorting methods that only combine the main order of rows and distance judgment.

[0085] by Figure 2 Taking the image shown as an example, the existing method of page block sorting is used to sort page blocks 210 to 290, and the sorting result is as follows. Figure 6 As shown. The page block sorting method described in this embodiment is used to sort page blocks 210 to 290, and the resulting sorting result is as follows. Figure 7 As shown. By Figure 6 and Figure 7 The comparison shows that for page blocks that are not on the same line but have related content and are close in distance, the existing page block sorting methods do not classify them as related page blocks, which increases the difficulty of subsequent page block content analysis. However, the page block sorting method disclosed in the embodiments of this application can accurately sort page blocks.

[0086] Example 2

[0087] This application discloses a device for sorting page blocks, such as... Figure 8 As shown, it includes:

[0088] The page block spatial information sequence generation module 810 is used to generate a page block spatial information sequence based on the spatial information of at least two page blocks to be sorted, wherein the at least two page blocks to be sorted are obtained by page block cutting of a target image.

[0089] The feature encoding sequence acquisition module 820 is used to extract and encode features from the page block spatial information sequence through the encoding network module of the pre-trained neural network model, and obtain the feature encoding sequence corresponding to the page block spatial information sequence, wherein each page block feature vector in the feature encoding sequence corresponds one-to-one with the page block to be sorted.

[0090] The sorting module 830 is used to perform sorting prediction on the feature encoding sequence through the decoding network module of the neural network model to obtain the sorting result of each of the unsorted page blocks.

[0091] In some embodiments of this application, the decoding network module is built based on a pointer network, and the sorting module 830 is further used for:

[0092] The decoding network module of the neural network model performs sorting prediction on the feature encoding sequence based on the attention mechanism to obtain the sorting results of each of the unsorted page blocks.

[0093] In some embodiments of this application, the step of using the decoding network module of the neural network model to predict the sorting of the feature encoding sequence based on an attention mechanism to obtain the sorting results of each of the unsorted page blocks includes:

[0094] Average pooling is performed on the block feature vectors in the feature encoding sequence to obtain the global feature vector of the feature encoding sequence;

[0095] Using the global feature vector, the dynamically determined first vector, and the feature encoding sequence as input to the decoding network module, the decoding network module of the neural network model performs N rounds of sorting prediction on the feature encoding sequence, obtaining the sorting result of one of the unsorted page blocks output in each round of sorting prediction; wherein, the value of N is equal to the number of unsorted page blocks minus one; when performing the first round of sorting prediction, the first vector is the zero vector; when performing the Mth round of sorting prediction, the first vector is obtained by weighted summation of the feature vectors of each page block with corresponding weights, wherein the weights corresponding to the feature vectors of each page block are: the attention coefficients that match the feature vectors of the page blocks calculated when performing the (M-1)th round of sorting prediction, where M is an integer greater than 1 and less than or equal to N;

[0096] The sorting results of the one unsorted page block obtained from the sorting prediction in each round are aggregated to obtain the sorting results of the at least two unsorted page blocks, wherein the sorting result of the one unsorted page block obtained from the sorting prediction in each round is used to: indicate the unsorted page block at the sorting position of the page block matching the current round.

[0097] In some embodiments of this application, the global feature vector, the dynamically determined first vector, and the feature encoding sequence are used as inputs to the decoding network module. The decoding network module of the neural network model performs a round of sorting prediction on the feature encoding sequence, including:

[0098] The dynamically determined first vector and the global feature vector are concatenated to obtain the second vector used for ranking prediction in the current round.

[0099] The correlation score between the feature vector of each block in the feature encoding sequence and the second vector is calculated through the attention mechanism network in the decoding network module of the neural network model.

[0100] Based on the relevance score, obtain the attention coefficient that matches the feature vector of the corresponding page block;

[0101] The feature vector of the page block that matches the attention coefficient with the largest value is determined as the target page block feature vector, and the page block to be sorted corresponding to the target page block feature vector is determined as the sorting result of the page block to be sorted in the current round of sorting prediction output.

[0102] In some embodiments of this application, the process of calculating the correlation score between the feature vector of each block in the feature encoding sequence and the second vector through the attention mechanism network in the decoding network module of the neural network model includes:

[0103] The second vector is calculated using the attention mechanism network in the decoding network module of the neural network model to obtain a single query for the current round of ranking prediction; and the key value of each page block feature vector in the feature encoding sequence is calculated using the attention mechanism network to obtain the key value of each page block feature vector in the current round of ranking prediction.

[0104] For each of the page block feature vectors, the correlation score between the page block feature vector and the second vector is determined by the following method:

[0105] In response to the output of the sorting result of the page block to be sorted corresponding to the page block feature vector, the correlation score between the page block feature vector and the second vector is set to a negative value, so that the attention coefficient of the page block feature vector matching is 0;

[0106] In response to the fact that the sorting result of the page block to be sorted corresponding to the page block feature vector is not output, the correlation score between the page block feature vector and the second vector is calculated based on the key value corresponding to the page block feature vector and the single query.

[0107] In some embodiments of this application, such as Figure 9 As shown, the device further includes:

[0108] The neural network model training module 800 is used to train a neural network model based on several training samples. Each training sample's sample data is a sequence of page block spatial information, which consists of the spatial information of at least two page blocks to be sorted. These two page blocks are obtained by cutting a sample image into page blocks. Each training sample's sample label is a sequence of true values ​​for the page block sorting positions corresponding to the corresponding page block spatial information sequence. The information in the true value sequence for the page block sorting positions and the information at the corresponding positions in the corresponding page block spatial information sequence belong to the same page block to be sorted.

[0109] The trained neural network model includes:

[0110] For each training sample, the following ranking prediction operation is performed to obtain the prediction loss for the training sample:

[0111] The encoding network module of the neural network model performs feature extraction and encoding processing on the spatial information sequence of the page blocks of the training samples to obtain the feature encoding sequence corresponding to the spatial information sequence of the page blocks, wherein each page block feature vector in the feature encoding sequence corresponds one-to-one with the page blocks to be sorted.

[0112] The feature encoding sequence is sorted and predicted by the decoding network module of the neural network model to obtain the predicted sorting result value of each of the unsorted page blocks;

[0113] The prediction loss of the training samples is calculated based on the predicted sorting results of each unsorted panel in the training samples and the sample labels of the training samples.

[0114] Based on the prediction loss of all the training samples, the model loss of the neural network model is calculated, and the neural network model is iteratively trained with the goal of optimizing the model loss.

[0115] The page block sorting apparatus disclosed in this application is used to implement the page block sorting method described in Embodiment 1 of this application. The specific implementation of each module of the apparatus will not be repeated here, but can be found in the specific implementation of the corresponding steps in the method embodiment.

[0116] The page block sorting apparatus disclosed in this application generates a page block spatial information sequence based on the spatial information of at least two page blocks to be sorted, wherein the at least two page blocks to be sorted are obtained by segmenting a target image into page blocks; the encoding network module of a pre-trained neural network model performs feature extraction and encoding processing on the page block spatial information sequence to obtain a feature encoding sequence corresponding to the page block spatial information sequence, wherein each page block feature vector in the feature encoding sequence corresponds one-to-one with the page block to be sorted; the decoding network module of the neural network model performs sorting prediction on the feature encoding sequence to obtain the sorting result of each page block to be sorted, which helps to improve the accuracy of page block sorting.

[0117] The page block sorting apparatus disclosed in this application learns the complex mapping relationship between the spatial information of page blocks and the sorting position of page blocks through a neural network model and applies it to page block sorting. It can adapt to more page block distribution situations and greatly improves the accuracy of page block sorting for complex pages compared to sorting methods that only combine the main order of rows and distance judgment.

[0118] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus embodiments, since they are fundamentally similar to the method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0119] The above provides a detailed description of a method for sorting page blocks provided by this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method of this application and its core idea. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the idea of ​​this application. Therefore, the content of this specification should not be construed as a limitation of this application.

[0120] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0121] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components in the electronic device according to the embodiments of this application. This application can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such a program implementing this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0122] For example, Figure 10 An electronic device capable of implementing the methods according to this application is illustrated. The electronic device may be a PC, mobile terminal, personal digital assistant, tablet computer, etc. The electronic device conventionally includes a processor 1010 and a memory 1020, and program code 1030 stored in the memory 1020 and executable on the processor 1010. When the processor 1010 executes the program code 1030, it implements the methods described in the above embodiments. The memory 1020 may be a computer program product or a computer-readable medium. The memory 1020 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. The memory 1020 has a storage space 10201 for the program code 1030 of a computer program for performing any of the method steps described above. For example, the storage space 10201 for the program code 1030 may include various computer programs for implementing the various steps in the above methods. The program code 1030 is computer-readable code. These computer programs can be read from or written to one or more computer program products. These computer program products include program code carriers such as hard disks, compact discs (CDs), memory cards, or floppy disks. The computer program includes computer-readable code that, when executed on an electronic device, causes the electronic device to perform the methods according to the embodiments described above.

[0123] This application also discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the page block sorting method as described in Embodiment 1 of this application.

[0124] Such a computer program product can be a computer-readable storage medium, which can have the same characteristics as... Figure 10The memory 1020 in the illustrated electronic device is similarly arranged as storage segments, storage spaces, etc. Program code can be stored, for example, in a compressed form on the computer-readable storage medium. The computer-readable storage medium is typically as shown in the reference... Figure 11 The portable or fixed storage unit is described above. Typically, the storage unit includes computer-readable code 1030', which is code read by a processor. When executed by the processor, this code implements the various steps in the method described above.

[0125] The terms "an embodiment," "embodiment," or "one or more embodiments" as used herein mean that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of this application. Furthermore, please note that the examples of the phrase "in one embodiment" do not necessarily all refer to the same embodiment.

[0126] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0127] In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method of ordering a layout block, characterized by, The method comprises the following steps: generating a layout block spatial information sequence according to spatial information of at least two layout blocks to be sorted, wherein the at least two layout blocks to be sorted are obtained by performing layout block cutting on a target image; extracting and encoding the layout block spatial information sequence through an encoding network module of a pre-trained neural network model to obtain a feature code sequence corresponding to the layout block spatial information sequence, wherein each layout block feature vector in the feature code sequence corresponds to one of the layout blocks to be sorted; predicting the order of the layout blocks to be sorted through a decoding network module of the neural network model to obtain a sorting result of each of the layout blocks to be sorted; the decoding network module is based on a pointer network, and the predicting the order of the layout blocks to be sorted through the decoding network module of the neural network model to obtain a sorting result of each of the layout blocks to be sorted comprises: performing average pooling on the layout block feature vectors in the feature code sequence to obtain a global feature vector of the feature code sequence; taking the global feature vector, a first vector determined dynamically, and the feature code sequence as inputs of the decoding network module, and performing N rounds of order prediction on the feature code sequence through the decoding network module of the neural network model to obtain a sorting result of one of the layout blocks to be sorted in each round of order prediction; wherein N is equal to the number of the layout blocks to be sorted minus one; when performing the first round of order prediction, the first vector is a zero vector; when performing the Mth round of order prediction, the first vector is obtained by weighting and summing all the layout block feature vectors with corresponding weights, wherein the weight corresponding to each layout block feature vector is an attention coefficient matched with the layout block feature vector calculated when performing the M-1th round of order prediction, and M is an integer greater than 1 and less than or equal to N; aggregating the sorting results of one of the layout blocks to be sorted obtained in each round of order prediction to obtain a sorting result of the at least two layout blocks to be sorted, wherein the sorting result of one of the layout blocks to be sorted obtained in each round of order prediction is used to indicate the layout block to be sorted at the layout block sorting position matched with the current round.

2. The method of claim 1, wherein, taking the global feature vector, a first vector determined dynamically, and the feature code sequence as inputs of the decoding network module, and performing one round of order prediction on the feature code sequence through the decoding network module of the neural network model comprises: splicing the first vector determined dynamically and the global feature vector to obtain a second vector used for current round order prediction; calculating the relevance scores of each layout block feature vector in the feature code sequence and the second vector through an attention mechanism network in the decoding network module of the neural network model; obtaining an attention coefficient matched with the corresponding layout block feature vector according to the relevance scores; The layout block feature vector with the maximum attention coefficient is matched to determine a target layout block feature vector, and the layout block corresponding to the target layout block feature vector is determined as a sorting result of the layout block to be sorted in the current round of sorting prediction.

3. The method of claim 2, wherein, The correlation score of each layout block feature vector in the feature encoding sequence and the second vector is calculated by an attention mechanism network in a decoding network module of the neural network model, including: The second vector is calculated by an attention mechanism network in a decoding network module of the neural network model to obtain a single query for the current round of sorting prediction, and each layout block feature vector in the feature encoding sequence is calculated by the attention mechanism network to obtain a key value corresponding to each layout block feature vector in the current round of sorting prediction; For each layout block feature vector, the correlation score of the layout block feature vector and the second vector is determined by the following method: In response to the sorting result of the layout block corresponding to the layout block feature vector being output, the correlation score of the layout block feature vector and the second vector is set to a negative value, so that the attention coefficient matched with the layout block feature vector is 0; In response to the sorting result of the layout block corresponding to the layout block feature vector being not output, the correlation score of the layout block feature vector and the second vector is calculated according to the key value corresponding to the layout block feature vector and the single query.

4. The method according to any one of claims 1 to 3, characterized in that, Before the feature extraction and encoding processing of the layout block spatial information sequence by the encoding network module of the pre-trained neural network model, the method further includes: Training a neural network model based on a plurality of training samples, wherein the sample data of each training sample is a layout block spatial information sequence composed of spatial information of at least two layout blocks to be sorted, and the two layout blocks to be sorted are obtained by layout block cutting on a sample image; and the sample label of each training sample is a layout block sorting position real value sequence corresponding to the corresponding layout block spatial information sequence, wherein the information at the corresponding position in the layout block spatial information sequence and the corresponding position in the layout block sorting position real value sequence belong to the same layout block to be sorted. The training of the neural network model includes: For each training sample, the following sorting prediction operations are performed to obtain a prediction loss of the training sample: The feature extraction and encoding processing of the layout block spatial information sequence of the training sample by the encoding network module of the neural network model obtains a feature encoding sequence corresponding to the layout block spatial information sequence, wherein each layout block feature vector in the feature encoding sequence corresponds to the layout block to be sorted one by one; The sorting prediction of the feature encoding sequence by the decoding network module of the neural network model obtains a sorting result prediction value of each layout block to be sorted. According to the ranking result prediction value of each to-be-sequenced layout block in the training sample and the sample label of the training sample, a prediction loss of the training sample is calculated; According to the prediction loss of all the training samples, a model loss of the neural network model is calculated, and the neural network model is iteratively trained with the goal of optimizing the model loss.

5. An apparatus for page block ordering, the apparatus comprising: Comprise: A layout block spatial information sequence generation module is configured to generate a layout block spatial information sequence according to spatial information of at least two to-be-sequenced layout blocks, wherein the at least two to-be-sequenced layout blocks are obtained by performing layout block cutting on a target image; A feature encoding sequence acquisition module is configured to acquire a feature encoding sequence corresponding to the layout block spatial information sequence by performing feature extraction and encoding processing on the layout block spatial information sequence through an encoding network module of a pre-trained neural network model, wherein each layout block feature vector in the feature encoding sequence corresponds to one of the to-be-sequenced layout blocks; A sequencing module is configured to obtain a ranking result of each to-be-sequenced layout block by performing sequencing prediction on the feature encoding sequence through a decoding network module of the neural network model; The decoding network module is based on a pointer network, and the sequencing module is further configured to: perform average pooling processing on the layout block feature vectors in the feature encoding sequence to obtain a global feature vector of the feature encoding sequence; use the global feature vector, a dynamically determined first vector, and the feature encoding sequence as inputs of the decoding network module, and perform N rounds of sequencing prediction on the feature encoding sequence through the decoding network module of the neural network model to obtain a ranking result of one of the to-be-sequenced layout blocks output by each round of sequencing prediction; wherein N is equal to the number of to-be-sequenced layout blocks minus one; when performing the first round of sequencing prediction, the first vector is a zero vector; when performing the Mth round of sequencing prediction, the first vector is obtained by weighting and summing each layout block feature vector with a corresponding weight, wherein the weight corresponding to each layout block feature vector is an attention coefficient matched with the layout block feature vector calculated when performing the M-1th round of sequencing prediction, and M is an integer greater than 1 and less than or equal to N; aggregate the ranking result of one of the to-be-sequenced layout blocks obtained by each round of sequencing prediction to obtain a ranking result of the at least two to-be-sequenced layout blocks, wherein the ranking result of one of the to-be-sequenced layout blocks obtained by each round of sequencing prediction is used to indicate the to-be-sequenced layout block at the layout block sequencing position matched with the current round.

6. An electronic device comprising a memory, a processor, and program code stored on the memory and executable on the processor, wherein, The processor executes the program code to implement the layout block sequencing method of any one of claims 1 to 4.

7. A computer-readable storage medium having stored thereon a program code, characterized in that, The program code is executed by the processor to implement the steps of the layout block sequencing method of any one of claims 1 to 4.

Images