Text frame sorting method and text frame sorting device for text image

Abstract

The present application relates to a method, device, computer equipment and storage medium for sorting text boxes of text images, which can be applied to sorting text boxes in videos. The method includes: based on the text boxes contained in the target text image, determining the spatial position features used to characterize the positional relationship between each text box; for each text box, extracting the visual features and text features corresponding to the text box, and performing feature fusion processing to obtain the node features of the nodes mapped by each text box in the graph structure; based on the node features of each node in the graph structure, updating the initial node connection features between each node determined based on the spatial position features to obtain updated node connection features; based on the node connection relationship between each node determined by the updated node connection features, sorting the text boxes represented by each node in the graph structure. The above method can accurately sort the text boxes in the target text image.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, computer equipment, and storage medium for sorting text boxes of text images. Background Art

[0002] With the development of science and technology, information has shown explosive growth. When faced with massive amounts of OCR (Optical Character Recognition) text, using manpower to obtain the content information is very limited and costly.

[0003] In the prior art, when performing OCR text recognition, the text on the image is usually detected and located, and then the characters on the detected text are recognized to obtain the recognized text content. The recognized text content is then displayed in order from top to bottom and from left to right based on the spatial position.

[0004] However, the text sorting results obtained by sorting and displaying based on spatial location features are less accurate. Summary of the Invention

[0005] Based on this, it is necessary to provide a text box sorting method, device, computer equipment and storage medium for text images that can improve the accuracy of text sorting in order to address the above technical problems.

[0006] A text frame sorting method for a text image, the method comprising:

[0007] Determining, based on text boxes contained in a target text image, a spatial position feature corresponding to the target text image, wherein the spatial position feature is used to characterize a positional relationship between text boxes in the target text image;

[0008] For each text box, extract the visual features and text features corresponding to the text box, perform feature fusion processing on the visual features and text features corresponding to the same text box, and obtain the node features of the node mapped by each text box in the graph structure;

[0009] Based on the node features of each node in the graph structure, feature updating is performed on the initial node edge features between the nodes determined based on the spatial position features to obtain updated node edge features;

[0010] Based on the node connection relationship determined by the updated node edge feature, the text boxes represented by each node in the graph structure are sorted.

[0011] A text frame sorting device for a text image, the device comprising:

[0012] A first feature extraction module is configured to determine a spatial position feature corresponding to the target text image based on text frames contained in the target text image, wherein the spatial position feature is used to characterize a positional relationship between text frames in the target text image;

[0013] The second feature extraction module is used to extract the visual features and text features corresponding to each text box, perform feature fusion processing on the visual features and text features corresponding to the same text box, and obtain the node features of the node mapped by each text box in the graph structure;

[0014] A feature updating module, configured to update the initial node edge features between the nodes determined based on the spatial position features based on the node features of each node in the graph structure, to obtain updated node edge features;

[0015] The text box sorting module is used to sort the text boxes represented by each node in the graph structure based on the node connection relationship determined by the updated node edge feature.

[0016] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0017] Determining, based on text boxes contained in a target text image, a spatial position feature corresponding to the target text image, wherein the spatial position feature is used to characterize a positional relationship between text boxes in the target text image;

[0018] For each text box, extract the visual features and text features corresponding to the text box, perform feature fusion processing on the visual features and text features corresponding to the same text box, and obtain the node features of the node mapped by each text box in the graph structure;

[0019] Based on the node features of each node in the graph structure, feature updating is performed on the initial node edge features between the nodes determined based on the spatial position features to obtain updated node edge features;

[0020] Based on the node connection relationship determined by the updated node edge feature, the text boxes represented by each node in the graph structure are sorted.

[0021] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0022] Determining, based on text boxes contained in a target text image, a spatial position feature corresponding to the target text image, wherein the spatial position feature is used to characterize a positional relationship between text boxes in the target text image;

[0023] For each text box, extract the visual features and text features corresponding to the text box, perform feature fusion processing on the visual features and text features corresponding to the same text box, and obtain the node features of the node mapped by each text box in the graph structure;

[0024] Based on the node features of each node in the graph structure, feature updating is performed on the initial node edge features between the nodes determined based on the spatial position features to obtain updated node edge features;

[0025] Based on the node connection relationship determined by the updated node edge feature, the text boxes represented by each node in the graph structure are sorted.

[0026] The above-mentioned text box sorting method, device, computer equipment and storage medium of the text image respectively obtain the spatial position features corresponding to the positional relationship between the text boxes and the visual features and text features corresponding to each text box based on the text boxes contained in the target text image, thereby realizing multimodal feature extraction. Taking each text box as a node, the node features of the nodes mapped by each text box in the graph structure are obtained by feature fusion of visual features and text features, thereby realizing multimodal feature fusion from the node dimension. Based on the node features of each node in the graph structure, the initial node connection features between each node determined based on the spatial position features are updated to obtain updated node connection features, thereby realizing multimodal feature fusion, so that the node connection relationship can be accurately determined based on the updated node connection features, thereby realizing accurate sorting of the text boxes represented by each node. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A diagram illustrating an application environment of a text frame sorting method for a text image in one embodiment;

[0028] Figure 2 Schematic diagram of a flow chart of a method for sorting text boxes of a text image in one embodiment;

[0029] Figure 3 Schematic diagram of a flow chart of a method for sorting text boxes of a text image in another embodiment;

[0030] Figure 4 Schematic diagram of a flow chart of a method for sorting text frames of a text image in another embodiment;

[0031] FIG5( a ) is a schematic diagram of performing matrix expansion of different dimensions on an initial node feature matrix in one embodiment;

[0032] FIG5( b ) is a schematic diagram of obtaining a spliced three-dimensional matrix and performing normalization processing in one embodiment;

[0033] Figure 6 A schematic diagram of performing MLP processing on a splicing node edge feature matrix in one embodiment;

[0034] FIG7( a ) and FIG7 ( b ) are schematic diagrams of updating the initial node feature matrix in one embodiment;

[0035] Figure 8 Schematic diagram of a flow chart of a method for sorting text boxes of a text image in another embodiment;

[0036] Figure 9 A schematic diagram of determining a node connection relationship between nodes in a graph structure in one embodiment;

[0037] Figure 10 Schematic diagram of a flow chart of a method for sorting text boxes of a text image in one embodiment;

[0038] Figure 11 Schematic diagram of the training process of a text box sorting model in one embodiment;

[0039] Figure 12 A schematic diagram of a process for training data for a text box sorting model in one embodiment;

[0040] Figure 13 is a structural block diagram of a text frame sorting device for a text image in one embodiment;

[0041] Figure 14 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0043] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0044] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0045] Computer vision (CV) is the science of making machines "see." Specifically, it refers to using cameras and computers to replace the human eye in identifying, tracking, and measuring objects. Furthermore, it involves image processing, which is then transformed into images more suitable for human observation or transmission to instrumentation. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include techniques such as optical character recognition (OCR).

[0046] In this application, it is particularly related to text box sorting in layout analysis in OCR. Text box sorting refers to identifying the corresponding arrangement order of text boxes in a text image. It not only requires the use of computer vision technology to obtain the location area of the text box in the text image, but also requires the use of machine learning to determine the corresponding arrangement order of text boxes in the text image.

[0047] The text frame sorting method provided by this application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The text image of the terminal 102 is sent to the server 104. The server 104 determines the spatial position feature corresponding to the target text image based on the text box contained in the target text image. The spatial position feature is used to characterize the positional relationship between the text boxes in the target text image. For each text box, the visual features and text features corresponding to the text box are extracted. The visual features and text features corresponding to the same text box are subjected to feature fusion processing to obtain the node features of the nodes mapped by each text box in the graph structure. Based on the node features of each node in the graph structure, the initial node edge features between each node determined based on the spatial position feature are updated to obtain the updated node edge features. Based on the node connection relationship determined by the binary classification result of the updated node edge feature, the text boxes represented by each node in the graph structure are sorted. The server 104 sends the sorted text to the terminal 102 for display. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, portable wearable devices, and vehicle-mounted terminals, and the server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0048] In one embodiment, Figure 2 As shown, a text box sorting method for text images is provided. Specifically, the text box sorting method for text images provided by the present application can be implemented by a trained text box sorting model. The training process of the text box sorting model can be described in the following embodiments. The execution subject in the text box sorting method for text images can be a terminal device that can call the text box sorting model, or a server that can call the text box sorting model. Among them, a client can be installed in the terminal device, and the terminal device can implement the call to the text box sorting model through the client. In the following process, the text box sorting method for text images provided by the present application is specifically explained by taking the server as the execution subject as an example, which specifically includes the following steps:

[0049] Step 202 : Based on the text boxes included in the target text image, determine the spatial position features corresponding to the target text image. The spatial position features are used to characterize the positional relationship between the text boxes in the target text image.

[0050] The target text image refers to the processing object that needs to be subjected to text recognition and sorting. The number of target text images can be one or more, for example, multiple frames of images in a video. The processing process of multiple target text images is the same and they do not interfere with each other. For the convenience of description, the following embodiments are described using a single target text image as an example. The target text image can specifically be a general text image, a card text image, a bill text image, etc. General text images include office document images, network images, digital images, table text images, etc. Card text images include identity card images, bank card images, business license images, business card images, etc. Bill text images include financial bill images, medical bill images, etc. The target text image can be an image containing text provided by a terminal. The server receives the target text image uploaded by the terminal that needs to be subjected to text box recognition and sorting.

[0051] A text box refers to an area in a target text image that has a certain style and contains at least one character, wherein the style of the text box can be pre-defined by configuration data, such as defining the style of the text box as a rectangle. In the target image text, the number of text boxes can be two or more, which is used to sort the text boxes. Specifically, the server can recognize the characters in the target text image through a text recognition model, and divide the area where the continuous characters are located into a text box. The text recognition model can be any text recognition model that has been trained in the field. For example, the above-mentioned text recognition model can be an existing OCR model. Specifically, the OCR model scans the target image text, determines its shape by detecting the dark and light pattern, and then uses the character recognition method to translate the shape into computer text, and divides the area where the continuous characters are located into a text box.

[0052] Spatial position features are used to characterize the positional relationships between text boxes in a target text image. The spatial position features corresponding to the target text image specifically include the positional relationships between any text box and each other text box in the target text image. These positional relationships can specifically include horizontal distances, vertical distances, and size differences between text boxes. Upon obtaining detailed information about a text box in the target text image and its positional relationship with another text box, detailed information corresponding to the other text box can be obtained based on the positional relationship. Spatial position plays a crucial role in the reading of text in an image. For example, the human eye prioritizes reading text within the same region. Text in a normal image is generally read from left to right and top to bottom, so spatial position features are crucial for determining text reading order. Specifically, the server extracts the positional information of each text box in the target text image, groups all text boxes into pairs, and performs positional relationship analysis on these combined text boxes to determine the positional relationships between the text boxes corresponding to the combinations. By pooling and analyzing the resulting positional relationships, the spatial position features corresponding to the target text image are obtained.

[0053] Specifically, the server identifies text boxes contained in the target text image and, based on the text boxes, determines the spatial position features corresponding to the target text image. Furthermore, after acquiring the target text image, the server inputs the target text image into a pre-trained text recognition model and divides the area containing the recognized continuous text into a text box to determine the text boxes contained in the target text image. The server then combines the text boxes with other text boxes, determines the positional relationship between each text box and the other text boxes, and obtains the spatial position features corresponding to the target text image by summarizing the positional relationships.

[0054] Step 204 : for each text box, extract the visual features and text features corresponding to the text box, perform feature fusion processing on the visual features and text features corresponding to the same text box, and obtain the node features of the node mapped by each text box in the graph structure.

[0055] Among them, visual features refer to the image features of the area where the text box is located. Image features mainly include color features, texture features, shape features and spatial relationship features of the image. Among them, color features are a global feature that describes the surface properties of the scene corresponding to the image or image area; texture features are also a global feature that also describes the surface properties of the scene corresponding to the image or image area; there are two types of shape features, one is contour features and the other is regional features. The contour features of the image are mainly aimed at the outer boundary of the object, while the regional features of the image are related to the entire shape area. Spatial relationship features refer to the mutual spatial position or relative direction relationship between multiple targets segmented from the image. Specifically, the server can extract visual features of the text box through a visual feature extraction network. The visual feature extraction network can be specifically a CNN (Convolutional Neural Network). The CNN network extracts visual features of the text box and obtains multiple feature maps corresponding to the area where the text box is located. The feature maps can exist in the form of a matrix. The image features of the text box area extracted by the CNN network are scattered features. The scattered features refer to multiple types of features. The specific number of features can be determined based on the specific structure of the CNN network. The number of filter channels, convolution kernel size and convolution stride in the CNN network can all be determined according to the actual application scenario and are not restricted.

[0056] In a specific application, the multiple feature maps of a text box obtained by a CNN network can be referred to as multiple pixel attribute feature matrices. Each feature map corresponds to a pixel attribute feature matrix, and each pixel attribute feature matrix is obtained by extracting features from the entire text box region. A pixel attribute feature matrix contains a single type of image feature in the text box region. Therefore, multiple pixel attribute feature matrices contain multiple types of image features for the text box region. These multiple types of image features are uncorrelated and, therefore, are discrete image features. The pixel attribute feature matrices primarily contain image features such as corners, edges, curves, and lines in the text box region. These image features are derived from the pixel values of each pixel in the text box region. Therefore, one pixel attribute feature matrix can contain corner features, one pixel attribute feature matrix can contain edge features, one pixel attribute feature matrix can contain curve features, and one pixel attribute feature matrix can also contain line features. The feature elements in the pixel attribute feature matrix contain attribute information for each pixel of the target object. This attribute information can include image information such as corners, edges, curves, and lines.

[0057] In another specific application, the selection of CNN includes but is not limited to U-net (a segmentation network), and visual feature extraction is performed using U-net of FCN (Fully Convolutional Networks). Generally, CNN networks add some fully connected layers at the end of the network, which obtain category probability information after softmax processing. FCN, on the other hand, replaces the following fully connected layers with convolutions to obtain a two-dimensional feature map, which is then followed by softmax to obtain the classification information of each pixel, thus solving the image segmentation problem. Compared with FCN networks, U-net networks use a different feature fusion method. Unlike FCN's point-by-point addition, U-Net uses a method of splicing features together in the channel dimension to form richer features. Visual feature extraction through the U-net network can quickly and effectively obtain richer features.

[0058] Text features are used to represent the basic units of text. They characterize the text content and have the ability to distinguish the target text from other texts. In Chinese text, characters, words, or phrases can be used as text feature items for text feature extraction. Text feature extraction specifically involves converting unstructured raw text into structured information that can be recognized and processed by computers. This involves scientifically abstracting the text and establishing a mathematical model to describe and replace it, enabling computers to recognize the text through calculations and operations on this model. Specifically, text feature extraction can be achieved using statistical-based feature extraction methods, semantic-based feature extraction methods, and other methods.

[0059] In a specific application, a server extracts text features using a text feature extraction network. The text feature extraction network can specifically be an RNN (Recurrent Neural Network) network, which is a neural network designed to process sequence data. Sequence data reflects the state or degree of change of an object or phenomenon over time, with subsequent data being related to previous data. Compared to conventional neural networks, RNNs are more capable of processing sequentially changing data. For example, sequence data can specifically be a text sequence, where the meaning of a word can vary depending on the context. By inputting the text sequence in a text box into the RNN network, the RNN can extract text features corresponding to the text in the text box. The RNN can be used to select an RNN, including but not limited to LSTM (Long Short-Term Memory). LSTM networks are a special type of RNN network designed to address the vanishing and exploding gradient problems during long sequence training. Simply put, compared to conventional RNNs, LSTM networks control the transmission state through gated states, retaining information that requires long-term memory and forgetting unimportant information. Therefore, LSTM networks perform better on longer sequences.

[0060] Feature fusion of visual and textual features refers to the process of optimizing the combination of different feature vectors using a feature fusion network. Specifically, the extracted visual and textual features can be controlled to have the same feature dimensions during the extraction process, for example, both are 256-dimensional, both are 128-dimensional, etc. By inputting the visual and textual features into a feature fusion network for feature fusion, they can be combined into a single feature with greater discriminative power than the input features. The feature fusion network can be implemented using methods such as concat and add, and the specific method can be selected based on the actual scenario needs.

[0061] A node is a connection point in a graph. In computer science, a graph is a data structure composed of two components: vertices and edges. A graph structure G can be described by its set of nodes V and edges E. By mapping text boxes in the target text image as nodes into the graph structure, each text box can be accurately represented by the nodes in the graph. The feature fusion result of the visual features and text features corresponding to the text box is the node feature corresponding to the node mapped in the graph structure.

[0062] Specifically, the server extracts the visual features and text features corresponding to each text box, performs feature fusion processing on the visual features and text features corresponding to the same text box, and obtains the feature fusion results corresponding to each text box. The server maps each text box in the target text image to the corresponding node in the graph structure, and uses the feature fusion results corresponding to each text box as the node features of the corresponding node.

[0063] Step 206 : Based on the node features of each node in the graph structure, the initial node edge features between the nodes determined based on the spatial position features are updated to obtain updated node edge features.

[0064] Node features are used to represent the corresponding characteristics of each node. Before updating, node features are only related to the characteristics of the node itself. The server fuses visual features with text features to represent the node features corresponding to the node mapped by the text box, achieving multi-dimensional feature expression.

[0065] The initial node edge feature is used to characterize the connection between nodes. Based on the spatial position feature, the positional relationship between any two text boxes can be obtained. Since text boxes are mapped to nodes in the graph structure, the initial node edge feature between each node in the graph structure can be obtained based on the spatial position feature.

[0066] The feature update of the initial node edge feature refers to the process of performing corresponding feature processing on the initial node edge feature according to the specified processing flow based on the influencing feature to obtain a new node edge feature. Among them, the influencing feature can be the node feature of each node in the graph structure, or it can be a combination of the node feature of each node in the graph structure and other features. It can be selected or set according to the needs of the actual scenario. The specified processing flow can specifically limit the feature processing order and processing method, such as feature fusion, feature linearization, feature normalization, and the processing order such as which one is processed first and which one is processed later. Updating the node edge feature is the result of updating the feature of the initial node edge feature.

[0067] In a specific application, feature updates can be achieved through graph neural networks. Graph neural networks generalize convolution operations from traditional data to graph data. The core idea is to learn a function mapping by which a node in the graph can aggregate its own features with those of its neighbors to generate a new representation of the node. Specifically, the server fuses visual features with text features and uses these fused features as node features in the graph neural network structure. This allows for the sorting of text boxes by determining the connectivity between nodes.

[0068] Step 208 : sorting the text boxes representing each node in the graph structure based on the node connection relationship determined by updating the node edge features.

[0069] Node connection relationships include node-to-node connections and node-to-node disconnections. Based on the updated node edge features, it is possible to achieve the fusion of visual features, text features, and spatial location features to obtain a very accurate feature expression.

[0070] Specifically, the server determines whether the relationship between each two nodes is connected or not based on the updated node edge features, and determines the connection objects of each node in turn based on the nodes with connected relationships, thereby realizing the concatenation of nodes, and sorting the text boxes based on the node concatenation results. The results obtained by sorting the text boxes can specifically include the serial numbers corresponding to the text boxes. After the server sorts the text boxes, it can also perform subsequent processing such as layout analysis based on the serial numbers of the text boxes. In addition, the server can also feed back the text box information including the serial numbers to the terminal, and the terminal splices and displays the contents of the text boxes based on the text box serial numbers.

[0071] The above-mentioned text box sorting method of text image realizes multimodal feature extraction by respectively obtaining the spatial position features corresponding to the positional relationship between text boxes and the visual features and text features corresponding to each text box based on the text boxes contained in the target text image. Taking each text box as a node, the node features of the nodes mapped by each text box in the graph structure are obtained by fusing the visual features with the text features, thereby realizing multimodal feature fusion from the node dimension. Based on the node features of each node in the graph structure, the initial node connection features between each node determined based on the spatial position features are updated to obtain updated node connection features, thereby realizing multimodal feature fusion, so that the node connection relationship can be accurately determined based on the updated node connection features, thereby realizing accurate sorting of the text boxes represented by each node.

[0072] In one embodiment, Figure 3As shown, based on the node features of each node in the graph structure, the initial node edge features between the nodes determined based on the spatial position features are updated to obtain updated node edge features, including steps 302 to 304.

[0073] Step 302: perform feature embedding and normalization processing on the spatial position features to obtain an initial node-edge feature matrix.

[0074] Feature embedding refers to the process of converting data into fixed-size feature representations for easier processing and computation. The primary purpose of embedding is to reduce the dimensionality of sparse features. This dimensionality reduction can be compared to a fully connected layer (without an activation function), where the weight matrix of the embedding layer is used to calculate the dimensionality. By embedding spatial position features, the processed spatial position features can be resized to the desired dimension, facilitating subsequent processing such as feature concatenation.

[0075] Normalization is the process of normalizing data. By changing the distribution of the original data, the weights of each feature dimension's influence on the objective function are made consistent. Normalization algorithms include linear transformation, logarithmic transformation, and inverse cotangent transformation, which can be selected based on the actual scenario. Specifically, normalization can constrain the values of feature data to be within the range [0, 1].

[0076] The spatial position feature corresponds to the entire target text image, including the positional relationship between all text boxes. Based on the feature embedding and normalization of the spatial position feature, the initial node-edge feature matrix can also represent the connection between all nodes in the graph structure.

[0077] In a specific application, the server inputs the output of the visual feature extraction network and the output of the text feature extraction network into the feature fusion network for feature fusion processing, and obtains the feature n of each node in the graph structure. i ∈R 1×256 , all nodes in the graph structure correspond to R n×256 The matrix, r ij It is the spatial location feature.

[0078] e′ ij =Normal(Er ij )

[0079] Among them, E is the embedding operation, for r ij Perform embedding operation, and then embed r ij Then perform the Normalization operation to obtain e′ ij ∈R1×256 , which is the initial node feature matrix.

[0080] Step 304 : Based on the initial node feature matrix formed by the node features corresponding to each node in the graph structure, the initial node-edge feature matrix is updated to obtain an updated node-edge feature matrix.

[0081] The initial node matrix is a matrix composed of node features corresponding to each node in the graph structure. Specifically, by fusing visual and text features, the feature dimension can be limited. For example, the node feature dimension can be set to 256. If the number of nodes in the text box (i.e., the graph structure) is n, then the initial node feature matrix composed of node features corresponding to each node in the graph structure is an n*256 matrix.

[0082] The initial node edge feature matrix is updated based on the initial node feature matrix. Specifically, the initial node feature matrix and the initial node edge feature matrix are fused, or the initial node feature matrix and the initial node edge feature matrix are fused, and other processing is performed after the fusion, such as MLP (Multi-layer Perceptron).

[0083] Specifically, the server performs feature embedding and normalization processing on the spatial position features in sequence to obtain an initial node-edge feature matrix, and obtains an initial node feature matrix based on the node features corresponding to each node in the graph structure. The server updates the initial node-edge feature matrix based on the initial node feature matrix to obtain an updated node-edge feature matrix.

[0084] Step 306 : sorting the text boxes represented by each node in the graph structure based on the node connection relationship determined by each updated node edge feature in the updated node edge feature matrix.

[0085] In this embodiment, by embedding and normalizing the spatial position features, an initial node edge feature matrix can be obtained, which is convenient for subsequent processing with the initial node feature matrix, thereby facilitating effective and rapid updating of the initial node edge feature matrix.

[0086] In one embodiment, based on an initial node feature matrix composed of node features corresponding to each node in a graph structure, an initial node edge feature matrix is updated to obtain an updated node edge feature matrix, including: performing matrix splicing processing on the initial node feature matrix and the initial node edge feature matrix to obtain a spliced node edge feature matrix; performing multi-layer perception processing on the spliced node edge feature matrix to obtain an updated node edge feature.

[0087] Matrix concatenation refers to the process of concatenating two or more matrices according to certain rules. The initial node-edge feature matrix and the initial node feature matrix can be matrices with the same number of rows and columns. By concatenating the initial node-edge feature matrix and the initial node feature matrix, node-to-node connection relationships can be combined.

[0088] Multilayer perceptron processing is based on the multilayer perceptron (MLP), a feed-forward artificial neural network that maps a set of input vectors to a set of output vectors. An MLP can be viewed as a directed graph consisting of multiple layers of nodes, each fully connected to the next. In addition to the input nodes, each node is a neuron with a nonlinear activation function. Compared to a single-layer perceptron, an MLP has multiple output terminals, with an output layer and hidden layers between the input and output terminals. Through multilayer perceptron processing, multidimensional data can be effectively integrated.

[0089] Specifically, the server inputs the initial node feature matrix and the initial node edge feature matrix into the graph neural network based on the feature fusion network. The graph neural network performs matrix splicing processing on the initial node feature matrix and the initial node edge feature matrix to obtain a spliced node edge feature matrix. The server performs multi-layer perception processing on the spliced node edge feature matrix based on the graph neural network to obtain and output the updated node edge features.

[0090] In this embodiment, by performing matrix splicing and multi-layer perception processing, the feature changes in the matrix splicing expansion and multi-layer perception reduction process are utilized to achieve full fusion of feature data and obtain a more accurate updated node-edge feature matrix.

[0091] In one embodiment, Figure 4 As shown, the initial node feature matrix is a two-dimensional matrix. Based on the node features of each node in the graph structure, the initial node edge features between each node determined based on the spatial position features are updated. The updated node edge features include:

[0092] Step 402: perform feature embedding and normalization processing on the spatial position features to obtain an initial node-edge feature matrix.

[0093] Step 404 : Expand the initial node feature matrix into matrices of different dimensions to obtain a first three-dimensional matrix and a second three-dimensional matrix.

[0094] Step 406 : normalize the spliced three-dimensional matrix formed by the first three-dimensional matrix and the second three-dimensional matrix to obtain a spliced node feature matrix.

[0095] Step 408 : Perform matrix concatenation processing on the concatenated node feature matrix and the initial node edge feature matrix to obtain a concatenated node edge feature matrix.

[0096] Step 410: Perform multi-layer perception processing on the spliced node edge feature matrix to obtain updated node edge features.

[0097] Among them, matrix expansions of different dimensions can be obtained based on different matrix expansion parameters. As shown in Figure 5(a) and Figure 5(b), the initial node feature matrix is a two-dimensional matrix of (n, 256). First, the two-dimensional matrix of (n, 256) is transformed into a three-dimensional matrix of (n, 1, 256) and a three-dimensional matrix of (1, n, 256). Based on the matrix expansion parameter expand(-1, n, -1), the three-dimensional matrix of (n, 1, 256) is expanded to (n, n, 25 6), based on the matrix expansion parameter expand(n,-1,-1), the three-dimensional matrix of (1, n, 256) is expanded to the three-dimensional matrix of (n, n, 256), thereby expanding the matrix from different dimensions to obtain two (n, n, 256), and then splicing the two (n, n, 256) three-dimensional matrices to obtain the three-dimensional matrix of (n, n, 256*2). By normalizing the three-dimensional matrix of (n, n, 256*2), we get (n 2 , 256*2) two-dimensional matrix, that is, the splicing node feature matrix.

[0098] like Figure 6 As shown, the splicing node feature matrix (n 2 , 256*2) and the initial node edge feature matrix (n 2 , 256) are spliced together to obtain (n 2 , 256*3) splicing node edge feature matrix, by (n 2 , 256*3) matrix is processed by MLP to obtain (n 2 , 256) of the updated node-edge feature matrix.

[0099] In a specific application, the specific processing process is as follows:

[0100] e′ ij =Normal(Er ij )

[0101] e ij =n i ||n j ||e′ ij

[0102] e ij =M(e ij )

[0103] Among them, r ij is the spatial position feature, E is the embedding operation, and ij Perform embedding operation, and then embed r ij Then perform the Normalization operation to obtain e′ ij ∈R 1×256 , which is the initial node-edge feature matrix. i ,n j ,e′ ij The three are connected in series to obtain the node edge feature e of each node in the graph neural network ij ∈R 1×(256*3) M represents MLP. After MLP network processing, e ij Convert to e ij ∈R 1×256 , that is, update the node-edge feature matrix.

[0104] In this embodiment, the initial node feature matrix is expanded and spliced in different dimensions and normalized to obtain (n 2 , 256*2) matrix, (n 2 , n*n rows in the (256*2) matrix can represent the combination of each two nodes, thereby realizing the pairwise combination of feature data from the perspective of node features, and then splicing the initial node edge feature matrix obtained based on the spatial position feature, so that the spliced (n 2 , 256*3) in the updated node-edge feature matrix can respectively characterize the node features and spatial position features of the combination of each two nodes. Based on this, the updated node-edge feature matrix after MLP processing can accurately express the feature information corresponding to the combination of each two nodes.

[0105] In one embodiment, based on an initial node feature matrix composed of node features corresponding to each node in a graph structure, an initial node edge feature matrix is updated to obtain an updated node edge feature matrix, including: inputting the initial node feature matrix composed of node features corresponding to each node and the initial node edge feature matrix into a graph neural network; in the graph neural network, based on the initial node feature matrix, the initial node edge feature matrix is updated to obtain an updated node edge feature matrix and output it.

[0106] Among them, the graph neural network (GNN) is a neural network that runs directly on the graph structure. GNN will update all input nodes and node edges to obtain updated nodes and edges. The input data of the graph neural network is the initial node feature matrix composed of the node features corresponding to each node, such as (n, 256), and the initial node edge feature matrix, such as (n2 , 256), for the initial node edge feature matrix, such as (n 2 , 256) can be specifically referred to the above embodiment and will not be described again here.

[0107] In this embodiment, the server inputs the initial node feature matrix and the initial node edge feature matrix composed of the node features corresponding to each node into the graph neural network. The graph neural network updates the initial node edge feature matrix based on the initial node feature matrix, obtains the updated node edge feature matrix and outputs it. Based on the graph neural network, the update logic of the node edge feature matrix can be built into the graph neural network. By training the graph neural network, the accurate parameters of each processing stage can be quickly obtained, which effectively improves the data processing efficiency and enables batch processing of data.

[0108] In one embodiment, the graph neural network is a graph neural network group formed by connecting at least two graph neural networks in sequence, and the output data of the previous graph neural network in the graph neural network group is the input data of the next graph neural network; the text box sorting method of the text image also includes: in the graph neural network, based on the updated node edge features, the initial node feature matrix is updated to obtain the updated node feature matrix and output it.

[0109] Among them, the graph neural network group is composed of at least two graph neural networks connected in sequence, and the output data of the previous graph neural network in the graph neural network group is the input data of the next graph neural network. Taking the graph neural network group including the first GNN layer and the second GNN layer connected in sequence as an example, the input data of the first GNN layer is the initial node feature matrix and the initial node edge feature matrix, and the output data of the first GNN layer is the input data of the second GNN layer, that is, the updated node feature matrix and the updated node edge feature matrix. The specific update process of the initial node edge feature matrix can be found in the above embodiment and will not be repeated here. The update process for the initial node feature matrix is as follows:

[0110]

[0111] in, represents the i-th node at the l-th time step (i.e., the l-th GNN layer), σ represents the relu activation function, and W l represents a fully connected layer. represents the normalized edge weight.

[0112]

[0113] After updating the initial node feature matrix based on the first GNN layer, the node feature matrix input to the second GNN layer is the updated node feature matrix, which in turn causes the second GNN layer to perform a secondary update on the updated node edge feature matrix based on the updated node feature matrix. It is understood that the more GNN layers there are, the more iterative updates are performed. The number of GNN layers in a graph neural network group can be determined based on the specific iterative update effect or actual application scenario.

[0114] In this embodiment, by connecting at least two graph neural networks in sequence to form a graph neural network group, multiple iterative updates of the node feature matrix and the node edge feature matrix can be achieved, thereby improving the accuracy of updating the node edge feature matrix.

[0115] In one embodiment, the text box sorting method of a text image also includes: updating the node features based on the updated node edge features to obtain updated node features; performing feature splicing processing on the updated node features and the updated node edge features, and performing multi-layer perception processing on the obtained splicing features to obtain secondary updated node edge features.

[0116] Among them, the node features are updated based on the updated node edge features. The specific process can be seen in Figure 7(a) and Figure 7(b). After obtaining the updated node edge features e ij ∈R 1×256 Afterwards, as shown in Figure 7(a), the node-edge feature matrix (n 2 , 256) to perform linear regression and obtain the matrix (n 2 , 1), then the matrix (n 2 1) Perform normalization according to the parameter view(n, n, 1) to obtain the matrix (n, n, 1), and then add the element values of the matrix (n, n, 1) and the custom matrix, where the custom matrix is a (n, n, 1) matrix. The diagonal elements in the custom matrix are infinite, so that the element values corresponding to the elements representing the node's own relationship in the matrix after the element values are accumulated are infinite, so as to reduce the impact on the judgment of the subsequent node connection relationship. The matrix after the element values are accumulated is the normalized edge weight Furthermore, as shown in Figure 7(b), by updating the node edge features (n 2 , 256) is normalized according to the parameter view(n, n, 1) to obtain the matrix (n, n, 256), and then the matrix after the element value accumulation is Perform full connection processing to obtain the matrix (n, 256) corresponding to the full connection processing result, and finally perform linear regression through the relu activation function to obtain the updated node feature (n, 256).

[0117] Furthermore, based on the node connection relationship determined by the updated node edge feature, the text boxes representing each node in the graph structure are sorted, including: based on the node connection relationship determined by the secondary updated node edge feature, the text boxes representing each node in the graph structure are sorted.

[0118] In this embodiment, by iteratively updating the initial node features, it is convenient to use the updated node features, and the updated node edge features are updated a second time, thereby improving the accuracy of the updated node edge features. Based on the node connection relationship determined by the second updated node edge features, the text boxes representing each node in the graph structure are sorted to obtain a more accurate text box sorting result.

[0119] In one embodiment, Figure 8 As shown, based on the node connection relationship determined by updating the node edge features, the text boxes representing each node in the graph structure are sorted, including:

[0120] In step 802 , binary classification is performed on each node edge feature in the updated node edge feature matrix to filter out the target node edge feature corresponding to each node, where the classification probability corresponding to the target node edge feature meets the probability condition.

[0121] Step 804 : Determine the connection object corresponding to each node based on the node connection relationship determined by the target node edge feature.

[0122] Step 806 : The connection object corresponding to each node sorts the text boxes in the target text image.

[0123] In which, each row of the updated node edge feature matrix can represent the node edge feature between each two nodes, and a binary classification process is performed on the node edge feature corresponding to each row in the matrix to determine the probability that a connection relationship exists between the two nodes corresponding to the node edge feature, and to determine whether a connection relationship exists or does not exist between the two nodes corresponding to the node edge feature. In which, the existence of a connection relationship indicates that the two text boxes are adjacent in the reading order, and the absence of a connection relationship indicates that the two text boxes are non-adjacent in the reading order. The probability condition can be determined based on the training process of the binary classification model, and the probability condition can specifically be greater than a set probability value.

[0124] Each node has connections with every other node, but each node should only have two connections. Specifically, when there are n nodes, each target node has connections with the remaining n-1 nodes, but there are only two of the remaining n-1 nodes, one before the target node and one after the target node.

[0125] In this embodiment, based on the binary classification process, it is possible to accurately determine the connection objects corresponding to the nodes, thereby achieving accurate sorting of the text boxes in the target text image.

[0126] In one embodiment, text boxes in a target text image are sorted based on the connection object corresponding to each node, including: determining an initial text box in the target text image based on the coordinates of each text box in the target text image; and sorting the text boxes in the target text image based on the node corresponding to the initial text box and the connection object corresponding to each node.

[0127] In order to find the order between nodes, that is, to determine the relationship between nodes through edges, we focus on the classification of edges. Divided into two categories, Represents the relationship between nodes. In a reading environment, there are two situations for the relationship between two text boxes. One is that the two text boxes are adjacent in the reading order, and the other is that the two text boxes are non-adjacent in the reading order. A node is a text box (not the initial node and the tail node). It has an upper node and a lower node in the reading order. The probability of whether the node is directly connected to other nodes can be calculated, and the two points with the highest probability can be calculated for the intermediate nodes. In this way, as long as a starting node is found, all nodes can be linked in order, and the result is the correct reading order. The difference between the starting node and the intermediate node is that for the intermediate node, we are looking for the two largest nodes adjacent to it, while for the starting node, we only need to find the largest node adjacent to it. After observing a large amount of data, it was found that according to the reading order of the human eye, the starting node is often at the top and leftmost in the target text image. Therefore, if Figure 9 As shown, based on the coordinates of each text box in the target text image, the coordinate data corresponding to the top and leftmost text box is determined as the initial text box start in the target text image, and then all nodes are linked in sequence, and the text boxes are sorted based on the correspondence between the text boxes and the nodes.

[0128] In this embodiment, by determining the initial text box, the starting node can be quickly located and then connected in sequence to increase the node linking speed, thereby quickly achieving the sorting of the text boxes.

[0129] In one embodiment, the visual features are visual feature vectors, and the text features are text feature vectors; for each text box, the visual features and text features corresponding to the text box are extracted, and the visual features and text features corresponding to the same text box are subjected to feature fusion processing to obtain the node features of the nodes mapped by each text box in the graph structure, including: based on the coordinate information of the target text box, extracting the visual information in the area corresponding to the coordinate information, performing mean pooling processing on the visual information, and obtaining the visual feature vector corresponding to the target text box; identifying the text in the target text box, determining the mapping number sequence corresponding to the target text box based on the mapping number corresponding to each text, performing feature embedding processing on the mapping number sequence, and obtaining the text feature vector corresponding to the target text box.

[0130] Specifically, the visual features are the result obtained by processing the image using a U-net network based on FCNs (fully connected convolutional neural networks). First, the U-net is used to process the entire target text image to identify the text box and obtain the annotation information corresponding to the text box. The annotation information consists of three parts: reading order, text box coordinate sequence, and text content in the text box. There are multiple text boxes in a target text image, and each text box has an annotation information. According to the area formed by the coordinate sequence of the text box, the feature data in the area where the text box is located is extracted, and then the extracted feature data is subjected to mean pooling operation to convert the visual features in each text box into visual feature vectors of the same length. Among them, the length of the visual feature vector can be 256 dimensions.

[0131] Text features are extracted using an RNN-based LSTM network. First, the text content in the text box is mapped using a Chinese character dictionary (such as a commonly used 3,500-character Chinese character dictionary), converting the original characters in the text box to the corresponding character numbers in the dictionary. A two-dimensional matrix is constructed based on the character mapping numbers in all text boxes in the target text image. Each row of data in this two-dimensional matrix is then fed into an LSTM network for embedding, ultimately yielding the text features corresponding to each text box. The text feature vector can be 256-dimensional.

[0132] In the above embodiment, by extracting and processing the visual features and text features accordingly, it is possible to ensure that the obtained visual feature vector and text feature vector have the same length, which facilitates feature fusion.

[0133] In one embodiment, determining the mapping sequence number sequence corresponding to the target text box based on the mapping sequence number corresponding to each character includes: determining the initial mapping sequence number sequence corresponding to the target text box based on the mapping sequence number corresponding to each character; when the sequence length of the initial mapping sequence number sequence is less than the target sequence length, filling the initial mapping sequence number sequence based on a preset sequence number to obtain the mapping sequence number sequence corresponding to the target text box; wherein the target sequence length corresponds to the number of characters in the text box containing the most characters.

[0134] In a specific application, the target text image contains 20 text boxes, among which the number of characters in the text box with the longest text content is 20. Then it can be determined that the length of the mapping sequence corresponding to each text box is 20. If the number of characters in one of the text boxes is 5, the length of the initial mapping sequence obtained through sequence mapping is 5, then the remaining 15-bit sequence numbers are padded with zeros, such as (2, 15, 42, 31, 5) is padded to (2, 15, 42, 31, 5, 0 ...

[0135] In this embodiment, by padding the initial mapping number sequence whose sequence length is less than the target sequence length, it is possible to ensure that the length of each mapping number sequence is consistent, thereby avoiding the situation where the mapping number sequence length is inconsistent and the accuracy of the text feature vector corresponding to the target text box is affected when the mapping number sequence is subsequently subjected to feature embedding processing.

[0136] In one embodiment, the text box is in the shape of a matrix. The spatial position features include position features corresponding to each text box combination, where the text box combinations are obtained by combining text boxes in the target text image in pairs. The text box sorting method for the text image further includes: for each text box combination, based on the reference point coordinates of each text box in the text box combination, determining a first position feature corresponding to the text box combination, and based on the width and height data of each text box in the text box combination, determining a second position feature corresponding to the text box combination; and performing feature splicing processing on the first position feature and the second position feature to obtain a position feature corresponding to the text box combination.

[0137] The reference point coordinates are for each text box's reference point. A reference point is a relative position within a text box. For example, the reference point can be the top left corner or the bottom right corner of the text box. This can be set based on actual needs. The width and height data specifically include the height and width of the text box. Since the text box is a matrix, determining the reference point coordinates and width and height data can determine the area corresponding to the text box within the target text image.

[0138] By combining the text boxes in the target text image in pairs, a text box combination can be obtained. For each text box combination, the reference point coordinates and width and height data are analyzed to obtain the first position features and second position features of different dimensions, and then the position features corresponding to each text box combination are obtained. By aggregating the position features corresponding to each text box combination, the spatial position features corresponding to the target text image can be obtained.

[0139] In this embodiment, by performing positional relationship analysis based on the reference point coordinates and the width and height data, the positional relationship between two text boxes in the text box combination can be characterized and the accuracy of the positional relationship can be improved.

[0140] In one embodiment, based on the reference point coordinates of each text box in the targeted text box combination, a first position feature corresponding to the targeted text box combination is determined, including: based on the reference point coordinates of each text box in the targeted text box combination, determining the horizontal distance and vertical distance between the reference points; normalizing the horizontal distance and the vertical distance and performing data splicing to obtain the first position feature.

[0141] The horizontal distance is the difference between the horizontal coordinates of the reference points of the first and second text boxes in the text box combination, and the vertical distance is the difference between the vertical coordinates of the reference points of the first and second text boxes in the text box combination. Normalizing the horizontal and vertical distances involves multiplying and dividing them by a normalization coefficient. Data concatenation is the process of concatenating the normalized horizontal and vertical distances.

[0142] In a specific application, the text box in the text box combination is t i With t j Taking the process of determining the positional relationship between x i ,y i ,x j ,y j t i With t j The coordinates of the upper left point are the reference point coordinates, and the processing process of the first position feature is as follows:

[0143] Δx ij =x j -x i

[0144] Δy ij =y j -y i

[0145]

[0146] Δx ij t i With t j Horizontal distance, Δy ij t i With t j , d is the normalization coefficient, and || is the concatenation operation.

[0147] In this embodiment, by normalizing the horizontal distance and the vertical distance respectively and then performing data splicing on the normalized results, it is possible to characterize the positional relationship between the reference points and ensure that similar data is not interfered with, thereby improving the accuracy of the positional relationship.

[0148] In one embodiment, based on the width and height data of each text box in the text box combination, a second position feature corresponding to the targeted text box combination is determined, including: selecting target data from the width and height data of each text box in the text box combination, and calculating the ratio results of unselected width and height data to the target data respectively; and performing data splicing on the ratio results to obtain the second position feature.

[0149] The text box combination consists of two text boxes, each containing height and width data. Therefore, the total number of width and height data for each text box in the text box combination is four. One of the four width and height data is selected as the target data, and the ratios of the remaining three data to the target data are calculated to obtain three ratio results. The second position feature is the concatenation of the three ratio results.

[0150] In a specific application, the text box in the text box combination is t i With t j Taking the process of determining the positional relationship between x i ,y i ,x j ,y j t i With t j The coordinates of the upper left point are the reference point coordinates, h i ,w i ,h j ,w j t i With t j The height and width of the second position feature are processed as follows:

[0151]

[0152] Among them, || is the concatenation operation.

[0153] The position feature obtained by combining the first position feature and the second position feature is:

[0154]

[0155] The position feature between each two text boxes is represented by r ij ∈R 1×5 , this vector of length 5.

[0156] In this embodiment, the positional relationship analysis is performed based on the width and height data, which can accurately characterize the size relationship between every two text boxes, thereby improving the accuracy of the positional relationship.

[0157] The present application also provides an application scenario, which can specifically be office scene document recognition. It performs structured recognition on documents in office scenes, and classifies and organizes the document content according to information such as titles, pictures, texts, and tables, and supports separating the text content and layout format in the document and outputting them in sequence.

[0158] This application scenario applies the above text box sorting method for text images. Specifically, the application of the text box sorting method for text images in this application scenario is as follows:

[0159] The text box sorting method based on multimodal text images is implemented based on the text box sorting model, which includes three networks: CNN, RNN and GNN. Figure 10 As shown, the visual features of the text regions in the image are extracted by a CNN network, while the text features in the image are extracted by an RNN network. The spatial relationship features between the text regions are also calculated. The visual and text features are first fused together, and then the fused features are used to form nodes in the GNN, while the spatial relationship features are used to form edges in the GNN. A two-layer GNN network is used to update the nodes and edges in the entire image. The edges output by the GNN are then classified into two categories: edges where two nodes are directly adjacent are classified into one category, and edges where two nodes are not adjacent are classified into another category. Ultimately, the relationship between all points is determined.

[0160] For CNN, we use FCN's U-net as an example for visual feature extraction, and for RNN, we use LSTM as an example for text feature extraction. It is understood that the options for CNN include but are not limited to U-net, and the options for RNN also include but are not limited to LSTM.

[0161] The training data and test data of each network in the text box sorting model are composed of sample text images and annotation information. Each sample text image corresponds to its own annotation information. Based on the sample text images, a text box sorting model for text box sorting is obtained.

[0162] The format of the annotation information is:

[0163]

[0164] Where filename is the image name, height and width are the image height and width, and annotations is the annotation information within the image. Annotations consists of three parts: reading_order (reading order), box (a sequence of text box coordinates, including the coordinates of the top left corner, top right corner, bottom left corner, and bottom right corner), and text (the text content within the text box). An image can contain multiple text boxes, each with an annotation.

[0165] During the training of the text box sorting model, the annotation information needs to be converted into a suitable groundtruth format, such as Figure 11 As shown in the example. By processing the sorting information in the annotation information, we can get Define the loss function:

[0166]

[0167] like Figure 12 As shown, After processing by the fully connected layer, we get CE represents the cross entropy function, which calculates the loss of the ground truth and the predicted value, and then uses the optimizer to optimize the network to obtain a text box sorting model that can sort text boxes.

[0168] The processing process of the text box sorting model mainly includes visual feature extraction, text feature extraction, spatial position feature extraction, GNN data update and edge classification.

[0169] During the visual feature extraction process, visual features are obtained by processing the target text image using a U-net network based on an FCN network. The U-net first processes the entire target text image, identifying each text box within the target text image and determining its coordinates. The visual feature data within the text box is then extracted. Mean pooling is then performed on this extracted visual feature data, converting the visual features within each text box into visual feature vectors of the same length.

[0170] During the text feature extraction process, the features are extracted using an RNN-based LSTM network. First, the text within the text box is converted using a commonly used 3,500-character Chinese dictionary, mapping the original characters in the text to the corresponding character numbers in the dictionary. The characters in all text boxes in the target text image are mapped to serial numbers, forming a two-dimensional matrix. Each row of data in the two-dimensional matrix is then fed into the LSTM network for embedding, ultimately obtaining the text features corresponding to each text box.

[0171] In the process of extracting spatial position features, spatial position features represent the spatial position relationship between text boxes. First, determine the text box t i With t j The relationship between x i ,y i ,x j ,y j t i With t j The coordinates of the upper left point, h i ,w i ,h j ,w j t i With t j height and width.

[0172] Δx ij =x j -x i

[0173] Δy ij =y j -y i

[0174]

[0175] Δx ij t i With t j Horizontal distance, Δy ij t i With t j The vertical distance between each two text boxes is represented by r ij ∈R 1×5 , which is a vector of length 5. After calculating the spatial position relationship between all text boxes in the target text image and other text boxes, we get a , n is the number of all text boxes in the target text image.

[0176] In the process of GNN data update and edge classification, the characteristic of the GNN network is that each network layer will update all nodes and edges. The output of the visual feature extraction network and the output of the text feature extraction network are input into the feature fusion network for processing to obtain the feature n of each node in the GNN. i ∈R 1×256 , all nodes in GNN correspond to R n×256 The matrix, r ij ∈R 1×5 It is the spatial location feature.

[0177] e′ ij =Normal(Er ij )

[0178] E is the embedding operation, ij Perform embedding operation. Then, the embedded r ij Then perform the Normalization operation to obtain e′ ij ∈R 1×256 . ij ∈R 1×256 ,n i ∈R 1×256 Input the first layer of GNN network, and convert n i ,e′ ij ,n j The three are connected in series to obtain the edge e in the GNN network ij ∈R 1×(256+256+256) .

[0179] e ij =n i ||e′ ij ||n j

[0180] e ij =M(e ij )

[0181] M represents MLP, after MLP network processing, e ij Convert to e ij ∈R 1×256 .

[0182] The process of node iteration in the first layer of GNN network is:

[0183]

[0184] represents the i-th node at the first time step, σ represents the relu activation function, W 1 Represents the fully connected layer of the first layer of the GNN network. Represents the normalized edge weights in the first layer of the GNN network.

[0185]

[0186] The process of node iteration in the l-th layer GNN is:

[0187]

[0188] The output after multiple layers of GNN iteration is

[0189] All Divided into two categories, Represents the relationship between nodes. In a reading context, the relationship between two text boxes can be divided into two categories: one in which the two text boxes are adjacent in the reading order, and one in which the two text boxes are non-adjacent. A node, or text box (not the initial or final node), has a parent node and a parent node in the reading order. We can calculate the probability of a node being directly connected to other nodes, and for intermediate nodes, we find the two nodes with the highest probability. Once we find a starting node, we can link all nodes together in order, and the result is the correct reading order. The difference between a starting node and an intermediate node is that for an intermediate node, we find the two largest nodes adjacent to it, while for a starting node, we only find the largest node adjacent to it. Observing extensive data reveals that, according to human reading order, the starting node is often at the top and leftmost of an image. Therefore, based on the coordinates of each text box in the target text image, we determine the text box with the coordinate data corresponding to the top and leftmost as the starting text box in the target text image. Then, we link all nodes together in order, and sort the text boxes based on the correspondence between text boxes and nodes.

[0190] Through the above processing process, it can provide effective auxiliary functions for layout analysis in OCR, improve the accuracy of OCR recognition services, and be applied to OCR layout analysis algorithms and OCR image content recognition algorithms to sort discrete text information in images according to human reading order for more efficient reprocessing.

[0191] It should be understood that, although the various steps in the flowcharts of the above-mentioned embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily performed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts of the above-mentioned embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0192] In one embodiment, Figure 13 As shown, a text box sorting device 1300 for a text image is provided. The device can be implemented as a software module or a hardware module, or a combination of both to form a part of a computer device. The device specifically includes: a first feature extraction module 1302, a second feature extraction module 1304, a feature update module 1306, and a text box sorting module 1308, wherein:

[0193] A first feature extraction module 1302 is configured to determine a spatial position feature corresponding to the target text image based on text frames contained in the target text image, wherein the spatial position feature is used to characterize the positional relationship between the text frames in the target text image;

[0194] The second feature extraction module 1304 is used to extract the visual features and text features corresponding to each text box, perform feature fusion processing on the visual features and text features corresponding to the same text box, and obtain the node features of the node mapped by each text box in the graph structure;

[0195] A feature updating module 1306 is configured to update the initial node-edge features between the nodes determined based on the spatial position features based on the node features of the nodes in the graph structure, to obtain updated node-edge features;

[0196] The text box sorting module 1308 is configured to sort the text boxes represented by each node in the graph structure based on the node connection relationship determined by the updated node edge feature.

[0197] In one embodiment, the feature update module includes an initial node-edge feature matrix determination module and a first update module, wherein:

[0198] The initial node-edge feature matrix determination module is used to perform feature embedding and normalization processing on the spatial position features to obtain an initial node-edge feature matrix;

[0199] The first updating module is configured to update the initial node-edge feature matrix based on an initial node feature matrix composed of node features corresponding to each node in the graph structure to obtain an updated node-edge feature matrix.

[0200] In one embodiment, the first updating module includes a matrix splicing module and a multi-layer perception module, wherein:

[0201] The matrix splicing module is used to perform matrix splicing processing on the initial node feature matrix and the initial node edge feature matrix to obtain a spliced node edge feature matrix;

[0202] The multi-layer perception module is used to perform multi-layer perception processing on the splicing node edge feature matrix to obtain updated node edge features.

[0203] In one embodiment, the initial node feature matrix is a two-dimensional matrix; the matrix splicing module includes a matrix expansion unit, a normalization unit, and a splicing unit, wherein:

[0204] The matrix expansion unit is used to perform matrix expansion of different dimensions on the initial node feature matrix to obtain a first three-dimensional matrix and a second three-dimensional matrix;

[0205] The normalization unit is used to perform normalization processing on the spliced three-dimensional matrix formed by the first three-dimensional matrix and the second three-dimensional matrix to obtain a spliced node feature matrix;

[0206] The splicing unit is used to perform matrix splicing processing on the splicing node feature matrix and the initial node edge feature matrix to obtain a splicing node edge feature matrix.

[0207] In one embodiment, the first update module includes an input module and a processing and output module, wherein:

[0208] The input module is used to input the initial node feature matrix composed of the node features corresponding to each node and the initial node edge feature matrix into the graph neural network;

[0209] The processing and output module is used to update the initial node edge feature matrix based on the initial node feature matrix in the graph neural network, obtain an updated node edge feature matrix and output it.

[0210] In one embodiment, the graph neural network comprises at least two graph neural networks connected in sequence to form a graph neural network group, wherein the output data of the previous graph neural network in the graph neural network group serves as the input data of the next graph neural network; the text frame sorting device for text images further comprises a second updating module;

[0211] The second updating module is used to update the initial node feature matrix in the graph neural network based on the updated node edge features, obtain an updated node feature matrix and output it.

[0212] In one embodiment, the text frame sorting device for a text image further includes a node feature updating module and a secondary updating module, wherein:

[0213] The node feature updating module is used to update the node feature based on the updated node edge feature to obtain an updated node feature;

[0214] The secondary updating module is configured to perform feature splicing processing on the updated node features and the updated node edge features, and perform multi-layer perception processing on the obtained splicing features to obtain secondary updated node edge features;

[0215] The text box sorting module is further configured to sort the text boxes represented by each node in the graph structure based on the node connection relationship determined by the second updated node edge feature.

[0216] In one embodiment, the text box sorting module includes a screening unit, a connection object determination unit, and a sorting unit, wherein:

[0217] The screening unit is used to perform binary classification processing on each node edge feature in the updated node edge feature matrix, and screen out the target node edge feature corresponding to each node, and the classification probability corresponding to the target node edge feature meets the probability condition;

[0218] The connection object determination unit is configured to determine a connection object corresponding to each node based on the node connection relationship determined by the edge feature of the target node;

[0219] The sorting unit is used to sort the text boxes in the target text image based on the connection object corresponding to each node.

[0220] In one embodiment, the sorting unit is also used to determine the initial text box in the target text image based on the coordinates of each text box in the target text image; and sort the text boxes in the target text image based on the nodes corresponding to the initial text box and the connection objects corresponding to each node.

[0221] In one embodiment, the visual feature is a visual feature vector, and the text feature is a text feature vector;

[0222] The second feature extraction module includes a visual feature vector extraction module and a text feature vector extraction module, wherein:

[0223] The visual feature vector extraction module is used to extract visual information within an area corresponding to the coordinate information of the target text box based on the coordinate information of the target text box, and perform mean pooling processing on the visual information to obtain a visual feature vector corresponding to the target text box;

[0224] The text feature vector extraction module is used to identify the text in the target text box, determine the mapping number sequence corresponding to the target text box based on the mapping number corresponding to each text, perform feature embedding processing on the mapping number sequence, and obtain the text feature vector corresponding to the target text box.

[0225] In one embodiment, the text feature vector extraction module is further used to determine an initial mapping sequence number sequence corresponding to a target text box based on the mapping sequence number corresponding to each character; when the sequence length of the initial mapping sequence number sequence is less than the target sequence length, the initial mapping sequence number sequence is padded based on a preset sequence number to obtain a mapping sequence number sequence corresponding to the target text box; wherein the target sequence length corresponds to the number of characters in the text box containing the most characters.

[0226] In one embodiment, the spatial position features include position features corresponding to each text box combination, and the text box combinations are obtained by combining two text boxes in the target text image;

[0227] The text frame sorting device for text images further includes a position feature determination module and a position feature splicing module, wherein:

[0228] The position feature determination module is configured to determine, for each text box combination, a first position feature corresponding to the text box combination based on the reference point coordinates of each text box in the text box combination, and determine a second position feature corresponding to the text box combination based on the width and height data of each text box in the text box combination;

[0229] The position feature splicing module is used to perform feature splicing processing on the first position feature and the second position feature to obtain the position feature corresponding to the targeted text box combination.

[0230] In one embodiment, the position feature determination module is also used to determine the horizontal distance and vertical distance between reference points based on the reference point coordinates of each text box in the targeted text box combination; the horizontal distance and the vertical distance are normalized and data spliced to obtain the first position feature.

[0231] In one embodiment, the position feature determination module is further used to select target data from the width and height data of each text box in the text box combination, and calculate the ratio results of unselected width and height data to the target data respectively; and perform data splicing on the ratio results to obtain a second position feature.

[0232] Specific embodiments of the text box sorting device for text images can be found in the embodiments of the text box sorting method for text images described above and will not be repeated here. Each module in the aforementioned text box sorting device for text images can be implemented in whole or in part via software, hardware, or a combination thereof. Each of these modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0233] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 14 As shown. The computer device includes a processor, a memory and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store text box sorting data of text images. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a text box sorting method for text images is implemented.

[0234] Those skilled in the art will understand that Figure 14 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0235] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0236] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above-mentioned method embodiments when executed by a processor.

[0237] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of each of the above-described method embodiments.

[0238] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0239] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0240] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A text frame sorting method for a text image, characterized in that: The method comprises: Determining, based on text boxes contained in a target text image, a spatial position feature corresponding to the target text image, wherein the spatial position feature is used to characterize a positional relationship between text boxes in the target text image; For each text box, extract the visual features and text features corresponding to the text box, perform feature fusion processing on the visual features and text features corresponding to the same text box, and obtain the node features of the node mapped by each text box in the graph structure; Based on the node features of each node in the graph structure, feature updating is performed on the initial node edge features between the nodes determined based on the spatial position features to obtain updated node edge features; Performing binary classification processing on each node edge feature in the updated node edge feature, filtering out the target node edge feature corresponding to each node, and the classification probability corresponding to the target node edge feature meets the probability condition; Determine the connection object corresponding to each node based on the node connection relationship determined by the edge feature of the target node; The text boxes in the target text image are sorted based on the connection objects corresponding to each node.

2. The method according to claim 1, characterized in that The updating of the initial node edge features between the nodes determined based on the spatial position features based on the node features of each node in the graph structure to obtain updated node edge features includes: Performing feature embedding and normalization processing on the spatial position features to obtain an initial node-edge feature matrix; Based on an initial node feature matrix composed of node features corresponding to each node in the graph structure, the initial node edge feature matrix is updated to obtain an updated node edge feature matrix.

3. The method according to claim 2, characterized in that The initial node feature matrix formed based on the node features corresponding to each node in the graph structure is updated to obtain an updated node edge feature matrix, including: Performing matrix splicing processing on the initial node feature matrix and the initial node edge feature matrix to obtain a spliced node edge feature matrix; Multi-layer perception processing is performed on the splicing node edge feature matrix to obtain updated node edge features.

4. The method according to claim 3, characterized in that The initial node feature matrix is a two-dimensional matrix; The matrix splicing process of the initial node feature matrix and the initial node edge feature matrix to obtain a spliced node edge feature matrix includes: Expand the initial node feature matrix into different dimensions to obtain a first three-dimensional matrix and a second three-dimensional matrix; Normalizing a spliced three-dimensional matrix formed by the first three-dimensional matrix and the second three-dimensional matrix to obtain a spliced node feature matrix; The splicing node feature matrix and the initial node edge feature matrix are subjected to matrix splicing processing to obtain a splicing node edge feature matrix.

5. The method according to claim 2, characterized in that The initial node feature matrix formed based on the node features corresponding to each node in the graph structure is updated to obtain an updated node edge feature matrix, including: Inputting the initial node feature matrix composed of the node features corresponding to each node and the initial node edge feature matrix into the graph neural network; In the graph neural network, the initial node edge feature matrix is updated based on the initial node feature matrix to obtain an updated node edge feature matrix and output it.

6. The method according to claim 5, characterized in that The graph neural network is a graph neural network group formed by sequentially connecting at least two graph neural networks, wherein the output data of the previous graph neural network in the graph neural network group serves as the input data of the next graph neural network; The method further comprises: In the graph neural network, the initial node feature matrix is updated based on the updated node edge features to obtain and output an updated node feature matrix.

7. The method according to claim 1, characterized in that The updating of the node-edge feature is a secondary updating of the node-edge feature; the method further includes: Based on the updated node edge feature, the node feature is updated to obtain an updated node feature; The updated node features and the updated node edge features are subjected to feature splicing processing, and the obtained splicing features are subjected to multi-layer perception processing to obtain secondary updated node edge features.

8. The method according to claim 1, characterized in that The sorting of text boxes in the target text image based on the connection object corresponding to each node includes: Determining an initial text frame in the target text image based on the coordinates of each text frame in the target text image; The text boxes in the target text image are sorted based on the nodes corresponding to the initial text boxes and the connection objects corresponding to each node.

9. The method according to any one of claims 1 to 8, characterized in that The visual feature is a visual feature vector, and the text feature is a text feature vector; The step of extracting visual features and text features corresponding to each text box includes: Based on the coordinate information of the target text box, extracting visual information within the area corresponding to the coordinate information, performing mean pooling processing on the visual information, and obtaining a visual feature vector corresponding to the target text box; Identify the characters in the target text box, determine the mapping number sequence corresponding to the target text box based on the mapping number corresponding to each character, perform feature embedding processing on the mapping number sequence, and obtain the text feature vector corresponding to the target text box.

10. The method according to claim 9, characterized in that The determining of the mapping sequence corresponding to the target text box based on the mapping sequence corresponding to each character includes: Based on the mapping sequence number corresponding to each character, determine the initial mapping sequence number corresponding to the target text box; When the sequence length of the initial mapping sequence number is less than the target sequence length, the initial mapping sequence number is filled based on the preset sequence number to obtain a mapping sequence number sequence corresponding to the target text box; The target sequence length corresponds to the number of characters in the text box containing the most characters.

11. The method according to any one of claims 1 to 8, characterized in that The spatial position features include position features corresponding to each text box combination, and the text box combination is obtained by combining two text boxes in the target text image; The method further comprises: For each text box combination, based on the reference point coordinates of each text box in the text box combination, determining a first position feature corresponding to the text box combination; and based on the width and height data of each text box in the text box combination, determining a second position feature corresponding to the text box combination; The first position feature and the second position feature are subjected to feature splicing processing to obtain a position feature corresponding to the targeted text box combination.

12. The method according to claim 11, characterized in that The determining, based on the reference point coordinates of each text box in the targeted text box combination, a first position feature corresponding to the targeted text box combination includes: Determine the horizontal distance and vertical distance between the reference points based on the reference point coordinates of each text box in the targeted text box combination; The horizontal distance and the vertical distance are normalized and data are spliced to obtain a first position feature.

13. The method according to claim 11, characterized in that The determining, based on the width and height data of each text box in the text box combination, a second position feature corresponding to the text box combination includes: Selecting target data from the width and height data of each text box in the text box combination, and calculating the ratio of the unselected width and height data to the target data; The ratio results are subjected to data splicing to obtain a second position feature.

14. A text frame sorting device for a text image, characterized in that: The device comprises: A first feature extraction module is configured to determine a spatial position feature corresponding to the target text image based on text frames contained in the target text image, wherein the spatial position feature is used to characterize a positional relationship between text frames in the target text image; The second feature extraction module is used to extract the visual features and text features corresponding to each text box, perform feature fusion processing on the visual features and text features corresponding to the same text box, and obtain the node features of the node mapped by each text box in the graph structure; A feature updating module, configured to update the initial node edge features between the nodes determined based on the spatial position features based on the node features of each node in the graph structure, to obtain updated node edge features; A text box sorting module, configured to sort the text boxes represented by each node in the graph structure based on the node connection relationship determined by the updated node edge feature; The text box sorting module includes a screening unit, a connection object determination unit, and a sorting unit, wherein: The screening unit is used to perform binary classification processing on each node edge feature in the updated node edge feature matrix, and screen out the target node edge feature corresponding to each node, and the classification probability corresponding to the target node edge feature meets the probability condition; The connection object determination unit is configured to determine a connection object corresponding to each node based on the node connection relationship determined by the edge feature of the target node; The sorting unit is used to sort the text boxes in the target text image based on the connection object corresponding to each node.

15. The device according to claim 14, characterized in that The feature update module includes an initial node-edge feature matrix determination module and a first update module, wherein: The initial node-edge feature matrix determination module is used to perform feature embedding and normalization processing on the spatial position features to obtain an initial node-edge feature matrix; The first updating module is configured to update the initial node-edge feature matrix based on an initial node feature matrix composed of node features corresponding to each node in the graph structure to obtain an updated node-edge feature matrix.

16. The device according to claim 15, characterized in that The first updating module includes a matrix splicing module and a multi-layer perception module, wherein: The matrix splicing module is used to perform matrix splicing processing on the initial node feature matrix and the initial node edge feature matrix to obtain a spliced node edge feature matrix; The multi-layer perception module is used to perform multi-layer perception processing on the splicing node edge feature matrix to obtain updated node edge features.

17. The device according to claim 16, characterized in that The initial node feature matrix is a two-dimensional matrix; the matrix splicing module includes a matrix expansion unit, a normalization unit, and a splicing unit, wherein: The matrix expansion unit is used to perform matrix expansion of different dimensions on the initial node feature matrix to obtain a first three-dimensional matrix and a second three-dimensional matrix; The normalization unit is used to perform normalization processing on the spliced three-dimensional matrix formed by the first three-dimensional matrix and the second three-dimensional matrix to obtain a spliced node feature matrix; The splicing unit is used to perform matrix splicing processing on the splicing node feature matrix and the initial node edge feature matrix to obtain a splicing node edge feature matrix.

18. The device according to claim 15, characterized in that The first update module includes an input module and a processing and output module, wherein: The input module is used to input the initial node feature matrix composed of the node features corresponding to each node and the initial node edge feature matrix into the graph neural network; The processing and output module is used to update the initial node edge feature matrix based on the initial node feature matrix in the graph neural network, obtain an updated node edge feature matrix and output it.

19. The device according to claim 18, characterized in that The graph neural network comprises at least two graph neural networks connected in sequence to form a graph neural network group, wherein the output data of the previous graph neural network in the graph neural network group serves as the input data of the next graph neural network; the text frame sorting device for text images further comprises a second updating module; The second updating module is used to update the initial node feature matrix in the graph neural network based on the updated node edge features, obtain an updated node feature matrix and output it.

20. The device according to claim 14, wherein The updating node edge feature is a secondary updating of the node edge feature; the text frame sorting device for text images further includes a node feature updating module and a secondary updating module, wherein: The node feature updating module is used to update the node feature based on the updated node edge feature to obtain an updated node feature; The secondary updating module is used to perform feature splicing processing on the updated node features and the updated node edge features, and perform multi-layer perception processing on the obtained splicing features to obtain secondary updated node edge features.

21. The device according to claim 14, characterized in that The sorting unit is further configured to determine an initial text box in the target text image based on the coordinates of each text box in the target text image; and sort the text boxes in the target text image based on the nodes corresponding to the initial text box and the connection objects corresponding to each node.

22. The device according to any one of claims 14 to 21, characterized in that The visual feature is a visual feature vector, and the text feature is a text feature vector; The second feature extraction module includes a visual feature vector extraction module and a text feature vector extraction module, wherein: The visual feature vector extraction module is used to extract visual information within an area corresponding to the coordinate information of the target text box based on the coordinate information of the target text box, and perform mean pooling processing on the visual information to obtain a visual feature vector corresponding to the target text box; The text feature vector extraction module is used to identify the text in the target text box, determine the mapping number sequence corresponding to the target text box based on the mapping number corresponding to each text, perform feature embedding processing on the mapping number sequence, and obtain the text feature vector corresponding to the target text box.

23. The device according to claim 22, characterized in that The text feature vector extraction module is further used to determine an initial mapping sequence number sequence corresponding to a target text box based on the mapping sequence number corresponding to each character; when the sequence length of the initial mapping sequence number sequence is less than the target sequence length, the initial mapping sequence number sequence is padded based on a preset sequence number to obtain a mapping sequence number sequence corresponding to the target text box; wherein the target sequence length corresponds to the number of characters in the text box containing the most characters.

24. The device according to any one of claims 14 to 21, characterized in that The spatial position features include position features corresponding to each text box combination, and the text box combination is obtained by combining two text boxes in the target text image; The text frame sorting device for a text image further includes a position feature determination module and a position feature splicing module, wherein: The position feature determination module is configured to determine, for each text box combination, a first position feature corresponding to the text box combination based on the reference point coordinates of each text box in the text box combination, and determine a second position feature corresponding to the text box combination based on the width and height data of each text box in the text box combination; The position feature splicing module is used to perform feature splicing processing on the first position feature and the second position feature to obtain the position feature corresponding to the targeted text box combination.

25. The device according to claim 23, characterized in that The position feature determination module is also used to determine the horizontal distance and vertical distance between reference points based on the reference point coordinates of each text box in the targeted text box combination; normalize the horizontal distance and the vertical distance and perform data splicing to obtain the first position feature.

26. The device according to claim 23, characterized in that The position feature determination module is further configured to select target data from the width and height data of each text box in the text box combination, and calculate the ratio results of the unselected width and height data to the target data; and perform data splicing on the ratio results to obtain a second position feature.

27. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 13 are implemented.

28. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

29. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

Images