Text content processing methods, apparatus, computer equipment and storage media

Abstract

This application relates to a text content processing method, apparatus, computer device, and storage medium, belonging to the field of artificial intelligence technology. The method includes: acquiring individual character blocks in a target text object, where a character block is a region within the target text object containing continuous text content; acquiring a first feature for each character block, the first feature including image features, positional features, and semantic features of the corresponding character block; acquiring a feature matrix based on the first features of each character block; acquiring an adjacency matrix for each character block based on the feature matrix, the adjacency matrix indicating the order of the character blocks; and sorting and combining the text content corresponding to each character block based on the adjacency matrix to obtain the text processing result of the target text object. This method processes text content based on a multimodal global perspective, thereby improving the accuracy of text word order restoration.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a text content processing method, apparatus, computer device, and storage medium. Background Technology

[0002] Today, with the increasing development of artificial intelligence, AI technology is being applied more and more widely in life, including restoring the text order extracted from images.

[0003] In related technologies, text boxes corresponding to text in an image are identified, their coordinates are determined, and cluster analysis is performed on them. First, the text boxes are clustered according to their y-coordinates to identify text boxes belonging to the same row. Then, the text boxes in the same row are sorted according to their x-coordinates to output the order of the text boxes.

[0004] However, the above-mentioned method for restoring text order only determines the order of text blocks by the coordinates of the text boxes corresponding to each text block. Therefore, when the text layout is not completely regular, it is impossible to accurately determine the order between text blocks, thus affecting the accuracy of text order restoration. Summary of the Invention

[0005] This application provides a text content processing method, apparatus, computer device, and storage medium. The technical solution is as follows:

[0006] On the one hand, a text content processing method is provided, the method comprising:

[0007] Obtain individual character blocks from a target text object, where each character block is a region in the target text object containing continuous text content;

[0008] Obtain the first feature of each character block, wherein the first feature includes the image feature, position feature and semantic feature of the corresponding character block;

[0009] A feature matrix is ​​obtained based on the first feature of each of the character blocks; the feature matrix contains a second feature obtained by fusing the first features of every two character blocks in each character block;

[0010] Based on the feature matrix, the adjacency matrix of each character block is obtained, and the adjacency matrix is ​​used to indicate the order between the character blocks;

[0011] Based on the adjacency matrix, the text content corresponding to each character block is sorted and combined to obtain the text processing result of the target text object.

[0012] On one hand, a text content processing apparatus is provided, the apparatus comprising:

[0013] The character block acquisition module is used to acquire each character block in the target text object, wherein the character block is a region in the target text object that contains continuous text content;

[0014] The first feature acquisition module is used to acquire the first feature of each character block, wherein the first feature includes the image feature, position feature and semantic feature of the corresponding character block;

[0015] The matrix acquisition module is used to acquire a feature matrix based on the first feature of each of the character blocks; the feature matrix contains a second feature obtained by fusing the first features of every two character blocks in each character block;

[0016] An adjacency matrix acquisition module is used to acquire the adjacency matrix of each character block based on the feature matrix, wherein the adjacency matrix is ​​used to indicate the order between the character blocks;

[0017] The result acquisition module is used to sort and combine the text content corresponding to each character block based on the adjacency matrix to obtain the text processing result of the target text object.

[0018] In one possible implementation, the first feature acquisition module includes:

[0019] The splicing feature acquisition submodule is used to acquire the splicing features of each character block. The splicing features include the features obtained by splicing the image features, position features and semantic features of the corresponding character block.

[0020] The first feature acquisition submodule is used to acquire the first feature of each character block based on the splicing features of each character block.

[0021] In one possible implementation, the first feature acquisition submodule includes:

[0022] The similarity acquisition unit is used to acquire the similarity between the splicing features corresponding to every two character blocks in each character block;

[0023] The graph construction unit is used to construct graph data based on the splicing features of each character block and the similarity between the splicing features of each pair of character blocks.

[0024] The neighbor feature acquisition unit is used to perform neighbor fusion on the splicing features of each character block based on the graph data to obtain the neighbor fusion features corresponding to each character block.

[0025] The first feature acquisition unit is used to acquire the first feature of each character block based on the corresponding neighbor fusion feature of each character block.

[0026] In one possible implementation, the proximity feature acquisition unit is used for:

[0027] Based on the graph data, obtain K neighboring character blocks of the target character block; the neighboring character blocks are K character blocks whose similarity between their corresponding splicing features and the splicing features of the target character block satisfies the adjacent condition; the target character block is any one of the character blocks; K is a positive integer;

[0028] The splicing features of the target character block are fused with the splicing features of K adjacent character blocks to obtain the neighbor fusion features of the target character block.

[0029] In one possible implementation, the first feature acquisition unit is used to,

[0030] By using a fully connected network, the corresponding neighbor fusion features of each character block are aggregated to obtain the first feature of each character block.

[0031] In one possible implementation, the result acquisition module includes:

[0032] The sequence acquisition submodule is used to obtain the target sequence by traversing the adjacency matrix; the target sequence includes the arrangement order of the various word blocks;

[0033] The result acquisition submodule is used to arrange and combine the text content corresponding to each character block based on the arrangement order of each character block, so as to obtain the text processing result of the target text object.

[0034] In one possible implementation, the adjacency matrix acquisition module includes:

[0035] The acquisition submodule is used to input the feature matrix into the classification network to obtain the adjacency matrix of each character block.

[0036] On the other hand, embodiments of this application provide a computer device including a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to implement the text content processing method as described above.

[0037] On the other hand, embodiments of this application provide a computer-readable storage medium storing at least one computer program, which is loaded and executed by a processor to implement the text content processing method as described above.

[0038] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A terminal's processor reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the terminal to perform the text content processing methods provided in various alternative implementations of the above aspect.

[0039] The technical solution provided in this application may include the following beneficial effects:

[0040] In the scheme shown in the embodiments of this application, based on the multimodal features corresponding to each extracted character block, a first feature corresponding to each character block is obtained. By fusing the first features of every two character blocks in each character block, a second feature is generated, and a feature matrix containing each second feature is formed. Then, based on the feature matrix, an adjacency matrix for indicating the order between character blocks is determined. This provides a global consideration for determining the order, avoiding the problem that when the character blocks in the target text object are irregularly distributed, there may be a character block and multiple character blocks that may have a different order, making it impossible to determine whether the character block is actually in front of or behind the character block. The text content can be processed based on a global consideration under multimodal conditions, thereby improving the accuracy of text word order restoration.

[0041] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0042] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0043] Figure 1 This is a flowchart illustrating a text content processing method according to an exemplary embodiment;

[0044] Figure 2 This is a diagram illustrating a model training and result prediction framework according to an exemplary embodiment.

[0045] Figure 3 This is a flowchart illustrating a text content processing method according to an exemplary embodiment;

[0046] Figure 4 yes Figure 3 The illustrated embodiment is a framework diagram of a feature splicing process;

[0047] Figure 5 yes Figure 3The illustrated embodiment relates to a framework diagram of a feature fusion process;

[0048] Figure 6 This is a schematic diagram illustrating text content processing in a medical record according to an exemplary embodiment;

[0049] Figure 7 yes Figure 6 The illustrated embodiment is a schematic diagram of a target medical record;

[0050] Figure 8 This is a block diagram illustrating a text content processing apparatus according to an exemplary embodiment;

[0051] Figure 9 This is a structural block diagram of a computer device according to an exemplary embodiment;

[0052] Figure 10 This is a structural block diagram of a computer device according to an exemplary embodiment. Detailed Implementation

[0053] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0054] It should be understood that "multiple" as used in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0055] This application provides a text content processing method that can improve the accuracy of text content sorting.

[0056] Figure 1 This is a flowchart illustrating a text content processing method according to an exemplary embodiment. This text content processing method can be executed by a computer device. For example, the computer device may include at least one of a terminal or a server. Figure 1 As shown, the text content processing method includes the following steps:

[0057] Step 101: Obtain each character block in the target text object. A character block is a region in the target text object that contains continuous text content.

[0058] In this embodiment of the application, each character block is obtained from the target text object.

[0059] The target text object can be an image or text content comprising at least two character blocks. A character block can be a region within the target text object containing continuous text content.

[0060] For example, the target text object can be a medical record text image generated on a computer after scanning the medical record text, or it can be image content containing at least two text blocks, or it can be a webpage containing at least two text blocks, or it can be a page of a document.

[0061] In one possible implementation, if the target text object is a text document, the individual character blocks can be directly obtained, or the text document can be converted to an image format and the individual character blocks can be extracted. If the target text object is an image object, text detection can be performed on the image and the individual character blocks can be extracted.

[0062] Step 102: Obtain the first feature of each character block. The first feature includes the image feature, position feature and semantic feature of the corresponding character block.

[0063] In this embodiment of the application, in response to obtaining each character block, feature extraction is performed on each character block to obtain the image features, position features and semantic features corresponding to each character block. Based on the obtained image features, position features and semantic features, a first feature corresponding to each character block can be generated.

[0064] In one possible implementation, the image features corresponding to each character block include features obtained by extracting image features from the center position of the character block, positional features include vertex position features of the text detection box containing the character block generated when performing text detection on the character block, and semantic features include features obtained after performing text recognition on the text content in the character block.

[0065] The process of feature extraction from each character block falls under the category of artificial intelligence (AI). AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine capable of reacting in a manner similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities. AI technology is a comprehensive discipline involving a wide range of fields, encompassing both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly include computer vision, speech processing, natural language processing, and machine learning / deep learning. The display device containing image acquisition components shown in this application primarily involves computer vision and machine learning / deep learning technologies.

[0066] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instruction-based learning.

[0067] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, and map building.

[0068] Step 103: Obtain the feature matrix based on the first feature of each character block; the feature matrix contains the second feature obtained by fusing the first features of every two character blocks in each character block.

[0069] In this embodiment of the application, by performing feature fusion on the first feature corresponding to each character block, a second feature can be obtained after fusing the first features of every two character blocks.

[0070] Step 104: Based on the feature matrix, obtain the adjacency matrix of each character block. The adjacency matrix is ​​used to indicate the order between each character block.

[0071] In this embodiment of the application, an adjacency matrix is ​​obtained to indicate the order between any two character blocks based on the second feature obtained by fusing the first feature of each pair of character blocks in the feature matrix.

[0072] Step 105: Sort and combine the text content corresponding to each character block based on the adjacency matrix to obtain the text processing result of the target text object.

[0073] In this embodiment of the application, based on the sequential arrangement relationship between any two character blocks contained in the adjacency matrix, the text content corresponding to each character block is arranged and combined to obtain the text processing result of the target text object.

[0074] In one possible implementation, the text processing result of the target text object is the sorted text content generated by arranging and combining individual character blocks.

[0075] The resulting text processing can be output as text or displayed directly on the target text object.

[0076] In summary, in the scheme shown in the embodiments of this application, based on the multimodal features corresponding to each extracted character block, a first feature corresponding to each character block is obtained. By fusing the first features of every two character blocks in each character block, a second feature is generated, and a feature matrix containing each second feature is formed. Then, based on the feature matrix, an adjacency matrix for indicating the order between character blocks is determined. This provides a global consideration for determining the order, avoiding the problem that when the character blocks in the target text object are irregularly distributed, there may be a character block and multiple character blocks that may have different order, making it impossible to determine whether the character block is actually in front of or behind the character block. The text content can be processed based on a global consideration under multimodal conditions, thereby improving the accuracy of text word order restoration.

[0077] The above embodiments can be applied in the medical field to restore and sort the text in blocks of medical medical record images or texts, thereby arranging the various blocks in the medical record into a complete text content that conforms to the word order for output. At the same time, the processed text content can also be displayed for easy viewing by users.

[0078] It can also be applied to document layout analysis, where it can perform text restoration and sorting on documents with complex layouts, outputting the sorted text content of the document, thus facilitating the analysis of text documents.

[0079] Alternatively, it can be applied to the process of text information extraction, where a target text object is extracted into blocks, and the text content within the blocks is identified and sorted. The results can then be output to other applications on the terminal for analysis of the extracted and sorted text content.

[0080] In the scheme described in the above embodiments of this application, feature extraction is performed on each character block to obtain multimodal features of each character block. These multimodal features are then concatenated to generate concatenated features corresponding to each character block. These concatenated features are input into a feature fusion network. The feature fusion network obtains adjacent character blocks corresponding to each character block and accumulates the concatenated features of each adjacent character block onto the concatenated features of the corresponding character block, extracting neighbor fusion features corresponding to each character block. These neighbor fusion features are then aggregated to obtain a first feature corresponding to each character block. The first features of every two character blocks are then fused to obtain a second feature corresponding to every two character blocks. A feature matrix corresponding to each character block is generated based on this second feature. This feature matrix determines the adjacency matrix of each character block, indicating the order between every two character blocks. Finally, the character blocks are sorted to obtain the text processing result of the target text object, thereby improving the accuracy of the sorting between character blocks. In an exemplary scheme, the text content processing system involved in the above embodiments of this application includes a model training and update stage and a result prediction stage. Figure 2 This is a diagram illustrating a model training and result prediction framework according to an exemplary embodiment. For example... Figure 2 As shown, during the model training and update phase, the model training device 210 updates the word order restoration model, which includes the feature extraction network, feature fusion network, and classification network, using each set of sample word blocks. The updated word order restoration model can be uploaded to the cloud or a database for use. During the result prediction phase, the prediction device 220 directly restores the word order between the text content in each word block based on the trained word order restoration model and the input word blocks.

[0081] The model training device 210 and prediction device 220 can be computer devices with machine learning capabilities. For example, the computer device can be a personal computer, server, or fixed scientific research equipment, or it can be a mobile computer device such as a tablet computer or e-book reader. This application embodiment does not limit the specific type of the model training device 210.

[0082] Optionally, the model training device 210 and the prediction device 220 can be the same device, or they can be different devices. Furthermore, when the model training device 210 and the prediction device 220 are different devices, they can be of the same type, such as both being personal computers; or they can be of different types, such as the model training device 210 being a server, and the prediction device 220 being a fixed medical device or similar equipment. This application does not limit the specific types of the model training device 210 and the prediction device 220.

[0083] Figure 3 This is a flowchart illustrating a text content processing method according to an exemplary embodiment. The text content processing method can be executed by a computer device. For example, the computer device may be... Figure 2 The model training device in the text can also be a prediction device, or it can be both a model training device and a prediction device. For example... Figure 3 As shown, the text content processing method includes the following steps:

[0084] Step 301: Obtain each character block in the target text object.

[0085] In this embodiment of the application, the computer device acquires individual character blocks in the target text object.

[0086] In one possible implementation, the computer device obtains individual character blocks from the target text object by performing text detection on the target text object.

[0087] In this context, a character block is a region within a target text object that contains continuous text content. Text detection can detect the positional information of each character block on an image or on a document, and it can also detect the region image corresponding to the area where each character block is located on the image or on the document.

[0088] For example, the region image can be a region image generated by performing text detection, which surrounds the individual word blocks.

[0089] Step 302: Obtain the splicing features of each character block.

[0090] In this embodiment of the application, the computer device acquires the position information corresponding to each character block obtained based on text detection and the region image corresponding to each character block, and performs image feature extraction, position feature extraction and semantic feature extraction on each character block respectively.

[0091] Among them, the splicing features can include the features obtained by splicing the image features, positional features and semantic features of the corresponding character blocks.

[0092] In one possible implementation, features are extracted from each character block using a feature extraction network.

[0093] The feature extraction network can include convolutional neural networks, positional embeddings, and word embeddings. Convolutional neural networks are used for image feature extraction, positional embeddings are used to determine the positional features of word blocks, and word embeddings are used for semantic feature extraction. In the feature extraction network, the image features, positional features, and semantic features corresponding to each word block can be concatenated to generate concatenated features, and the feature extraction network outputs the concatenated features corresponding to each word block.

[0094] In one possible implementation, the convolutional neural network includes a cropped residual module. The convolutional neural network is used to obtain image features corresponding to the center position of an image region, which are then used as the image features corresponding to the character blocks.

[0095] For example, the region images corresponding to each character block can be input into a convolutional neural network, and then the resolution of the region images can be scaled up to 512. 512, then the normalized image corresponding to this region, as well as the scaling factor scale_x in the x-direction and the scaling factor scale_y in the y-direction, are obtained. Then, the normalized image is fed into the cropped ResNet residual module to obtain a resolution of 128. 128. Feature map with 64 channels. If the image region is a rectangular text detection box, then the coordinates of the four vertices of the image region, i.e. (x1, y1, w1, h1), are calculated according to (128 / 512). scale_x and (128 / 512) The scale is applied by `scale_y` to obtain the coordinates of the text detection boxes corresponding to each character block on the lowest-level feature map, i.e., (x1', y1', w1', h1'). Finally, the image features at the position (x1'+w1' / 2, y1'+ h1' / 2) in the lowest-level feature map, i.e., the features of the center point of the text detection box, are taken as image features, thus obtaining an image feature set containing the image features corresponding to each character block, and the feature dimension of the image features corresponding to each character block can be 64.

[0096] In one possible implementation, the positional features of each character block are determined by using the positional information of each character block obtained from the text detection output.

[0097] For example, when the position information corresponding to each character block is the position information of the text detection box corresponding to each character block, if the text detection box is a rectangle, then the coordinate information of the vertex corresponding to the text detection box is used as the position information corresponding to the character block.

[0098] For example, if the coordinate information of the vertex corresponding to the text detection box is (x1, y1, w1, h1), then (x1, y1, w1, h1) can be used as the position feature vector corresponding to the character block, so as to obtain a set of position features containing the position features corresponding to each character block, and the feature dimension of the position feature corresponding to each character block can be 4.

[0099] In one possible implementation, the region images corresponding to each character block obtained from the text detection output are input into a text recognition network. The text recognition network recognizes the text content corresponding to each character block. Then, word embedding in natural language processing tools is used to process the recognized text content corresponding to each character block, mapping the text content into real-valued vectors. The mapped real-valued vectors are determined as word vectors, and the semantic features corresponding to each character block are obtained based on the word vectors.

[0100] Natural Language Processing (NLP) is an important area within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it has a close connection with linguistic research. NLP technologies typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0101] For example, by obtaining the word vectors corresponding to each character block, a set of semantic features containing the semantic features corresponding to each character block is obtained, and the feature dimension of the semantic features corresponding to each character block can be 64.

[0102] In one possible implementation, the image feature vector, positional feature vector, and semantic feature vector corresponding to the character blocks obtained through feature extraction are concatenated to generate a concatenated feature vector.

[0103] The concatenated feature vector corresponding to each character block can be a 132-dimensional feature vector.

[0104] For example, Figure 4 This is a framework diagram of a feature splicing process involved in an embodiment of this application. For example... Figure 4 As shown, the feature network includes a convolutional neural network 41, a position embedding 42, and a word embedding 43. The image features extracted by the multi-layer convolutional neural network can be obtained through the convolutional neural network 41. The final image features are obtained through the residual network. The position features corresponding to the character blocks can be extracted based on the position embedding 42. The semantic features corresponding to the character blocks can be extracted based on the word embedding 43. The image features, position features, and semantic features are concatenated to generate the splicing features 44 corresponding to the character blocks.

[0105] Step 303: Obtain the similarity between the splicing features corresponding to every two character blocks in each character block.

[0106] In this embodiment of the application, the computer device obtains the similarity between the splicing features corresponding to every two character blocks in each character block.

[0107] In one possible implementation, the similarity between concatenated features is calculated based on a vector space model.

[0108] For example, the feature similarity between the concatenation matrices of every two characters in each block can be calculated using Euclidean distance, or the feature similarity between the concatenation matrices of every two characters in each block can be calculated using cosine similarity, or feature similarity can be calculated based on other similarity calculation methods.

[0109] For example, if there are block A, block B and block C, the feature similarity between block A and block B, the feature similarity between block B and block C, and the feature similarity between block A and block C are obtained respectively.

[0110] In one possible implementation, in response to the feature extraction network outputting the concatenated features corresponding to each character block, the concatenated features corresponding to each character block are input into the feature fusion network, and then the similarity between the concatenated features corresponding to every two character blocks in each character block is calculated.

[0111] The feature fusion network can include graph neural networks and fully connected neural networks.

[0112] Step 304: Construct graph data based on the splicing features of each character block and the similarity between the splicing features of every two character blocks.

[0113] In this embodiment of the application, the computer device can treat each character block as a node of the graph data, and the splicing feature corresponding to each character block as the node feature corresponding to each node. Then, the similarity between the splicing features corresponding to each two character blocks is used as the edge data (also known as edge weight) connecting each node in the graph data.

[0114] In one possible implementation, individual character blocks identified on the same target text object can be constructed as graph data.

[0115] Step 305: Based on the graph data, perform neighbor fusion on the splicing features of each character block to obtain the corresponding neighbor fusion features of each character block.

[0116] In this embodiment of the application, based on the constructed graph data, KNN (k-Nearest Neighbor) can be used to perform neighbor fusion on the splicing features of each character block to obtain the neighbor fusion features corresponding to each character block.

[0117] In one possible implementation, based on graph data, the K neighboring character blocks of the target character block are obtained, and the concatenation features of the target character block are fused with the concatenation features of the K neighboring character blocks to obtain the proximity fusion features of the target character block.

[0118] Among them, adjacent character blocks can be K character blocks whose similarity between the corresponding splicing features and the splicing features of the target character block satisfies the adjacent condition; the target character block is any one of the character blocks, and K can be a positive integer greater than 0.

[0119] The adjacent condition can be that the similarity between the target character block and other character blocks is within a specified threshold range, or it can be that the similarity between the target character block and other character blocks is sorted in descending order, and the similarity arrangement position is within a specified range.

[0120] In one possible implementation, the splicing features of the target character block and the splicing features of the K adjacent character blocks are summed. Then, the graph data with the splicing features after feature summation is input into a multilayer perceptron (MLP) for feature extraction to obtain the neighbor fusion features of the target character block.

[0121] Graph neural networks can include multilayer perceptrons.

[0122] For example, if the graph data includes blocks A, B, C, and D, the feature similarities between the concatenated feature vector a corresponding to block A and the concatenated feature vector b corresponding to block B, c corresponding to block C, and d corresponding to block D can be obtained from the graph data as x, y, and z, respectively, where x is greater than z, which is greater than y. If K is 2, then the two blocks with the higher feature similarity, i.e., x and z, corresponding to blocks B and D, are selected as adjacent blocks of block A. Then, the concatenated feature vectors of block A, B, and D are fused to obtain the concatenated feature vector (a, b, d) after feature accumulation for block A. The above calculation is performed on each block in the graph data to obtain the concatenated feature vector after feature accumulation for each block. Then, the graph data with each node feature as its corresponding concatenated feature vector is input into a multilayer perceptron to obtain the neighbor fusion features corresponding to each block.

[0123] In one possible implementation, the constructed graph data is input into a graph neural network to obtain the neighbor fusion features corresponding to each character block.

[0124] Step 306: Based on the neighbor fusion features corresponding to each character block, obtain the first feature of each character block.

[0125] In one possible implementation, a fully connected neural network is used to aggregate the features of each character block's corresponding neighbor fusion features to obtain the first feature of each character block.

[0126] Among them, the neighbor fusion features of each character block can be aggregated by a fully connected neural network (FCN), so that the dimensionality of the neighbor fusion features corresponding to each character block can be reduced to obtain the first feature of each character block.

[0127] For example, if the neighbor fusion feature vector corresponding to each character block is 132-dimensional, then the neighbor fusion feature vector corresponding to each character block can be aggregated to 64-dimensional through a fully connected neural network.

[0128] In one possible implementation, the first feature of each character block is obtained based on the splicing features of each character block.

[0129] Step 307: Obtain the feature matrix based on the first feature of each character block.

[0130] In this embodiment of the application, the computer device obtains the first feature corresponding to each pair of character blocks in each character block, fuses the first features corresponding to each pair of character blocks to generate the second feature corresponding to each pair of character blocks, and obtains the feature matrix corresponding to the image data based on the second feature corresponding to each pair of character blocks.

[0131] The feature matrix contains the second feature obtained by fusing the first features of every two character blocks in each character block.

[0132] For example, Figure 5 This is a framework diagram of a feature fusion process according to an embodiment of this application. Figure 5 As shown, the concatenated features corresponding to each character block output by the feature extraction network are input into the feature fusion network 50. First, a graph data 51 containing the concatenated features of character blocks on the same target text object is constructed based on the concatenated features corresponding to each character block. Then, KNN is used to determine the adjacent character blocks corresponding to each character block and to determine the neighbor fusion features corresponding to each character block. The neighbor fusion features corresponding to each character block are input into a fully connected neural network. The nodes corresponding to each character block are processed by a fully connected neural network 52. The fully connected neural network can aggregate the neighbor fusion feature vectors corresponding to each character block to 64 dimensions to obtain the first feature 531 corresponding to each character block. The first features 531 corresponding to each pair of character blocks are fused to generate the second feature 532 corresponding to each pair of character blocks. If the graph data contains N character blocks in total, an N-dimensional feature can be generated in the end. The feature matrix of N, and the feature vector corresponding to each feature point in the feature matrix is ​​a 128-dimensional second feature vector.

[0133] Step 308: Based on the feature matrix, obtain the adjacency matrix of each character block.

[0134] In this embodiment of the application, based on the feature matrix output by the feature fusion network, an adjacency matrix can be obtained to indicate the order between every two character blocks in each character block.

[0135] The adjacency matrix can be used to indicate the order of individual blocks.

[0136] In one possible implementation, the feature matrix is ​​input into a classification network, which processes the feature matrix to obtain the adjacency matrix of each character block.

[0137] The classification network can be a binary classification network, where each feature point in the feature matrix corresponds to the second feature of two character blocks. Based on the classification network, the feature value corresponding to each feature point can be determined. This feature value is used to indicate the order between the two corresponding character blocks.

[0138] In one possible implementation, Monte Carlo sampling can be used to randomly select pairs of character blocks during the training of the classification network. Randomly selecting pairs of character blocks using Monte Carlo sampling improves both the efficiency and randomness of the classification network training.

[0139] For example, when the second feature is obtained by concatenating the first features of each of the word blocks A and B, the order of word blocks A and B can be predicted based on the classification network. If word block A comes before word block B, the feature value at the corresponding feature point in the adjacency matrix can be set to 1. If word block B comes before word block A, the feature value at the corresponding feature point in the adjacency matrix can be set to 0.

[0140] In one possible implementation, in response to a block size of N, the adjacency matrix is ​​N. A binary classification matrix of N.

[0141] Step 309: Sort and combine the text content corresponding to each character block based on the adjacency matrix to obtain the text processing result of the target text object.

[0142] In one possible implementation, the target sequence is obtained by traversing the adjacency matrix; based on the arrangement order of each character block, the text content corresponding to each character block is arranged and combined to obtain the text processing result of the target text object.

[0143] The number of times the adjacency matrix is ​​traversed is determined by the number of each character block. When there are k character blocks, the adjacency matrix needs to be traversed k-1 times.

[0144] For example, during the first adjacency matrix traversal, the character block that is first in the target sequence is determined. During the second adjacency matrix traversal, the character block that is second in the target sequence is determined. This process continues until the (k-1)th adjacency matrix traversal, at which point the character block that is k-1 in the target sequence is determined, and the traversal stops. The last remaining character block is the character block that is k-th in the target sequence.

[0145] In one possible implementation, the word blocks that need to be traversed sequentially for the kth time are determined based on the state matrix.

[0146] The state matrix can be used to record whether the position sorting comparison of each word block that needs to be compared during the k-th traversal has been completed at the current time.

[0147] For example, during the first traversal, it is necessary to determine the first character block in the target sequence. When the character blocks that need to be sorted include block 1, block 2, and block 3, the initial state matrix indicates that each character block is in an incomplete sorting state. Therefore, any one of block 1, block 2, and block 3 is temporarily selected as the first character block in the target sequence. If block 2 is selected, the value corresponding to block 2 in the state matrix is ​​set to 0, indicating that block 1 has been processed in this round of traversal. Then, in the next moment, any one of block 1 and block 3 is selected as the first character block. The order of block 1 and block 2 is determined by the adjacency matrix. If block 1 is selected and it precedes block 2, the first position in the target sequence is updated to block 1. If block 1 follows block 2, the first position in the target sequence remains block 2. Similarly, the value corresponding to block 1 in the state matrix is ​​set to 0. Then, block 3 is selected, and its order with the first block is determined. Based on the result, the first block in the target sequence is updated, and its state value in the state matrix is ​​set to 0. The state values ​​of other blocks are reset to 1. The second round of traversal is then performed to determine the second block in the target sequence.

[0148] For example, to transform the adjacency matrix into a target sequence S of text box order, first, traverse the adjacency matrix and select an unused block A as the seed point at the beginning of the current target sequence. Then, continue traversing the adjacency matrix, selecting an unused block B, and comparing it with block A. If block B precedes block A, then block B is placed at the beginning of the current target sequence. This process is repeated for the remaining blocks until the block at the beginning of the sequence is found, and then that block is placed in the final target sequence S. The above steps are repeated until all corresponding blocks in the adjacency matrix have been used, at which point the traversal stops, and the final target sequence S is obtained.

[0149] In one possible implementation, the text content corresponding to the character blocks is output or displayed according to the order of the character blocks corresponding to the target sequence.

[0150] In summary, in the scheme shown in the embodiments of this application, based on the multimodal features corresponding to each extracted character block, a first feature corresponding to each character block is obtained. By fusing the first features of every two character blocks in each character block, a second feature is generated, and a feature matrix containing each second feature is formed. Then, based on the feature matrix, an adjacency matrix for indicating the order between character blocks is determined. This provides a global consideration for determining the order, avoiding the problem that when the character blocks in the target text object are irregularly distributed, there may be a character block and multiple character blocks that may have different order, making it impossible to determine whether the character block is actually in front of or behind the character block. The text content can be processed based on a global consideration under multimodal conditions, thereby improving the accuracy of text word order restoration.

[0151] Figure 6 This is a schematic diagram illustrating text content processing in a medical record according to an exemplary embodiment. For example... Figure 6As shown, firstly, the word order restoration model 62 is trained in the model training device. The specific model training process includes acquiring a sample medical record image 61 containing at least two sample word blocks, inputting the sample medical record image 61 into the word order restoration model 62, and firstly inputting the sample medical record image into the feature extraction network 63. The feature extraction network 63 performs text detection and text recognition on the word blocks. Through text detection, the position information of the text detection box corresponding to each sample word block and the word block image corresponding to the text detection box can be determined. Through text recognition, the semantic information corresponding to each sample word block can be determined. Based on block images, image features can be extracted using an image feature extraction network, i.e., a convolutional neural network. Positional features corresponding to sample blocks can be extracted using position embedding based on the positional information of the text detection boxes. Semantic features can be obtained through word embedding using the semantic information of the sample blocks. Then, the image features, positional features, and semantic features corresponding to each sample block are concatenated to generate concatenated features for each sample block. These concatenated features are then input into a feature fusion network 64. First, a sample image data is constructed for each sample block belonging to the same medical record image. The nodes in this sample image data represent each sample block, the node features are the concatenated features corresponding to each sample block, and the edges are the connections between corresponding nodes. The feature similarity corresponding to the concatenated features is determined using the KNN algorithm based on the constructed sample image data. This algorithm obtains the neighboring characters corresponding to each sample character block, accumulates the concatenated features of the neighboring characters and the concatenated features of the sample character blocks, and then performs MLP feature extraction on the accumulated concatenated features to determine the neighbor fusion features corresponding to each sample character block. The sample image data containing these neighbor fusion features is then input into a fully connected neural network to aggregate the neighbor fusion features, obtaining the first feature corresponding to each sample character block. The first features of each pair of sample character blocks are then fused to determine the second feature corresponding to each sample character block. Based on the second features of each sample character block, a feature matrix containing these second features is generated. This feature matrix is ​​then input into a classification network 65, which outputs a predicted sample adjacency matrix. Based on the predicted sample adjacency matrix, a predicted target sequence corresponding to the sample character block is determined. Based on this predicted target sequence, the text content corresponding to each sample character block is arranged and combined to obtain the text content after restoring the word order of the predicted sample character blocks in the sample medical record. The loss function value is obtained based on the predicted text content and the text content after word order restoration by manual annotation of sample medical records, and the word order restoration model is updated based on the loss function value.

[0152] After the word order restoration model 62 is trained, it is sent to the prediction device. When a user needs to perform word order reordering on the text content in the target medical record image 66, the following... Figure 7 This is a schematic diagram of a target medical record according to an embodiment of this application. Figure 7 As shown, the target medical record image 71 includes multiple word blocks that need to be sorted. The target medical record image 66 is input into the word order restoration model 62, and the text content 67 of the target medical record image 66 after word order arrangement is output through the steps of the above embodiment.

[0153] For example, such as Figure 7 As shown, the target medical record image 71 includes various text blocks, each corresponding to the following text content: "Name: XXX", "Gender", "Female", "Age: 55", "Ward", "Bed", "Clinical Diagnosis", "Right Upper Lobe Nodule, Diabetes, Hypertension", "Requested by Submitting Physician", "Examination Site and Name", "Chest CT", "Chest CT", "Examination Time", "2020-12-03", "16:20:56", "Examination Method", "XXXXXXX", "Radiological Findings", "XXXXXXX, XXXXXX", "Radiological Diagnosis", "XXXXXXX, XXXXXX", "XXXX", and "XXXXX, XXX". By inputting this target medical record image 71 as follows... Figure 6In the word order restoration model shown, the output text content 67 is obtained by arranging the text content corresponding to each of the above word blocks in a determined word order. This text content 67 can be: "XX Hospital Diagnostic Report (CT) Name: XXX Gender: Female Age: 55 Ward Bed Clinical Diagnosis: Right upper lobe nodule, diabetes, hypertension Submitting Physician's Request: Chest CT Examination Site and Name Chest CT Examination Time: 2020-12-03 16:20:56 Examination Method: XXXXXXX Radiological Manifestations: XXXXXXX, XXXXXX Radiological Diagnosis: XXXXXXX, XXXXXX XXXX" The text content obtained by the word order restoration model, which arranges the various blocks on the target medical record image 71 in sequence, can effectively solve the problems of text offset and many-to-many matching. For example, the text content of the answer to "Requested by the referring physician" on the target medical record image 71 should be "Chest CT". However, since the blocks corresponding to "Requested by the referring physician" and "Chest CT" on the target medical record 71 are arranged vertically, not in the conventional left-to-right text order, the word order restoration model can be used to determine that the block following "Requested by the referring physician" is "Chest CT". Alternatively, when the block corresponding to "Examination site and name" is located between the blocks corresponding to "Chest CT" and "Examination time", it is impossible to determine the block following "Examination site and name" by simple position judgment. In this case, the word order restoration model can be used to determine that the block following "Examination site and name" is "Chest CT". Alternatively, when the block corresponding to "examination location and name" is located between the blocks corresponding to "chest CT" and "brain CT", it is impossible to determine the block following the block corresponding to "examination location and name" through simple positional judgment, nor is it possible to determine the block following the block corresponding to "examination location and name" through semantic judgment. Instead, a word order restoration model can be used to make a global judgment on the block following the block corresponding to "examination location and name", thus determining that the block following the block corresponding to "examination location and name" is the block corresponding to "chest CT".

[0154] In summary, in the scheme shown in the embodiments of this application, based on the multimodal features corresponding to each extracted character block, a first feature corresponding to each character block is obtained. By fusing the first features of every two character blocks in each character block, a second feature is generated, and a feature matrix containing each second feature is formed. Then, based on the feature matrix, an adjacency matrix for indicating the order between character blocks is determined. This provides a global consideration for determining the order, avoiding the problem that when the character blocks in the target text object are irregularly distributed, there may be a character block and multiple character blocks that may have different order, making it impossible to determine whether the character block is actually in front of or behind the character block. The text content can be processed based on a global consideration under multimodal conditions, thereby improving the accuracy of text word order restoration.

[0155] Figure 8 This is a block diagram illustrating a text content processing apparatus according to an exemplary embodiment, such as... Figure 8 As shown, this text content processing device can be implemented entirely or partially within a computer device through hardware or a combination of hardware and software, in order to perform... Figure 1 or Figure 3 The text content processing apparatus may include all or part of the steps of the method shown in the corresponding embodiment.

[0156] The character block acquisition module 810 is used to acquire each character block in the target text object, wherein the character block is a region in the target text object that contains continuous text content;

[0157] The first feature acquisition module 820 is used to acquire the first feature of each character block, wherein the first feature includes the image feature, position feature and semantic feature of the corresponding character block;

[0158] The matrix acquisition module 830 is used to acquire a feature matrix based on the first feature of each of the character blocks; the feature matrix contains a second feature obtained by fusing the first features of every two character blocks in each character block;

[0159] The adjacency matrix acquisition module 840 is used to acquire the adjacency matrix of each character block based on the feature matrix, wherein the adjacency matrix is ​​used to indicate the order between the character blocks;

[0160] The result acquisition module 850 is used to sort and combine the text content corresponding to each character block based on the adjacency matrix to obtain the text processing result of the target text object.

[0161] In one possible implementation, the first feature acquisition module 820 includes:

[0162] The splicing feature acquisition submodule is used to acquire the splicing features of each character block. The splicing features include the features obtained by splicing the image features, position features and semantic features of the corresponding character block.

[0163] The first feature acquisition submodule is used to acquire the first feature of each character block based on the splicing features of each character block.

[0164] In one possible implementation, the first feature acquisition submodule includes:

[0165] The similarity acquisition unit is used to acquire the similarity between the splicing features corresponding to every two character blocks in each character block;

[0166] The graph construction unit is used to construct graph data based on the splicing features of each character block and the similarity between the splicing features of each pair of character blocks.

[0167] The neighbor feature acquisition unit is used to perform neighbor fusion on the splicing features of each character block based on the graph data to obtain the neighbor fusion features corresponding to each character block.

[0168] The first feature acquisition unit is used to acquire the first feature of each character block based on the corresponding neighbor fusion feature of each character block.

[0169] In one possible implementation, the proximity feature acquisition unit is used for:

[0170] Based on the graph data, obtain K neighboring character blocks of the target character block; the neighboring character blocks are K character blocks whose similarity between their corresponding splicing features and the splicing features of the target character block satisfies the adjacent condition; the target character block is any one of the character blocks; K is a positive integer;

[0171] The splicing features of the target character block are fused with the splicing features of K adjacent character blocks to obtain the neighbor fusion features of the target character block.

[0172] In one possible implementation, the first feature acquisition unit is used to,

[0173] By using a fully connected network, the corresponding neighbor fusion features of each character block are aggregated to obtain the first feature of each character block.

[0174] In one possible implementation, the result acquisition module 850 includes:

[0175] The sequence acquisition submodule is used to obtain the target sequence by traversing the adjacency matrix; the target sequence includes the arrangement order of the various word blocks;

[0176] The result acquisition submodule is used to arrange and combine the text content corresponding to each character block based on the arrangement order of each character block, so as to obtain the text processing result of the target text object.

[0177] In one possible implementation, the adjacency matrix acquisition module 840 includes:

[0178] The acquisition submodule is used to input the feature matrix into the classification network to obtain the adjacency matrix of each character block.

[0179] In summary, in the scheme shown in the embodiments of this application, based on the multimodal features corresponding to each extracted character block, a first feature corresponding to each character block is obtained. By fusing the first features of every two character blocks in each character block, a second feature is generated, and a feature matrix containing each second feature is formed. Then, based on the feature matrix, an adjacency matrix for indicating the order between character blocks is determined. This provides a global consideration for determining the order, avoiding the problem that when the character blocks in the target text object are irregularly distributed, there may be a character block and multiple character blocks that may have different order, making it impossible to determine whether the character block is actually in front of or behind the character block. The text content can be processed based on a global consideration under multimodal conditions, thereby improving the accuracy of text word order restoration.

[0180] Figure 9 A structural block diagram of a computer device 900 illustrated in an exemplary embodiment of this application is shown. This computer device can be implemented as a server as described above in this application. The computer device 900 includes a Central Processing Unit (CPU) 901, a system memory 904 including Random Access Memory (RAM) 902 and Read-Only Memory (ROM) 903, and a system bus 905 connecting the system memory 904 and the CPU 901. The computer device 900 also includes a mass storage device 906 for storing an operating system 909, application programs 910, and other program modules 911.

[0181] The mass storage device 906 is connected to the central processing unit 901 via a mass storage controller (not shown) connected to the system bus 905. The mass storage device 906 and its associated computer-readable media provide non-volatile storage for the computer device 900. That is, the mass storage device 906 may include computer-readable media (not shown) such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0182] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage media are not limited to the above-mentioned types. The system memory 904 and mass storage device 906 described above can be collectively referred to as memory.

[0183] According to various embodiments of this disclosure, the computer device 900 can also be connected to a remote computer on a network, such as the Internet. That is, the computer device 900 can be connected to a network 908 via a network interface unit 907 connected to the system bus 905, or it can use the network interface unit 907 to connect to other types of networks or remote computer systems (not shown).

[0184] The memory further includes at least one instruction, at least one program, code set, or instruction set, which are stored in the memory. The central processing unit 901 executes the at least one instruction, at least one program, code set, or instruction set to implement all or part of the steps in the text content processing methods shown in the above embodiments.

[0185] Figure 10A structural block diagram of a computer device 1000 provided in an exemplary embodiment of this application is shown. The computer device 1000 can be implemented as the aforementioned terminal, such as a smartphone, tablet computer, laptop computer, or desktop computer. The computer device 1000 may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other names.

[0186] Typically, computer device 1000 includes a processor 1001 and a memory 1002.

[0187] Processor 1001 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1001 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1001 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1001 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 1001 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0188] The memory 1002 may include one or more computer-readable storage media, which may be non-transitory. The memory 1002 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1002 are used to store at least one instruction, which is executed by the processor 1001 to implement all or part of the steps in the object attribute determination method provided in the method embodiments of this application.

[0189] In some embodiments, the computer device 1000 may optionally include a peripheral device interface 1003 and at least one peripheral device. The processor 1001, memory 1002, and peripheral device interface 1003 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1003 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 1004, a display screen 1005, a camera assembly 1006, an audio circuit 1007, and a power supply 1009.

[0190] Peripheral device interface 1003 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1001 and memory 1002. In some embodiments, processor 1001, memory 1002 and peripheral device interface 1003 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1001, memory 1002 and peripheral device interface 1003 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0191] In some embodiments, the computer device 1000 further includes one or more sensors 1010. The one or more sensors 1010 include, but are not limited to, an accelerometer 1011, a gyroscope 1012, a pressure sensor 1013, an optical sensor 1015, and a proximity sensor 1016.

[0192] Those skilled in the art will understand that Figure 10 The structure shown does not constitute a limitation on the computer device 1000, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0193] In one exemplary embodiment, a computer-readable storage medium is also provided for storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement all or part of the steps in the object attribute determination method described above. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, or optical data storage device, etc.

[0194] In one exemplary embodiment, a computer program product or computer program is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned actions. Figure 1 or Figure 3 All or part of the steps of the method shown in any embodiment.

[0195] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0196] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A text content processing method, characterized in that, The method includes: Obtain individual character blocks from a target text object, where each character block is a region in the target text object containing continuous text content; Obtain the first feature of each character block, wherein the first feature includes the image feature, position feature and semantic feature of the corresponding character block; Based on the first feature of each of the character blocks, a feature matrix is ​​obtained that contains the second feature obtained by fusing the first features of every two character blocks in each character block; Based on the feature matrix, the adjacency matrix of each character block is obtained, and the adjacency matrix is ​​used to indicate the order between the character blocks; Based on the adjacency matrix, the text content corresponding to each character block is sorted and combined to obtain the text processing result of the target text object.

2. The method according to claim 1, characterized in that, The step of obtaining the first feature of each of the word blocks includes: Obtain the splicing features of each character block, wherein the splicing features include the features obtained by splicing the image features, position features and semantic features of the corresponding character block; Based on the splicing features of each character block, the first feature of each character block is obtained.

3. The method according to claim 2, characterized in that, The step of obtaining the first feature of each character block based on the concatenation features of each character block includes: Obtain the similarity between the splicing features corresponding to every two character blocks in each character block; Based on the splicing features of each character block and the similarity between the splicing features of every two character blocks, graph data is constructed. Based on the graph data, the splicing features of each character block are fused by proximity to obtain the proximity fusion features corresponding to each character block. Based on the neighbor fusion features corresponding to each character block, the first feature of each character block is obtained.

4. The method according to claim 3, characterized in that, The step of performing neighbor fusion on the splicing features of each character block based on the graph data to obtain the corresponding neighbor fusion features for each character block includes: Based on the graph data, obtain K neighboring character blocks of the target character block; the neighboring character blocks are K character blocks whose similarity between their corresponding splicing features and the splicing features of the target character block satisfies the adjacent condition; the target character block is any one of the character blocks; K is a positive integer; The splicing features of the target character block are fused with the splicing features of K adjacent character blocks to obtain the neighbor fusion features of the target character block.

5. The method according to claim 3, characterized in that, The step of obtaining the first feature of each character block based on the corresponding proximity fusion features of each character block includes: By using a fully connected network, the corresponding neighbor fusion features of each character block are aggregated to obtain the first feature of each character block.

6. The method according to claim 1, characterized in that, The step of sorting and combining the text content corresponding to each character block based on the adjacency matrix to obtain the text processing result of the target text object includes: The target sequence is obtained by traversing the adjacency matrix; the target sequence includes the arrangement order of the various character blocks. Based on the arrangement order of the various character blocks, the text content corresponding to each character block is arranged and combined to obtain the text processing result of the target text object.

7. The method according to claim 1, characterized in that, The step of obtaining the adjacency matrix of each character block based on the feature matrix includes: The feature matrix is ​​input into the classification network to obtain the adjacency matrix of each character block.

8. A text content processing device, characterized in that, The device includes: The character block acquisition module is used to acquire each character block in the target text object, wherein the character block is a region in the target text object that contains continuous text content; The first feature acquisition module is used to acquire the first feature of each character block, wherein the first feature includes the image feature, position feature and semantic feature of the corresponding character block; The matrix acquisition module is used to acquire a feature matrix containing a second feature obtained by fusing the first features of each of the two character blocks, based on the first features of each of the character blocks. An adjacency matrix acquisition module is used to acquire the adjacency matrix of each character block based on the feature matrix, wherein the adjacency matrix is ​​used to indicate the order between the character blocks; The result acquisition module is used to sort and combine the text content corresponding to each character block based on the adjacency matrix to obtain the text processing result of the target text object.

9. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one computer program, which is loaded and executed by the processor to implement the text content processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the text content processing method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, The computer program product includes computer instructions that are executed by a processor to implement the text content processing method as described in any one of claims 1 to 7.

Images