Method, device, electronic device and storage medium for determining text element reading order

By extracting and fusing the position and semantic features of text elements, an element graph network is established, and the problem of determining the reading order in complex layout documents is solved, and the accurate order determination of text elements is achieved. It is suitable for a variety of layouts and granularities, improving information extraction accuracy and robustness.

CN114239598BActive Publication Date: 2024-12-03HANGZHOU HIKVISION DIGITAL TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111549632.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-17
Publication Date
2024-12-03
Estimated Expiration
2041-12-17

AI Technical Summary

Technical Problem

In the field of text recognition and information extraction, the semantic information of complex layout documents is easily disrupted, resulting in problems such as entity truncation in information extraction. How to determine the reading order of text elements has become a technical challenge that needs to be solved urgently.

Method used

By obtaining the text data to be analyzed, the positional features and semantic features of the text elements are extracted, combined with image features, a network of elements is established, and the reading order of text elements is determined using the decoder network.

Benefits of technology

Effectively determine the reading order of text elements, solves the problem of semantic information disruption in complex layout documents, improves the accuracy and robustness of information extraction, and is suitable for a variety of layouts and granularities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114239598B_ABST
    Figure CN114239598B_ABST
Patent Text Reader

Abstract

The embodiment of the present application provides a method, device, electronic device and storage medium for determining the reading order of text elements, obtaining text data to be analyzed, wherein the text data to be analyzed includes multiple text elements; parsing or feature extraction of the text elements in the text data to be analyzed, obtaining the positional features and semantic features of each text element in the text data to be analyzed; extracting image features of the text data to be analyzed, obtaining the image features of each text element in the text data to be analyzed; for each text element, fusing the semantic features and image features of the text element to obtain the fusion features of the text element; establishing an element graph network of the text data to be analyzed based on the positional features and fusion features of each text element; and determining the reading order of each text element according to the element graph network. It can be seen that the reading order of text elements can be determined through the present application.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, electronic device and storage medium for determining a reading order of text elements. Background Art

[0002] With the continuous development of artificial intelligence technology, the application of text recognition technology and information extraction technology based on artificial intelligence is becoming more and more extensive. In the field of text recognition technology, the recognition technology of regular text is already relatively mature, but there is a problem of reading order for right-to-left text; in the field of information extraction, the sequential splicing between various sections is often done from top to bottom and from left to right, but for complex layouts such as various magazines and resumes, such a reading order often disrupts the semantic information of the entire document, causing problems such as entity truncation in information extraction. Therefore, how to determine the reading order of text elements has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a method, device, electronic device and storage medium for determining the reading order of text elements, so as to achieve the determination of the reading order of text elements. The specific technical solution is as follows:

[0004] In a first aspect, an embodiment of the present application provides a method for determining a reading order of text elements, comprising:

[0005] Acquire text data to be analyzed, wherein the text data to be analyzed includes a plurality of text elements;

[0006] Parsing or feature extracting text elements in the text data to be analyzed to obtain position features and semantic features of each text element in the text data to be analyzed;

[0007] Extracting image features of the text data to be analyzed to obtain image features of each text element in the text data to be analyzed;

[0008] For each text element, the semantic features and image features of the text element are fused to obtain the fused features of the text element;

[0009] Based on the position features and fusion features of each of the text elements, an element graph network of the text data to be analyzed is established;

[0010] According to the element graph network, the reading order of each of the text elements is determined.

[0011] In a possible implementation manner, parsing or feature extracting the text elements in the text data to be analyzed to obtain the position features and semantic features of each text element in the text data to be analyzed includes:

[0012] Acquiring text elements from the text data to be analyzed, and obtaining the content and position of each text element in the text data to be analyzed;

[0013] For each text element, the semantic feature of the text element is determined based on the content of the text element, and the position feature of the text element is obtained based on the position of the text element.

[0014] In a possible implementation, the text data to be analyzed is in a portable document format PDF;

[0015] The acquiring of text elements from the text data to be analyzed to obtain the content and position of each text element in the text data to be analyzed includes:

[0016] Parsing the text elements of the text data to be analyzed by using a preset PDF parser to obtain the content and position of each text element in the text data to be analyzed;

[0017] Before extracting image features from the text data to be analyzed to obtain image features of each text element in the text data to be analyzed, the method further includes:

[0018] The text data to be analyzed is converted from PDF to image format.

[0019] In a possible implementation manner, the text data to be analyzed is in a picture format;

[0020] The acquiring of text elements from the text data to be analyzed to obtain the content and position of each text element in the text data to be analyzed includes:

[0021] The text elements of the text data to be analyzed are recognized using a preset optical character recognition (OCR) algorithm to obtain the content and position of each text element in the text data to be analyzed.

[0022] In a possible implementation manner, extracting image features from the text data to be analyzed to obtain image features of each text element in the text data to be analyzed includes:

[0023] The text data to be analyzed is input into a pre-trained deep learning model, and the image features of each text element in the text data to be analyzed are extracted from the feature output layer of the deep learning model.

[0024] In a possible implementation, establishing the element graph network of the text data to be analyzed based on the position features and fusion features of each of the text elements includes:

[0025] According to the fusion features of each of the text elements, node features of each node in the element graph network are established, wherein each of the text elements corresponds to a node in the element graph network;

[0026] According to the position features of each of the text elements, edge features of each node in the element graph network are established.

[0027] In a possible implementation manner, determining the reading order of each of the text elements according to the element graph network includes:

[0028] The element graph network is decoded using a pre-trained decoder network to obtain a reading order of each of the text elements.

[0029] In a possible implementation, for each text element, fusing the semantic features and image features of the text element to obtain the fused features of the text element includes:

[0030] The target application scenario features of the text data to be analyzed are input into a pre-trained feature fusion network, and for each text element, the semantic features and image features of the text element are weightedly fused through the feature fusion network to obtain the fusion features of the text element, wherein the target application scenario features include position features, semantic features and image features of the text elements in the text data to be analyzed, and wherein the weights used in the weighted fusion process of the feature fusion network are obtained by training sample application scenario features of sample text data under different application scenarios.

[0031] In a second aspect, an embodiment of the present application provides a device for determining a reading order of text elements, comprising:

[0032] A text data acquisition module, used to acquire text data to be analyzed, wherein the text data to be analyzed includes a plurality of text elements;

[0033] A semantic and position extraction module, used for parsing or feature extraction of text elements in the text data to be analyzed, and obtaining position features and semantic features of each text element in the text data to be analyzed;

[0034] An image feature extraction module is used to extract image features from the text data to be analyzed, and obtain image features of each text element in the text data to be analyzed;

[0035] A feature fusion module is used to fuse the semantic features and image features of each text element to obtain a fusion feature of the text element;

[0036] An element graph network construction module, used to establish an element graph network of the text data to be analyzed based on the position features and fusion features of each of the text elements;

[0037] The decoder module is used to determine the reading order of each of the text elements according to the element graph network.

[0038] In a possible implementation, the semantics and position extraction module includes:

[0039] A text element identification submodule is used to obtain text elements from the text data to be analyzed, and obtain the content and position of each text element in the text data to be analyzed;

[0040] A semantic feature extraction submodule is used to determine the semantic feature of each text element based on the content of the text element;

[0041] The position feature extraction submodule is used to obtain the position feature of each text element based on the position of the text element.

[0042] In a possible implementation, the text data to be analyzed is in a portable document format PDF;

[0043] The text element identification submodule is specifically used to: parse the text elements of the text data to be analyzed using a preset PDF parser to obtain the content and position of each text element in the text data to be analyzed;

[0044] The device also includes: a format conversion module, which is used to convert the text data to be analyzed from PDF to image format.

[0045] In a possible implementation manner, the text data to be analyzed is in a picture format;

[0046] The text element recognition submodule is specifically used to: use a preset optical character recognition (OCR) algorithm to recognize text elements in the text data to be analyzed, and obtain the content and position of each text element in the text data to be analyzed.

[0047] In a possible implementation, the image feature extraction module is specifically used to: input the text data to be analyzed into a pre-trained deep learning model, and extract image features of each text element in the text data to be analyzed from the feature output layer of the deep learning model.

[0048] In a possible implementation, the element graph network construction module is specifically used to: establish node features of each node in the element graph network based on fusion features of each of the text elements, wherein each of the text elements corresponds to a node in the element graph network; and establish edge features of each node in the element graph network based on position features of each of the text elements.

[0049] In a possible implementation, the decoder module is specifically used to decode the element graph network using a pre-trained decoder network to obtain a reading order of each of the text elements.

[0050] In a possible implementation, the feature fusion module is specifically used to: input the target application scenario features of the text data to be analyzed into a pre-trained feature fusion network, and for each text element, perform weighted fusion of the semantic features and image features of the text element through the feature fusion network to obtain the fusion features of the text element, wherein the target application scenario features include position features, semantic features and image features of the text elements in the text data to be analyzed, and wherein the weights used in the weighted fusion process of the feature fusion network are obtained by training sample application scenario features of sample text data under different application scenarios.

[0051] In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory;

[0052] The memory is used to store computer programs;

[0053] The processor is used to implement any method for determining the reading order of text elements described in the present application when executing the program stored in the memory.

[0054] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, it implements any method for determining the reading order of text elements described in the present application.

[0055] Beneficial effects of the embodiments of the present application:

[0056] The method, device, electronic device and storage medium for determining the reading order of text elements provided in the embodiments of the present application obtain text data to be analyzed, wherein the text data to be analyzed includes multiple text elements; the text elements in the text data to be analyzed are parsed or feature extracted to obtain the position features and semantic features of each text element in the text data to be analyzed; the image features of the text data to be analyzed are extracted to obtain the image features of each text element in the text data to be analyzed; for each text element, the semantic features and image features of the text element are fused to obtain the fusion features of the text element; based on the position features and fusion features of each text element, an element graph network of the text data to be analyzed is established; according to the element graph network, the reading order of each text element is determined; the reading order of the text elements can be determined. Of course, the implementation of any product or method of the present application does not necessarily require all the advantages described above to be achieved at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and a person skilled in the art can also obtain other drawings based on these drawings.

[0058] Figure 1 A first schematic diagram of a method for determining a reading order of text elements according to an embodiment of the present application;

[0059] Figure 2a A schematic diagram of a possible implementation of step S102 in an embodiment of the present application;

[0060] Figure 2b A schematic diagram of outputting the reading order of text elements in an embodiment of the present application;

[0061] Figure 3 A second schematic diagram of the method for determining the reading order of text elements according to an embodiment of the present application;

[0062] Figure 4 A third schematic diagram of the method for determining the reading order of text elements according to an embodiment of the present application;

[0063] Figure 5 A schematic diagram of a device for determining a reading order of text elements according to an embodiment of the present application

[0064] Figure 6 A schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0065] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0066] Clarifying the reading order of text elements is of great significance for machines to understand the entire text. In view of this, the present application embodiment provides a method for determining the reading order of text elements, see Figure 1 ,include:

[0067] S101, obtaining text data to be analyzed, wherein the text data to be analyzed includes a plurality of text elements.

[0068] The method for determining the reading order of text elements in the embodiment of the present application can be implemented by an electronic device. For example, the electronic device can be a smart terminal, a personal computer, or a server.

[0069] The text data to be analyzed is any data for which the reading order of text elements needs to be determined. The text data to be analyzed includes multiple text elements, wherein the granularity of the text elements may include character level, sentence level, paragraph level, and layout level, etc.; in one example, the granularity of each text element in the text data to be analyzed in the embodiment of the present application may be the same; in one example, the granularity of each text element in the text data to be analyzed in the embodiment of the present application may include multiple levels, for example, the granularity of some text elements is character level, the granularity of some text elements is sentence level, etc., all of which are within the protection scope of the embodiment of the present application.

[0070] S102, parsing or feature extraction of text elements in the text data to be analyzed, to obtain position features and semantic features of each text element in the text data to be analyzed.

[0071] The feature extraction method of text elements in the text data to be analyzed is used to extract features of the text elements, thereby obtaining the position features and semantic features of each text element in the text data to be analyzed.

[0072] In a possible implementation, feature extraction of text elements can be divided into two processes: obtaining text elements and extracting features. In one example, obtaining text elements includes PDF (Portable Document Format) parsing and text element recognition. Figure 2a The parsing or feature extraction of the text elements in the text data to be analyzed to obtain the position features and semantic features of each text element in the text data to be analyzed includes:

[0073] S1021, acquiring text elements from the text data to be analyzed, and obtaining the content and position of each text element in the text data to be analyzed.

[0074] The specific method of obtaining text elements can refer to the text element recognition method in the related art. For example, the text elements can be obtained by using a PDF parser, a deep learning method or a traditional image recognition method. By recognizing the text elements of the text data to be analyzed, the position of each text element in the text data to be analyzed can be obtained, and the text content in the position area can be obtained according to the position, that is, the content of the text element.

[0075] S1022, for each text element, determine the semantic feature of the text element based on the content of the text element, and obtain the position feature of the text element based on the position of the text element.

[0076] According to the content of the text element, the semantic features of the text element can be obtained; for example, the semantic features of the text element can be obtained by using word embedding methods such as word embedding, or by using language models in related technologies, such as BERT (Bidirectional Encoder Representation from Transformers) model, etc. to obtain the semantic features of the text element. When the granularity of the text element is a character or a word, the above method can be used directly to obtain its semantic vector; and when the granularity of the text element is a text line, a text block or a text layout, the above method can be used to obtain the feature vector of each character of the text element, and then the feature vectors of each character are averaged to obtain the semantic features of the text element, or the [cls] symbol can be directly added in front of the entire text element sequence, and the feature vector obtained by mapping this symbol can be used to represent the semantic features of the text element.

[0077] According to the position of the text element, the position feature of the text element can be obtained. For example, the pixel coordinates of the text element can be directly used as the position feature of the text element.

[0078] S103, extracting image features of the text data to be analyzed to obtain image features of each text element in the text data to be analyzed.

[0079] In one example, the image features of the text data to be analyzed can be extracted through a convolutional neural network. In a possible implementation, the image features of the text data to be analyzed are extracted to obtain the image features of each text element in the text data to be analyzed, including: inputting the text data to be analyzed into a pre-trained deep learning model, and extracting the image features of each text element in the text data to be analyzed from the feature output layer of the deep learning model.

[0080] The pre-trained deep learning model can be a target detection model for detecting text elements, such as FasterRCNN (Regions with Convolutional Neural Networks features), Mask RCNN, YOLO (YouOnly Live Once), SSD (Single Shot MultiBox Detector), TP (Text Perceptron), etc. By using the pre-trained deep learning model to detect text elements in the image of the text data to be analyzed, the position coordinate information of the text element can be obtained, and the corresponding vector can be extracted from the feature output layer of the deep learning model according to the position coordinate information as the image feature of the corresponding text element.

[0081] S104, for each text element, the semantic feature and the image feature of the text element are fused to obtain a fused feature of the text element.

[0082] The semantic features and image features of text elements are fused to obtain fused features. For example, the semantic features and image features of text elements can be fused by using the concat function, setting a multi-layer convolution adapter, or adding a gate function. The fused features contain both the visual information of the image and the semantic information of the text, and both the visual information and the semantic information can contribute to how to determine the reading order of the text elements. The fused features include both visual information and semantic information, which can further improve the accuracy of the final reading order of the text elements.

[0083] For different scenarios, the emphasis on semantic features and image features may be different. The weight ratio of semantic features and image features can be adaptively determined according to the target application scenario characteristics of the text data to be analyzed.

[0084] In a possible implementation, for each text element, the semantic features and image features of the text element are fused to obtain the fused features of the text element, including: inputting the target application scenario features of the text data to be analyzed into a pre-trained feature fusion network, and for each text element, performing weighted fusion on the semantic features and image features of the text element through the feature fusion network to obtain the fused features of the text element, wherein the target application scenario features include position features, semantic features and image features of the text elements in the text data to be analyzed, and wherein the weights used in the weighted fusion process of the feature fusion network are obtained by training sample application scenario features of sample text data under different application scenarios.

[0085] In one example, the target application scenario features of the text data to be analyzed may include the image features, position features, and semantic features of the text elements in the text data to be analyzed, and may also include the scene type of the text data to be analyzed. The feature fusion network may be a convolutional neural network, such as a small convolutional neural network, etc. The feature fusion network may include a weighted subnetwork or a gated subnetwork, etc., for adaptively allocating the weights of the semantic features and image features of the same text element; for text data in different application scenarios, the weight ratios of the semantic features and image features can be adaptively adjusted. The weights of the semantic features and image features in different application scenarios are learnable parameters, which can be obtained through training of sample application scenario features of sample text data in different application scenarios.

[0086] In an example, the training process of the feature fusion network may include: obtaining sample application scenario features of sample text data, the sample application scenario features including sample image features of the sample text data, position features and semantic features of each sample text element in the sample text data; outputting the sample application scenario features to the feature fusion network, and for each sample text element, obtaining a fusion feature obtained by weighted fusion of the position features and semantic features of the sample text element; establishing a sample element graph network of the sample text data based on the fusion features and position features of each sample text element; determining a predicted reading order of the sample text data based on the sample element graph network; calculating the loss based on the predicted reading order of the sample text data and the true reading order of the sample text data; adjusting the parameters of the feature fusion network based on the current loss; and continuing training until a preset number of training times is reached or the loss converges to obtain a pre-trained feature fusion network.

[0087] In one example, for application scenarios with high semantic relevance, the weight of semantic features in the weighted fusion process is greater than the weight of image features; for application scenarios with high layout relevance, the weight of semantic features in the weighted fusion process is less than the weight of image features. For example, for scenarios with strong semantic relevance between the front and back of newspapers and magazines, the proportion of semantic features is greater than that of image features; while for scenarios such as e-commerce product introduction homepages and bills, the semantic relevance between text elements is low, while the layout (i.e., image features) relevance is high, and image features account for a larger proportion.

[0088] For each text element, the semantic features and image features of the text element are fused according to the weight ratios of the semantic features and image features corresponding to the text data to be analyzed, so as to obtain the fused features of the text element; thus, better results can be achieved in various scenarios.

[0089] S105: Building an element graph network of the text data to be analyzed based on the position features and fusion features of each of the text elements.

[0090] The fusion features of each text element and the relationship between each text element (position feature) can form an element graph network. In an example, a graph convolutional neural network can be used to treat text elements as nodes in the element graph network, the fusion features of text elements as the features of the nodes, and the position feature relationship between each text element to form the edges between the nodes.

[0091] In a possible implementation, establishing the element graph network of the text data to be analyzed based on the position features and fusion features of each of the text elements includes:

[0092] Step 1: establishing node features of each node in the element graph network according to the fusion features of each of the text elements, wherein each of the text elements corresponds to a node in the element graph network;

[0093] Step 2: Establish edge features of each node in the element graph network based on the position features of each text element.

[0094] The fusion features of text elements are used as node features. According to the position features of text elements, the edges between nodes with related positions can be constructed. By using the graph convolutional neural network to perform convolution operations on edges and nodes at the same time and update them, the features of each level can be obtained, and finally the element graph network can be obtained.

[0095] S106: Determine the reading order of each of the text elements according to the element graph network.

[0096] The element graph network can be decoded using a decoder to obtain the reading order of each text element. The decoder used can include a graph network, a mathematical or topological method, a pointer network, etc. In an example, the output is a number representing the order, then Figure 2b As shown, the reading order of the first text element in the input element graph network is 2, the reading order of the second text element is 1, the reading order of the third text element is 3, the reading order of the fourth text element is 5, and the reading order of the fifth text element is 4, where BART (Bidirectional and Auto-Regressive Transformers) Decoder represents a decoder.

[0097] In addition to deep learning methods such as BART and graph networks, mathematical and graph theory methods can also be directly used, for example, various topological sorting methods or global path search methods, etc., all of which are within the scope of protection of this application. In one possible implementation, the reading order of each of the text elements is determined according to the element graph network, including: using a pre-trained decoder network to decode the element graph network to obtain the reading order of each of the text elements.

[0098] In one example, the sample element graph network and the true value of the reading order of each text element in the manually annotated sample element graph network are used as training samples to train the decoder network, thereby obtaining a pre-trained decoder network.

[0099] In the embodiments of the present application, the reading order of text elements can be determined, and the granularity of text elements includes: characters, sentences, paragraphs, layouts, etc., and the reading order of text elements in complex layout text data can be determined. Image features, position features and semantic features are used at the same time to construct an element graph network, which not only depends on the position of text elements, but also integrates image features and semantic features. Therefore, it can be applicable to various layouts such as complex multi-column, embedded, and surrounding shapes, which greatly improves the scope of application and performance. The present application can perform end-to-end optimization, does not require cumbersome post-processing and parameter adjustment, does not rely on too much manual intervention, can not only cover various granularities, but also can adapt to various layouts, and has high robustness and generalization capabilities.

[0100] In one possible implementation, see Figure 3 , the text data to be analyzed is in a portable document format PDF (Portable Document Format);

[0101] The acquiring of text elements from the text data to be analyzed to obtain the content and position of each text element in the text data to be analyzed includes:

[0102] S301, using a preset PDF parser to parse the text elements of the text data to be analyzed, and obtaining the content and position of each text element in the text data to be analyzed;

[0103] Before extracting image features from the text data to be analyzed to obtain image features of each text element in the text data to be analyzed, the method further includes:

[0104] S302: Convert the text data to be analyzed from PDF to image format.

[0105] For the scenario where the text data to be analyzed is a PDF, a preset PDF parser can be used to obtain the content and position of each text element in the text data to be analyzed.

[0106] In an embodiment of the present application, a method for determining the reading order of text elements in text data in PDF format is provided, and image features, position features and semantic features are used to construct an element graph network. It not only relies on the position of text elements, but also integrates image features and semantic features. Therefore, it can be applicable to various layouts such as complex multi-column, embedded, and surrounding shapes, thereby greatly improving the scope of application and performance.

[0107] In one possible implementation, see Figure 4 , the text data to be analyzed is in picture format;

[0108] The acquiring of text elements from the text data to be analyzed to obtain the content and position of each text element in the text data to be analyzed includes:

[0109] S401, using a preset OCR (Optical Character Recognition) algorithm to identify text elements in the text data to be analyzed, and obtaining the content and position of each text element in the text data to be analyzed.

[0110] For the text data to be analyzed in the image format, the OCR algorithm can be used to identify the content and position of the text elements in the image.

[0111] In an embodiment of the present application, a method for determining the reading order of text elements in text data in image format is provided, which can perform end-to-end optimization, does not require tedious post-processing and parameter adjustment, does not rely on too much manual intervention, can not only cover various granularities, but also can be adapted to various layouts, and has high robustness and generalization capabilities.

[0112] The present application also provides a device for determining the reading order of text elements. Figure 5 ,include:

[0113] The text data acquisition module 501 is used to acquire the text data to be analyzed, wherein the text data to be analyzed includes a plurality of text elements;

[0114] The semantic and position extraction module 502 is used to parse or extract features of the text elements in the text data to be analyzed, and obtain the position features and semantic features of each text element in the text data to be analyzed;

[0115] An image feature extraction module 503 is used to extract image features from the text data to be analyzed, and obtain image features of each text element in the text data to be analyzed;

[0116] The feature fusion module 504 is used to fuse the semantic features and image features of each text element to obtain a fusion feature of the text element;

[0117] An element graph network construction module 505 is used to establish an element graph network of the text data to be analyzed based on the position features and fusion features of each of the text elements;

[0118] The decoder module 506 is used to determine the reading order of each of the text elements according to the element graph network.

[0119] In a possible implementation, the semantics and position extraction module includes:

[0120] A text element identification submodule is used to obtain text elements from the text data to be analyzed, and obtain the content and position of each text element in the text data to be analyzed;

[0121] A semantic extraction submodule is used to determine the semantic features of each text element based on the content of the text element;

[0122] The position feature extraction submodule is used to obtain the position feature of each text element based on the position of the text element.

[0123] In a possible implementation, the text data to be analyzed is in a portable document format PDF;

[0124] The text element identification submodule is specifically used to: parse the text elements of the text data to be analyzed using a preset PDF parser to obtain the content and position of each text element in the text data to be analyzed;

[0125] The device also includes: a format conversion module, which is used to convert the text data to be analyzed from PDF to image format.

[0126] In a possible implementation manner, the text data to be analyzed is in a picture format;

[0127] The text element recognition submodule is specifically used to: use a preset optical character recognition (OCR) algorithm to recognize text elements in the text data to be analyzed, and obtain the content and position of each text element in the text data to be analyzed.

[0128] In a possible implementation, the image feature extraction module is specifically used to: input the text data to be analyzed into a pre-trained deep learning model, and extract image features of each text element in the text data to be analyzed from the feature output layer of the deep learning model.

[0129] In a possible implementation, the element graph network construction module is specifically used to: establish node features of each node in the element graph network based on fusion features of each of the text elements, wherein each of the text elements corresponds to a node in the element graph network; and establish edge features of each node in the element graph network based on position features of each of the text elements.

[0130] In a possible implementation, the decoder module is specifically used to decode the element graph network using a pre-trained decoder network to obtain a reading order of each of the text elements.

[0131] In a possible implementation, the feature fusion module is specifically used to: input the target application scenario features of the text data to be analyzed into a pre-trained feature fusion network, and for each text element, perform weighted fusion of the semantic features and image features of the text element through the feature fusion network to obtain the fusion features of the text element, wherein the target application scenario features include position features, semantic features and image features of the text elements in the text data to be analyzed, and wherein the weights used in the weighted fusion process of the feature fusion network are obtained by training sample application scenario features of sample text data under different application scenarios.

[0132] The embodiment of the present application also provides an electronic device, including: a processor and a memory;

[0133] The memory is used to store computer programs;

[0134] When the processor is used to execute the computer program stored in the memory, any method for determining the reading order of text elements described in the present application is implemented.

[0135] Optional, see Figure 6 The electronic device of the embodiment of the present application further includes a communication interface 602 and a communication bus 604 , wherein the processor 601 , the communication interface 602 , and the memory 603 communicate with each other via the communication bus 604 .

[0136] The communication bus mentioned in the above electronic device may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0137] The communication interface is used for communication between the above electronic device and other devices.

[0138] The memory may include RAM (Random Access Memory) or NVM (Non-Volatile Memory), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0139] The above-mentioned processor can be a general-purpose processor, including a CPU (Central Processing Unit), an NP (Network Processor), etc.; it can also be a DSP (Digital Signal Processing), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0140] The embodiment of the present application also provides a computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method for determining the reading order of text elements described in any one of the present application is implemented.

[0141] In another embodiment provided in the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any of the methods for determining the reading order of text elements described in the present application.

[0142] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk SolidState Disk (SSD)), etc.

[0143] It should be noted that, in this article, the technical features in each optional solution can be combined to form a solution as long as there is no contradiction, and these solutions are all within the scope disclosed in this application. Relational terms such as first and second, etc. are only used to distinguish an entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only include those elements, but also include other elements that are not clearly listed, or also include elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements limited by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements.

[0144] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the embodiments of the device, electronic device, and storage medium, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0145] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A method for determining the reading order of text elements, It is characterized in that include: Acquire text data to be analyzed, wherein the text data to be analyzed includes a plurality of text elements; Parsing or feature extracting text elements in the text data to be analyzed to obtain position features and semantic features of each text element in the text data to be analyzed; Extracting image features of the text data to be analyzed to obtain image features of each text element in the text data to be analyzed; For each text element, the semantic features and image features of the text element are fused to obtain the fused features of the text element; Based on the position features and fusion features of each of the text elements, an element graph network of the text data to be analyzed is established; Determining the reading order of each of the text elements according to the element graph network; The step of fusing the semantic features and image features of each text element to obtain the fusion features of the text element includes: The target application scenario features of the text data to be analyzed are input into a pre-trained feature fusion network, and for each text element, the semantic features and image features of the text element are weightedly fused through the feature fusion network to obtain the fusion features of the text element, wherein the target application scenario features include position features, semantic features and image features of the text elements in the text data to be analyzed, and the weights used in the weighted fusion process of the feature fusion network are obtained by training sample application scenario features of sample text data under different application scenarios.

2. The method according to claim 1, It is characterized in that The parsing or feature extraction of the text elements in the text data to be analyzed to obtain the position features and semantic features of each text element in the text data to be analyzed includes: Acquiring text elements from the text data to be analyzed, and obtaining the content and position of each text element in the text data to be analyzed; For each text element, the semantic feature of the text element is determined based on the content of the text element, and the position feature of the text element is obtained based on the position of the text element.

3. The method according to claim 2, It is characterized in that The text data to be analyzed is in portable document format PDF; The acquiring of text elements from the text data to be analyzed to obtain the content and position of each text element in the text data to be analyzed includes: Parsing the text elements of the text data to be analyzed by using a preset PDF parser to obtain the content and position of each text element in the text data to be analyzed; Before extracting image features from the text data to be analyzed to obtain image features of each text element in the text data to be analyzed, the method further includes: The text data to be analyzed is converted from PDF to image format.

4. The method according to claim 2, It is characterized in that The text data to be analyzed is in picture format; The acquiring of text elements from the text data to be analyzed to obtain the content and position of each text element in the text data to be analyzed includes: The text elements of the text data to be analyzed are recognized using a preset optical character recognition (OCR) algorithm to obtain the content and position of each text element in the text data to be analyzed.

5. The method according to claim 1, It is characterized in that The extracting of image features from the text data to be analyzed to obtain image features of each text element in the text data to be analyzed includes: The text data to be analyzed is input into a pre-trained deep learning model, and the image features of each text element in the text data to be analyzed are extracted from the feature output layer of the deep learning model.

6. The method according to claim 1, It is characterized in that The step of establishing an element graph network of the text data to be analyzed based on the position features and fusion features of each of the text elements includes: According to the fusion features of each of the text elements, node features of each node in the element graph network are established, wherein each of the text elements corresponds to a node in the element graph network; According to the position features of each of the text elements, edge features of each node in the element graph network are established.

7. The method according to claim 1, It is characterized in that Determining the reading order of each of the text elements according to the element graph network includes: The element graph network is decoded using a pre-trained decoder network to obtain a reading order of each of the text elements.

8. A device for determining the reading order of text elements, It is characterized in that include: A text data acquisition module, used to acquire text data to be analyzed, wherein the text data to be analyzed includes a plurality of text elements; A semantic and position extraction module, used for parsing or feature extraction of text elements in the text data to be analyzed, and obtaining position features and semantic features of each text element in the text data to be analyzed; An image feature extraction module is used to extract image features from the text data to be analyzed, and obtain image features of each text element in the text data to be analyzed; A feature fusion module is used to fuse the semantic features and image features of each text element to obtain a fusion feature of the text element; An element graph network construction module, used to establish an element graph network of the text data to be analyzed based on the position features and fusion features of each of the text elements; A decoder module, for determining a reading order of each of the text elements according to the element graph network; Among them, the feature fusion module is specifically used to input the target application scenario features of the text data to be analyzed into a pre-trained feature fusion network, and for each text element, the semantic features and image features of the text element are weightedly fused through the feature fusion network to obtain the fusion features of the text element, wherein the target application scenario features include the position features, semantic features and image features of the text elements in the text data to be analyzed, and the weights used in the weighted fusion process of the feature fusion network are obtained by training the sample application scenario features of sample text data under different application scenarios.

9. An electronic device, It is characterized in that including a processor and a memory; The memory is used to store computer programs; The processor is used to implement the method for determining the reading order of text elements described in any one of claims 1-7 when executing the program stored in the memory.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for determining the reading order of text elements according to any one of claims 1 to 7 is implemented.