Data processing method and apparatus, and document processing method and apparatus

Abstract

Provided in the embodiments of the present disclosure are a data processing method and apparatus, and a document processing method and apparatus. The data processing method comprises: determining from a visual object a plurality of visual layout elements and data representations of a plurality of pieces of visual data of the visual object; performing attention processing on the data representation of each piece of visual data and the data representations of the plurality of pieces of visual data, so as to obtain a context data representation of each piece of visual data; on the basis of the context data representation of each piece of visual data and an association relationship between each piece of visual data and the plurality of visual layout elements, determining an element representation of each visual layout element; and on the basis of the element representation of each visual layout element, determining, from among the plurality of visual layout elements, an associated visual layout element corresponding to each visual layout element. Therefore, element representations of visual layout elements including fine-grained context are obtained, such that the accuracy of predicting an association relationship between the visual layout elements is improved.

Description

Data processing methods and apparatus, document processing methods and apparatus

[0001] This disclosure claims priority to Chinese Patent Application No. 202411676218.9, filed with the China Patent Office on November 21, 2024, entitled “Data Processing Method and Apparatus, Document Processing Method and Apparatus”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of computer technology, and in particular to a data processing method and apparatus, and a document processing method and apparatus. Background Technology

[0003] Today, much information is stored in documents in pixel-based form (such as images and scanned PDFs, or Portable Document Format) rather than in hierarchical formats; however, structured formats are crucial for efficient database storage, standardized data processing, and downstream tasks. In particular, the structural information in hierarchical format files can further enhance the ability of large language models to handle long documents and knowledge-intensive tasks.

[0004] Document hierarchy parsing aims to reconstruct the hierarchical relationships between document layout elements, thereby converting PDF or image-based documents into semi-structured or structured documents that can be stored on computers, such as Markdown. Recent work has utilized pre-trained language models as encoders to enhance model performance. However, this model independently extracts the textual representation of each layout element, and its accuracy in predicting the hierarchical relationships between layout elements needs improvement. Summary of the Invention

[0005] In view of this, embodiments of this disclosure provide a data processing method and a document processing method. One or more embodiments of this disclosure also relate to a data processing apparatus, a document processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, to solve the technical defects in the prior art, such as low accuracy in predicting the hierarchical relationship between page elements due to the independent extraction of the text representation of each page element.

[0006] According to a first aspect of the present disclosure, a data processing method is provided, comprising:

[0007] The data representation of multiple visual layout elements in a visual object and multiple visual data of the visual object is determined, wherein the visual layout elements include element visual data, and the element visual data is the visual data corresponding to the visual layout elements determined from the multiple visual data according to the association relationship between the visual layout elements and each visual data.

[0008] Attention processing is performed on the data representations of each visual data and the data representations of the multiple visual data to obtain the contextual data representations of each visual data.

[0009] Based on the contextual data representation of each visual data and the association between each visual data and the plurality of visual layout elements, the element representation of each visual layout element is determined.

[0010] Based on the element representation of each visual layout element, the associated visual layout element corresponding to each visual layout element is determined from the plurality of visual layout elements, wherein the associated visual layout element corresponding to each visual layout element is a visual layout element that has a hierarchical relationship with each visual layout element.

[0011] According to a second aspect of the present disclosure, a document processing method is provided, comprising:

[0012] The data representation of multiple document layout elements and multiple text data in the target document is determined, wherein the document layout elements include element document data, and the element document data is the document data corresponding to the document layout elements determined from the multiple document data according to the association relationship between the document layout elements and each document data;

[0013] Attention processing is performed on the data representations of each document data and the data representations of the multiple document data to obtain the contextual data representations of each document data.

[0014] Based on the contextual data representation of each document data and the association between each document data and the multiple document layout elements, the element representation of each document layout element is determined;

[0015] Based on the element representation of each document layout element, the associated layout element corresponding to each document layout element is determined from the plurality of document layout elements, wherein the associated layout element corresponding to each document layout element is a document layout element that has a hierarchical relationship with each document layout element.

[0016] According to a third aspect of the present disclosure, a data processing apparatus is provided, comprising:

[0017] The data representation determination module is configured to determine the data representation of multiple visual layout elements in a visual object and multiple visual data of the visual object. The visual layout elements include element visual data, which is the visual data corresponding to the visual layout elements determined from the multiple visual data based on the association relationship between the visual layout elements and each visual data.

[0018] The data representation acquisition module is configured to perform attention processing on the data representations of each visual data and the data representations of the plurality of visual data to obtain the contextual data representations of each visual data.

[0019] The element representation determination module is configured to determine the element representation of each visual layout element based on the contextual data representation of each visual data and the association relationship between each visual data and the plurality of visual layout elements.

[0020] The associated element determination module is configured to determine the associated visual layout element corresponding to each visual layout element from the plurality of visual layout elements based on the element representation of each visual layout element, wherein the associated visual layout element corresponding to each visual layout element is a visual layout element that has a hierarchical relationship with each visual layout element.

[0021] According to a fourth aspect of the present disclosure, a document processing apparatus is provided, comprising:

[0022] The data representation determination module is configured to determine the data representation of multiple document layout elements in a target document and multiple document data of the target document, wherein the document layout elements include element document data, and the element document data is the document data corresponding to the document layout elements determined from the multiple document data according to the association relationship between the document layout elements and each document data;

[0023] The data representation acquisition module is configured to perform attention processing on the data representations of each document data and the data representations of the multiple document data to obtain the contextual data representations of each document data.

[0024] The element representation determination module is configured to determine the element representation of each document layout element based on the context data representation of each document data and the association relationship between each document data and the multiple document layout elements.

[0025] The associated element determination module is configured to determine the associated layout element corresponding to each document layout element from the plurality of document layout elements based on the element representation of each document layout element, wherein the associated layout element corresponding to each document layout element is a document layout element that has a hierarchical relationship with each document layout element.

[0026] According to a fifth aspect of the present disclosure, a computing device is provided, comprising:

[0027] Memory and processor;

[0028] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, they implement the steps of the above-mentioned data processing method and document processing method.

[0029] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided that stores a computer program / instructions, which, when executed by a processor, implement the steps of the above-described data processing method and document processing method.

[0030] According to a seventh aspect of the present disclosure, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method and document processing method.

[0031] This disclosure provides a data processing method in one embodiment. A visual object includes multiple visual layout elements, each composed of element visual data. This element visual data is visual data determined from the visual data and corresponding to the visual layout elements. Based on determining the multiple visual layout elements of the visual object and the data representations of the multiple visual data within the visual object, attention processing is performed on the data representations of each visual data and the data representations of the multiple visual data within the visual object to obtain fine-grained contextual data representations for each visual data. Based on the contextual data representations of the visual data corresponding to each visual layout element, the element representation of that visual layout element is obtained, thereby obtaining the element representations of each visual layout element. The element representations of each visual layout element include fine-grained contextual data representations. Based on the element representations of each visual layout element, from the multiple visual layout elements of the visual object, it is possible to determine the associated visual layout elements corresponding to each visual layout element and having a hierarchical relationship with the visual layout elements. By simultaneously considering the relationship between fine-grained data representations and coarse-grained visual layout elements, this method solves the problem of previous methods neglecting fine-grained data representations, improving the accuracy of predicting the hierarchical relationship between visual layout elements, i.e., improving the accuracy of predicting the associated layout elements corresponding to visual layout elements. Attached Figure Description

[0032] Figure 1 is a schematic diagram of a data processing method provided in an embodiment of this disclosure;

[0033] Figure 2 is a flowchart of a data processing method provided in an embodiment of this disclosure;

[0034] Figure 3 is a flowchart of a document processing method provided in an embodiment of this disclosure;

[0035] Figure 4a is a schematic diagram of the processing process of a document processing method applied to a multi-page document scenario according to an embodiment of this disclosure;

[0036] Figure 4b is a schematic diagram illustrating the relationship between document layout elements according to an embodiment of this disclosure;

[0037] Figure 5 is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of the present disclosure;

[0038] Figure 6 is a schematic diagram of the structure of a document processing apparatus provided in an embodiment of this disclosure;

[0039] Figure 7 is a structural block diagram of a computing device provided in an embodiment of this disclosure. Detailed Implementation

[0040] Numerous specific details are set forth in the following description to provide a full understanding of this disclosure. However, this disclosure can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this disclosure. Therefore, this disclosure is not limited to the specific implementations disclosed below.

[0041] The terminology used in one or more embodiments of this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this disclosure. The singular forms “a,” “the,” and “the” as used in one or more embodiments of this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this disclosure refers to and includes any or all possible combinations of one or more associated listed items.

[0042] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this disclosure, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this disclosure, and similarly, second may also be referred to as first. Depending on the context, the word “if” as used herein may be interpreted as “when”, “in response to a determination”, or “when…”.

[0043] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this disclosure are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0044] First, the terms and concepts involved in one or more embodiments of this disclosure will be explained.

[0045] Document layout elements: Document layout elements refer to the various visual and structural components that make up a document. These components collectively define the visual layout and structural organization of the document, such as headings, paragraphs, charts, and tables.

[0046] Document hierarchy parsing aims to reconstruct the hierarchical relationships between document layout elements, thereby converting PDF or image-formatted documents into semi-structured documents that can be stored on a computer, such as Markdown.

[0047] Text-Layout Language Model: This language model is designed for visual rich documents. It represents the two-dimensional position of text on the document based on the ordinary language model, which improves the ability of the language model to perform tasks such as document understanding and document extraction.

[0048] This disclosure provides a data processing method and a document processing method. This disclosure also relates to a data processing apparatus, a document processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0049] Referring to Figure 1, Figure 1 shows a scenario diagram of a data processing method provided according to an embodiment of the present disclosure.

[0050] Specifically, this data processing method is implemented using the application-side device 102 and the server 104. In the case of applying the data processing method to a document parsing scenario, the data processing method is described in detail.

[0051] The endpoint device 102 is used to send the target document to the server 104. For example, in the user interface of the endpoint device 102, the user can upload the target document that needs to be parsed by operating the "upload document" control.

[0052] In server 104, multiple document layout elements and multiple document data representations of the target document are determined. Each document layout element includes element document data, which is the document data corresponding to the document layout element determined from the multiple document data based on the association between the document layout element and each document data. Attention processing is performed on the data representations of each document data and the data representations of the multiple document data to obtain the contextual data representation of each document data. Based on the contextual data representations of each document data and the association between each document data and the multiple document layout elements, the element representation of each document layout element is determined. Based on the element representations of each document layout element, the associated layout elements corresponding to each document layout element are determined from the multiple document layout elements. The associated layout elements corresponding to each document layout element are document layout elements with a hierarchical relationship to each document layout element. Based on each document layout element and its associated layout elements, a semi-structured, editable document processing result can be output. This document processing result is then returned to the end device 102.

[0053] The edge device 102 may include a browser, an app (application), or a web application such as an H5 (Hypertext Markup Language 5) application, a lightweight application (also known as a mini-program), or a cloud application. The edge device may be developed based on a software development kit (SDK) provided by the server, such as a real-time communication (RTC) SDK. The edge device can be deployed in an electronic device and depends on the device's operation or certain apps within the device to run. The electronic device may have a display screen and support information browsing, such as a personal mobile terminal like a mobile phone, tablet, or personal computer. Various other types of applications can also be configured in the electronic device, such as human-computer interaction applications, model training applications, data processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, and social media platform software.

[0054] Server 104 can be understood as a server providing various services, including physical servers and cloud servers. Examples include servers providing communication services to multiple clients, servers supporting backend training of models used on clients, and servers processing data sent by clients. It's important to note that Server 104 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. Server 104 can also be a server in a distributed system, or a server integrated with blockchain. Server 104 can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0055] The data processing method provided in this disclosure includes a visual object comprising multiple visual layout elements, each composed of element visual data. This element visual data is determined from the visual data and corresponds to the visual layout element. Based on the determination of multiple visual layout elements of the visual object and the data representations of multiple visual data within the visual object, attention processing is performed on the data representations of each visual data and the data representations of multiple visual data within the visual object to obtain fine-grained contextual data representations of each visual data. Based on the contextual data representations of the visual data corresponding to each visual layout element, the element representation of that visual layout element is obtained, thereby obtaining the element representations of each visual layout element. The element representations of each visual layout element include fine-grained contextual data representations. Based on the element representations of each visual layout element, the associated visual layout elements corresponding to each visual layout element and having a hierarchical relationship with the visual layout element can be determined from the multiple visual layout elements of the visual object. By simultaneously considering the relationship between fine-grained data representations and coarse-grained visual layout elements, the problem of previous methods neglecting fine-grained data representations is solved, improving the accuracy of predicting the hierarchical relationship between visual layout elements, i.e., improving the accuracy of predicting the associated layout elements corresponding to visual layout elements.

[0056] Referring to Figure 2, which shows a flowchart of a data processing method provided in an embodiment of the present disclosure, the method specifically includes the following steps.

[0057] Step 202: Determine the data representation of multiple visual layout elements in the visual object and multiple visual data of the visual object, wherein the visual layout elements include element visual data, and the element visual data is the visual data corresponding to the visual layout elements determined from the multiple visual data according to the association relationship between the visual layout elements and each visual data.

[0058] Visual objects can be understood as a series of objects displayed in pixel form, such as documents and images. Documents can be files containing information such as text, images, and tables, such as Word documents and PDF files, without any restrictions.

[0059] When the visual object is a document, visual layout elements can be understood as document layout elements, such as headings, paragraphs, images, and tables. When visual layout elements are document layout elements, element visual data includes, but is not limited to, text content and text position. It should be noted that for image-type visual layout elements, their text content is represented by placeholders, such as the text content being "image information".

[0060] When the visual object is an image, the visual layout element can be understood as the entity identified in the image. For example, when the image is "an image of a puppy biting a tennis ball", the visual layout element can be the puppy and the tennis ball. The corresponding element visual data can include, but is not limited to, the visual data such as color, texture, and position that make up the visual layout element.

[0061] Specifically, taking visual objects as documents as an example, the data processing method provided in this disclosure embodiment will be described in detail.

[0062] In practical applications, when a user uploads a document through the client's user interface, the text detection and recognition model can first identify multiple document layout elements and extract the text content. Based on this, the identified text content is converted into computer-processable text tokens (text tokens are the basic units of text data, which can be further encoded and processed), thereby determining the data representation of multiple visual data of the visual object. That is, multiple text token representations are obtained through the encoding and processing of text tokens.

[0063] In one or more embodiments of this disclosure, since multi-page documents with extremely long text exist in practical applications, and the text length often exceeds the encoder's input limit in multi-page input scenarios, it is necessary to segment the text content and perform data processing on each visual object obtained after segmentation. Specific implementation methods are described below:

[0064] Before determining the data representation of multiple visual layout elements in the visual object and multiple visual data of the visual object, the method further includes:

[0065] Determine multiple initial visual layout elements in the initial visual object, and arrange the multiple initial visual layout elements according to a preset arrangement method;

[0066] The multiple initial visual layout elements are divided according to a preset division length to determine multiple candidate visual objects, and the multiple candidate visual objects are sequentially determined as the visual objects. Each candidate visual object includes multiple visual layout elements, which are obtained from the multiple initial visual layout elements after division.

[0067] The initial visual object can be understood as a multi-page document of long text, and the initial visual layout elements are all the visual layout elements in the multi-page document, that is, all the document layout elements in the multi-page document. The preset arrangement method can be understood as the reading order, such as from top to bottom or from left to right. Of course, for multiple initial visual layout elements with complex layouts, it is not limited to the order from top to bottom or from left to right, but the arrangement order of multiple initial visual layout elements is determined according to the actual and more reasonable reading order.

[0068] The preset division length can be set according to the actual situation (such as the input limit of the encoder), and there is no limit here.

[0069] Specifically, this disclosure provides a data processing method that uses an encoder to obtain contextual data representations of each visual data. Therefore, to avoid the text length of a multi-page document exceeding the encoder's input limit, multiple initial visual layout elements in the initial visual object are divided. This can be understood as grouping (or blocking) multiple initial visual layout elements. Each group obtained can be identified as a candidate visual object, and the initial visual layout elements within that group are visual layout elements. In other words, a candidate visual object includes multiple visual layout elements. Since the subsequent encoder's role is to obtain contextual data representations of the visual data, to ensure that the contextual data representations of the visual data contain contextual information related to the visual data, multiple document layout elements in the document can be arranged reasonably according to the reading order. Thus, when dividing multiple document layout elements according to a preset division length, the multiple document layout elements in the candidate visual objects obtained after division are related multiple document layout elements.

[0070] Of course, in practical applications, when dividing multiple document layout elements according to the preset division length, the division is carried out on a per-document layout element basis. That is, the text content of a document layout element should not be divided into different candidate visual objects. In other words, the text length in the candidate visual object should not exceed the preset division length to ensure that the integrity of each document layout element is not compromised.

[0071] Each candidate visual object may include one or more pages of document. For each candidate visual object, the element representation of each visual layout element is obtained through the data processing method provided in the embodiments of this disclosure, thereby obtaining the element representation of each visual layout element of the initial visual object. Based on the element representation of each visual layout element of the initial visual object, the associated visual layout element corresponding to each visual layout element in the initial visual object is determined.

[0072] The data processing method provided in this disclosure can be applied to multi-page document input scenarios with long text. By dividing the multi-page document, for each candidate visual object after division, the element representation of each visual layout element containing fine-grained text content is determined, thereby obtaining the element representation of each visual layout element of the initial visual object. This solves the multi-page problem in document hierarchy parsing and improves the generalization of document hierarchy parsing.

[0073] In one or more embodiments of this disclosure, by encoding the text content and its two-dimensional position within a document, the original layout and positional information of the text within the document can be preserved, thereby more accurately capturing the contextual relationship between visual data and other visual data. Specific implementation methods are described below:

[0074] The data representation of determining multiple visual layout elements in a visual object and multiple visual data of the visual object includes:

[0075] Determine multiple visual layout elements, multiple visual data, and the two-dimensional position of each visual layout element in the visual object;

[0076] Based on the relationship between each visual data and the plurality of visual layout elements, determine the element visual data corresponding to each visual layout element from the plurality of visual data;

[0077] The two-dimensional position of each visual layout element is determined as the two-dimensional position of the element visual data corresponding to each visual layout element.

[0078] Based on the two-dimensional position of the element visual data corresponding to each visual layout element, determine the two-dimensional position of multiple visual data of the visual object;

[0079] Encode multiple visual data of the visual object and the two-dimensional positions of the multiple visual data of the visual object to obtain a data representation of the multiple visual data of the visual object.

[0080] As can be seen from the above embodiments, the document can first be processed by a text detection and recognition model. Therefore, the two-dimensional position of each document layout element in the corresponding page of the document can be determined by the text detection and recognition model. Of course, the embodiments of this disclosure are not limited to the method of determining the two-dimensional position of visual layout elements. As long as the two-dimensional position of each visual layout element in the corresponding page of the document can be obtained, it is acceptable.

[0081] The two-dimensional position of each visual layout element is determined as the two-dimensional position of the visual data of the element associated with it. That is, the two-dimensional position of the visual data in each visual object is the two-dimensional position of the visual layout element to which the visual data belongs. By encoding the multiple visual data of the visual object and the two-dimensional positions of the multiple visual data of the visual object, the data representation of the multiple visual data is obtained.

[0082] Specifically, encoding multiple visual data of the visual object and the two-dimensional positions of the multiple visual data of the visual object to obtain a data representation of the multiple visual data of the visual object includes:

[0083] For any one of the plurality of visual data, the arbitrary visual data and its two-dimensional position are encoded to obtain the data representation of the arbitrary visual data.

[0084] Based on the data representation of any one of the visual data, data representations of multiple visual data of the visual object are obtained.

[0085] In practical applications, a visual object includes multiple visual layout elements. The visual data of each visual layout element is determined from the multiple visual data of the visual object. Therefore, the multiple visual data of the visual object can be determined based on the multiple visual data of each visual layout element. By encoding any visual data and the two-dimensional position of any visual data, the data representation of any visual data is obtained. Based on the data representation of any visual data, the data representation of each visual data can be determined, that is, the data representation of the multiple visual data of the visual object is obtained.

[0086] In the case of a visual object being a document, each text token is a comprehensive encoding of the text content and its two-dimensional location.

[0087] The data processing method provided in this disclosure obtains the data representation of each visual data by comprehensively encoding the visual data and the two-dimensional position of the visual data. It can preserve the original layout and position information of the text in the document, thereby more accurately capturing the contextual relationship between the visual data and other visual data.

[0088] In one or more embodiments of this disclosure, when the visual layout elements include multiple levels, the visual representation of the visual data may further include the layout position encoding of the visual data within the corresponding visual layout elements. Specific implementation methods are described below:

[0089] The data representation for determining multiple visual data of the visual object includes:

[0090] Determine the initial data representation of multiple visual data of the visual object;

[0091] Determine the relative position of each visual data in the visual layout element corresponding to each visual data, and determine the layout position code of each visual data based on the relative position;

[0092] Based on the initial data representation of each visual data and the layout position code, the data representation of multiple visual data of the visual object is determined.

[0093] The initial data representation can be understood as the data representation obtained by comprehensively encoding the visual data and the two-dimensional position of the visual data in the above embodiments; the layout position encoding can be understood as the representation obtained by encoding the relative position of the visual data in its corresponding visual layout elements.

[0094] By encoding the relative position of visual data within its corresponding visual layout elements, the layout position code of each visual data is determined. Through the initial data representation of the visual data and the layout position code of the visual data, the data representation of the visual data is determined. Based on this visual representation, attention processing is performed on the visual data and other visual data.

[0095] In practical applications, in multi-page document scenarios, the data representation of visual data is determined through initial visual data, layout position encoding, and page position encoding. The specific implementation method is as follows:

[0096] The data representation for determining multiple visual data of the visual object includes:

[0097] Determine the initial data representation of multiple visual data of the visual object;

[0098] Determine the page position code and layout position code corresponding to each visual data;

[0099] Based on the initial data representation of each visual data, the page position code, and the layout position code, the data representation of multiple visual data of the visual object is determined.

[0100] Among them, page location encoding is a representation obtained by encoding the location of the visual data on the page.

[0101] Specifically, the page number corresponding to the visual data is input into a linear layer. Within this linear layer, the page number corresponding to the visual data is encoded to obtain the page position code of the visual data. For any visual data, the data representation of any visual data is determined by comprehensively encoding the initial data representation, page position code, and layout position code of the obtained visual data. That is, the data representation of any visual data includes the data content information of the visual data (the initial data representation includes the data content information of the visual data), the page information (the page position code includes the page information of the visual data), and the position information in the layout structure (the layout position code includes the position information of the visual data in the layout structure).

[0102] The data processing method provided in this disclosure, in order to solve the multi-page and multi-level problems in document hierarchy parsing, after encoding the visual data and the two-dimensional position of the visual data, can also add additional position encoding, such as page position encoding and layout position encoding, to the obtained encoded representation, thereby obtaining a data representation containing richer position information.

[0103] Step 204: Perform attention processing on the data representations of each visual data and the data representations of the multiple visual data to obtain the contextual data representations of each visual data.

[0104] Specifically, attention processing is performed on the data representations of each visual data to obtain contextual data representations of each visual data that contain fine-grained contextual information.

[0105] In one or more embodiments of this disclosure, the contextual data representation of each visual data is determined by sequentially identifying each visual data as the target visual data to be processed, thereby obtaining the contextual data representation of the target visual data. Specific implementation methods are as follows:

[0106] The step of performing attention processing on the data representations of each visual data and the data representations of the multiple visual data to obtain the contextual data representations of each visual data includes:

[0107] Select any one of the plurality of visual data as the target visual data;

[0108] Attention processing is performed on the data representation of the target visual data and the data representations of the multiple visual data to obtain the contextual data representation of the target visual data.

[0109] The contextual data representation of each visual data is obtained based on the contextual data representation of the target visual data.

[0110] In this context, target visual data can be understood as any one of multiple visual data currently being processed.

[0111] Attention processing is performed on the data representation of the target visual data being processed and the data representations of multiple visual data in the visual object to obtain the contextual data representation of the target visual data being processed; thus, given that the contextual data representation of any visual data can be obtained, the contextual data representation of each visual data can be obtained.

[0112] Specifically, through an attention mechanism, multiple attention weights are determined for the data representation of the target visual data and the data representations of multiple visual data. Then, a weighted summation of the data representations of the visual data based on these attention weights is performed to obtain the contextual data representation of the target visual data. The specific implementation method is as follows:

[0113] The step of performing attention processing on the data representation of the target visual data and the data representations of the plurality of visual data to obtain the contextual data representation of the target visual data includes:

[0114] Attention processing is performed on the data representation of the target visual data and the data representations of the multiple visual data to determine multiple attention weights of the data representation of the target visual data and the data representations of the multiple visual data.

[0115] By using the multiple attention weights and weighted summation of the data representations of the multiple visual data, the contextual data representation of the target visual data is obtained.

[0116] Among them, attention weight can be used to represent the degree of association between the target visual data and multiple visual data; contextual data representation can be understood as a data representation that includes contextual information between the target visual data and multiple visual data.

[0117] Specifically, attention processing is performed on the data representation of the target visual data and the data representation of multiple visual data to measure the degree of correlation between the target visual data and multiple visual data, and multiple attention weights are determined accordingly; these attention weights actually reflect the relative importance or correlation between the target visual data and each visual data.

[0118] By using multiple attention weights to perform a weighted summation of the data representations of multiple visual data, a contextual data representation corresponding to the target visual data is obtained, which integrates the contextual information between the target visual data and multiple visual data.

[0119] The data processing method provided in this disclosure effectively captures the correlation between target visual data and multiple visual data through an attention mechanism, and generates a contextual data representation containing rich contextual information. This contextual data representation not only includes the features of the target visual data itself, but also incorporates the features of multiple related visual data, thereby providing richer information and a more comprehensive perspective.

[0120] Step 206: Determine the element representation of each visual layout element based on the contextual data representation of each visual data and the association between each visual data and the plurality of visual layout elements.

[0121] Specifically, since visual layout elements are composed of element visual data, the element representation of a visual layout element is determined based on the context data representation of the element visual data belonging to the same visual layout element, after obtaining the context data representation of each visual data.

[0122] The step of determining the element representation of each visual layout element based on the contextual data representation of each visual data and the association relationship between each visual data and the plurality of visual layout elements includes:

[0123] Based on the relationship between the visual data and the multiple visual layout elements, determine the multiple element visual data corresponding to each visual layout element;

[0124] Based on the contextual data representation of each visual data, determine the contextual data representation of the element visual data corresponding to each visual layout element.

[0125] The context data representation of the element visual data corresponding to each visual layout element is pooled to obtain the layout element representation of each visual layout element.

[0126] Since the element visual data corresponding to the visual layout element is obtained by dividing multiple visual data, that is, the element visual data is the visual data of the visual layout element to which it belongs, and the content has not changed, the context data representation of the element visual data in each visual layout element can be obtained by obtaining the context data representation of each visual data.

[0127] Specifically, the association between each visual data and the plurality of visual layout elements can be determined based on the two-dimensional position of the visual data. In the above embodiment, the two-dimensional position of the visual layout element is determined as the two-dimensional position of the visual layout element corresponding to the plurality of visual data. Therefore, visual data at the same two-dimensional position can be determined as the element visual data of the visual layout element at that two-dimensional position. Of course, the method of determining the association between each visual data and the plurality of visual layout elements is not limited in the embodiments of this disclosure.

[0128] In practice, taking a visual layout element as an example, the element visual data of the visual layout element is determined, and the context data representation of the element visual data corresponding to the visual layout element is pooled to obtain the element representation of the visual layout element.

[0129] In practical applications, taking visual layout elements as document layout elements as an example, when the document layout element is a paragraph, multiple text token representations within the paragraph are pooled to obtain the representation corresponding to the paragraph.

[0130] The data processing method provided in this disclosure determines the contextual data representation of multiple visual data corresponding to each visual layout element, and performs pooling processing on the contextual data representation of the element visual data corresponding to each visual layout element to obtain the element representation of each visual layout element. The element representation contains fine-grained context between visual data, thereby improving the accuracy of predicting the hierarchical relationship between visual layout elements.

[0131] Step 208: Based on the element representation of each visual layout element, determine the associated visual layout element corresponding to each visual layout element from the plurality of visual layout elements, wherein the associated visual layout element corresponding to each visual layout element is a visual layout element that has a hierarchical relationship with each visual layout element.

[0132] Specifically, after obtaining the element representation of each visual layout element, the element representation of each visual layout element can be input into the decoder to obtain the context element representation of each visual layout element. The relationship prediction network is then used to predict the relationship between the context element representation of each visual layout element, thereby determining the associated visual layout elements corresponding to each visual layout element.

[0133] In one or more embodiments of this disclosure, the element representations of multiple visual layout elements can be arranged according to the preset arrangement of the above embodiments. Based on the sequentially arranged element representations of the multiple visual layout elements, the associated visual layout elements corresponding to each visual layout element can be determined more accurately. Specific implementation methods are as follows:

[0134] The step of determining the associated visual layout element corresponding to each visual layout element from the plurality of visual layout elements based on the element representation of each visual layout element includes:

[0135] Determine the arrangement order of the plurality of visual layout elements;

[0136] Based on the arrangement order of the multiple visual layout elements, the element representations of each visual layout element are arranged to obtain the element representations of the multiple visual layout elements arranged in sequence.

[0137] Based on the element representations of the multiple visual layout elements arranged in the order, the associated visual layout elements corresponding to each visual layout element are determined from the multiple visual layout elements.

[0138] Among them, the associated visual layout elements corresponding to the visual layout elements can be understood as the visual layout elements at the parent node of the visual layout elements; the data processing method provided in this embodiment can determine the hierarchical relationship between each visual layout element through the element representation of multiple visual layout elements. The hierarchical relationship reflects the logic and organizational structure between visual layout elements. The associated visual layout elements of each visual layout element are determined through the parent node in the hierarchical relationship of each visual layout element.

[0139] Specifically, the arrangement order of multiple visual layout elements can be determined according to the above-mentioned preset arrangement method. This preset arrangement method can be a reading order, which is usually the order of reading the title and then reading the paragraphs under the title. Therefore, by arranging the element representations of each visual layout element according to the arrangement order of multiple visual layout elements, a set of element representations of multiple visual layout elements in sequence can be obtained. In this set, the element representations of each visual layout element are arranged according to their position in the reading order. The element representations above and below each visual layout element are usually element representations that are closely related to its content and have a certain hierarchical relationship.

[0140] Based on the element representations of multiple visual layout elements arranged in sequence, determine the associated visual layout elements corresponding to each visual layout element from among the multiple visual layout elements.

[0141] In practical applications, since some visual layout elements (such as the document title) use the entire document as their parent node, an additional learnable representation, the "global element feature," is used to determine the representation of the entire document. This allows visual layout elements whose parent node is the entire document to accurately identify the entire document as their parent node, avoiding the mistaken assumption that other visual layout elements are their parent nodes. The specific implementation method is as follows:

[0142] Before determining the associated visual layout element corresponding to each visual layout element from the plurality of visual layout elements arranged in the specified order, the method further includes:

[0143] Determine the global element features, and determine the arrangement order among the plurality of visual layout elements and the global element features;

[0144] Based on the arrangement order of the plurality of visual layout elements and the global element features, the element representations of each visual layout element and the global element features are arranged to obtain the sequentially arranged global element features and element representations of the plurality of visual layout elements.

[0145] The element representation of the plurality of visual layout elements arranged in the specified order, and the determination of the associated visual layout element corresponding to each visual layout element from the plurality of visual layout elements, include:

[0146] Based on the global element features arranged in the order and the element representations of the plurality of visual layout elements, the associated visual layout element corresponding to each visual layout element is determined from the plurality of visual layout elements.

[0147] Global element features can be understood as an additional learnable representation used to capture the global features of multiple visual layout elements.

[0148] The global element feature can be concatenated before the element features of multiple visual layout elements, and the multiple visual layout elements can be arranged according to their order to obtain the sequentially arranged global element features and the element representations of the multiple visual layout elements. Based on the sequentially arranged global element features and the element representations of the multiple visual layout elements, the associated visual layout elements corresponding to each visual layout element can be determined from the multiple visual layout elements. In this embodiment, the global element feature is placed before the element representations of the sequentially arranged multiple visual layout elements, which facilitates the extraction of global features.

[0149] The data processing method provided in this disclosure, through an additional learnable representation, can accurately predict the parent node of visual layout elements that take the entire document as their parent node, effectively avoiding the use of other visual layout elements as the parent node of such visual layout elements, and improving the accuracy and comprehensiveness of predicting the relationship between visual layout elements.

[0150] In one or more embodiments of this disclosure, the element representations of each visual layout element in a plurality of sequentially arranged visual layout elements are subjected to attention processing to obtain the contextual element representations of each visual layout element. Then, using the contextual element representations of each visual layout element, the associated visual layout elements corresponding to each visual layout element are determined. Specific implementation methods are as follows:

[0151] The element representation of the plurality of visual layout elements arranged in the specified order, and the determination of the associated visual layout element corresponding to each visual layout element from the plurality of visual layout elements, include:

[0152] Attention processing is performed on the element representations of each visual layout element in the ordered multiple visual layout elements, and on the element representations of the ordered multiple visual layout elements, to obtain the context element representations of each visual layout element.

[0153] Based on the context element representation of each visual layout element, the association relationship between each visual layout element is predicted, and the associated visual layout element corresponding to each visual layout element is determined.

[0154] Specifically, when performing attention processing on the element representations of multiple sequentially arranged visual layout elements to obtain the corresponding contextual element representations, it is possible to more accurately capture the correlation and interdependence between visual layout elements. Thus, based on the contextual element representations of each visual layout element and the prediction of the correlation between each visual layout element, more accurate prediction results can be obtained. That is, more accurate correlation between each visual layout element can be obtained, and based on the more accurate correlation between each visual layout element, the associated visual layout elements corresponding to each visual layout element can be determined.

[0155] Furthermore, when the visual object is obtained by dividing the initial visual object, attention processing is performed on the element representations of multiple visual layout elements of the initial visual object. By performing attention processing on the element representations of multiple sequentially arranged visual layout elements, global information of the initial visual object can be obtained.

[0156] The data processing method provided in this disclosure captures collective information from higher levels and global context, optimizes element representations globally, obtains context element representations, and further breaks down barriers between different visual objects, thereby enabling more accurate prediction results when the context element representations are input into a relationship prediction network.

[0157] In one or more embodiments of this disclosure, after determining the associated visual layout elements corresponding to each visual layout element, various visual data processing tasks can be performed by utilizing the relationships between the visual layout elements. Specific implementation methods are described below:

[0158] After determining the associated visual layout elements corresponding to each visual layout element, the process further includes:

[0159] Based on the associated visual layout elements corresponding to each visual layout element, perform visual data processing tasks.

[0160] The visual data processing tasks include, but are not limited to, converting visual objects into visual objects of a target format and generating a directory structure for the target objects, etc., which are not limited here.

[0161] For example, when the target format visual object is a semi-structured, editable document, and the visual object is a PDF file, by determining the associated page elements corresponding to each document page element, the document page elements in the PDF file can be displayed in Word according to the hierarchical relationship of each document page element.

[0162] Furthermore, this data processing method can help perform structured parsing of massive documents and can be applied to various downstream tasks such as extraction, retrieval, and question answering, without being limited to these specific tasks.

[0163] This disclosure provides a data processing method in one embodiment that, by determining the associated visual layout elements corresponding to each visual layout element, can further output structured documents. This solves the problem of efficient database storage where a large number of documents exist in pixel form and cannot be formatted into a structured format. By simultaneously considering the relationship between fine-grained data representation and coarse-grained visual layout elements, it solves the problem of previous methods neglecting fine-grained data representation and can improve the accuracy of predicting the associated layout elements corresponding to visual layout elements.

[0164] Referring to Figure 3, Figure 3 shows a flowchart of a document processing method provided in an embodiment of the present disclosure, which specifically includes the following steps.

[0165] Step 302: Determine the data representation of multiple document layout elements in the target document and multiple document data of the target document, wherein the document layout elements include element document data, and the element document data is the document data corresponding to the document layout elements determined from the multiple document data according to the association relationship between the document layout elements and each document data.

[0166] In this context, the target document can be understood as the visual object in the above embodiments, the document layout elements can be understood as the visual layout elements in the above embodiments, and the document data can be understood as the visual data in the above embodiments.

[0167] Step 304: Perform attention processing on the data representations of each document data and the data representations of the multiple document data to obtain the contextual data representations of each document data.

[0168] Step 306: Determine the element representation of each document layout element based on the contextual data representation of each document data and the association relationship between each document data and the multiple document layout elements.

[0169] Step 308: Based on the element representation of each document layout element, determine the associated layout element corresponding to each document layout element from the plurality of document layout elements, wherein the associated layout element corresponding to each document layout element is a document layout element that has a hierarchical relationship with each document layout element.

[0170] For specific implementation details, please refer to the above embodiments, which will not be repeated here.

[0171] The document processing method provided in this disclosure includes a target document comprising multiple document layout elements, each composed of element document data. The method determines the data representations of the multiple document layout elements and the multiple document data within the target document. By performing attention processing on the data representations of each document data and the data representations of the multiple document data within the target document, a fine-grained contextual data representation of each document data is obtained. Based on the contextual data representation of the element document data corresponding to each document layout element, the element representation of that document layout element is obtained, thereby obtaining the element representations of each document layout element. The element representations of each document layout element include fine-grained contextual data representations. Based on the element representations of each document layout element, the associated document layout elements corresponding to each document layout element are determined from the multiple document layout elements of the target document. By simultaneously considering the relationships between fine-grained data representations and coarse-grained document layout elements, this method solves the problem of previous methods neglecting fine-grained data representations, and can improve the accuracy of predicting the associated layout elements corresponding to document layout elements.

[0172] Figure 4a is a schematic diagram of the processing process of a document processing method applied to a multi-page document scenario according to an embodiment of this disclosure.

[0173] The overall process of the document processing method is as follows: The entire document, including the text and its two-dimensional positions, is input into the sparse text-layout encoder Esp, creating a fine-grained contextual representation for each input text. Then, through pooling operations, these fine-grained contextual representations are integrated into layout element representations of document layout elements, which are then input into a global decoder D at the document layout element granularity. The global decoder D can capture collective information from higher levels and the global context. Finally, the layout element representations output by the global decoder D (which can be understood as contextual layout element representations containing contextual information) are input into the relation prediction network to obtain the prediction result, which is the hierarchical relationship between document layout elements.

[0174] In one embodiment of this disclosure, a document processing method is described in detail using a multi-page document as an example.

[0175] This disclosure employs a sparse text-layout encoder Esp (hereinafter referred to as the encoder), which is derived from a powerful pre-trained text-layout language model that can generate layout element representations with rich fine-grained context.

[0176] Specifically, when the target document is a multi-page document (i.e., the initial visual object in the above embodiment), multiple document layout elements in the multi-page document are determined, and these multiple document layout elements are extracted in the reading order. For example, the multiple document layout elements are M (determined according to the actual situation), and E = {E1, E2, ..., EM}. In multi-page input, the text length usually exceeds the encoder's input limit L.

[0177] Therefore, a block-based sparse attention mechanism is adopted to maintain dense attention within blocks, thereby better utilizing the text-layout language model pre-trained on a single page. The multi-page document is decomposed into K blocks {C1,...,CK} (each block can be understood as candidate visual data in the above embodiments); each block contains the maximum number of document layout elements it can contain. For example, in Figure 4a, the first block includes layout elements 1-3, and the second block includes layout elements 4-j, ensuring that the total length of the text within each block does not exceed L and guaranteeing the integrity of the document layout elements.

[0178] Sparse attention is factored into dense attention within blocks, that is, attention processing is performed on the text representation within each block:

[0179] Where X is the text representation in the encoder (in practical applications, the encoder contains multiple layers of concatenated attention mechanisms, and the output of each layer serves as the input to the next layer; X can be understood as the text representation input to the attention mechanism in the encoder), xi It represents each text token in a multi-page document. It is x i The block it belongs to, softmax, refers to the softmax function, an activation function that can transform an unnormalized real vector (usually the model's raw output) into a normalized probability distribution.

[0180] and

[0181] Among them, W q W k and W v The weight matrix is ​​represented by d, which is the hidden layer size of the model. This attention method enriches the fine-grained context between text tokens while maintaining the controllability of computational cost.

[0182] To address the issues of multiple pages and multiple levels in document hierarchy parsing, two types of position encoding are added: page position encoding and internal page position encoding (i.e., page position encoding in the above embodiment).

[0183] The page location code is used to represent the page location of the input text token, and it is calculated as follows: e pg =Linear(sinPE(pn) i )), pn i Let be the page number of the i-th input text token, sinPE be the sine positional encoding function, and Linear be a linear layer.

[0184] The method for calculating the internal position code of the page is: e in =PosEmb1D(rp i ), rp i PosEmb1D is a one-dimensional positional encoding function, representing the relative position of the i-th input text token within its corresponding page element.

[0185] In summary, the representation of the i-th input text token is calculated as follows: Among them, t i It is the encoder's original text-layout representation (i.e., the initial data representation in the above embodiments); each layout element includes the representation of multiple text tokens within that layout element, for example, layout element 4 in Figure 4a includes the text token representations corresponding to ti to ti+3.

[0186] For each document layout element Ei, the element representation H of the text layout element is obtained by performing a pooling operation on its text representation. iSince some elements have the entire document as their parent node (such as the document title), an additional learnable representation H0 (i.e., the global element representation in the above embodiment) is used. The layout representations of the document layout elements (i.e., the element representations in the above embodiment) are concatenated and fed into a global decoder D based on a self-attention mechanism to generate the context layout representations of the document layout elements. (i.e., the context element representation in the above embodiments). The global decoder optimizes the layout representation of document layout elements globally and further breaks down the barriers between different blocks of the sparse encoder; that is, through the self-attention mechanism of the global decoder, the context layout representation of layout elements refers to the layout representation in the global scope, taking into account global information.

[0187] The hierarchical relationships between document layout elements are predicted using a dependency parsing method, which employs a bilinear layer: Here, Sigmoid is the sigmoid function, an activation function that converts the output of a neural network into probability values; then, based on the hierarchical relationship, it is calculated using argmax({p ij Let j = 0, 1, ..., M) predict the parent node of element Ei in the layout, where argmax is used to find the data with the highest probability as the prediction result. In the training process, this embodiment employs the cross-entropy loss function.

[0188] Referring to Figure 4b, Figure 4b shows a schematic diagram of the relationship between document layout elements provided in an embodiment of this disclosure.

[0189] Specifically, Figure 4b provides a clear and convenient description by using arrows to illustrate the relationships between document layout elements. For example, the text in the shaded area on the left is the section title, and the content on the right is the specific paragraph content under each section title. The direction of the arrows indicates the parent node of the document layout element. For example, the related document layout element corresponding to the first paragraph on the right is the first section title on the left, meaning that the first paragraph on the right is the specific content under the first section title on the left.

[0190] The document processing method provided in this disclosure effectively overcomes the challenges of multi-page and multi-granularity issues in document hierarchical parsing. By providing a sparse attention mechanism, it supports the application of text-layout language models to long document inputs, improving upon the problem that previous models did not fully utilize text-layout language models. Using a hierarchical encoder-decoder structure, it considers the relationships between fine-grained text and coarse-grained layout elements, solving the problem of previous methods neglecting fine-grained text representation, thereby improving the prediction accuracy of the document hierarchical parsing model. Experimental results show that this method is far superior to existing solutions in terms of output accuracy and can be applied to real-world scenarios as the underlying algorithm for document parsing products.

[0191] Corresponding to the above method embodiments, this disclosure also provides a data processing apparatus embodiment. Figure 5 shows a schematic diagram of the structure of a data processing apparatus provided in one embodiment of this disclosure. As shown in Figure 5, the apparatus includes:

[0192] The data representation determination module 502 is configured to determine the data representation of multiple visual layout elements in a visual object and multiple visual data of the visual object. The visual layout elements include element visual data, which is the visual data corresponding to the visual layout element determined from the multiple visual data according to the association relationship between the visual layout elements and each visual data.

[0193] The data representation acquisition module 504 is configured to perform attention processing on the data representations of each visual data and the data representations of the plurality of visual data to obtain the contextual data representations of each visual data.

[0194] The element representation determination module 506 is configured to determine the element representation of each visual layout element based on the context data representation of each visual data and the association relationship between each visual data and the plurality of visual layout elements.

[0195] The associated element determination module 508 is configured to determine the associated visual layout element corresponding to each visual layout element from the plurality of visual layout elements based on the element representation of each visual layout element, wherein the associated visual layout element corresponding to each visual layout element is a visual layout element that has a hierarchical relationship with each visual layout element.

[0196] Optionally, the data characterization acquisition module 504 is further configured to:

[0197] Select any one of the plurality of visual data as the target visual data;

[0198] Attention processing is performed on the data representation of the target visual data and the data representations of the multiple visual data to obtain the contextual data representation of the target visual data.

[0199] The contextual data representation of each visual data is obtained based on the contextual data representation of the target visual data.

[0200] Optionally, the data characterization acquisition module 504 is further configured to:

[0201] Attention processing is performed on the data representation of the target visual data and the data representations of the multiple visual data to determine the data representation of the target visual data and multiple attention weights of the data representations of the multiple visual data;

[0202] By using the multiple attention weights and weighted summation of the data representations of the multiple visual data, the contextual data representation of the target visual data is obtained.

[0203] Optionally, the element characterization determination module 506 is further configured to:

[0204] Based on the relationship between the visual data and the multiple visual layout elements, determine the element visual data corresponding to each visual layout element;

[0205] Based on the contextual data representation of each visual data, determine the contextual data representation of the element visual data corresponding to each visual layout element.

[0206] The context data representation of the element visual data corresponding to each visual layout element is pooled to obtain the element representation of each visual layout element.

[0207] Optionally, the data characterization determination module 502 is further configured to:

[0208] Determine multiple visual layout elements, multiple visual data, and the two-dimensional position of each visual layout element in the visual object;

[0209] Based on the relationship between each visual data and the plurality of visual layout elements, determine the element visual data corresponding to each visual layout element from the plurality of visual data;

[0210] The two-dimensional position of each visual layout element is determined as the two-dimensional position of the element visual data corresponding to each visual layout element.

[0211] Based on the two-dimensional position of the element visual data corresponding to each visual layout element, determine the two-dimensional position of multiple visual data of the visual object;

[0212] Encode multiple visual data of the visual object and the two-dimensional positions of the multiple visual data of the visual object to obtain a data representation of the multiple visual data of the visual object.

[0213] Optionally, the data characterization determination module 502 is further configured to:

[0214] For any one of the plurality of visual data, the arbitrary visual data and its two-dimensional position are encoded to obtain the data representation of the arbitrary visual data.

[0215] Based on the data representation of any one of the visual data, data representations of multiple visual data of the visual object are obtained.

[0216] Optionally, the associated element determination module 508 is further configured to:

[0217] Determine the arrangement order of the plurality of visual layout elements;

[0218] Based on the arrangement order of the multiple visual layout elements, the element representations of each visual layout element are arranged to obtain the element representations of the multiple visual layout elements arranged in sequence.

[0219] Based on the element representations of the multiple visual layout elements arranged in the order, the associated visual layout elements corresponding to each visual layout element are determined from the multiple visual layout elements.

[0220] Optionally, the associated element determination module 508 is further configured to:

[0221] Determine the global element features, and determine the arrangement order among the plurality of visual layout elements and the global element features;

[0222] Based on the arrangement order of the plurality of visual layout elements and the global element features, the element representations of each visual layout element and the global element features are arranged to obtain the sequentially arranged global element features and element representations of the plurality of visual layout elements.

[0223] Based on the global element features arranged in the order and the element representations of the plurality of visual layout elements, the associated visual layout element corresponding to each visual layout element is determined from the plurality of visual layout elements.

[0224] Optionally, the associated element determination module 508 is further configured to:

[0225] Attention processing is performed on the element representations of each visual layout element in the ordered multiple visual layout elements, and on the element representations of the ordered multiple visual layout elements, to obtain the context element representations of each visual layout element.

[0226] Based on the context element representation of each visual layout element, the association relationship between each visual layout element is predicted, and the associated visual layout element corresponding to each visual layout element is determined.

[0227] Optionally, the data characterization determination module 502 is further configured to:

[0228] Determine the initial data representation of multiple visual data of the visual object;

[0229] Determine the relative position of each visual data in the visual layout element corresponding to each visual data, and determine the layout position code of each visual data based on the relative position;

[0230] Based on the initial data representation of each visual data and the layout position code, the data representation of multiple visual data of the visual object is determined.

[0231] The device further includes an execution module configured to perform visual data processing tasks based on the associated visual layout elements corresponding to each visual layout element.

[0232] The device further includes: a division module configured to determine multiple initial visual layout elements in an initial visual object, arrange the multiple initial visual layout elements according to a preset arrangement method, divide the arranged multiple initial visual layout elements according to a preset division length, determine multiple candidate visual objects, and sequentially determine the multiple candidate visual objects as the visual objects, wherein each candidate visual object includes multiple visual layout elements, and the multiple visual layout elements are obtained from the divided multiple initial visual layout elements.

[0233] Optionally, the data characterization determination module 502 is further configured to:

[0234] Determine the initial data representation of multiple visual data of the visual object;

[0235] Determine the page position code and layout position code corresponding to each visual data;

[0236] Based on the initial data representation of each visual data, the page position code, and the layout position code, the data representation of multiple visual data of the visual object is determined.

[0237] The data processing apparatus provided in this embodiment includes a visual object comprising multiple visual layout elements, each composed of element visual data. The apparatus determines the multiple visual layout elements of the visual object and the data representations of the multiple visual data within the visual object. By processing the data representations of each visual data with the data representations of the multiple visual data within the visual object, a fine-grained contextual data representation of each visual data is obtained. Based on the contextual data representation of the element visual data corresponding to the visual layout element, the element representation of that visual layout element is obtained, thereby obtaining the element representation of each visual layout element. The element representation of each visual layout element includes a fine-grained contextual data representation. Based on the element representation of each visual layout element, the associated visual layout element corresponding to each visual layout element is determined from the multiple visual layout elements of the visual object. By simultaneously considering the relationship between fine-grained data representation and coarse-grained visual layout elements, the apparatus solves the problem of previous methods neglecting fine-grained data representation, thereby improving the accuracy of predicting the associated layout elements corresponding to visual layout elements.

[0238] The above is an illustrative scheme of a data processing apparatus according to this embodiment. It should be noted that the technical solution of this data processing apparatus and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing apparatus, please refer to the description of the technical solution of the data processing method described above.

[0239] Corresponding to the above method embodiments, this disclosure also provides a document processing apparatus embodiment. FIG6 shows a schematic diagram of the structure of a document processing apparatus provided in one embodiment of this disclosure. As shown in FIG6, the apparatus includes:

[0240] The data representation determination module 602 is configured to determine the data representation of multiple document layout elements in a target document and multiple document data of the target document, wherein the document layout elements include element document data, and the element document data is the document data corresponding to the document layout elements determined from the multiple document data according to the association relationship between the document layout elements and each document data;

[0241] The data representation acquisition module 604 is configured to perform attention processing on the data representations of each document data and the data representations of the plurality of document data to obtain the contextual data representations of each document data.

[0242] The element representation determination module 606 is configured to determine the element representation of each document layout element based on the context data representation of each document data and the association relationship between each document data and the plurality of document layout elements.

[0243] The associated element determination module 608 is configured to determine the associated page elements corresponding to each document page element from the plurality of document page elements based on the element representation of each document page element, wherein the associated page elements corresponding to each document page element are document page elements that have a hierarchical relationship with each document page element.

[0244] The above is an illustrative scheme of a document processing apparatus according to this embodiment. It should be noted that the technical solution of this document processing apparatus and the technical solution of the document processing method described above belong to the same concept. For details not described in detail in the technical solution of the document processing apparatus, please refer to the description of the technical solution of the document processing method described above.

[0245] Figure 7 shows a structural block diagram of a computing device 700 provided according to an embodiment of the present disclosure.

[0246] The computing device 700 includes:

[0247] Memory 710 and processor 720;

[0248] The memory 710 is used to store computer programs / instructions, and the processor 720 is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor 720, they implement the steps of the language processing model training method and the language processing method.

[0249] In one or more embodiments of this disclosure, the computing device can be understood as an integrated smart terminal, including but not limited to a server, desktop computer, PC (Personal Computer), all-in-one model machine, mobile phone, tablet computer or other portable smart terminal, etc., and the computing device may have the model described in the above embodiments of this disclosure pre-installed.

[0250] Specifically, this computing device can pre-install various types of models, including but not limited to models in natural language processing, visual processing, speech processing, code processing, and multimodal task processing, thus providing diverse model selection. In different product forms, this computing device can support one or more model usage methods, including but not limited to model training, model invocation, model fine-tuning, model deployment, model inference, and application. In some product forms, this computing device also supports model management, including but not limited to multi-type model management (supporting the management of discriminative, generative, and other types of models), model version control (supporting the control of different model versions), and model evaluation (evaluating model performance and effectiveness based on model evaluation tools). In other product forms, this computing device can also create applications based on models, providing API (Application Programming Interface) calling capabilities. Users can call models into created applications through the API interface, and application management tools are also provided to manage and monitor the applications.

[0251] Furthermore, the computing device may also include data management (supporting the creation and management of model tuning datasets), a training center (providing abundant training resources to help users learn and master AI (Artificial Intelligence) technology), and basic control capabilities (providing enterprise-level basic control capabilities to ensure the security and efficient operation of the system). Through the above functions, it provides a comprehensive and integrated device for AI development, training, deployment, and application.

[0252] Furthermore, the components of the computing device 700 include, but are not limited to, a memory 710 and a processor 720. The processor 720 and the memory 710 can be connected via a bus.

[0253] The computing device 700 may also include an access device that enables the computing device 700 to communicate with a database storing data via one or more networks. Examples of such networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device may include one or more of any type of wired or wireless network interface (e.g., network interface controller (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0254] In one embodiment of this disclosure, the aforementioned components of the computing device 700, as well as other components not shown in FIG. 7, may be interconnected, for example, via a bus. It should be understood that the computing device block diagram shown in FIG. 7 is merely for illustrative purposes and is not intended to limit the scope of this disclosure. Those skilled in the art can add or replace other components as needed.

[0255] The computing device 700 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 700 can also be a mobile or stationary server.

[0256] The processor 720 is used to execute the following computer program / instructions, which, when executed by the processor, implement the steps of the above-mentioned data processing method and document processing method.

[0257] The various embodiments in this disclosure are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computing device embodiments are basically similar to the data processing method and document processing method embodiments, so the description is relatively simple. Relevant parts can be referred to the descriptions of the data processing method and document processing method embodiments.

[0258] An embodiment of this disclosure also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method and document processing method.

[0259] The various embodiments in this disclosure are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computer-readable storage medium embodiments are basically similar to the data processing method and document processing method embodiments, so the description is relatively simple. Relevant parts can be referred to the descriptions of the data processing method and document processing method embodiments.

[0260] An embodiment of this disclosure also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the above-described data processing method and document processing method.

[0261] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the technical solutions of the data processing method and document processing method described above. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solutions of the data processing method and document processing method described above.

[0262] The foregoing has described specific embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0263] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0264] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this disclosure are not limited to the described order of actions, because according to the embodiments of this disclosure, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments of this disclosure.

[0265] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0266] The preferred embodiments disclosed above are merely illustrative of this disclosure. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments of this disclosure. These embodiments are selected and specifically described in this disclosure to better explain the principles and practical applications of the embodiments of this disclosure, thereby enabling those skilled in the art to better understand and utilize this disclosure. This disclosure is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method, comprising: The data representation of multiple visual layout elements in a visual object and multiple visual data of the visual object is determined, wherein the visual layout elements include element visual data, and the element visual data is the visual data corresponding to the visual layout elements determined from the multiple visual data according to the association relationship between the visual layout elements and each visual data. Attention processing is performed on the data representations of each visual data and the data representations of the multiple visual data to obtain the contextual data representations of each visual data. Based on the contextual data representation of each visual data and the association between each visual data and the plurality of visual layout elements, the element representation of each visual layout element is determined. Based on the element representation of each visual layout element, the associated visual layout element corresponding to each visual layout element is determined from the plurality of visual layout elements, wherein the associated visual layout element corresponding to each visual layout element is a visual layout element that has a hierarchical relationship with each visual layout element.

2. The data processing method according to claim 1, wherein performing attention processing on the data representations of each visual data and the data representations of the plurality of visual data to obtain the contextual data representations of each visual data includes: Select any one of the plurality of visual data as the target visual data; Attention processing is performed on the data representation of the target visual data and the data representations of the multiple visual data to obtain the contextual data representation of the target visual data. The contextual data representation of each visual data is obtained based on the contextual data representation of the target visual data.

3. The data processing method according to claim 2, wherein performing attention processing on the data representation of the target visual data and the data representations of the plurality of visual data to obtain the contextual data representation of the target visual data includes: Attention processing is performed on the data representation of the target visual data and the data representations of the multiple visual data to determine the data representation of the target visual data and multiple attention weights of the data representations of the multiple visual data; By using the multiple attention weights and weighted summation of the data representations of the multiple visual data, the contextual data representation of the target visual data is obtained.

4. The data processing method according to any one of claims 1-3, wherein determining the element representation of each visual layout element based on the contextual data representation of each visual data and the association relationship between each visual data and the plurality of visual layout elements includes: Based on the relationship between the visual data and the multiple visual layout elements, determine the element visual data corresponding to each visual layout element; Based on the contextual data representation of each visual data, determine the contextual data representation of the element visual data corresponding to each visual layout element. The context data representation of the element visual data corresponding to each visual layout element is pooled to obtain the element representation of each visual layout element.

5. The data processing method according to any one of claims 1-4, wherein determining the data representation of multiple visual layout elements in a visual object and multiple visual data of the visual object comprises: Determine multiple visual layout elements, multiple visual data, and the two-dimensional position of each visual layout element in the visual object; Based on the relationship between each visual data and the plurality of visual layout elements, determine the element visual data corresponding to each visual layout element from the plurality of visual data; The two-dimensional position of each visual layout element is determined as the two-dimensional position of the element visual data corresponding to each visual layout element. Based on the two-dimensional position of the element visual data corresponding to each visual layout element, determine the two-dimensional position of multiple visual data of the visual object; Encode multiple visual data of the visual object and the two-dimensional positions of the multiple visual data of the visual object to obtain a data representation of the multiple visual data of the visual object.

6. The data processing method according to claim 5, wherein encoding the multiple visual data of the visual object and the two-dimensional positions of the multiple visual data of the visual object to obtain a data representation of the multiple visual data of the visual object includes: For any one of the plurality of visual data, the arbitrary visual data and its two-dimensional position are encoded to obtain the data representation of the arbitrary visual data. Based on the data representation of any one of the visual data, data representations of multiple visual data of the visual object are obtained.

7. The data processing method according to any one of claims 1-6, wherein determining the associated visual layout element corresponding to each visual layout element from the plurality of visual layout elements based on the element representation of each visual layout element comprises: Determine the arrangement order of the plurality of visual layout elements; Based on the arrangement order of the multiple visual layout elements, the element representations of each visual layout element are arranged to obtain the element representations of the multiple visual layout elements arranged in sequence. Based on the element representations of the multiple visual layout elements arranged in the order, the associated visual layout elements corresponding to each visual layout element are determined from the multiple visual layout elements.

8. The data processing method according to claim 7, before determining the associated visual layout element corresponding to each visual layout element from the plurality of visual layout elements according to the element representation of the plurality of visual layout elements arranged in the order, further comprising: Determine the global element features, and determine the arrangement order among the plurality of visual layout elements and the global element features; Based on the arrangement order of the plurality of visual layout elements and the global element features, the element representations of each visual layout element and the global element features are arranged to obtain the sequentially arranged global element features and element representations of the plurality of visual layout elements. The element representation of the plurality of visual layout elements arranged in the specified order, and the determination of the associated visual layout element corresponding to each visual layout element from the plurality of visual layout elements, include: Based on the global element features arranged in the order and the element representations of the plurality of visual layout elements, the associated visual layout element corresponding to each visual layout element is determined from the plurality of visual layout elements.

9. The data processing method according to claim 7 or 8, wherein determining the associated visual layout element corresponding to each visual layout element from the plurality of visual layout elements according to the element representation of the plurality of visual layout elements arranged in the order comprises: Attention processing is performed on the element representations of each visual layout element in the ordered multiple visual layout elements, and on the element representations of the ordered multiple visual layout elements, to obtain the context element representations of each visual layout element. Based on the context element representation of each visual layout element, the association relationship between each visual layout element is predicted, and the associated visual layout element corresponding to each visual layout element is determined.

10. The data processing method according to any one of claims 1-4, wherein determining the data representation of multiple visual data of the visual object includes: Determine the initial data representation of multiple visual data of the visual object; Determine the relative position of each visual data in the visual layout element corresponding to each visual data, and determine the layout position code of each visual data based on the relative position; Based on the initial data representation of each visual data and the layout position code, the data representation of multiple visual data of the visual object is determined.

11. The data processing method according to any one of claims 1-10, further comprising, after determining the associated visual layout element corresponding to each visual layout element: Based on the associated visual layout elements corresponding to each visual layout element, perform visual data processing tasks.

12. The data processing method according to any one of claims 1-11, further comprising, before determining the data representation of the plurality of visual layout elements in the visual object and the plurality of visual data of the visual object: Determine multiple initial visual layout elements in the initial visual object, and arrange the multiple initial visual layout elements according to a preset arrangement method; The multiple initial visual layout elements are divided according to a preset division length to determine multiple candidate visual objects, and the multiple candidate visual objects are sequentially determined as the visual objects. Each candidate visual object includes multiple visual layout elements, which are obtained from the multiple initial visual layout elements after division.

13. The data processing method according to claim 12, wherein determining the data representation of multiple visual data of the visual object includes: Determine the initial data representation of multiple visual data of the visual object; Determine the page position code and layout position code corresponding to each visual data; Based on the initial data representation of each visual data, the page position code, and the layout position code, the data representation of multiple visual data of the visual object is determined.

14. The data processing method according to claim 12 or 13, wherein dividing the arranged initial visual layout elements according to a preset division length to determine multiple candidate visual objects includes: Determine the total length of visual data for each initial visual layout element, wherein the total length of visual data is the sum of the text lengths of all visual data contained in each initial visual layout element; The cumulative length is obtained by sequentially summing the total length of the visual data of the multiple initial visual layout elements after they are arranged. If the cumulative length does not exceed the preset division length, then the total visual data length of the next initial visual layout element will continue to be accumulated according to the arrangement order; If the cumulative length exceeds the preset division length, the accumulation stops, and the accumulated initial visual layout elements are divided into a candidate visual object.

15. A document processing method, comprising: The data representation of multiple document layout elements in a target document and multiple document data of the target document is determined, wherein the document layout elements include element document data, and the element document data is the document data corresponding to the document layout elements determined from the multiple document data according to the association relationship between the document layout elements and each document data; Attention processing is performed on the data representations of each document data and the data representations of the multiple document data to obtain the contextual data representations of each document data. Based on the contextual data representation of each document data and the association between each document data and the multiple document layout elements, the element representation of each document layout element is determined; Based on the element representation of each document layout element, the associated layout element corresponding to each document layout element is determined from the plurality of document layout elements, wherein the associated layout element corresponding to each document layout element is a document layout element that has a hierarchical relationship with each document layout element.

16. A data processing apparatus, comprising: The data representation determination module is configured to determine the data representation of multiple visual layout elements in a visual object and multiple visual data of the visual object. The visual layout elements include element visual data, which is the visual data corresponding to the visual layout elements determined from the multiple visual data based on the association relationship between the visual layout elements and each visual data. The data representation acquisition module is configured to perform attention processing on the data representations of each visual data and the data representations of the plurality of visual data to obtain the contextual data representations of each visual data. The element representation determination module is configured to determine the element representation of each visual layout element based on the contextual data representation of each visual data and the association relationship between each visual data and the plurality of visual layout elements. The associated element determination module is configured to determine the associated visual layout element corresponding to each visual layout element from the plurality of visual layout elements based on the element representation of each visual layout element, wherein the associated visual layout element corresponding to each visual layout element is a visual layout element that has a hierarchical relationship with each visual layout element.

17. A document processing apparatus, comprising: The data representation determination module is configured to determine the data representation of multiple document layout elements in a target document and multiple document data of the target document, wherein the document layout elements include element document data, and the element document data is the document data corresponding to the document layout elements determined from the multiple document data according to the association relationship between the document layout elements and each document data; The data representation acquisition module is configured to perform attention processing on the data representations of each document data and the data representations of the multiple document data to obtain the contextual data representations of each document data. The element representation determination module is configured to determine the element representation of each document layout element based on the context data representation of each document data and the association relationship between each document data and the multiple document layout elements. The associated element determination module is configured to determine the associated layout element corresponding to each document layout element from the plurality of document layout elements based on the element representation of each document layout element, wherein the associated layout element corresponding to each document layout element is a document layout element that has a hierarchical relationship with each document layout element.

18. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 15.

19. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 15.

20. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 15.

Images