Document processing model training method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a training method and device of a document processing model, electronic equipment, a storage medium and a program product, and relates to the technical field of artificial intelligence, in particular to the fields of large language models, image processing, document processing, text recognition and the like. The description module is used for describing structured information of a sample document image in a natural language mode; encoding the sample document image to obtain sample visual feature information; encoding the sample processing request to obtain sample language feature information; providing the sample language feature information, the sample visual feature information and the sample layout pseudo tag for a document processing model to obtain a candidate layout reasoning chain; and based on the difference between the candidate layout reasoning chain and the sample layout pseudo tag, calculating a preset loss function and carrying out back propagation on the document processing model so as to update the document processing model.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, particularly to the fields of large language models (LLM), image processing, document processing, and text recognition, specifically to a training method and apparatus for a document processing model, a document processing method and apparatus, an electronic device, a storage medium, and a program product. Background Technology

[0002] Artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies mainly include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0003] In recent years, in the field of document processing, in addition to the common dedicated OCR (Optical Character Recognition) systems, a method using a general visual language model has emerged based on the development of artificial intelligence technology. However, as the forms and types of documents become increasingly rich and diverse, these two common methods have gradually revealed various shortcomings in processing efficiency and accuracy, making it difficult to cope with the increasing complexity of documents and the ever-growing complex processing needs of users.

[0004] The methods described in this section are not necessarily methods that had been previously conceived or adopted. Unless otherwise specified, no method described in this section should be assumed to be prior art simply because it is included in this section. Similarly, unless otherwise specified, the issues mentioned in this section should not be considered to be accepted in any prior art. Summary of the Invention

[0005] This disclosure provides a method and apparatus for training a document processing model, a document processing method and apparatus, an electronic device, a storage medium, and a program product.

[0006] According to one aspect of this disclosure, a method for training a document processing model is provided, comprising: acquiring sample document images, sample processing requests, and sample layout pseudo-labels for training, wherein the sample layout pseudo-labels are used to describe the structured information of the sample document images in natural language; encoding the sample document images to obtain sample visual feature information; encoding the sample processing requests to obtain sample language feature information; providing the sample language feature information, sample visual feature information, and sample layout pseudo-labels to the document processing model to obtain a candidate layout inference chain output by the document processing model based on the structured information and in accordance with a predetermined inference chain format; and calculating a preset loss function based on the difference between the candidate layout inference chain and the sample layout pseudo-labels and performing backpropagation on the document processing model to update the document processing model.

[0007] According to one aspect of this disclosure, a document processing method is provided, comprising: acquiring a processing request for processing a target document and image data of the target document; and providing the processing request and image data to a document processing model trained by the method described above, so as to output a document processing result for responding to the processing request.

[0008] According to one aspect of this disclosure, a training apparatus for a document processing model is provided, comprising: a training sample acquisition module configured to acquire sample document images, sample processing requests, and sample layout pseudo-labels for training, wherein the sample layout pseudo-labels are used to describe the structured information of the sample document images in natural language; a first encoding module configured to encode the sample document images to obtain sample visual feature information; a second encoding module configured to encode the sample processing requests to obtain sample language feature information; a training result output module configured to provide the sample language feature information, sample visual feature information, and sample layout pseudo-labels to the document processing model to obtain a candidate layout inference chain output by the document processing model based on the structured information and in accordance with a predetermined inference chain format; and a model update module configured to calculate a preset loss function based on the difference between the candidate layout inference chain and the sample layout pseudo-labels and perform backpropagation on the document processing model to update the document processing model.

[0009] According to one aspect of this disclosure, a document processing apparatus is provided, comprising: an input data acquisition module configured to acquire a processing request for processing a target document and image data of the target document; and an output data providing module configured to provide the processing request and image data to a document processing model trained according to the apparatus as described above, so as to output a document processing result for feedback on the processing request.

[0010] According to another aspect of this disclosure, an electronic device is provided, including at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the methods described above in this disclosure.

[0011] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing a computer to perform the methods described above.

[0012] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the methods described above.

[0013] According to one or more embodiments of this disclosure, a document processing model with better document processing performance can be trained.

[0014] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0015] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0016] Figure 1 A schematic diagram of an exemplary system in which the various methods described herein can be implemented according to embodiments of this disclosure is shown.

[0017] Figure 2 A flowchart of a document processing method based on a large model according to an embodiment of the present disclosure is shown.

[0018] Figure 3 A schematic diagram of the activation conditions of a layout inference chain according to an embodiment of the present disclosure is shown.

[0019] Figure 4 A schematic diagram of specialized terms in a document processing method according to an embodiment of the present disclosure is shown.

[0020] Figure 5 A schematic diagram of a generation layout inference chain according to an embodiment of the present disclosure is shown.

[0021] Figure 6A structural block diagram of a large-model-based document processing apparatus according to an embodiment of the present disclosure is shown.

[0022] Figure 7 A flowchart of a method for generating inference data based on a large model according to an embodiment of the present disclosure is shown.

[0023] Figure 8 A schematic diagram of the activation conditions of a layout inference chain according to another embodiment of the present disclosure is shown.

[0024] Figure 9 A schematic diagram of a generation layout inference chain according to another embodiment of the present disclosure is shown.

[0025] Figure 10 A structural block diagram of a large-model-based inference data generation apparatus according to an embodiment of the present disclosure is shown.

[0026] Figure 11 A flowchart of a large-model-based text recognition method according to an embodiment of the present disclosure is shown.

[0027] Figure 12 A flowchart of a text recognition method according to another embodiment of this disclosure is shown.

[0028] Figure 13 A schematic diagram of correcting the initial character according to an embodiment of the present disclosure is shown.

[0029] Figure 14 A structural block diagram of a large-model-based text recognition device according to an embodiment of the present disclosure is shown.

[0030] Figure 15 A structural block diagram of a text recognition device according to another embodiment of the present disclosure is shown.

[0031] Figure 16 A schematic diagram of a document processing model according to an embodiment of the present disclosure is shown.

[0032] Figure 17 A flowchart illustrating a method for training a document processing model according to an embodiment of the present disclosure is shown.

[0033] Figure 18 A schematic diagram of a dynamic adjustment strategy for training data according to an embodiment of the present disclosure is shown.

[0034] Figure 19 A flowchart of a document processing method according to another embodiment of this disclosure is shown.

[0035] Figure 20 A structural block diagram of a training apparatus for a document processing model according to an embodiment of the present disclosure is shown.

[0036] Figure 21 A flowchart illustrating a method for training a document processing model according to another embodiment of this disclosure is shown.

[0037] Figure 22 A schematic diagram of obtaining sample layout pseudo-labels is shown according to an embodiment of the present disclosure.

[0038] Figure 23 A structural block diagram of a training apparatus for a document processing model according to another embodiment of the present disclosure is shown.

[0039] Figure 24 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0040] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0041] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0042] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof.

[0043] In related technologies, pipeline-based dedicated OCR systems generally adopt a technical approach of layout detection, element classification, content recognition, and result aggregation. Although they achieve high accuracy in pure OCR tasks, they suffer from several problems. For example, they can only "recognize" the content, not "understand" it, let alone answer specific questions. Secondly, the fixed workflow leads to wasted computation, as even simple tasks require a complete recognition process. Furthermore, cascading multiple modules can cause error accumulation, allowing errors from earlier modules to propagate to later ones. Additionally, they lack language understanding capabilities, cannot handle spelling errors or ambiguities, and the reasoning process is opaque, making it impossible to trace the basis of decisions.

[0044] While other general-purpose visual language models possess some document understanding and reasoning capabilities, their recognition accuracy is low, and they are prone to illusions when handling documents with complex layouts, leading to disordered recognition order. Furthermore, they suffer from the problem of a black-box reasoning process and an inability to adaptively adjust task complexity. Currently, the market lacks a unified document intelligence model that simultaneously possesses high-precision recognition and deep understanding reasoning capabilities, making it difficult to cope with the increasing complexity of documents and the ever-growing demand for complex processing from users.

[0045] To this end, this disclosure proposes a document processing method based on a large model, a reasoning data generation method based on a large model, a text recognition method based on a large model, and a training method for a document processing model. On the one hand, it can provide a more effective document processing mechanism, and on the other hand, it can obtain a document processing model with better document processing performance through training.

[0046] Before describing the methods of the embodiments of this disclosure in detail, firstly, in conjunction with Figure 1 The description includes exemplary systems in which the methods described herein may be implemented.

[0047] Figure 1 A schematic diagram of an exemplary system 100 in which the various methods and apparatus described herein can be implemented according to embodiments of this disclosure is shown. Reference Figure 1 The system 100 includes one or more client devices 101, 102, 103, 104, 105 and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105 and 106 can be configured to execute one or more applications.

[0048] In embodiments of this disclosure, server 120 may run one or more services or software applications that enable the various methods described herein to be performed.

[0049] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtual and virtual environments. In some embodiments, these services may be provided as web-based services or cloud services, such as to users of client devices 101, 102, 103, 104, 105 and / or 106 under a Software as a Service (SaaS) model.

[0050] exist Figure 1 In the configuration shown, server 120 may include one or more components that implement the functions performed by server 120. These components may include software components, hardware components, or combinations thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 can sequentially interact with server 120 using one or more client applications to utilize the services provided by these components. It should be understood that various different system configurations are possible and may differ from system 100. Therefore, Figure 1 This is an example of a system used to implement the various methods described herein, and is not intended to be limiting.

[0051] Users can use client devices 101, 102, 103, 104, 105, and / or 106 to provide target documents and processing requests for processing the target documents. The client devices can provide interfaces that allow users to interact with them. The client devices can also output information to the user through this interface. Although... Figure 1 Only six client devices are described, but those skilled in the art will understand that this disclosure can support any number of client devices.

[0052] Client devices 101, 102, 103, 104, 105, and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computer devices can run various types and versions of software applications and operating systems, such as Microsoft Windows, Apple iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as Google Chrome OS); or include various mobile operating systems, such as Microsoft Windows Mobile OS, iOS, Windows Phone, and Android. Portable handheld devices may include cellular phones, smartphones, tablets, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Gaming systems may include various handheld gaming devices, internet-enabled gaming devices, etc. Client devices are capable of executing various applications, such as various internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and can use various communication protocols.

[0053] Network 110 can be any type of network well known to those skilled in the art, and can support data communication using any of a variety of available protocols (including but not limited to TCP / IP, SNA, IPX, etc.). By way of example only, one or more networks 110 can be a local area network (LAN), an Ethernet-based network, a token ring network, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0054] Server 120 may include one or more general-purpose computers, special-purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for servers). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0055] The computing unit in server 120 can run one or more operating systems, including any of the aforementioned operating systems and any commercially available server operating system. Server 120 can also run any of a variety of additional server applications and / or middleware applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0056] In some implementations, server 120 may include one or more applications to analyze and merge data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and 106.

[0057] In some implementations, server 120 can be a server for a distributed system or a server integrated with blockchain. Server 120 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0058] System 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as visual features, layout information, etc. Databases 130 may reside in various locations. For example, a database used by server 120 may be local to server 120, or it may be located remotely to server 120 and may communicate with server 120 via a network-based or dedicated connection. Databases 130 may be of different types. In some embodiments, the database used by server 120 may be, for example, a relational database. One or more of these databases may store, update, and retrieve data from and from the databases in response to commands.

[0059] In some embodiments, one or more of the databases 130 may also be used by an application to store application data. The databases used by the application may be of different types, such as key-value stores, object stores, or regular stores supported by a file system.

[0060] Figure 1The system 100 can be configured and operated in various ways to enable the application of the various methods and apparatus described in this disclosure.

[0061] The various aspects of the methods according to embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.

[0062] In this paper, the large model is also referred to as the Large Language Model (LLM), which can be, for example, a deep neural network based on the Transformer architecture.

[0063] According to embodiments of this disclosure, a document processing method based on a large model is provided.

[0064] Figure 2 A flowchart of a large-model-based document processing method 200 according to an embodiment of the present disclosure is shown.

[0065] like Figure 2 As shown, the document processing method 200 based on a large model includes steps S201 to S205.

[0066] In step S201, a processing request for processing the target document and image data of the target document are obtained.

[0067] In the example, the target document, also known as the document to be processed, is such as a manuscript, report, invoice, contract, or brochure. The target document can be provided by the user to the large model, instructing it to perform various processing tasks, such as performing OCR (Optical Character Recognition) on specified portions of the document, extracting content from specified portions, or understanding the semantics of specified portions. Accordingly, processing requests can be provided to the large model via text or voice. Therefore, processing requests can include information associated with the user's intent, such as being presented as questions or instructions. These questions or instructions can be understood by the large model to execute the user-instructed processing. Simultaneously, to utilize the large model for processing the target document, it needs to be provided to the large model as image data. Image data can be obtained by the user through methods such as taking a photo or scanning. The large model can provide an interface for interaction with the user, allowing the user to send processing requests and image data to the large model via this interface.

[0068] In step S202, visual feature information for characterizing the target document is obtained based on the image data.

[0069] In the example, the image data can be processed via a visual encoder to obtain the corresponding visual feature information. Visual feature information is the representation of the target document at the visual feature vector level. The visual encoder could be a NaViT encoder, a Transformer architecture that combines natural language processing and visual processing, specifically designed to process image data and convert image information into visual feature vectors.

[0070] In step S203, based on the processing request, it is determined whether to activate the layout inference chain. The layout inference chain includes layout information of the target region in the target document associated with the processing request. The layout information of the target region is described in natural language.

[0071] In the example, the layout inference chain refers to the spatial specification written in natural language by the large model, which describes the spatial layout characteristics of a document. Therefore, the layout inference chain includes layout information describing these spatial layout characteristics. This layout inference chain is used by the large model itself during intermediate inference to obtain answers related to user questions or instructions. Taking a document containing multiple areas such as text, tables, and illustrations as an example, the layout information could include the bounding box coordinates of the text areas, the bounding box coordinates and row and column structure of the table areas, the bounding box coordinates of the illustration areas, and the reading order of all areas. Correspondingly, the target area can be a local area within the target document or the entire target document. The large model can determine the target area of ​​interest to the user by understanding and analyzing the processing request.

[0072] In the example, the large model does not always activate the layout inference chain; rather, it is activated only under specific conditions. Since the layout inference chain is used to describe the spatial layout characteristics of a document, it is only activated when the task corresponding to the request requires an understanding of these characteristics. Therefore, the layout inference chain has a conditional activation mechanism.

[0073] In step S204, in response to determining the activation of the layout inference chain, a layout inference chain is generated based on the processing request and visual feature information.

[0074] In the example, if the large model determines that the layout inference chain needs to be activated, it can initiate the process of generating the layout inference chain. To do this, the large model can feed the language feature information corresponding to the processing request and the visual feature information of the target document into the Transformer for attention, thereby achieving information fusion to generate the layout inference chain.

[0075] In step S205, based on the processing request, visual feature information, and layout inference chain, a document processing result is output to provide feedback on the processing request.

[0076] Therefore, on the one hand, by activating the layout inference chain logic on demand, redundant layout analysis is avoided for processing requests that do not require spatial information, thus improving document processing efficiency; on the other hand, by generating a natural language description of the target region layout inference chain for processing requests that require spatial positioning, the document structure understanding process becomes traceable and explainable, ensuring both the processing speed of simple tasks and improving the accuracy and reliability of the processing results for complex document tasks.

[0077] In some embodiments, such as in combination Figure 2 The method 200 shown may further include: determining the task type corresponding to the processing request, the task type including at least one of the first to fourth task types, the first task type being associated with identifying the corresponding content of the target region, the second task type being associated with locating the target region to extract the corresponding content, the third task type being associated with reasoning based on the corresponding content of the target region, and the fourth task type being associated with understanding the semantics of the target document as a whole.

[0078] In the example, the tasks corresponding to the processing requests can be categorized by type. The first task type focuses on identifying the recognition range and recognizing the corresponding content in the target area. For example, this could refer to reading information such as text and symbols within the target area, such as recognizing the amount on an invoice. The second task type aims to locate the target area and extract its content, such as finding a specific data table in a report and extracting its values. The third task type involves reasoning based on the content of the target area, such as determining the performance period of an obligation based on the stipulations of multiple clauses in a contract. The fourth task type focuses on understanding the overall semantics of the target document, such as grasping the ideas and intentions conveyed by the entire content of the document.

[0079] This task categorization design significantly improves the relevance and efficiency of document processing, while ensuring the accuracy and completeness of the results for different types of tasks.

[0080] In some embodiments, the above step of determining whether to activate the layout inference chain based on the processing request (such as in conjunction with...) Figure 2 Step S203 may include: determining to activate the layout inference chain in response to determining that the task type includes at least one of the first to third task types; and determining not to activate the layout inference chain in response to determining that the task type includes a fourth task type.

[0081] In the example, the layout inference chain can provide spatial layout characteristics described in natural language, enabling target area localization, content extraction, and analysis of logical relationships between areas. Therefore, determining whether to activate the layout inference chain can be based on whether the task requires this information. For the first task type, layout information can be used to lock the recognition range to avoid interference from irrelevant areas; the second task type inherently includes the requirement of target area localization, so layout information can serve as the basis for completing target area localization; the third task type, in the process of inferring the content of the target area, can use layout features such as the spatial distribution and hierarchical relationships of the areas to construct logical relationships, thus also requiring reliance on the information provided by activating the layout inference chain for task processing. In contrast, the fourth task type focuses on the overall semantic understanding of the target document, such as integrating the overall visual features of the target document to grasp the overall intent, therefore it does not rely on the specific layout details of local target areas, and thus does not need to activate the layout inference chain.

[0082] Therefore, by using this inference chain conditional activation method, the layout inference chain can be activated on demand, which reduces computational redundancy in scenarios where spatial structure analysis is not required, and provides a basis for task processing for tasks that rely on layout information, effectively improving the resource utilization efficiency and result accuracy of document processing.

[0083] Figure 3 A schematic diagram of the activation conditions of a layout inference chain according to an embodiment of the present disclosure is shown.

[0084] like Figure 3 As shown, the task types included in the processing request can include a first task type, a second task type, a third task type, and a fourth task type. When the task types included in the processing request are at least one of the first, second, and third task types, the task processing can be completed by activating the layout inference chain; when the task types included in the processing request are only the fourth task type, the layout inference chain can be left unactivated.

[0085] In some embodiments, the layout inference chain may include a first special term pair, which is used to indicate the start and end of the layout information of the target area.

[0086] In the example, the first specialized term can be a specialized identifier designed to achieve a structured definition of the layout information of the target region. For example, it can consist of a first specialized term at the predefined starting position and another first specialized term at the ending position, forming a semantically complementary pair of identifiers to clearly indicate the effective boundary of the target region layout information in the layout inference chain. When generating the layout inference chain, the first specialized term pairs can be embedded into the starting and ending positions of the target region layout information description, forming a standardized structure of starting specialized term, natural language description of the target region layout information, and ending specialized term.

[0087] This approach makes the information hierarchy of the layout inference chain clearer, significantly reducing the difficulty and error rate of parsing layout information in large models. It also improves the versatility and parsability of the layout inference chain, laying the foundation for the accuracy of document processing results.

[0088] In some embodiments, the layout information of the target region may include the coordinates of the bounding box, and the layout inference chain may also include a second special term and a third special term, wherein the second special term is used to indicate the coordinates of the bounding box, and the third special term is used to indicate the confidence level corresponding to the coordinates of the bounding box.

[0089] In the example, the coordinates of the bounding box can be parameters representing spatial location in the target area layout information, while the second and third specialized terms can be exclusive semantic identifiers indicating the coordinates of the bounding box. The second specialized term can be used to directly define the coordinate range of the bounding box; the third specialized term can be used to mark the confidence level directly corresponding to the aforementioned coordinates.

[0090] Therefore, by defining the coordinates of the bounding box, the efficiency and accuracy of large models in extracting target region data are greatly improved. At the same time, by introducing confidence, task processing errors can be reduced, further enhancing the accuracy and credibility of document processing results.

[0091] In some embodiments, the document processing result may include a fourth special term pair, which is used to indicate the start and end of the document processing result.

[0092] In the example, the document processing result can refer to the final feedback output made in response to the processing request, such as including recognition results, extracted data, answer understanding, inference conclusions, or overall semantic summaries. The fourth specialized term pair can be used to indicate the effective information range of the document processing result; for example, it can consist of a predefined fourth specialized term pair identifying the start position of the document processing result and another fourth specialized term pair identifying the end position of the document processing result. After generating the document processing result, the specific content of the document processing result can be defined by the fourth specialized term pair, so that the document processing result can form a well-defined, independent information unit.

[0093] This approach significantly reduces the difficulty of parsing document processing results for large models, avoids confusion between valid information and irrelevant content, and further ensures the accuracy of processing result transmission and parsing.

[0094] In some embodiments, intermediate reasoning information may also be included between the layout reasoning chain and the document processing result. The intermediate reasoning information includes a fifth special term pair, which is used to indicate the start and end of the intermediate reasoning information.

[0095] In the example, the intermediate reasoning information between the layout reasoning chain and the document processing result can serve as a carrier for the logical deduction process supporting the document processing result. The fifth special term pair can be designed to provide clear boundary markers for this logical deduction process, so as to avoid information confusion with the layout reasoning chain and the document processing result.

[0096] In this way, the layout reasoning chain, intermediate reasoning information, and document processing results can form a clear hierarchical information chain, which not only ensures the traceability of the intermediate reasoning process, but also avoids cross-interference between information from different modules, thereby improving the parsing efficiency of the large model for the overall processing flow.

[0097] Figure 4 A schematic diagram of specialized terms in a document processing method according to an embodiment of the present disclosure is shown.

[0098] like Figure 4 As shown, <layout> "and"< / layout> "This could be the first special term pair 401, used to indicate the start and end of the layout information of the target area, respectively." <bbox>"It could be the second special term 402, used to indicate the coordinates of the bounding box;" <confidence>"It could be the third special term 403, used to indicate the confidence level corresponding to the coordinates of the bounding box;" <answer> "and"< / answer> "It could be the fourth special term pair 404, used to indicate the beginning and end of the document processing result, respectively;" <think> "and"< / think> "It can be the fifth special term pair 405, which is used to indicate the beginning and end of intermediate reasoning information respectively.

[0099] In some embodiments, the above-described step of obtaining visual feature information for characterizing the target document based on image data (such as combining...) Figure 2 Step S202 may include: performing block preprocessing on the image data to obtain multiple sub-image data; and encoding the multiple sub-image data to obtain visual feature information, wherein the visual feature information includes multiple visual feature vectors, and the multiple visual feature vectors correspond one-to-one with the multiple sub-image data.

[0100] In the example, when acquiring visual feature information of a target document based on image data, block preprocessing can be used. Block preprocessing refers to structurally splitting the original image data according to preset rules, which can be based on image size, such as fixed-size blocks or adaptive semantic blocks. The resulting multiple sub-image data points each correspond to a local region of the target document. These sub-image data points can then be sequentially input into a pre-trained visual encoder (such as the NaViT encoder). The visual feature information, as a representation of the target document at the feature vector level, contains multiple visual feature vectors that correspond one-to-one with the sub-image data points.

[0101] This approach not only improves the efficiency of visual feature extraction but also strengthens the correlation between visual features and local areas of the target document, providing necessary feature support for layout inference chain generation and document processing result output.

[0102] In some embodiments, visual feature information can be projected into the language embedding space of a large model.

[0103] In the example, after obtaining the visual feature information of the target document based on image data, it can be projected into the language embedding space of a large model, for example via a multi-layer perceptron (MLP) for projection, to achieve alignment of visual modal information with language modal information, providing a unified feature interaction basis for multimodal information fusion and reasoning.

[0104] In this way, feature spaces of visual and linguistic modalities are aligned, breaking down barriers to cross-modal information interaction, ensuring the fusion of visual features and processing request semantics, improving the semantic understanding depth of large models of document visual information, and significantly enhancing the overall solution's cross-modal processing capabilities and the accuracy of document processing results.

[0105] In some embodiments, the above steps of generating a layout inference chain based on processing requests and visual feature information (such as combining) Figure 2 Step S204 may include: generating layout information of the target region based on the processing request and visual feature information; and generating a layout inference chain in natural language based on the layout information of the target region.

[0106] In the example, the layout information of the target area can be information representing attributes such as the spatial location and structural relationships of the target area, such as bounding box coordinates, area hierarchy, row and column structure, and reading order. Its generation requires processing requests and visual feature information. Processing requests can be used to clarify the scope and requirements of the target area that the user is interested in, providing direction for the filtering of layout information; while the visual feature information projected onto the language embedding space can provide visual basis for accurately identifying the spatial attributes of the target area.

[0107] This approach ensures the accuracy and completeness of layout information while also making the layout inference chain parsable. It provides logically clear and semantically explicit layout support for the generation of document processing results, significantly improving the efficiency of the large model in understanding the layout of the target area and the accuracy of the processing results.

[0108] In some embodiments, the processing request may be represented by linguistic feature information via encoding. Accordingly, the step of generating layout information of the target region based on the processing request and visual feature information may include: fusing the linguistic feature information and visual feature information to obtain layout feature information for representing the layout information of the target region; and decoding the layout feature information to obtain the layout information.

[0109] In the example, linguistic feature information can be a vector representation of the processed request. Layout feature information, which integrates linguistic and visual features, can be generated using, for example, the attention mechanism of a Transformer model. This layout feature information can then be decoded and transformed into layout information for the target region.

[0110] In this way, cross-modal fusion of linguistic and visual features is achieved, ensuring the relevance of layout information to user processing requests and its matching degree with document visual features, thereby improving the accuracy of layout information generation.

[0111] Figure 5 A schematic diagram of a generation layout inference chain according to an embodiment of the present disclosure is shown.

[0112] like Figure 5 As shown, the processing request can first undergo an encoding operation to transform it into linguistic feature information. Simultaneously, the visual feature information corresponding to the image data of the target document (which has already undergone linguistic embedding spatial projection) can serve as another input source, performing cross-modal feature fusion with the aforementioned linguistic feature information (e.g., via a Transformer attention mechanism) to generate layout feature information. Then, the layout feature information undergoes a decoding operation to transform it into layout information of the target region (such as bounding box coordinates, region structure, etc.). This layout information is then described using natural language, ultimately generating a layout inference chain in natural language form.

[0113] In some embodiments, the above-described document processing results (such as those based on processing requests, visual feature information, and layout inference chains) are output to provide feedback on the processing requests. Figure 2 Step S205 may include: fusing language feature information, visual feature information and layout feature information to obtain result feature information for characterizing the document processing result; and decoding the result feature information to obtain the document processing result.

[0114] In the example, during the stage of outputting the document processing results, the language feature information, visual feature information, and layout inference chain corresponding to the processing request can be fused to generate result feature information; then the result feature information can be converted into document processing results in natural language form.

[0115] This approach enables the synergy of multi-dimensional features, significantly improving the accuracy and reliability of the processing results while reducing processing errors caused by missing information dimensions.

[0116] In some embodiments, the layout inference chain can be output explicitly.

[0117] In the example, the layout inference chain can be output explicitly to achieve traceability and interpretability of the document processing process, providing reasonable reference information for users to consult or for subsequent verification. Explicit output can be relative to the implicit call to the inference chain within the larger model, meaning that the layout inference chain is presented externally as an independent and identifiable information unit along with intermediate inference information and the final document processing result.

[0118] This approach enhances the interpretability of the overall document processing solution, enabling users to clearly trace the layout and derivation of the final result, thereby increasing the credibility and persuasiveness of the result and further improving the practicality and robustness of the technical solution.

[0119] According to embodiments of this disclosure, a document processing apparatus based on a large model is also provided.

[0120] Figure 6 A structural block diagram of a large-model-based document processing apparatus 600 according to an embodiment of the present disclosure is shown.

[0121] like Figure 6 As shown, the document processing apparatus 600 may include a data acquisition module 601, a visual feature acquisition module 602, an activation determination module 603, an inference chain generation module 604, and a processing result output module 605. The data acquisition module 601 is configured to acquire a processing request for processing a target document, and image data of the target document; the visual feature acquisition module 602 is configured to acquire visual feature information characterizing the target document based on the image data; the activation determination module 603 is configured to determine whether to activate a layout inference chain based on the processing request, the layout inference chain including layout information of a target region in the target document associated with the processing request, the layout information of the target region being described in natural language; the inference chain generation module 604 is configured to generate a layout inference chain based on the processing request and the visual feature information in response to determining that the layout inference chain is activated; and the processing result output module 605 is configured to output a document processing result for feedback on the processing request based on the processing request, the visual feature information, and the layout inference chain.

[0122] In some embodiments, the document processing apparatus 600 may further include a task type determination module 606, configured to determine the task type corresponding to the processing request. The task type includes at least one of a first to a fourth task type. The first task type is associated with identifying the corresponding content of a target region, the second task type is associated with locating the target region to extract the corresponding content, the third task type is associated with reasoning based on the corresponding content of the target region, and the fourth task type is associated with understanding the semantics of the target document as a whole.

[0123] In some embodiments, the activation determination module 603 may include a first sub-determination module 6031 and a second sub-determination module 6032. The first sub-determination module 6031 may be configured to determine to activate the layout inference chain in response to determining that the task type includes at least one of a first task type to a third task type. The second sub-determination module 6032 may be configured to determine not to activate the layout inference chain in response to determining that the task type includes a fourth task type.

[0124] In some embodiments, the visual feature acquisition module 602 may include a preprocessing module 6021 and an encoding module 6022. The preprocessing module 6021 may be configured to perform block preprocessing on the image data to obtain multiple sub-image data. The encoding module 6022 may be configured to encode the multiple sub-image data to obtain visual feature information, which includes multiple visual feature vectors, and the multiple visual feature vectors correspond one-to-one with the multiple sub-image data.

[0125] In some embodiments, the inference chain generation module 604 may include a layout information generation module 6041 and an inference chain generation submodule 6042. The layout information generation module 6041 may be configured to generate layout information for a target region based on processing requests and visual feature information. The inference chain generation submodule 6042 may be configured to generate a layout inference chain in natural language based on the layout information of the target region.

[0126] In some embodiments, the processing request can be represented by linguistic feature information via encoding. Accordingly, the layout information generation module 6041 may include a fusion module 6041a and a layout information generation submodule 6041b. The fusion module 6041a can be configured to fuse linguistic feature information and visual feature information to obtain layout feature information representing the layout information of the target region. The layout information generation submodule 6041b can be configured to decode the layout feature information to obtain layout information.

[0127] In some embodiments, the processing result output module 605 may include a result feature information generation module 6051 and a processing result generation module 6052. The result feature information generation module 6051 may be configured to fuse language feature information, visual feature information, and layout feature information to obtain result feature information characterizing the document processing result. The processing result generation module 6052 may be configured to decode the result feature information to obtain the document processing result.

[0128] According to embodiments of this disclosure, a method for generating inference data based on a large model is provided.

[0129] Figure 7 A flowchart of a large-model-based inference data generation method 700 according to an embodiment of the present disclosure is shown.

[0130] like Figure 7 As shown, the inference data generation method 700 based on a large model includes steps S701 to S704.

[0131] In step S701, a processing request for processing the target document and visual feature information for characterizing the target document are obtained.

[0132] In the example, the target document, also known as the document to be processed, is such as a manuscript, report, invoice, contract, or brochure. The target document can be provided by the user to the large model, instructing it to perform various processing tasks, such as performing OCR (Optical Character Recognition) on specified parts of the document, extracting content from specified parts, or understanding the semantics of specified parts. Accordingly, processing requests can be provided to the large model via text or voice. Therefore, processing requests can include information associated with the user's intent, such as being presented as questions or instructions. These questions or instructions can be understood by the large model to execute the user-instructed processing. Simultaneously, to utilize the large model for processing the target document, visual feature information representing the target document needs to be provided to the large model. Visual feature information can be the representation of the target document at the visual feature vector level. The large model can provide an interface for interaction with the user, allowing the user to send processing requests and visual feature information to the large model through this interface.

[0133] In step S702, it is determined whether at least one of the processing request or visual feature information meets the preset activation conditions for activating the layout inference chain, which is associated with the layout information of the target document.

[0134] In the example, the layout inference chain refers to the spatial specification written in natural language by the large model, which describes the spatial layout characteristics of the target document. Therefore, the layout inference chain includes layout information describing these spatial layout characteristics. This layout inference chain is used by the large model itself during intermediate inference to obtain answers related to user questions or instructions. Taking a document containing multiple areas such as text, tables, and illustrations as an example, the layout information could include the bounding box coordinates of the text areas, the bounding box coordinates and row and column structure of the table areas, the bounding box coordinates of the illustration areas, and the reading order of all areas.

[0135] In the example, the large model does not always activate the layout inference chain; rather, it activates only under specific conditions. Since the layout inference chain is used to describe the spatial layout characteristics of a document, it is only activated when the task corresponding to the request or the expression of visual feature information requires an understanding of these spatial layout characteristics. Therefore, the layout inference chain has a conditional activation mechanism.

[0136] In step S703, in response to at least one of the processing request or visual feature information satisfying a preset activation condition, the layout information of the target region in the target document associated with the processing request is determined based on the processing request and the visual feature information.

[0137] In step S704, a layout inference chain is generated in natural language based on the layout information of the target area. The document processing result used to respond to the processing request is obtained based on the layout inference chain.

[0138] In the example, if either the processing request or the visual feature information meets a preset activation condition, the process of generating a layout inference chain can be initiated. To this end, the large model can feed the linguistic feature information corresponding to the processing request and the visual feature information of the target document into the Transformer for attention, thereby achieving information fusion to generate the layout inference chain. The target region can be a local area within the target document or the entire target document. The large model can determine the target region of interest to the user by understanding and analyzing the processing request.

[0139] Therefore, on the one hand, by activating the layout inference chain logic on demand, redundant layout analysis is avoided for processing requests that do not require spatial information, thus improving document processing efficiency; on the other hand, by generating a natural language description of the target region layout inference chain for processing requests that require spatial positioning, the inference data generation process is made traceable and interpretable, which not only ensures the processing speed of simple tasks, but also improves the accuracy and reliability of the processing results of complex tasks.

[0140] In some embodiments, the step of determining whether at least one of the processing request or visual feature information satisfies the preset activation conditions for activating the layout inference chain (such as in combination with...) Figure 7 Step S702 may include: determining whether the processing request is used to perform character-level recognition on a target region in a target document; and in response to determining that the processing request is used to perform character-level recognition, determining that the processing request meets a preset activation condition.

[0141] In the example, when determining whether the preset activation conditions of the layout inference chain are met based on the processing request, it can be determined whether the processing request is used for character-level recognition. Character-level recognition can be OCR recognition. Since character-level recognition requires using layout information to lock the recognition range, it can be achieved by activating the layout inference chain.

[0142] In this way, the necessary layout information can be provided for character-level recognition tasks, while avoiding the waste of resources caused by activating the layout inference chain in unnecessary scenarios, thus effectively improving the accuracy and processing efficiency of character recognition.

[0143] In some embodiments, the step of determining whether at least one of the processing request or visual feature information satisfies the preset activation conditions for activating the layout inference chain (such as in combination with...) Figure 7 Step S702 may include: determining whether the processing request includes a spatial locator, wherein the spatial locator indicates a target region in the target document; and in response to determining that the processing request includes a spatial locator, determining that the processing request meets a preset activation condition.

[0144] In the example, spatial locators can refer to words in the processing request that have directional or regional location attributes, such as expressions like "top-left table," "bottom signature bar," "right-side clause area," or "line number." These words can clearly point to the spatial location of the target area within the target document. In other words, spatial locators can directly associate with the spatial location of the target area. For example, the large model can first perform semantic understanding on the processing request to filter out spatial locators that contain expressions such as directional or regional location attributes. When a spatial locator is detected in the processing request, it can be determined that the processing request requires the spatial layout information of the target area, thus determining that the processing request meets the preset activation conditions, and triggering the activation process of the layout inference chain.

[0145] Therefore, by capturing spatial layout requirements in the processing request through spatial locator words, the activation conditions are accurately matched, which effectively improves the targeting of the layout inference chain activation, thereby further improving the efficiency of inference data generation.

[0146] In some embodiments, the step of determining whether at least one of the processing request or visual feature information satisfies the preset activation conditions for activating the layout inference chain (such as in combination with...) Figure 7 Step S702 may include: determining whether the processing request is used to obtain multi-element relationships, wherein the multi-element relationships include the mutual relationships between multiple target elements in the target document, and the target region includes multiple target elements; and in response to determining that the processing request includes multi-element relationships, determining that the processing request meets preset activation conditions.

[0147] In the example, target elements can be content units within a document that possess independent semantics or functions, such as table cells, clause entries, or illustration captions. Multi-element relationships refer to the logical associations between multiple target elements in the target document, such as the mapping between item names and corresponding data values ​​in a report, the correspondence between payment terms and amount terms in a contract, the comparison between two item names in a table, or the comparison of differences in total amounts. The target area can encompass these multiple target elements to be associated. Since obtaining multi-element relationships requires the spatial layout information of multiple target elements, the spatial information of the target area contained in the layout inference chain can provide the necessary data for obtaining multi-element relationships. In other words, when it is determined that the processing request points to the acquisition of multi-element relationships, it can be determined that the execution of the request can rely on the spatial layout information of the target elements. Therefore, it can be determined that the processing request meets the preset activation conditions, and the activation process of the layout inference chain can be triggered.

[0148] By leveraging the dependency attributes of multi-element relationships on spatial layout, it is possible to accurately determine whether a processing request meets the preset activation conditions, effectively improving the accuracy of multi-element relationship extraction and the efficiency of inference data generation.

[0149] In some embodiments, the step of determining whether at least one of the processing request or visual feature information satisfies the preset activation conditions for activating the layout inference chain (such as in combination with...) Figure 7 Step S702 may include: determining whether the visual feature information represents a mixed layout, wherein the mixed layout indicates that the target document includes at least one of multiple layout regions or multiple element types; and in response to determining that the visual feature information represents a mixed layout, determining that the processing request meets a preset activation condition.

[0150] In the example, the layout area can be a content block within a document with independent spatial boundaries (such as a text area, table area, or illustration area), and the element type can be different morphological units carrying information within the document (such as text, tables, etc.). Because the target document with a mixed layout contains multiple independent layout areas or various element types, its content relationships can be defined based on the spatial layout, that is, based on the layout information used to describe the spatial layout characteristics in the layout inference chain. Therefore, when visual feature information characterizes the mixed layout, the activation process of the layout inference chain can be triggered.

[0151] Therefore, using the layout of the target document as the activation condition for determining the layout inference chain not only provides the necessary spatial layout basis for processing mixed layout documents, but also effectively improves the accuracy of generating complex layout inference data.

[0152] In some embodiments, the step of determining whether at least one of the processing request or visual feature information satisfies the preset activation conditions for activating the layout inference chain (such as in combination with...) Figure 7 Step S702 may include: determining whether the processing request is used to understand the semantics of the target document as a whole or to identify a specified single element; and in response to determining that the processing request is used to understand the semantics of the target document as a whole or to identify a specified single element, determining that the processing request does not meet the preset activation conditions.

[0153] In the example, understanding the semantics of the target document as a whole can refer to the goal of the processing request being to grasp the main idea, summary intent, or to make open-ended inferences about the content of the target document (such as evaluation or suggestions). This relies on the overall integration of the target document and does not require focusing on local spatial distribution. Identifying a specific single element can refer to the processing request explicitly targeting a single target element in the document (such as a single text, a single symbol, or a single piece of data), which does not require the assistance of spatial layout information. Therefore, processing requests used to understand the semantics of the target document as a whole or to identify a specific single element can be set to not meet activation conditions to avoid invalid activation of the layout inference chain and waste of computational resources.

[0154] This approach enables the filtering of tasks that do not require layout information, effectively avoiding computational redundancy in unnecessary scenarios. It reduces resource consumption during the inference data generation process, makes the activation of the layout inference chain more targeted, and further improves resource utilization efficiency.

[0155] Figure 8 A schematic diagram of the activation conditions of a layout inference chain according to another embodiment of the present disclosure is shown.

[0156] like Figure 8 As shown, examples are provided summarizing the cases where the preset activation conditions are met and not met. One case can be determined based on the processing request. Processing requests can include four categories: 1) character-level recognition of the target region; 2) the processing request includes spatial locators; 3) obtaining multi-element relationships; and 4) understanding the overall semantics of the target document or recognizing a specified single element. When the processing request is character-level recognition of the target region, includes spatial locators, or obtains multi-element relationships, the processing request can be determined to meet the preset activation conditions; when the processing request is understanding the overall semantics of the target document or recognizing a specified single element, the processing request can be determined to not meet the preset activation conditions. Another case can be determined based on visual feature information. Visual feature information can characterize a mixed layout; when the visual feature information characterizes a mixed layout, the processing request can be determined to meet the preset activation conditions.

[0157] In some embodiments, visual feature information may be obtained based on the encoding of image data of the target document, and the visual feature information may be projected into the language embedding space of a large model.

[0158] In the example, image data of the target document can be acquired through methods such as taking a picture or scanning. This image data can be input into a pre-trained visual encoder (such as the NaViT encoder). Through the encoder's feature extraction mechanism, the spatial visual information and local detail features of the image data are transformed into high-dimensional visual feature vectors, forming visual feature information to comprehensively represent the visual attributes of the target document. After acquiring the visual feature information of the target document based on image data, it can be projected into the language embedding space of a large model to achieve cross-spatial alignment of visual modal information and language modal information, providing a unified feature interaction foundation for multimodal information fusion and inference.

[0159] In this way, feature space alignment between visual and linguistic modalities is achieved, breaking down barriers to cross-modal information interaction, improving the semantic understanding depth of large models on visual information of target documents, and significantly enhancing the overall solution's cross-modal processing capabilities and the accuracy of layout inference chain generation.

[0160] In some embodiments, the processing request may be characterized by encoding linguistic feature information. Accordingly, the steps described above for determining the layout information of the target region in the target document associated with the processing request based on the processing request and visual feature information (such as combining...) Figure 7 Step S703 may include: fusing language feature information and visual feature information to obtain layout feature information for representing the layout information of the target area; and decoding the layout feature information to obtain layout information.

[0161] In the example, linguistic feature information can be a vector representation of the processed request. Layout feature information, which integrates linguistic and visual features, can be generated using, for example, the attention mechanism of a Transformer model. This layout feature information can then be decoded and transformed into layout information for the target region.

[0162] In this way, cross-modal fusion of linguistic and visual features is achieved, ensuring the relevance of layout information to user processing requests and its matching degree with document visual features, thereby improving the accuracy of layout information generation.

[0163] In some embodiments, the layout information may include multiple sub-layout information items describing the spatial attributes of the target region. Accordingly, the steps described above for generating a layout inference chain in a natural language manner based on the layout information of the target region (such as combining...) Figure 7 Step S704 may include: filling multiple sub-layout information into a structured output template used to generate a layout inference chain to obtain a layout inference chain.

[0164] In the example, each sub-layout information can correspond to an independent dimension of spatial attributes, so that multiple sub-layout information pieces together constitute a complete description of the spatial features of the target region. The structured output template can be a predefined natural language expression framework. This structured output template can preset the padding values ​​corresponding one-to-one with the dimensions of the sub-layout information to ensure that the resulting text conforms to the parsing habits of large models and human understanding logic.

[0165] Therefore, by using template-based filling, efficient integration of sub-layout information is achieved, which not only ensures the uniformity of the output format and semantic accuracy of the layout inference chain, but also significantly reduces the omission of key spatial attributes. At the same time, it makes the layout inference chain more parsable, effectively improving the generation efficiency and reliability of the generation results.

[0166] In some embodiments, the plurality of sub-layout information may include first sub-layout information and second sub-layout information, wherein the first sub-layout information is associated with the number of layout regions included in the target document and the second sub-layout information is associated with the reading order of the target document.

[0167] In the example, the first sub-layout information may be related to the number of layout regions in the target document; the second sub-layout information may be related to the reading order of the target document. The reading order may refer to the content traversal order that conforms to the general information browsing logic, such as the top-to-bottom or left-to-right order of Chinese documents.

[0168] In this way, the layout inference chain possesses both a quantitative representation of spatial structure and a logical attribute related to semantic reading, significantly improving the depth of the large model's understanding of the target document space and semantic relationships, while ensuring the information integrity of the layout inference chain.

[0169] In some embodiments, the plurality of sub-layout information may further include at least one of the following: an element list of at least one element in the target area, a spatial relationship between at least one element, or a semantic role of at least one element.

[0170] In the example, the list of elements can be a list representation of all elements within the target area; spatial relationships can refer to the orientation and arrangement of these elements, such as relative orientation descriptions like cell A being directly below cell B; semantic roles can be the functional attributes that each element undertakes within the target area, such as title, body text, footnote, or chart title.

[0171] Therefore, by supplementing the sub-layout information of the element dimension, the layout inference chain achieves a dual representation of the macro-spatial structure and the micro-elemental relationship, which greatly improves the understanding depth of the large model of the information hierarchy of the target area and effectively ensures the accuracy of the layout inference chain generation.

[0172] Figure 9 A schematic diagram of a generation layout inference chain according to another embodiment of the present disclosure is shown.

[0173] like Figure 9 As shown, linguistic feature information 901 and visual feature information 902 can be fused to obtain layout feature information 903, which represents the layout information of the target region. Then, the layout feature information 903 can be decoded to obtain layout information 904. The layout information 904 can include multiple sub-layout information describing the spatial attributes of the target region. Further, the multiple sub-layout information contained in the layout information 904 can be filled into a structured output template for generating a layout inference chain 905 to obtain the layout inference chain 905. In this process, the multiple sub-layout information contained in the layout information 904 can include first sub-layout information 9041 associated with the number of layout regions, second sub-layout information 9042 associated with the reading order, an element list 9043 of at least one element in the target region, spatial positional relationships 9044 between at least one element, and semantic roles 9045 for each of at least one element.

[0174] In some embodiments, the layout inference chain can be generated explicitly.

[0175] In the example, the layout inference chain can be output explicitly to achieve traceability and interpretability of the document processing process, providing reasonable reference information for users to consult or for subsequent verification. Explicit output can be relative to the implicit call to the inference chain within the larger model, meaning that the layout inference chain is presented externally as an independent and identifiable information unit along with intermediate inference information and the final document processing result.

[0176] This approach enhances the interpretability of the overall document processing solution, enabling users to clearly trace the layout and derivation of the final result, thereby increasing the credibility and persuasiveness of the result and further improving the practicality and robustness of the technical solution.

[0177] According to embodiments of this disclosure, an apparatus for generating inference data based on a large model is also provided.

[0178] Figure 10 A structural block diagram of a large-model-based inference data generation apparatus 1000 according to an embodiment of the present disclosure is shown.

[0179] like Figure 10 As shown, the inference data generation device 1000 may include a processing information acquisition module 1001, an activation condition judgment module 1002, a layout information determination module 1003, and an inference data generation module 1004. The processing information acquisition module 1001 is configured to acquire a processing request for processing a target document, and visual feature information for characterizing the target document; the activation condition judgment module 1002 is configured to determine whether at least one of the processing request or visual feature information satisfies a preset activation condition for activating a layout inference chain, wherein the layout inference chain is associated with the layout information of the target document; the layout information determination module 1003 is configured to, in response to at least one of the processing request or visual feature information satisfying the preset activation condition, determine the layout information of a target region in the target document associated with the processing request based on the processing request and the visual feature information; and the inference data generation module 1004 is configured to generate a layout inference chain in natural language based on the layout information of the target region, wherein the document processing result used to respond to the processing request is obtained based on the layout inference chain.

[0180] In some embodiments, the activation condition determination module 1002 may include an identification determination module 1002-1 and a first activation module 1002-2. The identification determination module 1002-1 may be configured to determine whether a processing request is used for character-level recognition of a target region in a target document. The first activation module 1002-2 may be configured to determine that the processing request meets a preset activation condition in response to determining that the processing request is used for character-level recognition.

[0181] In some embodiments, the activation condition determination module 1002 may further include a spatial locator determination module 1002-3 and a second activation module 1002-4. The spatial locator determination module 1002-3 may be configured to determine whether the processing request includes a spatial locator, where the spatial locator indicates a target region in the target document. The second activation module 1002-4 may be configured to determine that the processing request meets a preset activation condition in response to determining that the processing request includes a spatial locator.

[0182] In some embodiments, the activation condition determination module 1002 may further include an element relationship acquisition module 1002-5 and a third activation module 1002-6. The element relationship acquisition module 1002-5 may be configured to determine whether the processing request is used to acquire multi-element relationships, where multi-element relationships include the relationships between multiple target elements in the target document, and the target region includes multiple target elements. The third activation module 1002-6 may be configured to determine that the processing request satisfies a preset activation condition in response to determining that the processing request includes multi-element relationships.

[0183] In some embodiments, the activation condition determination module 1002 may further include a mixed layout module 1002-7 and a fourth activation module 1002-8. The mixed layout module 1002-7 may be configured to determine whether visual feature information represents a mixed layout, wherein the mixed layout indicates that the target document includes at least one of multiple layout regions or multiple element types. The fourth activation module 1002-8 may be configured to determine, in response to determining that the visual feature information represents a mixed layout, that the processing request meets a preset activation condition.

[0184] In some embodiments, the activation condition judgment module 1002 may further include an understanding and recognition module 1002-9 and an inactivation determination module 1002-10. The understanding and recognition module 1002-9 may be configured to determine whether the processing request is used to understand the semantics of the target document as a whole or to identify a specified individual element. The inactivation determination module 1002-10 may be configured to, in response to determining that the processing request is used to understand the semantics of the target document as a whole or to identify a specified individual element, determine that the processing request does not meet a preset activation condition.

[0185] In some embodiments, the processing request can be represented by linguistic feature information via encoding. Accordingly, the layout information determination module 1003 may further include a fusion module 1003-1 and a decoding module 1003-2. The fusion module 1003-1 can be configured to fuse linguistic feature information and visual feature information to obtain layout feature information representing the layout information of the target region. The decoding module 1003-2 can be configured to decode the layout feature information to obtain the layout information.

[0186] In some embodiments, the layout information may include multiple sub-layout information for describing the spatial attributes of the target area. Accordingly, the inference data generation module 1004 may include a filling module 1004-1. The filling module 1004-1 may be configured to fill the multiple sub-layout information into a structured output template for generating a layout inference chain to obtain a layout inference chain.

[0187] According to embodiments of this disclosure, a text recognition method based on a large model is provided.

[0188] Figure 11 A flowchart of a large-model-based text recognition method 1100 according to an embodiment of the present disclosure is shown.

[0189] like Figure 11 As shown, the text recognition method 1100 based on a large model includes steps S1101 to S1104.

[0190] In step S1101, image data of the target document to be text-recognized is acquired.

[0191] In the example, the target document is the document to be processed, such as a manuscript, report, invoice, contract, or brochure. The target document can be provided by the user to the large model, instructing it to perform various processing tasks, such as performing OCR (Optical Character Recognition) on specified parts of the document, extracting content from specified parts of the document, or understanding the semantics of specified parts of the document. Image data can be obtained by the user through methods such as taking a photo or scanning. The large model can provide an interface for interaction with the user, allowing the user to send processing requests and image data to the large model via this interface.

[0192] In step S1102, based on the image data, a first recognition operation based on visual decoding is performed on the target document to obtain a first candidate recognition result. The first candidate recognition result includes a first candidate character, and the first candidate character has a first confidence level.

[0193] In the example, visual decoding can be performed based on OCR recognition. The first candidate recognition result can be a set of characters obtained from preliminary recognition, and the first candidate character can be a single character unit in the set. In step S1103, based on the image data and the first candidate recognition result, a second recognition operation based on language decoding is performed on the target document to obtain a second candidate recognition result. The second recognition operation uses the first candidate recognition result as contextual information for obtaining the second candidate recognition result. The second candidate recognition result includes a second candidate character, and the second candidate character has a second confidence level.

[0194] In this example, language decoding is an alternative to visual decoding. Its principle lies in using the results of visual decoding as semantic contextual information, thereby achieving language decoding. Similarly, the second candidate recognition result can be a set of characters recognized simultaneously with the first candidate recognition result, and the second candidate character can be a single character unit within that set. Visual decoding and language decoding can be performed concurrently.

[0195] In step S1104, based on the first confidence level of the first candidate character in the first candidate recognition result, it is determined whether to select the first candidate recognition result or the second candidate recognition result as the recognition result for text recognition of the target document.

[0196] In the example, the first confidence level directly reflects the reliability of visual decoding. A high confidence level indicates that the recognition effect of the corresponding character is good, while a low confidence level may require correction using language logic.

[0197] Therefore, this embodiment provides a dual-path decoding mechanism that performs visual decoding and language decoding simultaneously and collaboratively, combined with dynamic confidence filtering. This not only preserves the accuracy of visual recognition for clearer characters, but also corrects recognition errors in ambiguous scenarios with the help of language context, effectively improving the reliability and scene adaptability of text recognition.

[0198] In some embodiments, the step of determining whether to select the first candidate recognition result or the second candidate recognition result as the recognition result for text recognition of the target document based on the first confidence level of the first candidate character in the first candidate recognition result (such as combining...) Figure 11 Step S1104 may include: determining the first candidate character in the first candidate recognition result as the recognition result in response to the first confidence level of the first candidate character being greater than or equal to the first threshold; and determining the second candidate character in the second candidate recognition result as the recognition result in response to the first confidence level of the first candidate character being less than the first threshold.

[0199] In the example, the first confidence level of each first candidate character in the first candidate recognition result can be compared with the first threshold one by one. When the first confidence level is greater than or equal to the first threshold, it means that the visual decoding result of the first candidate character has sufficient reliability and does not need to be corrected by language decoding. At this time, the first candidate character can be directly determined as the final recognition result. When the first confidence level is less than the first threshold, it indicates that the visual recognition of the first candidate character is inaccurate or uncertain. Since the second candidate recognition result is the result after combining the language context logic decoding, it may be more accurate than the first candidate recognition result. Therefore, the corresponding second candidate character can be determined as the final recognition result.

[0200] By using thresholding, we can avoid redundant language decoding of high-confidence characters and compensate for the visual recognition defects of low-confidence characters through language correction. This effectively improves the accuracy and adaptability of text recognition, while ensuring a balance between recognition efficiency and reliability in different resolution scenarios.

[0201] In some embodiments, the step of determining whether to select the first candidate recognition result or the second candidate recognition result as the recognition result for text recognition of the target document based on the first confidence level of the first candidate character in the first candidate recognition result (such as combining...) Figure 11 Step S1104 may further include: in response to the first confidence level of the first candidate character being less than the first threshold and greater than or equal to the second threshold, determining the weight ratio of the first confidence level to the sum of the first confidence level and the second confidence level; and determining the first candidate character or the second candidate character as the recognition result based on the magnitude of the weight ratio.

[0202] In the example, the second threshold can be a confidence level lower than the first threshold; for example, if the first threshold is 0.8, the second threshold can be 0.5. The weight ratio can be the ratio of the first confidence level to the sum of the first and second confidence levels, to quantify the reliability weight of the visual recognition result relative to the language decoding result. When the first confidence level of the first candidate character is detected to be less than the first threshold but greater than or equal to the second threshold, it may mean that the first confidence level of the first candidate character is moderate. In this case, the weighted fusion method of this embodiment is used. A weight ratio threshold can be preset. If the weight ratio is greater than or equal to the threshold, it indicates that the visual matching reliability of the first candidate character is better than the semantic coherence reliability of the second candidate character, and the first candidate character can be selected as the recognition result. If the weight ratio is less than the threshold, it indicates that language decoding based on semantic coherence can better compensate for the uncertainty of visual recognition, and the second candidate character can be selected as the recognition result.

[0203] By using weighted ratios, the limitations of relying solely on visual or language decoding are avoided, while maximizing the advantages of both decoding methods, effectively improving the accuracy of text recognition in complex scenarios.

[0204] Figure 12 A flowchart of a text recognition method 1200 according to another embodiment of the present disclosure is shown.

[0205] like Figure 12 As shown, the text recognition method 1200 includes steps S1201 to S1203.

[0206] In step S1201, an initial recognition result for text recognition of the target document is obtained according to the text recognition method described above. The initial recognition result includes at least one initial character.

[0207] In step S1202, it is determined whether each of the at least one initial character is reasonable in the context, which includes the at least one initial character.

[0208] In the example, after obtaining the initial recognition result, the recognition accuracy can be further optimized through contextual rationality verification and correction. This compensates for potential local character errors in the text recognition during the dual-path decoding stage, as described above, ensuring the semantic coherence and overall accuracy of the recognition result. Rationality determination can be based on linguistic logic, such as grammatical rules, semantic associations, and common collocations, to judge the compatibility of a single initial character with the preceding and following initial characters. For example, when "document" is misrecognized as "document file," placing "file" after "document" in the character sequence lacks linguistic rationality.

[0209] In step S1203, in response to determining that any of the at least one initial character is not reasonable in the context, the initial character is corrected to obtain a first updated recognition result.

[0210] In the example, when any initial character is detected as not being reasonable in the context, a correction process can be initiated to obtain the first updated recognition result.

[0211] In this way, the shortcomings of the preceding recognition process in considering the coherence of character sequences are not compensated for, and the semantic-driven correction improves the linguistic standardization and accuracy of the recognition results, effectively enhancing the adaptability of text recognition methods in complex scenarios.

[0212] In some embodiments, the step of determining whether each of the at least one initial character is reasonable in the context (such as in combination) Figure 12 Step S1202 may include: determining the current initial character to be scored for reasonableness, the current initial character having a predetermined number of preceding initial characters and following initial characters; determining the reasonableness score of the current initial character based on the preceding initial characters and following initial characters; determining that the current initial character is reasonable in the context in response to determining that the reasonableness score is greater than or equal to a third threshold; and determining that the current initial character is not reasonable in the context in response to determining that the reasonableness score is less than the third threshold.

[0213] In the example, the current initial character can refer to the single initial character unit to be scored for reasonableness; the predetermined number of preceding and following initial characters can be the character sequences adjacent to the current initial character, for example, 3 to 5. The preceding and following initial characters can constitute a local context window for the current initial character to ensure that the validation fully utilizes the semantic and syntactic relationships between characters. The reasonableness score can be an indicator that quantifies the degree of fit between the current initial character and the local context.

[0214] This approach enables accurate assessment of character validity, avoiding the limitations of single-character verification and providing a clear basis for subsequent targeted corrections, thus effectively improving the semantic coherence and overall accuracy of text recognition results.

[0215] In some embodiments, the above-described step of correcting the initial characters in response to determining that any one of the at least one initial characters is not reasonable in the context, to obtain a first updated recognition result (such as in combination with...) Figure 12 The described step S1203) may include: determining at least one candidate character that is glyph - similar to the current initial character; determining the one among the current initial character and the at least one candidate character that has the highest rationality score in the context; determining the one with the highest rationality score as the replacement character; and replacing the current initial character with the replacement character.

[0216] In an example, at least one candidate character that is glyph - similar may refer to a set of characters that are similar to the current initial character in terms of stroke structure, morphological outline, etc. (such as "术” and "木”, "己” and "已”, etc.). When selecting candidate characters, the glyph features of the current initial character can be extracted, and characters with high feature matching degrees can be screened out by calculating the feature similarity between the two to form at least one candidate character that is glyph - similar.

[0217] In an example, the current initial character and each candidate character can be respectively substituted into the above - mentioned context window, and the rationality score of each character in this context can be determined one by one. The character with the highest score can be screened out and determined as the replacement character. In the initial recognition result, the replacement character can be used to replace the current initial character that is not reasonable in the context, and the remaining initial characters with rationality remain unchanged to obtain the first updated recognition result.

[0218] In this way, it not only compensates for the misjudgment defect caused by similar glyphs in visual recognition but also ensures the semantic coherence between the corrected character and the overall text, effectively improving the accuracy and language normativity of the text recognition result and enhancing the robustness of the method in complex glyph recognition scenarios.

[0219] Figure 13 Shows a schematic diagram of correcting an initial character according to an embodiment of the present disclosure.

[0220] As Figure 13 Shown in the figure, in the initial recognition result "文挡理解” (Document Understanding), when the current initial character 1301 is "挡” (block), the previous initial character 1302 is "文” (article), and the subsequent initial character 1303 is "理解” (understanding). When it is determined that the current initial character 1301 "挡” is not reasonable in the context, at least one candidate character 1304 that is glyph - similar to the current initial character 1301 "挡” can be determined first, such as "档” (file), "裆” (crotch), "铛” (bell), etc. Then, it can be determined that the one among the current initial character 1301 "挡” and these candidate characters 1304 that has the highest rationality score is "档” (file), and this character is determined as the replacement character 1305. Finally, the current initial character 1301 "挡” can be replaced with the replacement character 1305 "档” (file) to obtain the first updated recognition result "文档理解” (Document Understanding).

[0221] In some embodiments, as Figure 12 The method 1200 described above may further include: obtaining a request for checking the first update recognition result, the request including the first update recognition result and a prompt word for indicating whether to check for preset problems in the first update recognition result; and determining whether to correct the first update recognition result based on the request.

[0222] In an example, the secondary check and correction process for the first update recognition result can be to receive a customized check request to specifically solve the recognition problems not covered by the above automatic correction, so as to achieve precise optimization of the recognition result. The prompt word can be a check instruction provided in natural language form, such as "Check for semantic contradictions", "Check for misspelling of professional terms", etc., for clearly indicating the preset problems that the large model needs to detect. The preset problems can be associated with various types of text quality defects, such as spelling mistakes, glyph confusion, etc.

[0223] In an example, after receiving the prompt word for checking and correction, the large model can first perform semantic parsing on the prompt word. Then, based on the parsed preset problems, the large model can perform a targeted check on the first update recognition result. If no preset problems indicated by the prompt word are found in the first update recognition result, it can be determined that no correction is needed and the original first update recognition result remains unchanged; if a preset problem is detected, it can be determined that the correction process for the first update recognition result needs to be started.

[0224] In this way, the correction process is perfected from automatic correction by the large model to personalized correction, which not only makes up for the deficiencies of automatic correction but also meets the personalized correction requirements for the recognition result, further enhancing the scenario adaptability and overall accuracy of the text recognition result.

[0225] In some embodiments, the preset problems may include at least one of whether there is a spelling mistake in the first update recognition result, whether there is glyph confusion in the first update recognition result, and whether the context of the first update recognition result is coherent.

[0226] In an example, a spelling mistake is, for example, miswriting "receive" as "recieve". Glyph confusion is, for example, between "0" and "O", "1" and "I", "辩" and "辨", "己" and "已", etc. Context coherence can be associated with semantic coherence, for example, "文挡理解" (document understanding in Chinese) does not have context coherence.

[0227] In this way, it not only precisely covers the common error types at the character level and text level in text recognition but also improves the accuracy of defect recognition through targeted check logic, effectively covering the omitted errors in the previous correction process, further strengthening the language normativity and context logical coherence of the text recognition result, and adapting to the recognition requirements of more scenario-based texts.

[0228] In some embodiments, the step of determining whether to correct the first updated identification result based on a request may include: in response to determining that there is a preset problem in the first updated identification result, correcting the first updated identification result to obtain a second updated identification result.

[0229] In the example, the second updated recognition result can be the text result formed after the first updated recognition result is corrected for preset problems. Since it contains the reasonable content that has been corrected in the first updated recognition result, it can be optimized only for the detected preset problem part, avoiding redundant modification of the overall recognized text.

[0230] In this way, targeted optimization is achieved, which avoids new errors caused by blind modification, and ensures the linguistic standardization, semantic coherence and overall accuracy of the corrected text. This further improves the error of the entire text recognition process and makes the recognition results more suitable for the needs of actual application scenarios.

[0231] According to embodiments of this disclosure, a text recognition device based on a large model is also provided.

[0232] Figure 14 A structural block diagram of a large-model-based text recognition device 1400 according to an embodiment of the present disclosure is shown.

[0233] like Figure 14 As shown, the text recognition device 1400 may include an image data acquisition module 1401, a first recognition module 1402, a second recognition module 1403, and a recognition result determination module 1404.

[0234] Image data acquisition module 1401 is configured to acquire image data of the target document to be text recognized; first recognition module 1402 is configured to perform a first recognition operation based on visual decoding on the target document based on the image data to obtain a first candidate recognition result, the first candidate recognition result including a first candidate character, the first candidate character having a first confidence level; second recognition module 1403 is configured to perform a second recognition operation based on language decoding on the target document based on the image data and the first candidate recognition result to obtain a second candidate recognition result, the second recognition operation using the first candidate recognition result as context information for obtaining the second candidate recognition result, the second candidate recognition result including a second candidate character, the second candidate character having a second confidence level; and recognition result determination module 1404 is configured to determine, based on the first confidence level of the first candidate character in the first candidate recognition result, to select either the first candidate recognition result or the second candidate recognition result as the recognition result for text recognition of the target document.

[0235] In some embodiments, the recognition result determination module 1404 may include a first sub-recognition result determination module 1404-1 and a second sub-recognition result determination module 1404-2. The first sub-recognition result determination module 1404-1 may be configured to determine the first candidate character in the first candidate recognition results as the recognition result in response to a first confidence level of the first candidate character being greater than or equal to a first threshold. The second sub-recognition result determination module 1404-2 may be configured to determine the second candidate character in the second candidate recognition results as the recognition result in response to a first confidence level of the first candidate character being less than the first threshold.

[0236] In some embodiments, the recognition result determination module 1404 may further include a weight ratio determination module 1404-3 and a third sub-recognition result determination module 1404-4. The weight ratio determination module 1404-3 may be configured to determine the weight ratio of the first confidence level to the sum of the first confidence level and the second confidence level in response to a first confidence level of the first candidate character being less than a first threshold and greater than or equal to a second threshold. The third sub-recognition result determination module 1404-4 may be configured to determine either the first candidate character or the second candidate character as the recognition result based on the magnitude of the weight ratio.

[0237] Figure 15 A structural block diagram of a text recognition device 1500 according to another embodiment of the present disclosure is shown.

[0238] like Figure 15 As shown, the text recognition device 1500 may include an initial recognition result acquisition module 1501, a reasonableness determination module 1502, and an initial character correction module 1503. The initial recognition result acquisition module 1501 is configured to, according to the text recognition device described above (such as in combination with...), Figure 14 The large model-based text recognition device 1400 acquires an initial recognition result for text recognition of a target document, the initial recognition result including at least one initial character; the rationality determination module 1502 is configured to determine whether each of the at least one initial character is rational in the context, the context including at least one initial character; and the initial character correction module 1503 is configured to correct the initial character in response to determining that any of the at least one initial character is not rational in the context, so as to obtain a first updated recognition result.

[0239] In some embodiments, the rationality determination module 1502 may include a current initial character determination module 1502-1, a rationality score determination module 1502-2, a first rationality judgment module 1502-3, and a second rationality judgment module 1502-4. The current initial character determination module 1502-1 may be configured to determine the current initial character to be scored for rationality, the current initial character having a predetermined number of preceding and following initial characters. The rationality score determination module 1502-2 may be configured to determine the rationality score of the current initial character based on the preceding and following initial characters. The first rationality judgment module 1502-3 may be configured to determine that the current initial character is rational in the context in response to determining that the rationality score is greater than or equal to a third threshold. The second rationality judgment module 1502-4 may be configured to determine that the current initial character is not rational in the context in response to determining that the rationality score is less than the third threshold.

[0240] In some embodiments, the initial character correction module 1503 may include a candidate character determination module 1503-1, a highest score determination module 1503-2, a replacement character determination module 1503-3, and a replacement module 1503-4. The candidate character determination module 1503-1 may be configured to determine at least one candidate character that is glyphically similar to the current initial character. The highest score determination module 1503-2 may be configured to determine the one with the highest reasonableness score in the context among the current initial character and at least one candidate character. The replacement character determination module 1503-3 may be configured to determine the one with the highest reasonableness score as the replacement character. The replacement module 1503-4 may be configured to replace the current initial character with the replacement character.

[0241] In some embodiments, the text recognition device 1500 may further include a check request acquisition module 1504 and a recognition result correction module 1505. The check request acquisition module 1504 may be configured to acquire a request for checking a first updated recognition result, the request including the first updated recognition result and a prompt indicating whether the first updated recognition result has preset problems. The recognition result correction module 1505 may be configured to determine whether to correct the first updated recognition result based on the request.

[0242] In some embodiments, the identification result correction module 1505 may further include a correction submodule 1505-1. The correction submodule 1505-1 may be configured to correct the first updated identification result to obtain a second updated identification result in response to determining that a preset problem exists in the first updated identification result.

[0243] Figure 16 A schematic diagram of a document processing model 1600 according to an embodiment of the present disclosure is shown.

[0244] like Figure 16 As shown, the document processing model 1600 may include a visual encoder 1601, an MLP projector 1602, and a large model 1603. The visual encoder 1601 can be used to encode the image data of the target document to obtain visual feature information. This visual feature information can be projected into the language embedding space of the large model 1603 via the MLP projector 1602 to achieve alignment between visual modal information and language modal information. The large model 1603 according to embodiments of this disclosure provides a layout inference chain generation mechanism, which can autonomously decide whether to activate it based on the task type. For example, for tasks requiring spatial positioning, a specific layout description can be automatically generated, while for overall understanding tasks, layout inference can be skipped and the processing request can be answered directly. Since various aspects of the large model 1603 according to embodiments of this disclosure have been described in detail above in conjunction with various embodiments, they will not be repeated here.

[0245] According to embodiments of this disclosure, a method for training a document processing model is also provided.

[0246] Figure 17 A flowchart of a training method 1700 for a document processing model according to an embodiment of the present disclosure is shown.

[0247] like Figure 17 As shown, training method 1700 includes steps S1701 and S1702.

[0248] In step S1701, a sample training dataset for training is obtained. The sample training dataset includes a first subset associated with the recognition task, a second subset associated with the question-answering task, and a third subset associated with the understanding task. The proportions of data among the first, second, and third subsets are dynamically adjusted across multiple training phases.

[0249] In the examples, recognition tasks can refer to recognizing content within a document, such as OCR (Optical Character Recognition). Question-answering tasks can refer to answering user questions based on the document's content, i.e., document-based question answering, such as locating specific areas in a document to extract relevant content. Comprehension tasks can refer to understanding and reasoning about the document's content, i.e., reading comprehension, such as inferring from specific content within a document or understanding the overall semantics of the document.

[0250] In the example, these three types of tasks are not independent during the training of the document processing model, but rather can reinforce each other. For instance, regarding the recognition and question-answering tasks, on the one hand, they can share fine-grained visual perception capabilities; on the other hand, the recognition task can enhance character-level recognition to aid the localization of the question-answering task, while the question-answering task can enhance spatial localization to aid the layout understanding of the recognition task. Regarding the question-answering and understanding tasks, on the one hand, they can share semantic understanding capabilities; on the other hand, the question-answering task can enhance information extraction to aid the induction of the understanding task, while the understanding task can enhance context to aid the reasoning of the question-answering task. Regarding the recognition and understanding tasks, on the one hand, they can share document structure understanding; on the other hand, the recognition task can provide accurate content as a foundation for the understanding task, while the understanding task can provide semantic priors for error correction in the recognition task.

[0251] In the example, the proportions of data among the subsets corresponding to these three tasks are not constant during the training of the document processing model, but are dynamically adjusted at multiple stages of training. For example, these multiple stages can be divided into at least early, middle, and late stages, so that different tasks are emphasized at different stages. For instance, the early stage can focus on recognition tasks to establish basic recognition capabilities, the middle stage can focus on balanced training among various tasks, and the late stage can focus on more advanced tasks, such as understanding tasks.

[0252] In step S1702, following the sequence of the multiple stages, multi-task collaborative learning is performed on the document processing model based on the corresponding data proportion in each stage, so that the document processing model can simultaneously learn the recognition task, question answering task, and understanding task. The document processing model can process at least one of the recognition task, question answering task, and understanding task by generating a layout inference chain associated with the layout information of the target document.

[0253] In the example, the document processing model is able to generate layout inference chains during task processing. As mentioned earlier, a layout inference chain is a spatial specification written in natural language by the document processing model, used to describe the spatial layout characteristics of a document. That is, the layout inference chain includes layout information describing these spatial layout characteristics. This layout inference chain is used by the document processing model itself during intermediate inference to obtain answers associated with user questions or instructions. Therefore, to train the document processing model to possess this ability during task processing, multi-task collaborative learning will be applied to the document processing model, enabling it to simultaneously learn recognition tasks, question-answering tasks, and understanding tasks.

[0254] In the example, as mentioned earlier, the data ratios among the respective subsets of these three tasks are dynamically adjusted in each training stage; that is, each stage has a corresponding data ratio for each of the three tasks. Therefore, as the training process progresses, the document processing model will undergo multi-task collaborative learning based on the corresponding data ratios in each stage, following the sequence of the multiple training stages.

[0255] Therefore, the training method for the document processing model according to embodiments of this disclosure provides an effective multi-task collaborative learning mechanism. By dynamically adjusting the amount of training data for each of the recognition, question-answering, and understanding tasks in each training stage, the document processing model can learn these multiple tasks simultaneously, thereby optimizing each task synchronously in the same gradient flow. Since the training method of this disclosure enables multiple tasks to mutually reinforce each other and generate synergy, compared to traditional single-task training mechanisms, this synergy can effectively improve the overall training effect of the document processing model.

[0256] In some embodiments, the first subset may have a first quantity, the second subset may have a second quantity, and the third subset may have a third quantity. The first quantity, the second quantity, and the third quantity may have an initial baseline ratio, such that at least one of the first quantity, the second quantity, or the third quantity is dynamically adjusted based on the initial baseline ratio in each stage.

[0257] In the example, the initial baseline ratio of the first, second, and third quantities can indicate the initial training data allocation for the three tasks: recognition, question answering, and comprehension, such as 40%:40%:20%. For example, the recognition task may include recognizing text, tables, formulas, charts, etc.; the question answering task may include locating and extracting information or finding information; and the comprehension task may include forming summaries, making inferences, or performing comparative analysis.

[0258] In the example, as the training process progresses through different stages, a dynamic adjustment strategy can be adopted for the training data allocation of these three tasks. For instance, in the early stages of training, since the focus can be on the recognition task to establish basic recognition capabilities, the first quantity corresponding to the recognition task can be increased, while the third quantity corresponding to the understanding task can be decreased. In the middle stages of training, since the focus can be on balanced training, the training data allocation can be restored to the initial allocation, that is, the first, second, and third quantities can be maintained at the initial baseline ratio. In the later stages of training, since the focus can be on higher-level tasks, such as the understanding task, the first quantity corresponding to the recognition task can be decreased, while the third quantity corresponding to the understanding task can be increased.

[0259] By dynamically adjusting the training data, a multi-task collaborative learning mechanism can be more effectively integrated into the training iteration process. By adopting appropriate training data ratios at different training stages, the gradient contributions of the three types of tasks—identification, question answering, and understanding—can be balanced in a way that suits local conditions, thereby improving the training effect of the document processing model.

[0260] In some embodiments, the aforementioned multiple stages may be divided according to a predetermined ratio based on the total number of training epochs. These multiple stages may include a first stage to a third stage, such that the first quantity gradually decreases from the first stage to the third stage, while the third quantity gradually increases from the first stage to the third stage.

[0261] In the example, the early stage of training can be the first phase, which may include the first 30% of the total training rounds; the middle stage can be the second phase, which may include the middle 30% to 70% of the total training rounds; and the later stage can be the third phase, which may include the remaining rounds. Following this division, the predetermined ratio of the first to third stages based on the total training rounds could be 30%:40%:30%.

[0262] In the example, in the early stages of training, the ratio of the first, second, and third quantities can be adjusted to 60%:30%:10%. In the middle stages of training, the ratio can be restored to the initial baseline ratio of 40%:40%:20%. In the later stages of training, the ratio can be adjusted again to 30%:40%:30%. Following this exemplary dynamic adjustment strategy, the first quantity can gradually decrease from the first stage to the third stage, while the third quantity can gradually increase from the first stage to the third stage. The second quantity can essentially maintain the ratio from the initial baseline.

[0263] By dividing the training into different stages according to a predetermined ratio based on the total number of rounds, we can accurately quantify the overall training process and integrate the multi-task collaborative learning mechanism of different stages into the training process. This will facilitate the application of corresponding adjustment strategies for the training data allocation at each stage, thereby improving the training effect of the document processing model.

[0264] Figure 18 A schematic diagram of a dynamic adjustment strategy for training data according to an embodiment of the present disclosure is shown.

[0265] like Figure 18 As shown, the first subset of the dataset associated with the recognition task can have a first number R1, the second subset of the dataset associated with the question-answering task can have a second number R2, and the third subset of the dataset associated with the understanding task can have a third number R3. The training process can include a first stage, a second stage, and a third stage. The first stage, the second stage, and the third stage can be divided according to a predetermined ratio based on the total number of rounds in the training process. Taking a total of 40 rounds as an example, the first stage can include rounds 1 to 12, the second stage can include rounds 13 to 28, and the third stage can include rounds 29 to 40.

[0266] In the example, the initial baseline ratio of the first quantity R1, the second quantity R2, and the third quantity R3 is described as 40%:40%:20%. The proportion of the first quantity R1 in the first stage can increase from 40% to 60%, the proportion in the second stage can decrease back to 40%, and the proportion in the third stage can decrease again from 40% to 30%. Therefore, the first quantity R1 can gradually decrease from the first stage to the third stage. Conversely, the proportion of the third quantity R3 in the first stage can decrease from 20% to 10%, the proportion in the second stage can increase back to 20%, and the proportion in the third stage can increase again from 20% to 30%. Therefore, the third quantity R3 can gradually increase from the first stage to the third stage. The proportion of the second quantity R2 can be maintained at approximately 40% across the three stages. In this way, a dynamic adjustment strategy based on the training data can be applied at various stages of the overall training process to improve the training effect of the document processing model.

[0267] In some embodiments, the aforementioned multiple stages may have different learning strategies, such that character-level recognition, layout inference chain generation, and knowledge transfer between different tasks are learned sequentially according to the order of the multiple stages.

[0268] In the example, different learning strategies can be designed as the training process progresses, i.e., according to the sequence of multiple training stages. Taking the early, middle, and late stages as examples, since the early training objective can be to establish basic recognition capabilities, character-level recognition such as OCR recognition can be learned, such as learning to accurately recognize characters, tables, formulas, etc. Since the training objective in the middle stage can be more balanced, and considering that the basic recognition capabilities have been basically established by the early training, the learning of generating layout reasoning chains can be introduced in this stage, such as learning to generate structured layout descriptions. In the late stage of training, the training objective can be aimed at enhancing comprehension capabilities, so further mixed training of recognition tasks, question-answering tasks, and comprehension tasks can be carried out to learn knowledge transfer between different tasks.

[0269] By applying explicit course learning strategies at each stage of the overall training process, it is beneficial to integrate multi-task collaborative learning mechanisms into the training process, thereby improving the training effect of document processing models.

[0270] In some embodiments, the training method of the document processing model (such as combining) Figure 17 The training method 1700 may further include: fine-tuning the trained document processing model based on reinforcement learning (RLHF) techniques derived from human feedback to optimize the quality of the layout inference chain.

[0271] In the example, the total number of rounds in the overall training process can also include a phase for this fine-tuning, which can be performed after the early, middle, and late stages described above. In this case, training can include four phases, in which, in the fourth phase, RLHF reinforcement can be further introduced on top of the full-task hybrid training in the third phase to optimize the quality of the layout inference chain and the accuracy of the answer.

[0272] Since the basic multi-task hybrid capability of the document processing model can be built in the preceding training phase, the RLHF technology can help to further improve the training effect of the document processing model, thereby enabling it to have better layout inference chain generation capabilities.

[0273] In some embodiments, training may be performed based on a preset loss function, which may include a joint loss function. The joint loss function may include a first cross-entropy loss associated with the recognition task, a second cross-entropy loss associated with the question-answering task, a third cross-entropy loss associated with the understanding task, and a fourth cross-entropy loss associated with the quality of the layout inference chain.

[0274] In the example, the joint loss function can be represented as the sum of four losses, namely L_ocr + λ1×L_qa + λ2×L_comprehension + λ3×L_reasoning_chain, where L_ocr represents the first cross-entropy loss associated with the recognition task, L_qa represents the second cross-entropy loss associated with the question-answering task, L_comprehension represents the third cross-entropy loss associated with the comprehension task, and L_reasoning_chain represents the fourth cross-entropy loss associated with the quality of the layout reasoning chain, and λ1, λ2, and λ3 are the weights of the second, third, and fourth cross-entropy losses, respectively.

[0275] By constructing the joint loss function as described above, the multi-task collaborative learning strategy can be effectively applied to the training iteration, enabling the three types of tasks—identification, question answering, and understanding—to be optimized synchronously in the same gradient flow, thereby helping to improve the training effect of the document processing model.

[0276] In some embodiments, the first cross-entropy loss may have a larger loss weight than the second to fourth cross-entropy losses.

[0277] In the example, the weights λ1, λ2, and λ3 corresponding to the second to fourth cross-entropy losses can all be less than 1, while the first cross-entropy loss can have a weight of 1. This means that the recognition task can be the main driver, while other tasks are used as additional drivers, so as to optimize multiple tasks synchronously in the same gradient flow.

[0278] In this way, the construction of a joint loss function can effectively realize a multi-task collaborative learning strategy, thereby helping to improve the training effect of the document processing model.

[0279] In some embodiments, the fourth cross-entropy loss may include a first sub-loss associated with the format correctness of the layout inference chain, a second sub-loss associated with the relevance of the layout inference chain, and a third sub-loss associated with the simplicity of the layout inference chain.

[0280] In the example, the fourth cross-entropy loss can be represented as the sum of three sub-losses, namely, α×L_format + β×L_relevance + γ×L_conciseness, where L_format represents the first sub-loss associated with the format correctness of the layout inference chain, L_relevance represents the second sub-loss associated with the relevance of the layout inference chain, and L_conciseness represents the third sub-loss associated with the conciseness of the layout inference chain, and α, β, and γ are the weights of the first, second, and third sub-losses, respectively.

[0281] In the example, the correctness of the layout inference chain format can include whether the layout inference chain follows a preset format (such as whether it includes ""). <layout> "and"< / layout> (Word pairs). The relevance of the layout reasoning chain can include whether the layout reasoning chain contains the information needed to answer the question. The conciseness of the layout reasoning chain can include whether the layout reasoning chain contains a lengthy description that will be penalized.

[0282] By further refining the fourth cross-entropy loss, which is associated with the quality of the layout inference chain, into three sub-losses—format correctness, relevance, and conciseness—these multiple aspects can be optimized simultaneously in the same gradient flow, thereby effectively improving the training performance of the document processing model.

[0283] Figure 19 A flowchart of a document processing method 1900 according to another embodiment of the present disclosure is shown.

[0284] like Figure 19 As shown, document processing method 1900 includes steps S1901 and S1902.

[0285] In step S1901, a processing request for processing the target document and image data of the target document are obtained.

[0286] In the example, the target document, also known as the document to be processed, is such as a manuscript, report, invoice, contract, or brochure. The processing request can include information associated with the user's intent, presented as a question or instruction. These questions or instructions can be understood by the document processing model to execute the user-instructed processing. Simultaneously, the target document needs to be provided to the document processing model as image data. This image data can be obtained by the user through methods such as taking a photo or scanning. The document processing model can provide an interface for interaction with the user, allowing the user to send processing requests and image data to the document processing model via this interface.

[0287] In step S1902, the processing request and image data are provided to the document processing model trained according to the method described above, so as to output the document processing result for responding to the processing request.

[0288] In the example, if the document processing model determines that the layout inference chain needs to be activated during processing, it can obtain the document processing result that responds to the processing request based on the generated layout inference chain. The processing request can involve various different tasks, such as recognition tasks, question answering tasks, and understanding tasks.

[0289] In the document processing method according to the embodiments of this disclosure, since the document processing model has effectively learned the generation of layout inference chains through the above training method and has also performed multi-task collaborative learning, the layout inference chains associated with the task can be activated as needed, thereby intelligently completing various complex end-to-end document processing tasks.

[0290] According to embodiments of this disclosure, a training apparatus for a document processing model is also provided.

[0291] Figure 20 A structural block diagram of a training apparatus 2000 for a document processing model according to an embodiment of the present disclosure is shown.

[0292] like Figure 20 As shown, the training device 2000 may include a dataset acquisition module 2001 and a collaborative learning module 2002. The dataset acquisition module 2001 is configured to acquire a sample training dataset for training. The sample training dataset includes a first subset associated with a recognition task, a second subset associated with a question-answering task, and a third subset associated with a comprehension task. The data ratios among the first, second, and third subsets are dynamically adjusted across multiple training phases. The collaborative learning module 2002 is configured to perform multi-task collaborative learning on the document processing model in each phase, based on the corresponding data ratios, so that the document processing model simultaneously learns the recognition, question-answering, and comprehension tasks. The document processing model can process at least one of the recognition, question-answering, and comprehension tasks by generating a layout inference chain associated with the layout information of the target document.

[0293] In some embodiments, the training device 2000 may further include a fine-tuning module 2003. The fine-tuning module 2003 may be configured to fine-tune the trained document processing model based on RLHF technology to optimize the quality of the layout inference chain.

[0294] According to embodiments of this disclosure, a document processing apparatus is also provided. This document processing apparatus can perform document processing using a document processing model trained via the training apparatus described above. Accordingly, the document processing apparatus may include: an input acquisition module configured to acquire a processing request for processing a target document, and image data of the target document; and an output providing module configured to provide the processing request and image data to the training apparatus described above (e.g., in combination with...). Figure 20 The document processing model trained by the training device 2000 is used to output document processing results for responding to processing requests.

[0295] According to embodiments of this disclosure, another method for training a document processing model is also provided.

[0296] Figure 21 A flowchart of a training method 2100 for a document processing model according to another embodiment of the present disclosure is shown.

[0297] like Figure 21 As shown, training method 2100 includes steps S2101 to S2105.

[0298] In step S2101, sample document images, sample processing requests, and sample layout pseudo-labels are obtained for training. The sample layout pseudo-labels are used to describe the structured information of the sample document images in natural language.

[0299] In the example, the sample documents used for training can be manuscripts, reports, invoices, contracts, brochures, etc., and correspondingly, the sample document images can be obtained by methods such as taking a photo or scanning the sample document. Sample processing requests can be used to instruct specific tasks, such as recognition tasks, question-answering tasks, understanding tasks, etc., in the form of questions or instructions. Sample layout pseudo-labels can be generated based on a dedicated layout detection model. This layout detection model can be a pre-trained model specifically designed to identify corresponding elements (such as text, tables, illustrations, etc.) from images. The layout detection model can output structured information such as element type, bounding box coordinates, and confidence score, and remains frozen throughout the training process, not participating in subsequent gradient updates. Accordingly, structured information such as element type, bounding box coordinates, and confidence score can be converted into a natural language description in the sample layout pseudo-labels.

[0300] In step S2102, the sample document image is encoded to obtain sample visual feature information.

[0301] In the example, the sample document image can be processed via a visual encoder, such as the NaViT encoder, to obtain the corresponding sample visual feature information. A visual encoder can be specifically designed to process image data, converting image information from the image data into visual feature vectors.

[0302] In step S2103, the sample processing request is encoded to obtain sample language feature information.

[0303] In the example, a natural language processing request can be encoded into a language feature vector representation of sample language features via a large model. Encoding can include processes such as word segmentation and table lookup.

[0304] In step S2104, the sample language feature information, sample visual feature information, and sample layout pseudo-labels are provided to the document processing model to obtain the candidate layout inference chain output by the document processing model based on structured information and in accordance with a predetermined inference chain format.

[0305] In the example, the sample visual feature information can be aligned with the dimension of the sample language feature information through projection to enable unified input and attention computation for visual and language modalities.

[0306] In this example, as mentioned earlier, the layout inference chain refers to a spatial specification written in natural language by the document processing model, used to describe the spatial layout characteristics of a document. That is, the layout inference chain includes layout information describing these spatial layout characteristics. The candidate layout inference chains generated through training are required to include a predetermined inference chain format, such as lexical pairs indicating the start and end of the layout information. Therefore, the training method in this embodiment, which trains the document processing model to output candidate layout inference chains based on structured information provided by sample layout pseudo-labels, is specifically designed for learning to generate layout inference chains. During this training process, the document processing model is forced to conform to this predetermined inference chain format.

[0307] In step S2105, based on the difference between the candidate layout inference chain and the sample layout pseudo-label, a preset loss function is calculated and backpropagation is performed on the document processing model to update the document processing model.

[0308] In the example, the pre-defined loss function can be constructed jointly based on multiple tasks, such as recognition, question answering, and understanding. That is, the sample layout pseudo-labels can be used as ground truth to compare the trained candidate layout inference chain with them. In this way, the parameters of the document processing model are updated through multiple iterations until training is complete.

[0309] The training method for the document processing model according to the embodiments of this disclosure implements a mechanism based on the generation of learning layout inference chains to complete the training process. In this mechanism, sample layout pseudo-labels can be used to automatically learn how to correctly and effectively generate layout inference chains, thereby enabling the document processing model to intelligently use layout inference chains to assist in document processing, thereby improving the document processing effect of the document processing model.

[0310] In some embodiments, obtaining sample layout pseudo-labels in step S2101 may include: performing layout detection on the sample document image to generate sample layout pseudo-labels.

[0311] In the example, a dedicated layout detection model, as described above, can be used to perform layout detection on the sample document image. The sample layout pseudo-labels obtained through layout detection can describe the structured information such as the type and position of elements in the sample document image in natural language.

[0312] In this way, the spatial layout characteristics of sample document images can be hinted at by the structured information in the sample layout pseudo-labels, which can be used as ground truth to guide the document processing model to learn how to generate layout inference chains, thereby effectively realizing the training process of the document processing model.

[0313] In some embodiments, the above-described layout detection of the sample document image to generate sample layout pseudo-tags may include: detecting at least one element included in the sample document image; determining the reading order of the sample document image based on the at least one element; and generating a descriptive statement for each element based on the reading order, wherein the descriptive statement includes the type and position of the element.

[0314] In the example, suppose the sample document image contains two elements: text and table. For example, the text element "'type': 'text', 'bbox': [10,10,200,50], 'confidence': 0.95" indicates that the element type is text, the bounding box coordinates are [10,10,200,50], and the confidence score is 0.95. The table element "'type': 'table', 'bbox': [10,60,600,400], 'confidence': 0.98" indicates that the element type is table, the bounding box coordinates are [10,60,600,400], and the confidence score is 0.98. Next, the reading order of these two elements can be determined, such as text first, then table. Then, descriptive statements for the text element and descriptive statements for the table element can be generated. That is, at this point, the information obtained from the above detection needs to be converted into natural language descriptions. For example, the description of a text element is "1. Type: text, position (10,10,200,50), description: text paragraph", and the description of a table element is "2. Type: table, position (10,60,600,400), description: table, 2 rows × 3 columns".

[0315] Through this layout detection process, sample layout pseudo-labels can be accurately generated, which describe the structured information such as the type and position of elements in the sample document image in natural language. This can then provide ground values ​​for iterative optimization during the training process of the document processing model.

[0316] In some embodiments, sample layout pseudo-labels may include preset lexical pairs for indicating the start and end of the sample layout pseudo-labels.

[0317] In the example, the preset lexical pair can include " <layout> "and"< / layout> The word pair of ", where " <layout> "It can be used to indicate the start of pseudo-labels in the sample layout,"< / layout> "Can be used to indicate the end of the pseudo-label of the sample layout.

[0318] By specifying specific preset terms to characterize the start and end of pseudo-labels in the sample layout, it is possible to force the document processing model to follow a specific inference chain format during training.

[0319] Figure 22 This is a schematic diagram illustrating the acquisition of sample layout pseudo-labels according to an embodiment of the present disclosure.

[0320] like Figure 22 As shown, all elements in the sample document image 2200 can be detected, such as the first element 2201 and the second element 2202. Through this detection, the type and position of each of the first element 2201 and the second element 2202 can be obtained. For example, the type of the first element 2201 is text, and its position is (A...). 1x A 1y B 1x B 1y The second element, 2202, is of type table and is located in (A). 2x A 2y B 2x B 2y ), where A 1x A 1y B can represent the coordinates of the top-left corner of the bounding box of the first element 2201. 1x B 1y This can represent the coordinates of the bottom right corner of the bounding box of the first element 2201; A 2x A 2y B can represent the coordinates of the top-left corner of the bounding box of the second element 2202. 2x B 2y This can represent the coordinates of the bottom right corner of the bounding box of the second element 2202.

[0321] Following this, the reading order of the sample document image 2200 can be determined based on the first element 2201 and the second element 2202: the first element 2201 first, followed by the second element 2202. According to this reading order, descriptive statements can be generated for the first element 2201 and the second element 2202 respectively. In the example, the first descriptive statement for the first element 2201 can be "1. Type: text, position (A 1x A 1y B 1x B 1y ), Description: text paragraph", the second descriptive statement for the second element 2202 can be "2. Type: table, position (A 2x A 2y B 2x B 2y Description: Table, 2 rows x 3 columns. Therefore, the sample layout pseudo-tags can include this first description statement and the second description statement, and can also be accessed via " <layout> "and"< / layout> The lexical pair encloses the first and second descriptive statements to indicate the start and end of the sample layout pseudo-labels. Thus, sample layout pseudo-labels can be generated by performing layout detection on the sample document image 2200.

[0322] In some embodiments, the training method of the document processing model (such as combining) Figure 21 The training method 2100 for the document processing model may further include: based on the sample document image, performing semantic simplification or redundancy removal on the sample layout pseudo-labels to obtain the first optimized sample layout pseudo-labels for use in the document processing model.

[0323] In the example, a large model can be used to refine the pseudo-labels of the sample layout, making them more natural, accurate, and concise. That is, the pseudo-labels of the sample layout can be semantically simplified or have redundancy removed to achieve a more natural, accurate, and concise refinement. To this end, prompts can be constructed to use the large model to perform the above refinement process. For example, these prompts can include role-setting content, such as "You are a document layout analysis expert. Given a document image and preliminary layout analysis results, please optimize the description to make it more natural, accurate, and concise," and can also include requirements for the large model to refine, such as "Use natural language, avoid being too mechanical," and "Describe only key information, remove redundancy," etc.

[0324] By refining the pseudo-labels of the sample layout, we can obtain true values ​​with higher standards. This makes it easier for the document processing model to iteratively optimize towards these true values ​​during the training process, thereby improving the training effect.

[0325] In some embodiments, the training method of the document processing model (such as combining) Figure 21 The training method 2100 for the document processing model may further include: determining whether there is an error in at least one of the sample layout pseudo-labels or the first optimized sample layout pseudo-labels based on the sample document image; and correcting the error in response to determining that there is an error in at least one of the sample layout pseudo-labels or the first optimized sample layout pseudo-labels to obtain a second optimized sample layout pseudo-label for use in the document processing model.

[0326] In the example, a large model can also be used to check for pseudo-labels in the sample layout or the first optimized sample layout. To this end, the prompts used to instruct the large model can also include specifying which aspects need to be checked, such as "check if it is maintained..." <layout> ...< / layout> The format includes phrases such as "If there are obvious errors in the preliminary analysis, please correct them."

[0327] By providing a mechanism to examine samples, the quality of the ground truth during training can be further optimized, which in turn helps the document processing model to iteratively optimize towards the ground truth during the training process, thereby improving training performance.

[0328] In some embodiments, to further improve sample quality, a certain percentage of the sampled samples can be manually reviewed to establish a gold standard. Additionally, low-quality samples can be filtered out based on multiple dimensions of quality assessment, such as format accuracy, content accuracy, and conciseness.

[0329] In some embodiments, the document processing model may include a projection layer for projecting sample visual feature information of sample document images into the language embedding space of the document processing model. Accordingly, the training method of the document processing model (such as combining...) Figure 21 The training method 2100 for the document processing model may further include: acquiring sample image and text training data for training the projection layer, the sample image and text training data being aligned across visual and linguistic modes; and training the projection layer to project sample visual feature information into the linguistic embedding space based on the sample image and text training data, so as to learn the cross-modal alignment between visual and linguistic modes.

[0330] In the example, visual-language alignment pre-training can be performed before the layout inference chain learning. To do this, the visual encoder and large model parts of the document processing model can be frozen, and only the projection layer can be trained, with the goal of aligning visual feature information and the language embedding space. The sample image-text training data can come from general image-text pairs, for example, which may include a scanned image of a document and its full-text OCR recognition. Therefore, the sample image-text training data is cross-modal aligned between vision and language.

[0331] By pre-training the visual-linguistic alignment capability of the projection layer, the document processing model's ability to process both visual and linguistic features can be enhanced, thereby improving the training performance of the document processing model.

[0332] In some embodiments, the preset loss function described above may include a joint loss function. This joint loss function may include a first cross-entropy loss associated with the recognition task, a second cross-entropy loss associated with the question-answering task, a third cross-entropy loss associated with the understanding task, and a fourth cross-entropy loss associated with the quality of the inference chain.

[0333] In the example, the recognition task may include recognizing text, tables, formulas, charts, etc.; the question-answering task may include locating and extracting information or searching for information; and the understanding task may include forming a summary, making inferences, or performing comparative analysis. The joint loss function can be expressed as the sum of four losses, namely, L_ocr + λ1×L_qa + λ2×L_comprehension + λ3×L_reasoning_chain, where L_ocr represents the first cross-entropy loss associated with the recognition task, L_qa represents the second cross-entropy loss associated with the question-answering task, L_comprehension represents the third cross-entropy loss associated with the understanding task, and L_reasoning_chain represents the fourth cross-entropy loss associated with the quality of the layout reasoning chain. λ1, λ2, and λ3 are the weights of the second, third, and fourth cross-entropy losses, respectively.

[0334] By constructing the joint loss function as described above, the multi-task collaborative learning strategy can be effectively applied to the training iteration, enabling the three types of tasks—identification, question answering, and understanding—to be optimized synchronously in the same gradient flow, thereby helping to improve the training effect of the document processing model.

[0335] In some embodiments, the first cross-entropy loss may have a larger loss weight than the second to fourth cross-entropy losses.

[0336] In the example, the weights λ1, λ2, and λ3 corresponding to the second to fourth cross-entropy losses can all be less than 1, while the first cross-entropy loss can have a weight of 1. This means that the recognition task can be the main driver, while other tasks are used as additional drivers, so as to optimize multiple tasks synchronously in the same gradient flow.

[0337] In this way, the construction of a joint loss function can effectively realize a multi-task collaborative learning strategy, thereby helping to improve the training effect of the document processing model.

[0338] In some embodiments, the training method of the document processing model (such as combining) Figure 21 The training method 2100 for the document processing model may further include: fine-tuning the trained document processing model based on reinforcement learning (RLHF) techniques derived from human feedback to optimize the quality of the candidate layout inference chain. The reward function used in the RLHF technique may include multiple evaluation items for assessing answer accuracy, reasoning rationality, and conciseness, each with a preset weight.

[0339] In the example, the reward function can comprehensively consider reward factors from multiple dimensions, where the weight of the evaluation item used to assess the accuracy of the answer can be set to 0.7, the weight of the evaluation item used to assess the rationality of the reasoning can be set to 0.2, and the weight of the evaluation item used to assess the simplicity can be set to 0.1.

[0340] By leveraging the reward mechanism of the reward function, the training effect of the document processing model can be further improved by targeting more fine-grained training objectives.

[0341] In some embodiments, training methods via the document processing model described above (such as combining) can be used. Figure 21 The document processing model trained using the training method 2100 is used for document processing. Therefore, various aspects of the corresponding document processing method can be referenced as follows: Figure 19 The document processing method 1900 shown here will not be elaborated further.

[0342] Since the document processing model has effectively learned the generation of layout inference chains through the above training method, it can intelligently complete various complex end-to-end document processing tasks by means of layout inference chains.

[0343] According to embodiments of this disclosure, a training apparatus for a document processing model is also provided.

[0344] Figure 23 A structural block diagram of a training apparatus 2300 for a document processing model according to another embodiment of the present disclosure is shown.

[0345] like Figure 23 As shown, the training device 2300 may include a training sample acquisition module 2301, a first encoding module 2302, a second encoding module 2303, a training result output module 2304, and a model update module 2305. The training sample acquisition module 2301 is configured to acquire sample document images, sample processing requests, and sample layout pseudo-labels for training. The sample layout pseudo-labels are used to describe the structured information of the sample document images in natural language. The first encoding module 2302 is configured to encode the sample document images to obtain sample visual feature information. The second encoding module 2303 is configured to encode the sample processing requests to obtain sample language feature information. The training result output module 2304 is configured to provide the sample language feature information, sample visual feature information, and sample layout pseudo-labels to the document processing model to obtain a candidate layout inference chain output by the document processing model based on structured information and according to a predetermined inference chain format. The model update module 2305 is configured to calculate a preset loss function based on the difference between the candidate layout inference chain and the sample layout pseudo-labels and perform backpropagation on the document processing model to update the document processing model.

[0346] In some embodiments, the training sample acquisition module 2301 may further include a pseudo-label generation module 2301-1. The pseudo-label generation module 2301-1 may be configured to perform layout detection on the sample document image to generate sample layout pseudo-labels.

[0347] In some embodiments, the pseudo-tag generation module 2301-1 may further include an element detection module 2301-1a, a reading order determination module 2301-1b, and a description statement generation module 2301-1c. The element detection module 2301-1a is configured to detect at least one element included in the sample document image. The reading order determination module 2301-1b is configured to determine the reading order of the sample document image based on at least one element. The description statement generation module 2301-1c is configured to generate a description statement for each element based on the reading order, the description statement including the element's type and position.

[0348] In some embodiments, the training device 2300 may further include a semantic optimization module 2306. The semantic optimization module 2306 is configured to perform at least one of semantic simplification or redundancy removal on sample layout pseudo-labels based on sample document images to obtain first optimized sample layout pseudo-labels for use in providing to the document processing model.

[0349] In some embodiments, the training apparatus 2300 may further include a sample error determination module 2307 and a sample error correction module 2308. The sample error determination module 2307 is configured to determine, based on a sample document image, whether an error exists in at least one of the sample layout pseudo-labels or a first optimized sample layout pseudo-label. The sample error correction module 2308 is configured to correct the error in response to determining that an error exists in at least one of the sample layout pseudo-labels or the first optimized sample layout pseudo-label, to obtain a second optimized sample layout pseudo-label for use in providing to the document processing model.

[0350] In some embodiments, the document processing model may include a projection layer for projecting sample visual feature information of sample document images into the language embedding space of the document processing model. Correspondingly, the training apparatus 2300 may also include a projection training data acquisition module 2309 and a projection training module 2310. The projection training data acquisition module 2309 is configured to acquire sample image-text training data for training the projection layer, the sample image-text training data being cross-modal aligned between vision and language. The projection training module 2310 is configured to train the projection layer to project sample visual feature information into the language embedding space based on the sample image-text training data, thereby learning cross-modal alignment between vision and language.

[0351] In some embodiments, the training device 2300 may further include a fine-tuning optimization module 2311. The fine-tuning optimization module 2311 may be configured to fine-tune the trained document processing model based on RLHF technology to optimize the quality of the candidate layout inference chain. The reward function used in RLHF technology may include multiple evaluation items for assessing the accuracy of the answer, the rationality of the reasoning, and the conciseness, each evaluation item having a preset weight.

[0352] According to embodiments of this disclosure, a document processing apparatus is also provided, which can perform document processing using a document processing model trained via the training apparatus of the document processing model described above. Accordingly, the document processing apparatus may include: an input data acquisition module configured to acquire a processing request for processing a target document and image data of the target document; and an output data providing module configured to provide the processing request and image data to the training apparatus as described above (e.g., in combination with...). Figure 23 The document processing model trained by the training device 2300 is used to output document processing results for responding to processing requests.

[0353] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0354] According to embodiments of this disclosure, an electronic device is also provided, including at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0355] According to embodiments of this disclosure, a non-transitory computer-readable storage medium storing computer instructions is also provided, wherein the computer instructions are used to cause a computer to perform the methods described above.

[0356] According to embodiments of this disclosure, a computer program product is also provided, including a computer program, wherein the computer program implements the method described above when executed by a processor.

[0357] refer to Figure 24 The present invention describes a structural block diagram of an electronic device 2400 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0358] like Figure 24 As shown, the electronic device 2400 includes a computing unit 2401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 2402 or a computer program loaded from a storage unit 2408 into a random access memory (RAM) 2403. The RAM 2403 may also store various programs and data required for the operation of the electronic device 2400. The computing unit 2401, ROM 2402, and RAM 2403 are interconnected via a bus 2404. An input / output (I / O) interface 2405 is also connected to the bus 2404.

[0359] Multiple components in electronic device 2400 are connected to I / O interface 2405, including: input unit 2406, output unit 2407, storage unit 2408, and communication unit 2409. Input unit 2406 can be any type of device capable of inputting information to electronic device 2400. Input unit 2406 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device, and may include, but is not limited to, a mouse, keyboard, touchscreen, trackpad, trackball, joystick, microphone, and / or remote control. Output unit 2407 can be any type of device capable of presenting information, and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 2408 may include, but is not limited to, a hard disk and an optical disk. The communication unit 2409 allows the electronic device 2400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers and / or chipsets, such as Bluetooth devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices and / or the like.

[0360] The computing unit 2401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 2401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 2401 performs the various methods and processes described above. For example, in some embodiments, the above methods can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 2408. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 2400 via ROM 2402 and / or communication unit 2409. When the computer program is loaded into RAM 2403 and executed by the computing unit 2401, one or more steps of the methods described above can be performed. Alternatively, in other embodiments, the computing unit 2401 can be configured to perform methods by any other suitable means (e.g., by means of firmware).

[0361] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0362] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0363] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0364] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0365] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0366] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0367] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0368] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the invention is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as the technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.< / confidence> < / bbox>

Claims

1. A method for training a document processing model, comprising: Obtain sample document images, sample processing requests, and sample layout pseudo-labels for training, wherein the sample layout pseudo-labels are used to describe the structured information of the sample document images in natural language. The sample document image is encoded to obtain sample visual feature information; The sample processing request is encoded to obtain sample language feature information; The sample language feature information, the sample visual feature information, and the sample layout pseudo-labels are provided to the document processing model to obtain a candidate layout inference chain output by the document processing model based on the structured information and in accordance with a predetermined inference chain format; and Based on the difference between the candidate layout inference chain and the sample layout pseudo-label, a preset loss function is calculated and backpropagation is performed on the document processing model to update the document processing model.

2. The method according to claim 1, wherein, The acquisition of sample layout pseudo-labels includes: Layout detection is performed on the sample document image to generate sample layout pseudo-labels.

3. The method according to claim 2, wherein, The step of performing layout detection on the sample document image to generate sample layout pseudo-tags includes: Detect at least one element included in the sample document image; Based on the at least one element, determine the reading order of the sample document images; and Based on the reading order, a description statement is generated for each element, wherein the description statement includes the type and position of the element.

4. The method according to claim 2 or 3, wherein, The sample layout pseudo-labels include preset word pairs used to indicate the start and end of the sample layout pseudo-labels.

5. The method according to any one of claims 1 to 4, wherein, The method further includes: Based on the sample document image, the sample layout pseudo-labels are semantically simplified or redundancy-removed at least once to obtain a first optimized sample layout pseudo-label for use in the document processing model.

6. The method according to claim 5, wherein, The method further includes: Based on the sample document image, determine whether there is an error in at least one of the sample layout pseudo-labels or the first optimized sample layout pseudo-labels; and In response to determining that there is an error in at least one of the sample layout pseudo-labels or the first optimized sample layout pseudo-labels, the error is corrected to obtain a second optimized sample layout pseudo-label for use in the document processing model.

7. The method according to any one of claims 1 to 6, wherein, The document processing model includes a projection layer, which projects the sample visual feature information of the sample document image into the language embedding space of the document processing model. The method further includes: Acquire sample image-text training data for training the projection layer, the sample image-text training data being cross-modal aligned between vision and language; and Based on the sample image and text training data, the projection layer is trained to project the visual feature information of the samples into the language embedding space in order to learn cross-modal alignment between vision and language.

8. The method according to any one of claims 1 to 7, wherein, The preset loss function includes a joint loss function, which includes a first cross-entropy loss associated with the recognition task, a second cross-entropy loss associated with the question-answering task, a third cross-entropy loss associated with the understanding task, and a fourth cross-entropy loss associated with the quality of the inference chain.

9. The method according to claim 8, wherein, The first cross-entropy loss has a larger loss weight compared to the second to fourth cross-entropy losses.

10. The method according to any one of claims 1 to 9, wherein, The method further includes: The document processing model, trained using reinforcement learning (RLHF) techniques based on human feedback, is fine-tuned to optimize the quality of the candidate layout inference chain. The reward function used in the RLHF technique includes multiple evaluation items for assessing answer accuracy, reasoning rationality, and conciseness, each with a preset weight.

11. A document processing method, comprising: Obtain a processing request for processing the target document, as well as the image data of the target document; as well as The processing request and the image data are provided to the document processing model trained by the method according to any one of claims 1 to 10 to output a document processing result for responding to the processing request.

12. A training device for a document processing model, comprising: The training sample acquisition module is configured to acquire sample document images, sample processing requests, and sample layout pseudo-labels for training, wherein the sample layout pseudo-labels are used to describe the structured information of the sample document images in natural language. The first encoding module is configured to encode the sample document image to obtain sample visual feature information; The second encoding module is configured to encode the sample processing request to obtain sample language feature information; The training result output module is configured to provide the sample language feature information, the sample visual feature information, and the sample layout pseudo-labels to the document processing model to obtain a candidate layout inference chain output by the document processing model based on the structured information and in accordance with a predetermined inference chain format; and The model update module is configured to calculate a preset loss function based on the difference between the candidate layout inference chain and the sample layout pseudo-label, and then perform backpropagation on the document processing model to update the document processing model.

13. A document processing apparatus, comprising: The input data acquisition module is configured to acquire a processing request for processing the target document, as well as the image data of the target document; as well as An output data providing module is configured to provide the processing request and the image data to a document processing model trained by the apparatus according to claim 12, so as to output a document processing result for feedback on the processing request.

14. An electronic device, comprising: At least one processor; as well as A memory that is communicatively connected to the at least one processor; in The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-11.

15. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-11.

16. A computer program product comprising a computer program, wherein, When the computer program is executed by a processor, it implements the method of any one of claims 1-11.

Images