Layout analysis model training method, layout analysis method and task platform

Abstract

The embodiment of the invention provides a training method of a layout analysis model, a layout analysis method and a task platform, and the training method of the layout analysis model comprises the steps: obtaining a first training sample set, training an initial layout analysis model based on the first training sample set, and obtaining a first layout analysis model, the first training sample set comprises a first training document and positive and negative samples corresponding to the first training document; a second training sample set is obtained, the first layout analysis model is trained based on the second training sample set, a second layout analysis model is obtained, and the second training sample set comprises a second training document and a task sample label corresponding to the second training document; a third training sample set is obtained, the second layout analysis model is trained based on the third training sample set, a target layout analysis model is obtained, and the third training sample set comprises a third training document and sample layout information corresponding to the third training document.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of artificial intelligence technology, and in particular to a training method for a layout analysis model, a layout analysis method, and a task platform. Background Technology

[0002] With the application of large-scale models, Retrieval-Augmented Generation (RAG) technology has become a major research direction for large-scale model applications. Unstructured documents (such as PDFs, DOCs, and PPTs) are important production data in RAG systems. Converting them into structured data for retrieval by the retrieval module within the RAG system has become a major research focus. The more successful the document parsing, the cleaner and more standardized the data in the RAG system, resulting in better performance.

[0003] Page layout analysis, a crucial step in document parsing, determines the effectiveness of subsequent parsing processes. The goal of page layout analysis is to detect key elements within a document, such as titles, body text, tables, illustrations, headers, and footers, and provide their location information. Traditional page layout analysis methods employ object detection, which has the limitation of failing to recognize new categories once the training categories are determined. Furthermore, current methods lack the alignment and fusion of textual and image features. Therefore, there is an urgent need for researchers to develop a novel training method for page layout analysis models to enhance their flexibility and generalization capabilities. Summary of the Invention

[0004] In view of the above, embodiments of this specification provide a training method for a layout analysis model, a layout analysis method, and a task platform. One or more embodiments of this specification also relate to a training device for a layout analysis model, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a method for training a layout analysis model is provided, comprising: Obtain a first training sample set, and train an initial layout analysis model based on the first training sample set to obtain a first layout analysis model. The first training sample set includes a first training document and positive and negative samples corresponding to the first training document. Obtain a second training sample set, and train the first layout analysis model based on the second training sample set to obtain a second layout analysis model. The second training sample set includes a second training document and task sample labels corresponding to the second training document. Obtain a third training sample set, and train the second layout analysis model based on the third training sample set to obtain the target layout analysis model. The third training sample set includes a third training document and sample layout information corresponding to the third training document.

[0006] According to a second aspect of the embodiments of this specification, a layout analysis method is provided, comprising: Obtain the document to be identified and at least one page element to be identified; Generate layout recognition prompt text based on the document image corresponding to the document to be recognized and each layout element to be recognized; The layout recognition prompt text is input into the target layout analysis model to obtain the layout recognition result output by the target layout analysis model, wherein the target layout analysis model is trained by the above training method.

[0007] According to a third aspect of the embodiments of this specification, a layout analysis method is provided, applied to cloud-based devices, comprising: A layout analysis request sent by a receiving end-side device, wherein the layout analysis request carries a document to be identified and at least one layout element to be identified; Generate layout recognition prompt text based on the document image corresponding to the document to be recognized and each layout element to be recognized; The layout recognition prompt text is input into the target layout analysis model to obtain the layout recognition result output by the target layout analysis model, wherein the target layout analysis model is trained by the above training method; The layout recognition result is sent to the terminal device.

[0008] According to a fourth aspect of the embodiments of this specification, a task platform is provided, including a request interface and a response unit; The request interface is used to receive a layout analysis request sent by the end device, wherein the layout analysis request carries a document to be identified and at least one layout element to be identified. The response unit is configured to generate layout recognition prompt text based on the document image to be recognized and each layout element to be recognized; input the layout recognition prompt text into the target layout analysis model to obtain the layout recognition result output by the target layout analysis model, wherein the target layout analysis model is trained by the above-described training method.

[0009] According to a fifth aspect of the embodiments of this specification, a training apparatus for a layout analysis model is provided, comprising: The first training module is configured to acquire a first training sample set and train an initial layout analysis model based on the first training sample set to obtain a first layout analysis model. The first training sample set includes a first training document and positive and negative samples corresponding to the first training document. The second training module is configured to acquire a second training sample set and train the first layout analysis model based on the second training sample set to obtain a second layout analysis model. The second training sample set includes a second training document and task sample labels corresponding to the second training document. The third training module is configured to acquire a third training sample set and train the second layout analysis model based on the third training sample set to obtain the target layout analysis model. The third training sample set includes a third training document and sample layout information corresponding to the third training document.

[0010] According to a sixth aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the above method.

[0011] According to a seventh aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0012] According to an eighth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0013] The training method for the layout analysis model provided in this specification uses three stages to complete the training. The first training stage focuses on training the alignment ability between visual and text features. The second training stage focuses on training the model's ability to handle multiple tasks such as image recognition, image annotation, and image classification. The third training stage focuses on training the model's ability to recognize and output a specified format. Using the method provided in this specification, a layout analysis model capable of recognizing image information, performing specified tasks, and outputting a specified format can be trained, thus giving the trained target layout analysis model better applicability. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating a training method for a layout analysis model provided in one embodiment of this specification; Figure 2This is a flowchart illustrating a layout analysis method provided in one embodiment of this specification; Figure 3 This is a flowchart illustrating a layout analysis method for cloud-based devices provided in one embodiment of this specification. Figure 4 This is a schematic diagram of a task platform provided in one embodiment of this specification; Figure 5 This is a schematic diagram of the structure of a training device for a layout analysis model provided in one embodiment of this specification; Figure 6 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0015] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0016] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0017] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0018] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0019] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundation model. It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0020] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios for large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0021] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0022] LargeVision-LanguageModels: Large-scale visual language models.

[0023] Page layout analysis: Identify elements such as titles, body text, illustrations, and tables in a document.

[0024] Optical Character Recognition.

[0025] ObjectDetection: Object detection.

[0026] With the rise of large-scale models, Retrieval-Augmented Generation (RAG) has become a major research direction for large-scale model applications. Unstructured documents (such as PDFs, DOCs, and PPTs) are important production data in RAG systems. How to transform them into structured data for retrieval by the retrieval module of the RAG system has become a major research direction. The more successful the document parsing, the cleaner and more standardized the data in the RAG system, and the better the results will be.

[0027] Page layout analysis, a crucial step in document parsing, aims to detect key elements such as titles, body text, tables, illustrations, headers, and footers, and provide their location information. Its effectiveness significantly impacts the overall quality of document parsing. Traditional page layout analysis methods employ object detection, which has limitations: once the training categories are determined, new categories cannot be detected during inference; furthermore, it lacks the ability to align and fuse text and image features.

[0028] Based on this, this specification provides a training method for a layout analysis model. This specification also relates to a layout analysis method, a training device for a layout analysis model, a task platform, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0029] See Figure 1 , Figure 1 A flowchart is shown of a training method for a layout analysis model according to an embodiment of this specification, which specifically includes the following steps.

[0030] Step 102: Obtain the first training sample set, and train the initial layout analysis model based on the first training sample set to obtain the first layout analysis model. The first training sample set includes the first training document and the positive and negative samples corresponding to the first training document.

[0031] The method provided in the embodiments of this specification uses a three-stage training method for the layout analysis model, employing different training sample data in three different training stages. The first stage can be understood as the pre-training stage, and the training data used in the first stage is called the first training sample set, which includes the first training document and the corresponding positive and negative samples of the first training document.

[0032] The first training document can be understood as the document used to recognize layout information. Positive samples corresponding to the first training document can be understood as the text recognition information corresponding to the first training document, while negative samples are other text recognition information besides the text recognition information corresponding to the first training document. The purpose of the first stage of training is to align visual encoded features and text encoded features.

[0033] In the first training phase, the initial layout analysis model is trained using a first training sample set through contrastive learning. Contrastive learning is a self-supervised learning method whose core idea is to extract useful features by having the model learn to distinguish between similar and dissimilar samples. During training, the model learns an embedding space in which similar samples are placed closer together, while dissimilar samples are placed further apart. Contrastive learning is a self-supervised learning method that does not require a large amount of labeled data, thus reducing the data acquisition cost of the first training sample set.

[0034] In one specific embodiment provided in this specification, the initial layout analysis model includes a visual encoder, a feature adapter, and a text decoder; The initial layout analysis model is trained based on the first training sample set to obtain the first layout analysis model, including: An initial layout analysis model is trained based on the first training sample set, and the model parameters of the visual encoder and the feature adapter are adjusted to obtain the first layout analysis model.

[0035] In practical applications, the method provided in the embodiments of this specification is required to train a target layout analysis model. The initial layout analysis model can be understood as a large-scale vision-language model. Throughout the three stages of model training, the model structure remains unchanged; that is, the initial layout analysis model includes a visual encoder, feature adapter, and text decoder. The model structure remains the same in subsequent training stages as well. The difference between the different training stages lies in the different model parameters of each component.

[0036] A visual encoder is used to extract layout visual information from a document. In one specific embodiment provided in this specification, VitDet can be used as the visual encoder. VitDet is a ViT (Vision Transformer) based object detection model that combines ViT and object detection to improve the performance of object detection tasks. Utilizing VitDet's local attention can significantly reduce the computational cost of high-resolution images.

[0037] The purpose of a feature adapter is to adapt the visual features encoded by the visual encoder to the text feature space. The feature adapter includes a randomly initialized single-layer cross-attention module that uses a set of trainable vectors as the query vector and the image features obtained from the visual encoder as the key vector. This compresses the visual feature sequence to a fixed length.

[0038] The text decoder can use commonly available text decoders on the market, such as the GPT model.

[0039] In the method provided in the embodiments of this specification, the layout analysis model includes a visual encoder and a text decoder, each containing a visual encoding space and a text encoding space. Since the visual encoding space and the text encoding space belong to different spaces before the model is trained, the visual features and text features converted using the visual encoding space and text encoding space cannot be effectively aligned. In the first training phase, the visual encoder and feature adapter in the layout analysis model need to be trained so that the feature adapter can cooperate with the visual encoder to align the visual features with the text features in the text decoder.

[0040] Specifically, based on the first training sample set, an initial layout analysis model is trained, and the model parameters of the visual encoder and the feature adapter are adjusted to obtain the first layout analysis model, including: Obtain the first training document image of the first training document, and the positive and negative samples corresponding to the first training document; The first training document image is input into the visual encoder to obtain the feature information of the first training document; The feature information of the first training document is input into the feature adapter to obtain the first training document adaptation feature information; The positive and negative samples are input into the text decoder to obtain positive and negative sample feature information. The contrastive loss function value is calculated based on the adaptation feature information of the first training document, the positive sample feature information, and the negative sample feature information. Adjust the model parameters of the visual encoder and the feature adapter according to the contrast loss function value until the first model training stopping condition is met, and obtain the first layout analysis model.

[0041] Furthermore, in another specific embodiment provided in this specification, the explanation is based on a first training document in the first training sample set. First, the first training document image, positive samples, and negative samples of the first training document are obtained.

[0042] In practical applications, the first training document can be a text document, PDF document, XML document, etc. To extract visual features from the document, it needs to be converted into a corresponding image, i.e., the first training document image. In practical applications, the first training document image corresponding to the first training document can also be directly saved in the first training sample set. Positive samples are the text content information in the first training document, and negative samples are other text content information besides the positive samples.

[0043] The first training document image is input into a visual encoder for visual encoding, where the visual encoder extracts the first training document feature information from the first training document image. This first training document feature information is then input into a feature adapter, which converts the first training document feature information into first training document-adaptive feature information aligned with the text feature information.

[0044] In addition, positive and negative samples need to be input into the text decoder. The text decoder maps the positive and negative samples into the text space to obtain positive and negative sample feature information. Then, the adaptation feature information of the first training document is also input into the text decoder, and together with the positive and negative sample feature information, the contrastive loss function value is calculated through contrastive learning.

[0045] Once the contrastive loss function value is obtained, the model parameters of the visual encoder and feature adapter can be adjusted using backpropagation. In the first training phase, the model parameters of the text decoder can be fixed, and only the model parameters of the visual encoder and feature adapter need to be adjusted.

[0046] Repeat the above model training process until the first model training stopping condition is met. This concludes the first training phase and yields the first layout analysis model. The first model training stopping condition can typically be set to the number of preset training rounds, such as 2 or 3 rounds.

[0047] The first layout analysis model is generated by comparing and training the initial layout analysis model with the first training sample set. It has the ability to align visual features with text features through feature adapters.

[0048] Step 104: Obtain the second training sample set, and train the first layout analysis model based on the second training sample set to obtain the second layout analysis model. The second training sample set includes the second training document and the task sample labels corresponding to the second training document.

[0049] After the first training phase, the second training phase begins. In this phase, the focus is on training the layout analysis model's ability to perform different tasks. This second training phase can also be called the multi-task pre-training phase.

[0050] The second training phase uses a second training sample set, which includes the second training document and its corresponding task sample labels. The task sample labels can be understood as the labels corresponding to different tasks performed on the second training document. For example, when the second training document performs an image description task, the task sample label is the image description text; when the second training document performs an image question-answering task, the second training document also includes the question text for the document, and its corresponding task sample label is the answer text; when the second training document performs an image annotation task, its corresponding task sample label is the sample annotation box information.

[0051] In the method provided in the embodiments of this specification, the second training sample set is not a training sample set for a single training task, but rather a fusion of training sample sets corresponding to multiple training tasks. The first layout analysis model is further trained using the second training sample set to obtain a second layout analysis model, enabling the second layout analysis model to possess better generalization capabilities, rather than only exhibiting good performance in a single image processing task.

[0052] In one specific embodiment provided in this specification, the first layout analysis model is trained based on the second training sample set to obtain the second layout analysis model, including: The first layout analysis model is trained based on the second training sample set, and the model parameters of the visual encoder, the feature adapter, and the text decoder are adjusted to obtain the second layout analysis model.

[0053] In the second training task, a general image processing task is used to pre-train the first layout analysis model. The aim is to enable the layout analysis model to effectively combine images and text. This process involves training both the visual encoder's ability to extract image layout information and the feature adapter's feature adaptation ability, as well as the text decoder's decoding ability. Therefore, in the second training phase, the model parameters in the visual encoder, feature adapter, and text decoder need to be adjusted simultaneously.

[0054] Specifically, the first layout analysis model is trained based on the second training sample set, and the model parameters of the visual encoder, the feature adapter, and the text decoder are adjusted to obtain the second layout analysis model, including: Determine the current training task, and obtain the second training document image and the task sample label corresponding to the second training document for the current training task; The second training document image is input into the visual encoder to obtain the feature information of the second training document; The feature information of the second training document is input into the feature adapter to obtain the adaptation feature information of the second training document; The adaptation feature information of the second training document is input into the text decoder to obtain the task prediction label; Calculate the task loss function value based on the task prediction label and the task sample label; The model parameters of the visual encoder, the feature adapter, and the text decoder are adjusted according to the task loss function value until the second model training stopping condition is met, thereby obtaining the second layout analysis model.

[0055] In this embodiment, the first layout analysis model is further trained based on the second training sample set. First, the current training task must be determined, and the corresponding second training document image and task sample label must be determined based on the current training task.

[0056] The second training document image is input into the visual encoder to obtain the feature information of the second training document. This feature information is then input into the feature adapter to obtain the adapted feature information of the second training document. The adapted feature information is then input into the text decoder, and after processing by the text decoder, task prediction labels are generated.

[0057] The task loss function value can be calculated based on the task prediction label and the task sample label. There are many methods for calculating the task loss function value provided in this manual, such as cross-entropy loss function, maximum loss function, average loss function, etc. This manual does not limit the specific method of loss function, and the actual application shall prevail.

[0058] Based on the task loss function value, backpropagate and adjust the model parameters of the first page analysis model. Specifically, adjust the model parameters in the visual encoder, feature adapter, and text decoder. Continue training the first page analysis model according to the above operations until the training stopping condition of the second model is met.

[0059] In the second training phase, training is performed using multiple image processing tasks, and the current training task needs to be determined in each training iteration. Determining the current training task includes: Obtain a preset training task set, wherein the preset training task set includes image description task, image question answering task, image annotation task, and image recognition task; Randomly select one task from the preset training task set as the current training task.

[0060] In this embodiment, a preset training task set can be pre-set, which includes multiple image processing tasks, including at least one of image description, image question answering, image annotation, and image recognition tasks. During each training session, one task can be randomly selected from the image processing tasks as the current training task.

[0061] In one specific embodiment provided in this specification, obtaining the second training document image and the task sample label corresponding to the second training document for the current training task includes: When the current training task is an image description task, the second training document image of the second training document and the sample image description corresponding to the second training document are obtained. In the case where the current training task is an image question answering task, the second training document image of the second training document and the sample image question and sample image answer corresponding to the second training document are obtained; When the current training task is an image annotation task, the second training document image of the second training document and the sample annotation box information corresponding to the second training document are obtained; When the current training task is an image recognition task, the second training document image and the sample text recognition information corresponding to the second training document are obtained.

[0062] In this embodiment, different image processing tasks have different corresponding task sample labels. In image description tasks, the task sample label is the description of the sample image; in image annotation tasks, the task sample label is the answer to the sample image; in image question-answering tasks, the task sample label is the sample image question and the sample image answer; in image recognition tasks, the task sample label is the sample text recognition information, and so on.

[0063] In practical applications, the types of image processing tasks are more diverse. The methods provided in the embodiments of this specification are not limited to these four tasks, but may also include other image processing tasks. The content of image processing tasks is not limited here, but shall be subject to actual application.

[0064] The method provided in the embodiments of this specification aims to enable the final layout analysis model to have the ability to recognize, describe, and segment images. Therefore, in the second stage of training, the layout analysis model is trained to have the relevant capabilities through multi-task joint training.

[0065] Step 106: Obtain the third training sample set, and train the second layout analysis model based on the third training sample set to obtain the target layout analysis model. The third training sample set includes the third training document and the sample layout information corresponding to the third training document.

[0066] After the above training, a second layout analysis model trained in the second training stage can be obtained. At this point, the third stage of model training begins. In the third training stage, supervised fine-tuning of the layout analysis model is required to enable it to perform layout analysis and output a specified layout format.

[0067] In the third training phase, a third training sample set is used, which includes the third training document and the corresponding sample layout information.

[0068] In practical applications, the text types of the first training text, the second training text, and the third training text are similar. For the explanation of the text types of the second and third training texts, please refer to the explanation of the first training text, which will not be repeated here.

[0069] The sample layout information can be understood as the annotation information of the layout in the third training document, which includes the element name, position information and other information of each element in the third training document.

[0070] For example, the sample layout information is {"Category": "Title", "Annotation Box": [x1, y1, x2, y2]}, where (x1, y1) are the coordinates of the top left corner of the annotation box, and (x2, y2) are the coordinates of the bottom right corner of the annotation box. The position of the annotation box in the third training document can be located by using the two coordinates of the top left and bottom right corners of the annotation box.

[0071] The layout analysis model can be trained using the third training sample set to recognize and output a specified format. In a specific embodiment provided in this specification, the second layout analysis model is trained based on the third training sample set to obtain the target layout analysis model, including: The second layout analysis model is trained based on the third training sample set, and the model parameters of the feature adapter and the text decoder are adjusted to obtain the target layout analysis model.

[0072] In the third training phase, the layout analysis model is trained using the third training sample set. The focus is on training the model's ability to output in a specified format. Therefore, in this phase, model training specifically refers to adjusting the model parameters of the feature adapter and text decoder in the model. After the training in the third phase is completed, the target layout analysis model can be obtained.

[0073] Specifically, the second layout analysis model is trained based on the third training sample set, and the model parameters of the feature adapter and the text decoder are adjusted to obtain the target layout analysis model, including: Obtain the third training document image and the sample layout information corresponding to the third training document; The third training document image is input into the visual encoder to obtain the feature information of the third training document; The feature information of the third training document is input into the feature adapter to obtain the adaptation feature information of the third training document; The third training document adaptation feature information is input into the text decoder to obtain the predicted layout information; Calculate the layout information loss function value based on the predicted layout information and the sample layout information; The model parameters of the feature adapter and the text decoder are adjusted according to the page layout information loss function value until the third model training stopping condition is met, thereby obtaining the target page layout analysis model.

[0074] In practical applications, the third training sample set includes the third training document and the sample layout information corresponding to the third training document. During the training process, the third training document image corresponding to the third training document is also used for training.

[0075] The third training document image is input into the visual encoder to obtain its feature information. This feature information is then input into the feature adapter to obtain its adapted feature information. Finally, this adapted feature information is input into the text decoder to obtain the predicted layout information.

[0076] At this point, the predicted layout information may differ from the actual sample layout information. Further calculation of the layout information loss function value is needed based on both the predicted and sample layout information. The model parameters of the feature adapter and text decoder are then adjusted based on the layout information loss function value until the third model training stopping condition is met. This third model training stopping condition can be a preset number of training epochs and / or the layout information loss function value being less than a preset loss threshold. This yields the final target layout analysis model.

[0077] The training method for the layout analysis model provided in this specification uses three stages to complete the training. The first training stage focuses on training the alignment ability between visual and text features. The second training stage focuses on training the model's ability to handle multiple tasks such as image recognition, image annotation, and image classification. The third training stage focuses on training the model's ability to recognize and output a specified format. Using the method provided in this specification, a layout analysis model capable of recognizing image information, performing specified tasks, and outputting a specified format can be trained, thus giving the trained target layout analysis model better applicability.

[0078] Figure 2 A flowchart illustrating an embodiment of the layout analysis method provided in this specification is shown, such as... Figure 2 As shown, the layout analysis method provided in this manual specifically includes the following steps.

[0079] Step 202: Obtain the document to be identified and at least one layout element to be identified.

[0080] Step 204: Generate layout recognition prompt text based on the document image corresponding to the document to be recognized and each layout element to be recognized.

[0081] Step 206: Input the layout recognition prompt text into the target layout analysis model to obtain the layout recognition result output by the target layout analysis model, wherein the target layout analysis model is trained by the above-mentioned layout analysis model training method.

[0082] The method provided in the embodiments of this specification can analyze the document layout. The document to be identified is the document whose layout needs to be identified using this method. The layout elements to be identified can be understood as elements that need to be identified within the document.

[0083] In existing layout recognition methods, the recognition function of a trained model is fixed. Different models are needed to identify different layout content from a document. For example, if a model is trained to recognize titles in a document, it cannot identify other elements. However, the layout analysis model trained using the above-described training method can identify elements within a layout based on the input elements to be recognized. Users can input the elements to be recognized according to their needs; these elements can be understood as those that need to be identified in the current recognition process.

[0084] For example, obtain the document image test.png to be recognized, and the three layout elements to be recognized from the document: "title", "table" and "image".

[0085] The layout recognition prompt text is generated based on the interface between the document to be recognized and the layout elements to be recognized. This layout recognition prompt text can be understood as the prompt that will be subsequently input into the target layout analysis model (also known as the AI ​​big model) in this embodiment. In the AI ​​big model, the prompt's role is to provide the AI ​​big model with contextual information about the input and the parameters input to the model. The prompt helps the big model better understand the intent of the input and respond accordingly, thus improving the interpretability and accessibility of the big model.

[0086] For example, taking the document image to be recognized as test.png and the layout elements to be recognized as "title", "table", and "image" as an example, the corresponding layout recognition prompt text is: { test.png You are a layout analysis expert. Given an image, please identify three types of elements in the document: "headings," "tables," and "images," and output their corresponding bounding boxes. Please output in the following JSON format: [{"Category": "***", "bbox": [x1, y1, x2, y2]}]}.

[0087] In this file, test.png is the document image of the document to be identified, and "title", "table" and "image" are the layout elements to be identified.

[0088] The layout recognition prompt text is input into the pre-trained target layout analysis model, which is trained as described above. The target layout analysis model can then output the layout recognition result based on the prompt text.

[0089] For example, based on the layout recognition prompt text in the example above, the corresponding layout recognition result can be output as follows: {[{"Category": "Title", "bbox": [x1, y1, x2, y2]}, {"Category": "Table", "bbox": [x3, y3, x4, y4]}, {"Category": "Image", "bbox": [x5, y5, x6, y6]}]}.

[0090] The layout analysis method provided in the embodiments of this specification can automatically identify the corresponding layout elements from the layout image of the document to be identified based on the layout elements to be identified input by the user, and output them in the format of annotation boxes. Compared with the current layout analysis model, it is more flexible and accurate.

[0091] Figure 3 This specification illustrates a flowchart of a layout analysis method for cloud-side devices according to an embodiment of the present invention, as shown below. Figure 3 As shown, the method specifically includes the following steps.

[0092] Step 302: Receive a layout analysis request sent by the receiving end device, wherein the layout analysis request carries a document to be identified and at least one layout element to be identified.

[0093] Step 304: Generate layout recognition prompt text based on the document image to be recognized and each layout element to be recognized.

[0094] Step 306: Input the layout recognition prompt text into the target layout analysis model to obtain the layout recognition result output by the target layout analysis model, wherein the target layout analysis model is trained by the above-mentioned layout analysis model training method.

[0095] Step 308: Send the layout recognition result to the terminal device.

[0096] In practical applications, the analysis of the document to be recognized requires a model, and the device deploying the model has good computing resources. Some edge devices may not have the corresponding processing capabilities. Therefore, the method provided in the embodiments of this specification can also be implemented on a cloud-side device. After obtaining the document to be recognized and at least one layout element to be recognized sent by the edge device, the cloud-side device performs the above steps to obtain the final layout recognition result and sends the layout recognition result to the edge device.

[0097] Figure 4 This is a schematic diagram of a task platform provided in one embodiment of this specification. The task platform includes a request interface 402 and a response unit 404, wherein: The request interface 402 is used to receive a layout analysis request sent by the end device, wherein the layout analysis request carries a document to be identified and at least one layout element to be identified. The response unit 404 is configured to generate layout recognition prompt text based on the document image to be recognized and each layout element to be recognized; input the layout recognition prompt text into the target layout analysis model to obtain the layout recognition result output by the target layout analysis model, wherein the target layout analysis model is trained using the above-mentioned layout analysis model training method.

[0098] Corresponding to the above method embodiments, this specification also provides embodiments of a training device for a layout analysis model. Figure 5This specification illustrates a schematic diagram of a training device for a layout analysis model according to one embodiment. Figure 5 As shown, the device includes: The first training module 502 is configured to acquire a first training sample set and train an initial layout analysis model based on the first training sample set to obtain a first layout analysis model, wherein the first training sample set includes a first training document and positive and negative samples corresponding to the first training document. The second training module 504 is configured to acquire a second training sample set and train the first layout analysis model based on the second training sample set to obtain a second layout analysis model. The second training sample set includes a second training document and task sample labels corresponding to the second training document. The third training module 506 is configured to acquire a third training sample set and train the second layout analysis model based on the third training sample set to obtain the target layout analysis model. The third training sample set includes a third training document and sample layout information corresponding to the third training document.

[0099] Optionally, the initial layout analysis model includes a visual encoder, a feature adapter, and a text decoder; The first training module 502 is further configured as follows: The initial layout analysis model is trained based on the first training sample set, and the model parameters of the visual encoder and the feature adapter are adjusted to obtain the first layout analysis model. The second training module 504 is further configured as follows: The first layout analysis model is trained based on the second training sample set, and the model parameters of the visual encoder, the feature adapter and the text decoder are adjusted to obtain the second layout analysis model. The third training module 506 is further configured as follows: The second layout analysis model is trained based on the third training sample set, and the model parameters of the feature adapter and the text decoder are adjusted to obtain the target layout analysis model.

[0100] Optionally, the first training module 502 is further configured as follows: Obtain the first training document image of the first training document, and the positive and negative samples corresponding to the first training document; The first training document image is input into the visual encoder to obtain the feature information of the first training document; The feature information of the first training document is input into the feature adapter to obtain the first training document adaptation feature information; The positive and negative samples are input into the text decoder to obtain positive and negative sample feature information. The contrastive loss function value is calculated based on the adaptation feature information of the first training document, the positive sample feature information, and the negative sample feature information. Adjust the model parameters of the visual encoder and the feature adapter according to the contrast loss function value until the first model training stopping condition is met, and obtain the first layout analysis model.

[0101] Optionally, the second training module 504 is further configured as follows: Determine the current training task, and obtain the second training document image and the task sample label corresponding to the second training document for the current training task; The second training document image is input into the visual encoder to obtain the feature information of the second training document; The feature information of the second training document is input into the feature adapter to obtain the adaptation feature information of the second training document; The adaptation feature information of the second training document is input into the text decoder to obtain the task prediction label; Calculate the task loss function value based on the task prediction label and the task sample label; The model parameters of the visual encoder, the feature adapter, and the text decoder are adjusted according to the task loss function value until the second model training stopping condition is met, thereby obtaining the second layout analysis model.

[0102] Optionally, the second training module 504 is further configured as follows: Obtain a preset training task set, wherein the preset training task set includes image description task, image question answering task, image annotation task, and image recognition task; Randomly select one task from the preset training task set as the current training task.

[0103] Optionally, the second training module 504 is further configured as follows: When the current training task is an image description task, the second training document image of the second training document and the sample image description corresponding to the second training document are obtained. In the case where the current training task is an image question answering task, the second training document image of the second training document and the sample image question and sample image answer corresponding to the second training document are obtained; When the current training task is an image annotation task, the second training document image of the second training document and the sample annotation box information corresponding to the second training document are obtained; When the current training task is an image recognition task, the second training document image and the sample text recognition information corresponding to the second training document are obtained.

[0104] Optionally, the third training module 506 is further configured as follows: Obtain the third training document image and the sample layout information corresponding to the third training document; The third training document image is input into the visual encoder to obtain the feature information of the third training document; The feature information of the third training document is input into the feature adapter to obtain the adaptation feature information of the third training document; The third training document adaptation feature information is input into the text decoder to obtain the predicted layout information; Calculate the layout information loss function value based on the predicted layout information and the sample layout information; The model parameters of the feature adapter and the text decoder are adjusted according to the page layout information loss function value until the third model training stopping condition is met, thereby obtaining the target page layout analysis model.

[0105] The training apparatus for the layout analysis model provided in this specification uses three stages to complete the training of the layout analysis model. The first training stage focuses on training the alignment ability between visual features and text features. The second training stage focuses on training the model's ability to handle multiple tasks such as image recognition, image annotation, and image classification. The third training stage focuses on training the model's ability to recognize and output a specified format. Using the method provided in this specification, a layout analysis model capable of recognizing image information, performing specified tasks, and outputting a specified format can be trained, thus giving the trained target layout analysis model better applicability.

[0106] The above is a schematic scheme of a training device for a layout analysis model according to this embodiment. It should be noted that the technical solution of this training device for a layout analysis model and the technical solution of the above-described training method for a layout analysis model belong to the same concept. For details not described in detail in the technical solution of the training device for a layout analysis model, please refer to the description of the technical solution of the above-described training method for a layout analysis model.

[0107] Figure 6 A structural block diagram of a computing device 600 according to one embodiment of this specification is shown. The components of the computing device 600 include, but are not limited to, a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and a database 650 is used to store data.

[0108] The computing device 600 also includes an access device 640, which enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 640 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0109] In one embodiment of this specification, the above-described components of the computing device 600 and Figure 6 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 6 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0110] The computing device 600 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 600 can also be a mobile or stationary server.

[0111] The processor 620 is used to execute the following computer program / instruction, which, when executed by the processor, implements the steps of the above-mentioned layout analysis model training method or layout analysis method.

[0112] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the computing device embodiments are relatively simple in description because they are fundamentally similar to the training method or implementation of the layout analysis method for the layout analysis model. Relevant details can be found in the descriptions of the training method or implementation of the layout analysis method for the layout analysis model.

[0113] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described training method or layout analysis method for the layout analysis model.

[0114] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the computer-readable storage medium embodiments are described simply because they are substantially similar to the training method or implementation of the layout analysis method for the layout analysis model. Relevant details can be found in the descriptions of the training method or implementation of the layout analysis method for the layout analysis model.

[0115] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described training method or layout analysis method for the layout analysis model.

[0116] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the technical solution of the above-mentioned training method for the layout analysis model or the technical solution of the layout analysis method. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the above-mentioned training method for the layout analysis model or the technical solution of the layout analysis method.

[0117] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0118] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0119] It should be noted that the above description describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments of this specification.

[0120] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0121] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A training method for a layout analysis model, comprising: Obtain a first training sample set, and train an initial layout analysis model based on the first training sample set to obtain a first layout analysis model. The first training sample set includes a first training document and positive and negative samples corresponding to the first training document. Obtain a second training sample set, and train the first layout analysis model based on the second training sample set to obtain a second layout analysis model. The second training sample set includes a second training document and task sample labels corresponding to the second training document. Obtain a third training sample set, and train the second layout analysis model based on the third training sample set to obtain the target layout analysis model. The third training sample set includes a third training document and sample layout information corresponding to the third training document.

2. The method as described in claim 1, wherein the initial layout analysis model comprises a visual encoder, a feature adapter, and a text decoder; The initial layout analysis model is trained based on the first training sample set to obtain the first layout analysis model, including: The initial layout analysis model is trained based on the first training sample set, and the model parameters of the visual encoder and the feature adapter are adjusted to obtain the first layout analysis model. The first layout analysis model is trained based on the second training sample set to obtain the second layout analysis model, including: The first layout analysis model is trained based on the second training sample set, and the model parameters of the visual encoder, the feature adapter and the text decoder are adjusted to obtain the second layout analysis model. The second layout analysis model is trained based on the third training sample set to obtain the target layout analysis model, including: The second layout analysis model is trained based on the third training sample set, and the model parameters of the feature adapter and the text decoder are adjusted to obtain the target layout analysis model.

3. The method as described in claim 2, wherein an initial layout analysis model is trained based on the first training sample set, and the model parameters of the visual encoder and the feature adapter are adjusted to obtain the first layout analysis model, comprising: Obtain the first training document image of the first training document, and the positive and negative samples corresponding to the first training document; The first training document image is input into the visual encoder to obtain the feature information of the first training document; The feature information of the first training document is input into the feature adapter to obtain the first training document adaptation feature information; The positive and negative samples are input into the text decoder to obtain positive and negative sample feature information. The contrastive loss function value is calculated based on the adaptation feature information of the first training document, the positive sample feature information, and the negative sample feature information. Adjust the model parameters of the visual encoder and the feature adapter according to the contrast loss function value until the first model training stopping condition is met, and obtain the first layout analysis model.

4. The method as described in claim 2, wherein the first layout analysis model is trained based on the second training sample set, and the model parameters of the visual encoder, the feature adapter, and the text decoder are adjusted to obtain the second layout analysis model, comprising: Determine the current training task, and obtain the second training document image and the task sample label corresponding to the second training document for the current training task; The second training document image is input into the visual encoder to obtain the feature information of the second training document; The feature information of the second training document is input into the feature adapter to obtain the adaptation feature information of the second training document; The adaptation feature information of the second training document is input into the text decoder to obtain the task prediction label; Calculate the task loss function value based on the task prediction label and the task sample label; The model parameters of the visual encoder, the feature adapter, and the text decoder are adjusted according to the task loss function value until the second model training stopping condition is met, thereby obtaining the second layout analysis model.

5. The method of claim 4, wherein determining the current training task includes: Obtain a preset training task set, wherein the preset training task set includes image description task, image question answering task, image annotation task, and image recognition task; Randomly select one task from the preset training task set as the current training task.

6. The method as described in claim 5, wherein obtaining the second training document image of the second training document corresponding to the current training task and the task sample label corresponding to the second training document includes: When the current training task is an image description task, the second training document image of the second training document and the sample image description corresponding to the second training document are obtained. In the case where the current training task is an image question answering task, the second training document image of the second training document and the sample image question and sample image answer corresponding to the second training document are obtained; When the current training task is an image annotation task, the second training document image of the second training document and the sample annotation box information corresponding to the second training document are obtained; When the current training task is an image recognition task, the second training document image and the sample text recognition information corresponding to the second training document are obtained.

7. The method as described in claim 2, wherein the second layout analysis model is trained based on the third training sample set, and the model parameters of the feature adapter and the text decoder are adjusted to obtain the target layout analysis model, comprising: Obtain the third training document image and the sample layout information corresponding to the third training document; The third training document image is input into the visual encoder to obtain the feature information of the third training document; The feature information of the third training document is input into the feature adapter to obtain the adaptation feature information of the third training document; The third training document adaptation feature information is input into the text decoder to obtain the predicted layout information; Calculate the layout information loss function value based on the predicted layout information and the sample layout information; The model parameters of the feature adapter and the text decoder are adjusted according to the page layout information loss function value until the third model training stopping condition is met, thereby obtaining the target page layout analysis model.

8. A layout analysis method, comprising: Obtain the document to be identified and at least one page element to be identified; Generate layout recognition prompt text based on the document image corresponding to the document to be recognized and each layout element to be recognized; The layout recognition prompt text is input into the target layout analysis model to obtain the layout recognition result output by the target layout analysis model, wherein the target layout analysis model is trained by the training method described in any one of claims 1-7.

9. A layout analysis method applied to cloud-based devices, comprising: A layout analysis request sent by a receiving end-side device, wherein the layout analysis request carries a document to be identified and at least one layout element to be identified; Generate layout recognition prompt text based on the document image corresponding to the document to be recognized and each layout element to be recognized; The layout recognition prompt text is input into the target layout analysis model to obtain the layout recognition result output by the target layout analysis model, wherein the target layout analysis model is trained by the training method described in any one of claims 1-7; The layout recognition result is sent to the terminal device.

10. A task platform, comprising a request interface and a response unit; The request interface is used to receive layout analysis requests sent by the end-side device, wherein... The layout analysis request carries the document to be identified and at least one layout element to be identified; The response unit is configured to generate layout recognition prompt text based on the document image to be recognized corresponding to the document to be recognized and each layout element to be recognized; input the layout recognition prompt text into the target layout analysis model to obtain the layout recognition result output by the target layout analysis model, wherein the target layout analysis model is trained by the training method described in any one of claims 1-7.

11. A training device for a layout analysis model, comprising: The first training module is configured to acquire a first training sample set and train an initial layout analysis model based on the first training sample set to obtain a first layout analysis model. The first training sample set includes a first training document and positive and negative samples corresponding to the first training document. The second training module is configured to acquire a second training sample set and train the first layout analysis model based on the second training sample set to obtain a second layout analysis model. The second training sample set includes a second training document and task sample labels corresponding to the second training document. The third training module is configured to acquire a third training sample set and train the second layout analysis model based on the third training sample set to obtain the target layout analysis model. The third training sample set includes a third training document and sample layout information corresponding to the third training document.

12. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 9.

13. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 9.

14. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 9.

Images