Document analysis method, device, storage medium and document analysis equipment

Abstract

The present application relates to the field of artificial intelligence technology, and discloses a document analysis method, device, storage medium and document analysis equipment, the method comprising: obtaining a document to be analyzed, identifying the document to be analyzed, and obtaining layout information to be analyzed of the document to be analyzed; analyzing the document to be analyzed based on the layout information to be analyzed by a preset large language model, and obtaining an analysis result; the preset large language model is obtained by model training based on the sample layout information of the sample document and the corresponding pre-training task, and the pre-training task is a task that enables the preset large language model to have the ability to understand the sample layout information after model training. Compared with the existing large language model that can only understand the plain text content, the preset large language model in the present application can also understand the layout information therein, thereby improving the model's understanding and analysis capabilities and improving the accuracy of subsequent downstream task analysis.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more particularly to a document analysis method, apparatus, storage medium, and document analysis equipment. Background Art

[0002] At present, with the rapid development of artificial intelligence technology, large language models (LLMs) have made significant progress in the field of natural language processing (NLP). These models are trained through deep learning algorithms and can understand and generate natural language. They are widely used in many fields such as machine translation, text summarization, and question-answering systems.

[0003] However, existing large language models can generally only understand plain text content when understanding documents. For complex documents containing various layout elements (such as titles, pictures or tables, etc.), they are often unable to analyze the layout information (that is, the spatial structural relationship between the layout elements, such as the position, size, format and order of the layout elements). As a result, the model's ability to understand and analyze documents is poor, affecting the accuracy of subsequent downstream task analysis. Summary of the invention

[0004] The main purpose of this application is to provide a document analysis method, apparatus, storage medium and document analysis equipment, aiming to solve the technical problem that the existing large language model is unable to analyze the layout information when performing document analysis, resulting in the model's poor ability to understand and analyze the document, affecting the accuracy of subsequent downstream task analysis.

[0005] To achieve the above object, the present application provides a document analysis method, which includes:

[0006] Acquire a document to be analyzed, identify the document to be analyzed, and obtain layout information to be analyzed of the document to be analyzed;

[0007] Analyzing the document to be analyzed based on the layout information to be analyzed by using a preset large language model to obtain an analysis result;

[0008] The preset large language model is obtained by model training according to sample layout information of the sample document and a corresponding pre-training task, wherein the pre-training task is a task that enables the preset large language model to have the ability to understand the sample layout information after model training.

[0009] In one embodiment, the pre-training task includes: at least one of a document level pre-training task, a region level pre-training task, and a segment level pre-training task;

[0010] The document level pre-training task is a task of globally learning the sample layout information;

[0011] The region-level pre-training task is a task of performing specific region learning on the sample layout information;

[0012] The segment-level pre-training task is a task of performing segment learning on the sample layout information.

[0013] In one embodiment, before the step of obtaining the document to be analyzed, the method further includes:

[0014] Acquire a sample document, identify the sample document, and obtain sample layout information of the sample document;

[0015] Constructing a corresponding pre-training task based on the sample layout information;

[0016] The initial large language model is trained using the sample layout information and the corresponding pre-training task to obtain a preset large language model.

[0017] In one embodiment, the pre-training task includes a document-level pre-training task, and the document-level pre-training task includes: a dense information description subtask and a text layout reconstruction subtask;

[0018] The step of constructing a corresponding pre-training task based on the sample layout information includes:

[0019] generating a dense description of the sample document according to the sample layout information, and constructing the dense information description subtask according to the dense description;

[0020] The text coordinates of the layout elements in the sample layout information are determined, and the text layout reconstruction subtask is constructed based on the text coordinates.

[0021] In one embodiment, the pre-training task includes a region-level pre-training task, and the region-level pre-training task includes: a layout element position determination subtask, a layout element type determination subtask, and a layout element content determination subtask;

[0022] The step of constructing a corresponding pre-training task based on the sample layout information includes:

[0023] Determining element coordinates, element types, and element contents of layout elements in the sample layout information;

[0024] Constructing the layout element position determination subtask and the layout element type determination subtask according to the element coordinates and the element type;

[0025] The layout element content determination subtask is constructed according to the element coordinates, the element type, and the element content.

[0026] In one embodiment, the pre-training task includes a segment-level pre-training task, and the segment-level pre-training task includes: a mask visual language subtask, a mask position instruction subtask, and a geometric layout analysis subtask;

[0027] The step of constructing a corresponding pre-training task based on the sample layout information includes:

[0028] Performing content masking on the layout elements in the sample layout information, and constructing the mask visual language subtask according to the sample layout information after content masking;

[0029] Performing position masking on the layout elements in the sample layout information, and constructing the mask position instruction subtask based on the position masked sample layout information;

[0030] The geometric distribution of layout elements in the sample layout information is determined, and the geometric layout analysis subtask is constructed according to the geometric distribution.

[0031] In one embodiment, the step of training the initial large language model by using the sample layout information and the corresponding pre-training task to obtain a preset large language model includes:

[0032] Receiving a fine-tuning instruction corresponding to the pre-training task, and determining an instruction type of the fine-tuning instruction;

[0033] Determine a target original document related to the fine-tuning instruction in the sample document based on the sample layout information and the instruction type;

[0034] The target original document is sorted, and an initial large language model is trained according to the sorting result to obtain a preset large language model.

[0035] In one embodiment, when the pre-training task includes a dense information description subtask, the fine-tuning instruction includes a first fine-tuning instruction, and the first fine-tuning instruction is an instruction for enabling the initial large language model to have text summarization capability; the method further includes:

[0036] Summarizing the target original document to obtain a dense description of the sample document, and training the initial large language model through the dense description to obtain a preset large language model;

[0037] When the pre-training task includes a text layout reconstruction subtask, the fine-tuning instruction includes a second fine-tuning instruction, and the second fine-tuning instruction is an instruction for enabling the initial large language model to have a text position learning capability; the method further includes:

[0038] The text coordinates of the target original document are determined, and the initial large language model is trained using the text coordinates and the target original document to obtain a preset large language model.

[0039] In one embodiment, when the pre-training task includes a layout element position determination subtask, the fine-tuning instruction includes a third fine-tuning instruction, and the third fine-tuning instruction is an instruction for enabling the initial large language model to have element position learning capability; the method further includes:

[0040] Determining element coordinates and element types of the target original document, and training the initial large language model according to the element coordinates and the element type to obtain a preset large language model;

[0041] When the pre-training task includes a layout element type determination subtask, the fine-tuning instruction includes a fourth fine-tuning instruction, and the fourth fine-tuning instruction is an instruction for enabling the initial large language model to have element type learning capability; the method further includes:

[0042] Determining element coordinates and element types of the target original document, and training the initial large language model according to the element types and the element coordinates to obtain a preset large language model;

[0043] When the pre-training task includes a layout element content determination subtask, the fine-tuning instruction includes a fifth fine-tuning instruction, and the fifth fine-tuning instruction is an instruction for enabling the initial large language model to have an element content learning capability of the layout element; the method further includes:

[0044] The element coordinates, element type and element content of the target original document are determined, and the initial large language model is trained according to the element coordinates, the element type and the element content to obtain a preset large language model.

[0045] In one embodiment, when the pre-training task includes a mask visual language subtask, the fine-tuning instruction includes a sixth fine-tuning instruction, and the sixth fine-tuning instruction is an instruction for enabling the initial large language model to have the ability to make predictions based on contextual text content; the method further includes:

[0046] Predicting the context text content of the target original document, and training an initial large language model using the obtained first prediction result to obtain a preset large language model;

[0047] When the pre-training task includes a mask position instruction subtask, the fine-tuning instruction includes a seventh fine-tuning instruction, and the seventh fine-tuning instruction is an instruction for enabling the initial large language model to have the ability to predict the association between text content and corresponding text positions; the method further includes:

[0048] Predicting the corresponding text position according to the text content of the target original document, and training the initial large language model through the obtained second prediction result to obtain a preset large language model;

[0049] When the pre-training task includes a geometric layout analysis subtask, the fine-tuning instruction includes an eighth fine-tuning instruction, and the eighth fine-tuning instruction is an instruction for enabling the initial large language model to have the ability to predict geometric layout relationships; the method further includes:

[0050] The geometric layout relationship between the target original documents is predicted, and the initial large language model is trained according to the geometric layout relationship prediction result to obtain a preset large language model.

[0051] In one embodiment, the step of acquiring a sample document, identifying the sample document, and obtaining sample layout information of the sample document includes:

[0052] Obtain a sample document, and determine whether the document type of the sample document is a preset document type;

[0053] If not, performing text analysis on the sample document using the initial large language model, and converting the document type of the sample document based on the text analysis result;

[0054] The sample document after the document type conversion is identified to obtain sample layout information of the sample document.

[0055] In one embodiment, when the document to be analyzed is a job seeker's resume containing layout elements such as text, pictures, tables, and titles, the analysis result is a result of the spatial structural relationship between the layout elements.

[0056] In addition, to achieve the above-mentioned purpose, the embodiment of the present application also proposes a document analysis device, the device comprising:

[0057] A document acquisition module, used to acquire a document to be analyzed, identify the document to be analyzed, and obtain layout information to be analyzed of the document to be analyzed;

[0058] A document analysis module, used to analyze the document to be analyzed based on the layout information to be analyzed by using a preset large language model to obtain an analysis result;

[0059] The preset large language model is obtained by model training according to sample layout information of the sample document and a corresponding pre-training task, wherein the pre-training task is a task that enables the preset large language model to have the ability to understand the sample layout information after model training.

[0060] In addition, to achieve the above-mentioned purpose, an embodiment of the present application further proposes a storage medium, on which a document analysis program is stored, and when the document analysis program is executed by a processor, the steps of the document analysis method described above are implemented.

[0061] In addition, to achieve the above-mentioned purpose, an embodiment of the present application also proposes a document analysis device, which includes: a memory, a processor, and a document analysis program stored in the memory and executable on the processor, and when the document analysis program is executed by the processor, the steps of the document analysis method described above are implemented.

[0062] The embodiments of the present application provide a document analysis method, apparatus, storage medium and document analysis device, the method comprising: obtaining a document to be analyzed, identifying the document to be analyzed, and obtaining layout information to be analyzed of the document to be analyzed; analyzing the document to be analyzed based on the layout information to be analyzed by a preset large language model to obtain an analysis result; the preset large language model is obtained by model training based on sample layout information of a sample document and a corresponding pre-training task, wherein the pre-training task is a task that enables the preset large language model to have the ability to understand the sample layout information after model training.

[0063] Since the present application can identify sample documents, obtain sample layout information of sample documents, perform model training through the sample layout information and corresponding pre-training tasks, and obtain a preset large language model capable of understanding sample layout information, when used, the document to be analyzed is first identified, the layout information to be analyzed of the document to be analyzed is obtained, and then the document to be analyzed is analyzed based on the layout information to be analyzed through the preset large language model to obtain the analysis result. Compared with the existing large language model that can only understand the plain text content, the preset large language model in the present application can also understand the layout information therein, thereby improving the model's ability to understand and analyze, and improving the accuracy of subsequent downstream task analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0065] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0066] Figure 1 A schematic diagram of the structure of a document analysis device in a hardware operating environment involved in an embodiment of the present application;

[0067] Figure 2 This is a flowchart of the first embodiment of the document analysis method of this application;

[0068] Figure 3 This is the overall structure diagram of the first embodiment of the document analysis method of this application;

[0069] Figure 4 This is a flowchart of the third embodiment of the document analysis method of this application;

[0070] Figure 5 This is a structural block diagram of the first embodiment of the document analysis device of this application.

[0071] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0072] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0073] Reference Figure 1 , Figure 1 A schematic diagram of the document analysis device structure of the hardware operating environment involved in the embodiment of the present application.

[0074] like Figure 1As shown, the document analysis device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include an interface for connecting a display screen (Display), and the user interface 1003 may also include a standard wired interface and a wireless interface. The wired interface of the user interface 1003 may be a USB interface in this application. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM) or a stable memory (Non-volatile Memory, NVM), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0075] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the document analysis device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0076] like Figure 1 As shown, the memory 1005 identified as a computer storage medium may include an operating system, a network communication module, a user interface module, and a document analysis program.

[0077] exist Figure 1 In the document analysis device shown, the network interface 1004 is mainly used to connect to the background server and communicate data with the background server; the user interface 1003 is mainly used to connect to the user device; the document analysis device calls the document analysis program stored in the memory 1005 through the processor 1001, and executes the document analysis method provided in the embodiment of the present application.

[0078] It should be noted that, with the rapid development of artificial intelligence technology, large language models (LLMs) have made significant progress in the field of natural language processing (NLP). These models are trained through deep learning algorithms and can understand and generate natural language. They are widely used in many fields such as machine translation, text summarization, and question-answering systems.

[0079] However, existing large language models can generally only understand plain text content when understanding documents. For complex documents containing various layout elements (such as titles, pictures or tables, etc.), they are often unable to analyze the layout information (that is, the spatial structural relationship between layout elements, such as the position, size, format and order of elements), resulting in the model's poor ability to understand and analyze documents, affecting the accuracy of subsequent downstream task analysis.

[0080] Therefore, in order to solve the above defects, this embodiment provides a document analysis method, which can identify sample documents, obtain sample layout information of sample documents, perform model training through the sample layout information and corresponding pre-training tasks, and obtain a preset large language model with the ability to understand the sample layout information. When used, the document to be analyzed is first identified to obtain the layout information to be analyzed of the document to be analyzed, and then the document to be analyzed is analyzed based on the layout information to be analyzed by the preset large language model to obtain the analysis result. Compared with the existing large language model that can only understand the plain text content, the preset large language model in this embodiment can also understand the layout information therein, thereby improving the model's understanding and analysis capabilities and improving the accuracy of subsequent downstream task analysis.

[0081] For ease of understanding, the following combination Figures 2 to 4 The document analysis method provided in the embodiment of the present application is introduced in detail.

[0082] Reference Figure 2 , Figure 2 The flowchart of the first embodiment of the document analysis method of the present application is provided. Figure 2 As shown, in this embodiment, the specific method includes:

[0083] Step S10: Acquire the document to be analyzed, identify the document to be analyzed, and obtain layout information to be analyzed of the document to be analyzed.

[0084] It is understandable that the method of this embodiment can be applied on a document analysis device, wherein the document analysis device can be any device for performing document analysis, such as a computer, etc., and this embodiment does not limit this. The execution subject of this method embodiment can adopt the document analysis device (referred to as the device), and this device is used as an example to illustrate this embodiment and the following embodiments.

[0085] It should be understood that the document to be analyzed can be any document that needs to be analyzed. The document to be analyzed can include not only text content but also various other layout elements, such as titles, pictures, and tables, etc., which are not limited in this embodiment. And because the document to be analyzed includes various layout elements, it can be understood that the document to be analyzed is a complex document compared to a general plain text document. For the convenience of subsequent explanation, in this embodiment, the document to be analyzed can be a job seeker's resume with layout elements such as text, titles (such as basic information, educational background and work experience, etc.), pictures (such as photos), and tables.

[0086] After obtaining the document to be analyzed, the document to be analyzed may be identified to obtain layout information of the document to be analyzed as the layout information to be analyzed. The layout information may be information related to the spatial structure of layout elements in the document to be analyzed, such as the position, size, format and order of the layout elements, which is not limited in this embodiment.

[0087] It should be noted that when identifying the document to be analyzed, any method of obtaining the layout information to be analyzed can be used. This embodiment uses optical character recognition (OCR) for recognition, which can extract text information from image documents and convert it into a format that can be recognized and processed by computers. Therefore, the document to be analyzed can be a document in image form.

[0088] During OCR recognition, a text box is usually marked on the document to be analyzed, and the recognized document content is displayed in the text box. Figure 3 , Figure 3 This is the overall structure diagram of the first embodiment of the document analysis method of this application, such as Figure 3 As shown, after the document to be analyzed is subjected to OCR recognition, a recognition result can be obtained, which may include several text boxes (i.e. Figure 3 Text box 1, text box 2, ..., text box N), and can also include the document content in each text box (i.e. Figure 3 According to the positions of these text boxes and the document contents in the text boxes, the layout information of the document to be analyzed can be obtained.

[0089] In actual use, the above-mentioned device may first identify the acquired document to be analyzed, and obtain layout information of the document to be analyzed as the above-mentioned layout information to be analyzed.

[0090] Step S20: Analyze the document to be analyzed based on the layout information to be analyzed by using a preset large language model to obtain an analysis result.

[0091] It is understandable that the preset large language model is obtained by model training based on the sample layout information of the sample document and the corresponding pre-training task, and the pre-training task is a task that enables the preset large language model to have the ability to understand the sample layout information after model training.

[0092] It is also understandable that the preset large language model in this embodiment can be a large language model capable of understanding sample layout information. The above-mentioned sample document can be any document used for training, such as a job applicant's resume, etc., and the sample document can also include the above-mentioned various layout elements. The above-mentioned sample layout information can be the layout information of the sample document, which can also be obtained through the above-mentioned OCR recognition.

[0093] Since the initial large language model is trained through the sample layout information of the sample document and the corresponding pre-training task in this embodiment to obtain the preset large model, and the pre-training task is a task to enable the preset large language model to have the ability to understand the sample layout information. Therefore, if the document to be analyzed is a job seeker resume containing layout elements such as text, pictures, tables and titles, the analysis result is the spatial structural relationship result between each of the layout elements. The spatial structural relationship can be understood as the spatial position relationship between layout elements such as text, pictures, tables and titles.

[0094] Furthermore, in this embodiment, in order to obtain the above-mentioned preset large language model, before the above-mentioned step S10, it also includes:

[0095] Step S01: obtaining a sample document, identifying the sample document, and obtaining sample layout information of the sample document;

[0096] Step S02: constructing a corresponding pre-training task based on the sample layout information;

[0097] Step S03: training an initial large language model using the sample layout information and the corresponding pre-training task to obtain a preset large language model.

[0098] It should be understood that the above-mentioned pre-training task can be any task that enables the initial large language model to have the ability to understand sample layout information after model training. For example, the task of understanding the position of layout elements in the sample document, etc., which is not limited in this embodiment. The above-mentioned initial large language model can be a large language model before training, which is not limited in this embodiment.

[0099] In actual use, the above device can first perform OCR recognition on the sample document to obtain a recognition result, which may include the text box position of the sample document and the document content in the text box. Then, these text boxes and the document content in the text boxes can be input into the document pre-training model (i.e. Figure 3 The document pre-training model can obtain the layout information of the document to be analyzed. The above document preset model can be a model that obtains layout information based on OCR recognition results, such as a layout-aware language model (LayoutLM), etc., which is not limited in this embodiment.

[0100] After the document pre-training model obtains the sample layout information, it can be used through the image mapping layer (i.e. Figure 3 Image map layer) and text and layout map layer (i.e. Figure 3 The Chinese text and layout mapping layer maps the sample layout information to the initial large language model (i.e. Figure 3 The image mapping layer may be a layer that maps image information in the sample layout information, and the text and layout mapping layer may be a layer that maps text and layout information of the text in the sample layout information.

[0101] After the mapping is completed, the corresponding pre-training task (i.e. Figure 3 Pre-training / instruction fine-tuning tasks), the initial large language model can be trained through sample layout information and corresponding pre-training tasks (i.e. Figure 3 Instruction embedding, Figure 3 Thinking chain reasoning of Chinese text and layout information and Figure 3 In the present embodiment, the document preset model can be understood as an encoder, the OCR recognition result is encoded, and the encoding result is then mapped to the initial large language model, so that the initial large language model obtains the sample layout information, and finally the model training is performed based on the preset training task.

[0102] This embodiment can identify sample documents, obtain sample layout information of sample documents, perform model training through the sample layout information and corresponding pre-training tasks, and obtain a preset large language model that has the ability to understand the sample layout information. When used, the document to be analyzed is first identified to obtain the layout information to be analyzed of the document to be analyzed, and then the document to be analyzed is analyzed based on the layout information to be analyzed through the preset large language model to obtain the analysis result. Compared with the existing large language model that can only understand the plain text content, the preset large language model in this embodiment can also understand the layout information therein, thereby improving the model's understanding and analysis capabilities and improving the accuracy of subsequent downstream task analysis (see Figure 3 An example of this is Figure 3 The mid- and downstream tasks are to ask: What is the highest educational level of this resume? The preset large language model can analyze and obtain: This is a table information query question. In the text boxes [2, 3, 15, 16], the query result is a master's degree. At the same time, the preset large language model can directly output: Master's degree).

[0103] Continue to refer to Figure 2 , and based on the above-mentioned first embodiment, a second embodiment of the document analysis method of the present application is proposed.

[0104] In this embodiment, in order to enable the initial large language model to have the ability to learn and understand the sample layout information of the sample document, in this embodiment, the pre-training task includes: at least one of a document level pre-training task, a region level pre-training task, and a segment level pre-training task;

[0105] The document level pre-training task is a task of globally learning the sample layout information;

[0106] The region-level pre-training task is a task of performing specific region learning on the sample layout information;

[0107] The segment-level pre-training task is a task of performing segment learning on the sample layout information.

[0108] It should be noted that the above-mentioned document-level pre-training tasks, region-level pre-training tasks, and segment-level pre-training tasks can be understood as tasks constructed according to different content levels of sample documents. Specifically, the above-mentioned document-level pre-training tasks can be tasks for basic global understanding of sample documents, which can make the initial large language model focus on the text content of sample documents. The above-mentioned region-level pre-training tasks can be tasks for learning specific semantically rich regions in sample documents. The above-mentioned semantically rich regions can be layout elements in the specific regions that contain key information, such as information carried in document titles, pictures, and tables. Since text can directly reflect its corresponding information, but layout elements such as pictures and tables cannot directly reflect it, the layout elements in this task can be understood as layout elements other than text. The above-mentioned segment-level pre-training tasks can be tasks for learning text segments and element layouts in sample documents. Text segments can be understood as document content in text boxes obtained through OCR recognition, and element layouts can be understood as the positional relationship between layout elements (i.e., the positional relationship between text boxes).

[0109] During actual training, at least one of the document-level pre-training tasks, region-level pre-training tasks, and fragment-level pre-training tasks can be used for training. In order to improve the training effect, this embodiment uses document-level pre-training tasks, region-level pre-training tasks, and fragment-level pre-training tasks as the above-mentioned pre-training tasks for illustration.

[0110] It should be emphasized that when constructing the pre-training tasks mentioned above, they can be constructed according to a certain preset construction order, wherein the preset construction order can be constructed in the order of document-level pre-training tasks, region-level pre-training tasks, and fragment-level pre-training tasks. This is because the document-level pre-training tasks, region-level pre-training tasks, and fragment-level pre-training tasks are tasks that learn from the whole to the part in sequence, and therefore can be constructed according to the preset construction order.

[0111] Further, in order to construct the above-mentioned document level pre-training task, in this embodiment, the pre-training task includes a document level pre-training task, and the document level pre-training task includes: a dense information description subtask and a text layout reconstruction subtask;

[0112] The step of constructing a corresponding pre-training task based on the sample layout information includes:

[0113] Step S021: Generate a dense description of the sample document according to the sample layout information, and construct the dense information description subtask according to the dense description.

[0114] It is understandable that the above dense description can be a general description of the sample document, which may include the basic information and basic content of the sample document, wherein the basic information may include information such as the document type of the sample document, and the basic content may be the key content of the sample document. For example, when the sample document is a job seeker's resume, the above dense description of the sample document may be: This document is a resume, the name is Zhang San, graduated from a certain university... When the sample document is a thesis, the above dense description of the sample document may be: This document is a thesis, the name is XX, and it mainly describes XX. The dense description can be obtained by manual annotation based on the sample layout information, and of course it can also be obtained by other means, which is not limited in this embodiment.

[0115] After obtaining the dense description, a corresponding fine-tuning instruction can be generated based on the dense description. For the convenience of subsequent description, the fine-tuning instruction of the dense description is recorded as the first fine-tuning instruction. The first fine-tuning instruction can be generated based on the dense description through a generative language model, and of course can also be generated through other methods, which is not limited in this embodiment.

[0116] It should be understood that the first fine-tuning instruction can be an instruction to enable the initial large language model to have the ability to densely describe the sample document, that is, an instruction to enable the initial large language model to have the ability to summarize text, for example: please read the following document and generate a document summary description within 1,000 words. The document summary description can be understood as the above-mentioned dense description. Finally, the above-mentioned dense information description subtask can be constructed according to the first fine-tuning instruction. Furthermore, during training, according to the dense information description subtask, the initial large language model can have the ability to summarize the basic information of the sample document, such as the document type and general content.

[0117] Step S022: Determine the text coordinates of the layout elements in the sample layout information, and construct the text layout reconstruction subtask based on the text coordinates.

[0118] It should be noted that the above text coordinates may be the coordinates of the position of the text in the layout element in the sample document in the sample document, and the text coordinates may be understood as the coordinates of the text box in which the layout element is the text obtained after the above OCR recognition.

[0119] In actual use, since the recognition result after OCR recognition can also include the coordinates of the text box, the text coordinates of the text in each layout element in the sample layout can be determined according to the recognition result, and can be set according to a certain preset template. The preset template can be in the format of "<{box}, {text}>", where {box} is the element coordinate and {text} is the corresponding layout element. For example: <[0, 5, 30, 15], Zhang San>; <[20, 3, 6, 88], Telephone>. It can be understood that in the job seeker's resume, the text coordinates of Zhang San in the sample document are [0, 5, 30, 15], and the text coordinates of Telephone in the sample document are [20, 3, 6, 88].

[0120] After obtaining the text coordinates of the above-mentioned texts, the corresponding fine-tuning instruction can be generated based on the text coordinates. For the convenience of subsequent description, the fine-tuning instruction of the text coordinates is recorded as the second fine-tuning instruction here.

[0121] The second fine-tuning instruction may be an instruction to enable the initial large language model to have the ability to learn the position of the text, that is, an instruction to have the ability to identify the position of the text in the sample document, for example: please generate all the layout information of this document. The layout information is the text position of the text in the sample document. Finally, the above-mentioned text layout reconstruction subtask can be constructed according to the second fine-tuning instruction. Furthermore, during training, according to the text layout reconstruction subtask, the initial large language model can have the ability to understand the layout of the text in the sample document.

[0122] Further, in order to construct the above-mentioned region-level pre-training task, in this embodiment, the pre-training task includes a region-level pre-training task, and the region-level pre-training task includes: a layout element position determination subtask, a layout element type determination subtask, and a layout element content determination subtask;

[0123] The step of constructing a corresponding pre-training task based on the sample layout information includes:

[0124] Step S023: Determine the element coordinates, element type and element content of the layout element in the sample layout information.

[0125] It is understandable that the above element coordinates may be the coordinates of the position of the layout element in the sample document. Different from the above text coordinates, the element coordinates in this embodiment are the coordinates of all layout elements, which may include the above text coordinates. The above element type may be the type of the layout element, which may be understood as the type of document content in the text box. In this embodiment, the above element type may include but is not limited to: title, picture, table, etc. The above element content may be the specific description content of the layout element in the sample document, and may also be understood as the document content in the text box, such as the content carried in the table.

[0126] In actual use, since the sample layout information of the sample document can be obtained after OCR recognition, the sample layout information can contain the element coordinates, element type and element content of each layout element.

[0127] Step S024: constructing the layout element position determination subtask and the layout element type determination subtask according to the element coordinates and the element type.

[0128] It should be understood that the above layout element position determination subtask may be a task for determining the position of a layout element. The above layout element type determination subtask may be a task for determining the type of a layout element.

[0129] In this embodiment, after obtaining the above-mentioned element coordinates and element types, corresponding fine-tuning instructions can be generated based on the element coordinates and element types. For the convenience of subsequent description, the fine-tuning instructions of the layout element position determination subtask are recorded as the third fine-tuning instructions, and the layout element type determination subtask is recorded as the fourth fine-tuning instructions.

[0130] The third fine-tuning instruction may be an instruction for enabling the initial large language model to learn the element position of the layout element. The element position may be the position of the above-mentioned semantically rich specific area, that is, the third fine-tuning instruction may be an instruction for identifying the position of the above-mentioned semantically rich specific area, for example: where can the document title be found, the corresponding answer may be: there are nine places in total, <[0, 5, 30, 15], Zhang San>; <[125, 44, 96, 33], educational background>. The fourth fine-tuning instruction may be an instruction for enabling the initial large language model to learn the element type of the layout element. The element type may be the type of the above-mentioned semantically rich specific area, that is, the fourth fine-tuning instruction may be an instruction for identifying the element type of the above-mentioned semantically rich specific area, for example: what type of layout element at [0, 5, 30, 15] is. Finally, the above-mentioned layout element position determination subtask may be constructed according to the third fine-tuning instruction, and the above-mentioned layout element type determination subtask may be constructed according to the fourth fine-tuning instruction. Furthermore, during training, determining the subtask according to the layout element position can enable the initial large language model to have the ability to locate the element position of a specific layout element in a sample document, and determining the subtask according to the layout element type can enable the initial large language model to have the ability to determine the element type of the layout element at a specific position.

[0131] Step S025: constructing the layout element content determination subtask according to the element coordinates, the element type and the element content.

[0132] It should be noted that the above-mentioned layout element content determination subtask may be a task of document content of layout elements in a sample document, such as values ​​of specific rows and columns in a table, querying cell values, footnotes in a document, and captions of charts, etc. In this embodiment, after obtaining the above-mentioned element coordinates, element type, and element content, corresponding fine-tuning instructions can be generated based on the element coordinates, element type, and element content. For the convenience of subsequent description, the fine-tuning instructions of the layout element content determination subtask are recorded as the fifth fine-tuning instructions.

[0133] The fifth fine-tuning instruction may be an instruction that enables the initial large language model to learn the element content of the layout element. The element content may be the content of the above-mentioned semantically rich specific area, that is, the fifth fine-tuning instruction may be an instruction that has the ability to identify the information carried by the semantically rich specific area, for example: what is the content of the row and column in the table. Finally, the above-mentioned layout element content determination subtask can be constructed according to the fifth fine-tuning instruction. Furthermore, the layout element content determination subtask can enable the initial large language model to have the ability to determine the element content of a specific layout element.

[0134] Further, in order to construct the above-mentioned segment-level pre-training task, in this embodiment, the pre-training task includes a segment-level pre-training task, and the segment-level pre-training task includes: a mask visual language subtask, a mask position instruction subtask, and a geometric layout analysis subtask;

[0135] The step of constructing a corresponding pre-training task based on the sample layout information includes:

[0136] Step S026: performing content masking on the layout elements in the sample layout information, and constructing the mask visual language subtask according to the sample layout information after content masking.

[0137] It is understandable that the above content masking may be to randomly mask the element content (ie, document content) of the text in the layout element. The above mask visual language subtask may be a task for determining the content of the text in the masked layout element.

[0138] In actual use, the content of the text in the layout elements in the sample layout information can be randomly masked, and the corresponding fine-tuning instructions can be generated based on the sample layout information after content masking. For the convenience of subsequent explanation, the fine-tuning instructions of the mask visual language subtask are recorded here as the sixth fine-tuning instructions.

[0139] The sixth fine-tuning instruction may be an instruction for enabling the initial large language model to have the ability to make predictions based on the context text content, that is, to analyze the context text and predict the masked content. For example, if the masked content is: academic intern, the corresponding sixth fine-tuning instruction may be, for example: what is the masked part of the text. The masked part can be understood as the above-mentioned masked content. Finally, the above-mentioned mask visual language subtask can be constructed according to the sixth fine-tuning instruction. Furthermore, during training, the initial large language model can be enabled to have the ability to make predictions based on the context text content according to the mask visual language subtask.

[0140] Step S027: performing position masking on the layout elements in the sample layout information, and constructing the mask position instruction subtask based on the position masked sample layout information.

[0141] It should be understood that the position masking may be to randomly mask the position of the text in the layout element, and specifically, the element coordinates corresponding to the text box of the text in the layout element may be set to 0, that is, to [0, 0, 0, 0]. The mask position instruction subtask may be a task for determining the text position of the text in the layout element after masking. The text position may be the coordinate position of the masked text.

[0142] In actual use, the position of the text in the layout elements in the sample layout information can be masked, that is, the text coordinates of a certain text can be modified to [0, 0, 0, 0], and the corresponding fine-tuning instructions can be generated according to the sample layout information after position masking. For the convenience of subsequent explanation, the fine-tuning instructions of the mask visual language subtask are recorded here as the seventh fine-tuning instructions.

[0143] The seventh fine-tuning instruction may be an instruction that enables the initial large language model to predict the relationship between the text content in the layout element and the corresponding text position. The text content may be the content of the text in the layout element, and the text position may be the coordinate position of the text content in the layout element. For example, if the text coordinates [56, 89, 558, 217] in <[56, 89, 558, 217], My Responsibilities> are modified to [0, 0, 0, 0], then the corresponding seventh fine-tuning instruction may be, for example: the text box is set to the position of 0, what is the original text box. Finally, the above-mentioned mask position instruction subtask can be constructed according to the seventh fine-tuning instruction. Furthermore, during training, according to the mask position instruction subtask, the initial large language model can have the ability to learn the relationship between the text content and the location of the text (i.e., the corresponding text position).

[0144] Step S028: Determine the geometric distribution of layout elements in the sample layout information, and construct the geometric layout analysis subtask according to the geometric distribution.

[0145] It should also be understood that the above-mentioned geometric distribution may be a task of geometric distribution relationship between layout elements, wherein the geometric distribution may be a distribution of distance and direction.

[0146] In actual use, the geometric distribution of the layout elements in the sample layout information can be determined, that is, the distance and direction between the layout elements can be determined. For example, the direction of "academic intern" relative to "2024.05-2024.06" in the sample document is diagonally downward, and the distance is 30. And the corresponding fine-tuning instructions are generated according to the geometric distribution. For the convenience of subsequent explanation, the fine-tuning instructions of the mask visual language subtask are recorded as the eighth fine-tuning instructions.

[0147] The eighth fine-tuning instruction may be an instruction that enables the initial large language model to predict the geometric distribution of layout elements. Based on the above example, the corresponding eighth fine-tuning instruction may be, for example, what is the direction and distance of "academic intern" relative to "2024.05-2024.06". Finally, the above-mentioned geometric layout analysis subtask can be constructed according to the eighth fine-tuning instruction. Furthermore, during training, the initial large language model can be equipped with the ability to predict and analyze the direction and distance between text boxes according to the geometric layout analysis subtask.

[0148] In this embodiment, by setting three levels of pre-training tasks, the three-layer structure of the document can be effectively and intelligently understood, which improves the preset large language model's ability to fully understand the layout of the document content, thereby enhancing the model's cross-modal understanding ability. At the same time, by setting the above three levels of pre-training tasks, compared to the existing ones that only focus on the text content in the document, this embodiment effectively integrates information such as document text, layout structure, and image features, improving the efficiency and scalability in the actual implementation scenarios of enterprises.

[0149] Reference Figure 4 , Figure 4 This is a flow chart of the third embodiment of the document analysis method of the present application. Based on the above embodiments, the third embodiment of the document analysis method of the present application is proposed.

[0150] like Figure 4 As shown, in this embodiment, the step of training the initial large language model through the sample layout information and the corresponding pre-training task to obtain the preset large language model includes:

[0151] Step S031: receiving a fine-tuning instruction corresponding to the pre-training task, and determining an instruction type of the fine-tuning instruction.

[0152] It should be noted that this embodiment trains the initial large language model based on the pre-training task, which is implemented based on the Chain of Thought (CoT). Specifically, it can be understood as asking questions to fine-tune the initial large language model according to the corresponding pre-training task. The order of questions can also be consistent with the construction order of the above pre-training tasks, and the corresponding answers are input into the initial large language model at the same time. The initial large language model compares its own prediction results with the corresponding answers to complete the training.

[0153] Furthermore, the fine-tuning instruction can be an instruction for asking questions to the initial large language model during training, and specifically can refer to the first fine-tuning instruction to the eighth fine-tuning instruction. After receiving the corresponding fine-tuning instruction, the instruction type of the fine-tuning instruction can be determined first. Since the pre-training task is divided into eight subtasks in the above embodiment, eight instruction types can be corresponding, such as: table question type, counting and reasoning question type or key value extraction question type, etc. This embodiment does not limit this.

[0154] Step S032: determining a target original document related to the fine-tuning instruction in the sample document based on the sample layout information and the instruction type;

[0155] Step S033: sorting the target original document, and training the initial large language model based on the sorting result to obtain a preset large language model.

[0156] It is understandable that the above-mentioned target original document may be the original text corresponding to the answer to the fine-tuning instruction in the sample document.

[0157] After determining the instruction type, it is possible to know what information is needed to answer the fine-tuning instruction. The above-mentioned device can then reason based on the instruction type, sample layout information and the fine-tuning instruction, and find the original text related to the fine-tuning instruction from the sample document as the above-mentioned target original document.

[0158] After obtaining the target original document, the initial large language model can organize the target original document, organize and generate the logical output that answers the fine-tuning instruction, and obtain the above-mentioned organization result. According to the organization result, the initial large language model can be trained to obtain the above-mentioned preset large language model.

[0159] Specifically, in this embodiment, when the pre-training task includes a dense information description subtask, the fine-tuning instruction includes a first fine-tuning instruction, and the first fine-tuning instruction is an instruction for enabling the initial large language model to have text summarization capability; the above step S033 also includes:

[0160] The target original document is summarized to obtain a dense description of the sample document, and the initial large language model is trained by the dense description to obtain a preset large language model.

[0161] It should be noted that if the dense information description subtask is currently being executed, the initial large language model may receive the above-mentioned first fine-tuning instruction, for example, the first fine-tuning instruction of "please read the following document and generate a document summary description within 1000 words" may be received;

[0162] After receiving the first fine-tuning instruction, the instruction type of the first fine-tuning instruction can be determined first, which can be a dense description generation type, and then it can be known that dense description generation is about to be performed. Then, based on the sample layout information and the corresponding instruction type, the target original document related to generating dense description can be obtained from the sample document, and the target original document can be summarized to obtain the corresponding dense description, such as "this document is a resume, the name is Zhang San, graduated from a certain university...";

[0163] The initial large language model is then trained using the dense description to obtain the above-mentioned preset large language model.

[0164] Further, in this embodiment, when the pre-training task includes a text layout reconstruction subtask, the fine-tuning instruction includes a second fine-tuning instruction, and the second fine-tuning instruction is an instruction for enabling the initial large language model to have text position learning capability; the above step S033 also includes:

[0165] The text coordinates of the target original document are determined, and the initial large language model is trained using the text coordinates and the target original document to obtain a preset large language model.

[0166] It is understandable that if the text layout reconstruction subtask is currently being executed, the initial large language model may receive the second fine-tuning instruction, for example, the second fine-tuning instruction of "please generate all layout information of this document" may be received;

[0167] After receiving the second fine-tuning instruction, the instruction type of the second fine-tuning instruction can be determined first, which can be a type for generating all layout information, and then it can be known that all layout information generation is about to be performed. Then, based on the sample layout information and the corresponding instruction type, the target original documents related to generating all layout information can be obtained from the sample document, and the text coordinates of each target original document can be determined, so as to obtain the corresponding global layout information, such as "<[0, 5, 30, 15], Zhang San>; <[20, 3, 6, 88], Telephone>; ...";

[0168] Afterwards, the initial large language model is trained using the target original document and the corresponding text coordinates (ie, the above-mentioned global layout information) to obtain a preset large language model.

[0169] Further, in this embodiment, when the pre-training task includes a layout element position determination subtask, the fine-tuning instruction includes a third fine-tuning instruction, and the third fine-tuning instruction is an instruction for enabling the initial large language model to have element position learning capability; the above step S033 also includes:

[0170] The element coordinates and element types of the target original document are determined, and the initial large language model is trained according to the element coordinates and the element types to obtain a preset large language model.

[0171] It should be understood that if the layout element position determination subtask is currently being executed, the initial large language model may receive the third fine-tuning instruction, for example, the third fine-tuning instruction of “where can I find the document title” may be received;

[0172] After receiving the third fine-tuning instruction, the instruction type of the third fine-tuning instruction can be determined first, which can be a layout element position determination type, and then it can be known that the layout element position determination is about to be performed. Then, based on the sample layout information and the corresponding instruction type, the target original document related to the layout element position generation can be obtained from the sample document, and the element coordinates and element type of each original document can be determined to obtain the corresponding layout element position, for example, "there are nine places in total, <[0, 5, 30, 15], Zhang San>; <[125, 44, 96, 33], Education Background>, ..."

[0173] Then, the initial large language model is trained using the element coordinates (i.e., [0, 5, 30, 15], [125, 44, 96, 33], etc. in the above example) and the element type (i.e., the document title in the above example) to obtain a preset large language model.

[0174] Further, in this embodiment, when the pre-training task includes a layout element type determination subtask, the fine-tuning instruction includes a fourth fine-tuning instruction, and the fourth fine-tuning instruction is an instruction for enabling the initial large language model to have element type learning capability; the above step S033 also includes:

[0175] The element coordinates and element types of the target original document are determined, and the initial large language model is trained according to the element types and the element coordinates to obtain a preset large language model.

[0176] It should be noted that if the layout element type determination subtask is currently being executed, the initial large language model may receive the fourth fine-tuning instruction, for example, the fourth fine-tuning instruction of "what is the type of the layout element at [0, 5, 30, 15]" may be received;

[0177] After receiving the fourth fine-tuning instruction, the instruction type of the fourth fine-tuning instruction may be determined first, and the type may be a layout element type determination type, so that it can be known that the type of the layout element is about to be determined. Then, based on the sample layout information and the corresponding instruction type, the target original document related to the type of the layout element may be obtained from the sample document, and the element type and the corresponding element coordinates of each original document may be determined, so as to obtain the corresponding layout element type, for example, "the type of the layout element at [0, 5, 30, 15] is the document title";

[0178] Then, the initial large language model is trained using the element coordinates (ie, [0, 5, 30, 15] in the above example) and the element type (ie, the document title in the above example) to obtain a preset large language model.

[0179] Further, in this embodiment, when the pre-training task includes a layout element content determination subtask, the fine-tuning instruction includes a fifth fine-tuning instruction, and the fifth fine-tuning instruction is an instruction for enabling the initial large language model to have the element content learning capability of the layout element; the above step S033 also includes:

[0180] The element coordinates, element type and element content of the target original document are determined, and the initial large language model is trained according to the element coordinates, the element type and the element content to obtain a preset large language model.

[0181] It is understandable that if the layout element content determination subtask is currently being executed, the initial large language model may receive the fifth fine-tuning instruction, for example, the fifth fine-tuning instruction of "what is the content of the Xth row and Xth column in the table" may be received;

[0182] After receiving the fifth fine-tuning instruction, the instruction type of the fifth fine-tuning instruction can be determined first, which can be the type of layout element content determination, and then it can be known that the content of the layout element is about to be determined. Then, based on the sample layout information and the corresponding instruction type, the target original document related to the generated layout element content can be obtained from the sample document, and the element type, element coordinates and element content of the original document can be determined, so as to obtain the corresponding layout element content, for example, "the content of the Xth row and the Xth column in the table is XX";

[0183] Then, the initial large language model is trained using the element coordinates (i.e., the coordinates of the table in the above example), the element type (i.e., the table in the above example), and the element content (i.e., the content of the table in the above example) to obtain a preset large language model.

[0184] Further, in this embodiment, when the pre-training task includes the mask visual language subtask, the fine-tuning instruction includes a sixth fine-tuning instruction, and the sixth fine-tuning instruction is an instruction for enabling the initial large language model to have the ability to make predictions based on the context text content; the above step S033 also includes:

[0185] The context text content of the target original document is predicted, and an initial large language model is trained according to the obtained first prediction result to obtain a preset large language model.

[0186] It is understandable that if the mask visual language subtask is currently being executed, the initial large language model may first perform content masking on the layout elements and receive the sixth fine-tuning instruction, for example, the sixth fine-tuning instruction of “what is the text of the masked part” may be received;

[0187] After receiving the sixth fine-tuning instruction, the instruction type of the sixth fine-tuning instruction can be determined first, which can be a content masking prediction type, and then the masking content prediction to be performed can be known. Then, based on the sample layout information and the corresponding instruction type, the target original document related to the generated layout element content can be obtained from the sample document, and the context content of the original document, that is, the context content of the masked part, can be determined. Then, prediction is performed based on the context content, and the obtained prediction result is used as the first preset result, for example: "academic intern";

[0188] Then, the initial large language model is trained using the first prediction result (i.e., the academic intern in the above example) to obtain a preset large language model.

[0189] Further, in this embodiment, when the pre-training task includes a mask position instruction subtask, the fine-tuning instruction includes a seventh fine-tuning instruction, and the seventh fine-tuning instruction is an instruction for enabling the initial large language model to have the ability to predict the association between the text content and the corresponding text position; the above step S033 also includes:

[0190] The corresponding text position is predicted according to the text content of the target original document, and the initial large language model is trained by using the obtained second prediction result to obtain a preset large language model.

[0191] It is understandable that if the mask position instruction subtask is currently being executed, the initial large language model may first perform position masking on the layout elements, that is, modify the text coordinates of a certain text to [0, 0, 0, 0], for example, modify the text coordinates [56, 89, 558, 217] in <[56, 89, 558, 217], my responsibilities> to [0, 0, 0, 0], and receive the seventh fine-tuning instruction, for example, the seventh fine-tuning instruction of "the text box is set to the position of 0, what is the original text box" may be received;

[0192] After receiving the seventh fine-tuning instruction, the instruction type of the seventh fine-tuning instruction can be determined first, which can be a position masking prediction type, and then the masking position prediction can be known. Then, based on the sample layout information and the corresponding instruction type, the target original document for generating the association relationship between the text content and the corresponding text position can be obtained from the sample document, and the text content and the corresponding text position of the original document, that is, the text position of the masked part, can be determined. Then, the text position is predicted according to the text content, and the obtained prediction result is used as the second preset result, for example: "the text box is set to the position of 0, and the original text box is [56, 89, 558, 217]";

[0193] Then, the initial large language model is trained using the second prediction result (i.e., the position where the text box is set to 0 in the above example, the original text box is [56, 89, 558, 217]) to obtain the preset large language model.

[0194] Further, in this embodiment, when the pre-training task includes a geometric layout analysis subtask, the fine-tuning instruction includes an eighth fine-tuning instruction, and the eighth fine-tuning instruction is an instruction for enabling the initial large language model to have the ability to predict geometric layout relationships; the above step S033 also includes:

[0195] The geometric layout relationship between the target original documents is predicted, and the initial large language model is trained according to the geometric layout relationship prediction result to obtain a preset large language model.

[0196] It should be understood that if the geometric layout analysis subtask is currently being executed, the initial large language model may first determine the element coordinates of each layout element and receive the eighth fine-tuning instruction, for example, the eighth fine-tuning instruction of “what is the direction and distance of “academic intern” relative to “2024.05-2024.06” may be received;

[0197] After receiving the eighth fine-tuning instruction, the instruction type of the eighth fine-tuning instruction can be determined first, which can be a geometric layout analysis type, and then it can be known that a prediction of a geometric layout relationship is about to be performed. Then, based on the sample layout information and the corresponding instruction type, the target original document for generating a geometric layout relationship can be obtained from the sample document, and the element coordinates of the original document can be determined. Then, the geometric layout relationship is predicted according to the element coordinates, and the obtained prediction result is used as the above-mentioned geometric layout relationship preset result, for example: "The direction of "academic intern" relative to "2024.05-2024.06" is obliquely downward, and the distance is 30";

[0198] Then, the initial large language model is trained based on the preset result of the geometric layout relationship (i.e., in the above example, the direction of "academic intern" relative to "2024.05-2024.06" is diagonally downward, and the distance is 30") to obtain the preset large language model.

[0199] At the same time, in actual use, since the preset large language model can understand the layout information of the document to be analyzed, it can then combine the layout information when answering, making the answer more accurate and improving the accuracy of subsequent downstream analysis tasks.

[0200] Furthermore, considering that the sample documents may not all be documents in image form, in this embodiment, the steps of obtaining the sample document, identifying the sample document, and obtaining the sample layout information of the sample document include:

[0201] Step S011: obtaining a sample document, and determining whether the document type of the sample document is a preset document type;

[0202] Step S012: If not, perform text analysis on the sample document using the initial large language model, and convert the document type of the sample document based on the text analysis result;

[0203] Step S013: Identify the sample document after the document type conversion, and obtain sample layout information of the sample document.

[0204] It should be noted that the above-mentioned preset document type can be a document type in the form of an image. In actual use, after obtaining a sample document, it can be first determined whether the sample document is a document type in the form of an image;

[0205] If so, it can be explained that the OCR method can be directly used for recognition, and then the sample document can be OCR recognized to obtain the layout information of the sample document;

[0206] If not, it can be explained as a document of other document types, such as a sample document of word type, etc., and then the sample document of word type can be directly input into the initial large language model. This is because the sample document of word type contains pure text content, which generally does not contain layout information. Therefore, the initial large language model can directly perform text analysis on the sample document to obtain the text content therein. As the above-mentioned text analysis result, when performing the analysis, it can also be asked questions according to the various fine-tuning instructions in the above-mentioned pre-training task to obtain the above-mentioned text analysis result. It should be noted that it may not contain fine-tuning instructions for asking questions about layout information. Since the questioning process can be consistent with the above-mentioned process of fine-tuning the first to eighth instructions, this embodiment will not be repeated.

[0207] After obtaining the above text analysis results, the text analysis results can be converted into a document type, and can be converted into a HyperText Markup Language (HTML) format, and then the sample document in HTML format can be converted into a sample document in image form, and then the sample document in image form can be used as a sample document after the above document type conversion, and it can be recognized by OCR to obtain the layout information of the sample document.

[0208] In addition, an embodiment of the present application further proposes a storage medium on which a document analysis program is stored. When the document analysis program is executed by a processor, the document analysis method as described above is implemented.

[0209] In addition, refer to Figure 5 , Figure 5 This is a structural block diagram of the first embodiment of the document analysis device of this application; Figure 5 As shown, the embodiment of the present application also proposes a document analysis device, which includes:

[0210] The document acquisition module 501 is used to acquire the document to be analyzed, identify the document to be analyzed, and obtain the layout information to be analyzed of the document to be analyzed;

[0211] A document analysis module 502, configured to analyze the document to be analyzed based on the layout information to be analyzed by using a preset large language model to obtain an analysis result;

[0212] The preset large language model is obtained by model training according to sample layout information of the sample document and a corresponding pre-training task, wherein the pre-training task is a task that enables the preset large language model to have the ability to understand the sample layout information after model training.

[0213] This embodiment can identify sample documents, obtain sample layout information of sample documents, perform model training through the sample layout information and corresponding pre-training tasks, and obtain a preset large language model capable of understanding sample layout information. When used, the document to be analyzed is first identified to obtain the layout information to be analyzed of the document to be analyzed, and then the document to be analyzed is analyzed based on the layout information to be analyzed through the preset large language model to obtain the analysis result. Compared with the existing large language model that can only understand the plain text content, the preset large language model in this embodiment can also understand the layout information therein, thereby improving the model's understanding and analysis capabilities and improving the accuracy of subsequent downstream task analysis.

[0214] As an implementation mode, the document acquisition module 501 is also used to acquire sample documents, identify the sample documents, and obtain sample layout information of the sample documents; construct corresponding pre-training tasks based on the sample layout information; train the initial large language model through the sample layout information and the corresponding pre-training tasks to obtain a preset large language model.

[0215] As an implementation manner, when the document to be analyzed is a job seeker's resume containing layout elements such as text, pictures, tables, and titles, the analysis result is a result of the spatial structural relationship between the layout elements.

[0216] Based on the above-mentioned first embodiment of the document analysis device of the present application, a second embodiment of the document analysis device of the present application is proposed.

[0217] In this embodiment, the pre-training task includes: at least one of a document-level pre-training task, a region-level pre-training task and a fragment-level pre-training task; the document-level pre-training task is a task for globally learning the sample layout information; the region-level pre-training task is a task for performing specific region learning on the sample layout information; the fragment-level pre-training task is a task for performing fragment learning on the sample layout information.

[0218] As an implementation mode, the pre-training task includes a document level pre-training task, and the document level pre-training task includes: a dense information description sub-task and a text layout reconstruction sub-task;

[0219] The document acquisition module 501 is also used to generate a dense description of the sample document according to the sample layout information, and construct the dense information description subtask according to the dense description; determine the text coordinates of the layout elements in the sample layout information, and construct the text layout reconstruction subtask based on the text coordinates.

[0220] As an implementation manner, the pre-training task includes a region-level pre-training task, and the region-level pre-training task includes: a layout element position determination subtask, a layout element type determination subtask, and a layout element content determination subtask;

[0221] The document acquisition module 501 is further used to determine the element coordinates, element type and element content of the layout elements in the sample layout information; construct the layout element position determination subtask and the layout element type determination subtask according to the element coordinates and the element type; and construct the layout element content determination subtask according to the element coordinates, the element type and the element content.

[0222] As an implementation manner, the pre-training task includes a segment-level pre-training task, and the segment-level pre-training task includes: a mask visual language subtask, a mask position instruction subtask, and a geometric layout analysis subtask;

[0223] The document acquisition module 501 is also used to perform content masking on the layout elements in the sample layout information, and construct the mask visual language subtask based on the sample layout information after content masking; perform position masking on the layout elements in the sample layout information, and construct the mask position instruction subtask based on the sample layout information after position masking; determine the geometric distribution of the layout elements in the sample layout information, and construct the geometric layout analysis subtask based on the geometric distribution.

[0224] Based on the above-mentioned embodiments of the document analysis device of the present application, a third embodiment of the document analysis device of the present application is proposed.

[0225] In this embodiment, the document acquisition module 501 is also used to receive the fine-tuning instruction corresponding to the pre-training task and determine the instruction type of the fine-tuning instruction; determine the target original document related to the fine-tuning instruction in the sample document based on the sample layout information and the instruction type; organize the target original document, and train the initial large language model through the organization result to obtain a preset large language model.

[0226] As an implementation mode, when the pre-training task includes a dense information description subtask, the fine-tuning instruction includes a first fine-tuning instruction, and the first fine-tuning instruction is an instruction for enabling the initial large language model to have a text summarization capability; the document acquisition module 501 is further used to summarize the target original document to obtain a dense description of the sample document, and train the initial large language model through the dense description to obtain a preset large language model;

[0227] When the pre-training task includes a text layout reconstruction subtask, the fine-tuning instruction includes a second fine-tuning instruction, and the second fine-tuning instruction is an instruction for enabling the initial large language model to have text position learning capabilities; the document acquisition module 501 is also used to determine the text coordinates of the target original document, and train the initial large language model through the text coordinates and the target original document to obtain a preset large language model.

[0228] As an implementation manner, when the pre-training task includes a layout element position determination subtask, the fine-tuning instruction includes a third fine-tuning instruction, and the third fine-tuning instruction is an instruction for enabling the initial large language model to have element position learning capability; the document acquisition module 501 is further used to determine the element coordinates and element types of the target original document, and train the initial large language model through the element coordinates and the element type to obtain a preset large language model;

[0229] When the pre-training task includes a layout element type determination subtask, the fine-tuning instruction includes a fourth fine-tuning instruction, and the fourth fine-tuning instruction is an instruction for enabling the initial large language model to have element type learning capability; the document acquisition module 501 is further used to determine the element coordinates and element types of the target original document, and train the initial large language model through the element types and the element coordinates to obtain a preset large language model;

[0230] When the pre-training task includes a layout element content determination subtask, the fine-tuning instruction includes a fifth fine-tuning instruction, and the fifth fine-tuning instruction is an instruction for enabling the initial large language model to have the element content learning ability of the layout element; the document acquisition module 501 is also used to determine the element coordinates, element type and element content of the target original document, and train the initial large language model through the element coordinates, the element type and the element content to obtain a preset large language model.

[0231] As an implementation manner, when the pre-training task includes a mask visual language subtask, the fine-tuning instruction includes a sixth fine-tuning instruction, and the sixth fine-tuning instruction is an instruction for enabling the initial large language model to have a prediction capability based on context text content; the document acquisition module 501 is further used to predict the context text content of the target original document, and train the initial large language model through the obtained first prediction result to obtain a preset large language model;

[0232] When the pre-training task includes a mask position instruction subtask, the fine-tuning instruction includes a seventh fine-tuning instruction, and the seventh fine-tuning instruction is an instruction for enabling the initial large language model to have the ability to predict the association between text content and the corresponding text position; the document acquisition module 501 is further used to predict the corresponding text position according to the text content of the target original document, and train the initial large language model through the obtained second prediction result to obtain a preset large language model;

[0233] When the pre-training task includes a geometric layout analysis subtask, the fine-tuning instruction includes an eighth fine-tuning instruction, and the eighth fine-tuning instruction is an instruction for enabling the initial large language model to have the ability to predict geometric layout relationships; the document acquisition module 501 is also used to predict the geometric layout relationships between the target original documents, and train the initial large language model through the geometric layout relationship prediction results to obtain a preset large language model.

[0234] As an implementation mode, the document acquisition module 501 is also used to acquire a sample document and determine whether the document type of the sample document is a preset document type; if not, perform text analysis on the sample document through an initial large language model, and convert the document type of the sample document based on the text analysis result; identify the sample document after the document type conversion, and obtain sample layout information of the sample document.

[0235] Other embodiments or specific implementations of the document analysis device described in this application can refer to the above-mentioned method embodiments and will not be repeated here.

[0236] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0237] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0238] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory image (ROM) / random access memory (RAM), a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0239] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A document analysis method, characterized in that: The method comprises: Acquire a document to be analyzed, identify the document to be analyzed, and obtain layout information to be analyzed of the document to be analyzed; Analyzing the document to be analyzed based on the layout information to be analyzed by using a preset large language model to obtain an analysis result; The preset large language model is obtained by model training according to sample layout information of the sample document and a corresponding pre-training task, wherein the pre-training task is a task that enables the preset large language model to have the ability to understand the sample layout information after model training.

2. The method according to claim 1, characterized in that The pre-training task includes: at least one of a document level pre-training task, a region level pre-training task, and a segment level pre-training task; The document level pre-training task is a task of globally learning the sample layout information; The region-level pre-training task is a task of performing specific region learning on the sample layout information; The segment-level pre-training task is a task of performing segment learning on the sample layout information.

3. The method according to claim 1, characterized in that Before the step of obtaining the document to be analyzed, the method further includes: Acquire a sample document, identify the sample document, and obtain sample layout information of the sample document; Constructing a corresponding pre-training task based on the sample layout information; The initial large language model is trained using the sample layout information and the corresponding pre-training task to obtain a preset large language model.

4. The method according to claim 3, characterized in that The pre-training task includes a document-level pre-training task, and the document-level pre-training task includes: a dense information description sub-task and a text layout reconstruction sub-task; The step of constructing a corresponding pre-training task based on the sample layout information includes: generating a dense description of the sample document according to the sample layout information, and constructing the dense information description subtask according to the dense description; The text coordinates of the layout elements in the sample layout information are determined, and the text layout reconstruction subtask is constructed based on the text coordinates.

5. The method according to claim 3, characterized in that The pre-training task includes a region-level pre-training task, and the region-level pre-training task includes: a layout element position determination subtask, a layout element type determination subtask, and a layout element content determination subtask; The step of constructing a corresponding pre-training task based on the sample layout information includes: Determining element coordinates, element types, and element contents of layout elements in the sample layout information; Constructing the layout element position determination subtask and the layout element type determination subtask according to the element coordinates and the element type; The layout element content determination subtask is constructed according to the element coordinates, the element type, and the element content.

6. The method according to claim 3, characterized in that The pre-training task includes a segment-level pre-training task, and the segment-level pre-training task includes: a mask visual language subtask, a mask position instruction subtask, and a geometric layout analysis subtask; The step of constructing a corresponding pre-training task based on the sample layout information includes: Performing content masking on the layout elements in the sample layout information, and constructing the mask visual language subtask according to the sample layout information after content masking; Performing position masking on the layout elements in the sample layout information, and constructing the mask position instruction subtask based on the position masked sample layout information; The geometric distribution of layout elements in the sample layout information is determined, and the geometric layout analysis subtask is constructed according to the geometric distribution.

7. The method according to claim 3, characterized in that The step of training the initial large language model by using the sample layout information and the corresponding pre-training task to obtain a preset large language model includes: Receiving a fine-tuning instruction corresponding to the pre-training task, and determining an instruction type of the fine-tuning instruction; Determine a target original document related to the fine-tuning instruction in the sample document based on the sample layout information and the instruction type; The target original document is sorted, and an initial large language model is trained according to the sorting result to obtain a preset large language model.

8. The method according to claim 7, characterized in that When the pre-training task includes a dense information description subtask, the fine-tuning instruction includes a first fine-tuning instruction, and the first fine-tuning instruction is an instruction for enabling the initial large language model to have text summarization capability; the method further includes: Summarizing the target original document to obtain a dense description of the sample document, and training the initial large language model through the dense description to obtain a preset large language model; When the pre-training task includes a text layout reconstruction subtask, the fine-tuning instruction includes a second fine-tuning instruction, and the second fine-tuning instruction is an instruction for enabling the initial large language model to have a text position learning capability; the method further includes: The text coordinates of the target original document are determined, and the initial large language model is trained using the text coordinates and the target original document to obtain a preset large language model.

9. The method according to claim 7, characterized in that When the pre-training task includes a layout element position determination subtask, the fine-tuning instruction includes a third fine-tuning instruction, and the third fine-tuning instruction is an instruction for enabling the initial large language model to have element position learning capability; the method further includes: Determining element coordinates and element types of the target original document, and training the initial large language model according to the element coordinates and the element type to obtain a preset large language model; When the pre-training task includes a layout element type determination subtask, the fine-tuning instruction includes a fourth fine-tuning instruction, and the fourth fine-tuning instruction is an instruction for enabling the initial large language model to have element type learning capability; the method further includes: Determining element coordinates and element types of the target original document, and training the initial large language model according to the element types and the element coordinates to obtain a preset large language model; When the pre-training task includes a layout element content determination subtask, the fine-tuning instruction includes a fifth fine-tuning instruction, and the fifth fine-tuning instruction is an instruction for enabling the initial large language model to have an element content learning capability of the layout element; the method further includes: The element coordinates, element type and element content of the target original document are determined, and the initial large language model is trained according to the element coordinates, the element type and the element content to obtain a preset large language model.

10. The method according to claim 7, characterized in that When the pre-training task includes a mask visual language subtask, the fine-tuning instruction includes a sixth fine-tuning instruction, and the sixth fine-tuning instruction is an instruction for enabling the initial large language model to have a prediction capability according to context text content; the method further includes: Predicting the context text content of the target original document, and training an initial large language model using the obtained first prediction result to obtain a preset large language model; When the pre-training task includes a mask position instruction subtask, the fine-tuning instruction includes a seventh fine-tuning instruction, and the seventh fine-tuning instruction is an instruction for enabling the initial large language model to have the ability to predict the association between text content and corresponding text positions; the method further includes: Predicting the corresponding text position according to the text content of the target original document, and training the initial large language model through the obtained second prediction result to obtain a preset large language model; When the pre-training task includes a geometric layout analysis subtask, the fine-tuning instruction includes an eighth fine-tuning instruction, and the eighth fine-tuning instruction is an instruction for enabling the initial large language model to have the ability to predict geometric layout relationships; the method further includes: The geometric layout relationship between the target original documents is predicted, and the initial large language model is trained according to the geometric layout relationship prediction result to obtain a preset large language model.

11. The method according to claim 3, characterized in that The step of acquiring a sample document, identifying the sample document, and obtaining sample layout information of the sample document includes: Obtain a sample document, and determine whether the document type of the sample document is a preset document type; If not, performing text analysis on the sample document using the initial large language model, and converting the document type of the sample document based on the text analysis result; The sample document after the document type conversion is identified to obtain sample layout information of the sample document.

12. The method according to any one of claims 1 to 11, characterized in that When the document to be analyzed is a resume of a job seeker containing layout elements such as text, picture, table and title, the analysis result is a result of the spatial structural relationship between the layout elements.

13. A document analysis device, characterized in that: The device comprises: A document acquisition module, used to acquire a document to be analyzed, identify the document to be analyzed, and obtain layout information to be analyzed of the document to be analyzed; A document analysis module, used to analyze the document to be analyzed based on the layout information to be analyzed by using a preset large language model to obtain an analysis result; The preset large language model is obtained by model training according to sample layout information of the sample document and a corresponding pre-training task, wherein the pre-training task is a task that enables the preset large language model to have the ability to understand the sample layout information after model training.

14. A storage medium, characterized in that: The storage medium stores a document analysis program, which, when executed by a processor, implements the steps of the document analysis method according to any one of claims 1 to 12.

15. A document analysis device, characterized in that: The document analysis device comprises: a memory, a processor, and a document analysis program stored in the memory and executable on the processor, wherein the document analysis program implements the steps of the document analysis method according to any one of claims 1 to 12 when executed by the processor.

Images