File analysis model training method and device

Abstract

Embodiments of the present application disclose a file analysis model training method and device. After obtaining an initial multi-modal model, a plurality of fine-tuning samples are obtained, and the multi-modal model is supervised fine-tuned according to the plurality of fine-tuning samples, and then a file analysis model is obtained according to the multi-modal model after the supervised fine-tuning. The fine-tuning samples include sample files and corresponding sample analysis data, the sample analysis data includes text information and semantic description of non-text key information, the text information is text modal extraction information of the sample file, the non-text key information is non-text modal extraction information of the sample file, the text information and the semantic description have corresponding attribute information, and the attribute information is used to represent the position and / or structure of the non-text key information corresponding to the text information or the semantic description in the sample file. Therefore, the file analysis model obtained by training can have both fine-grained analysis capability and semantic description capability.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to a method and apparatus for training a file parsing model. Background Technology

[0002] In the field of artificial intelligence, file parsing is commonly used to extract internal data from files (including compound files and video files) to transform unstructured data into structured data that machines can understand, thereby providing fundamental data support for downstream AI applications. Currently, file parsing is typically implemented through file parsing models. However, existing file parsing models struggle to balance fine-grained parsing capabilities with semantic description capabilities, making them unable to meet the file parsing requirements of AI applications. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a method and apparatus for training a file parsing model, so that the trained file parsing model can take into account both fine-grained parsing capabilities and semantic description capabilities.

[0004] In a first aspect, embodiments of the present invention aim to provide a method for training a file parsing model, the method comprising: Obtain the initial multimodal model; Multiple fine-tuning samples are obtained, wherein the fine-tuning samples include sample files and corresponding sample parsing data. The sample parsing data includes semantic descriptions of text information and non-text key information. The text information is the text modality extraction information of the sample file, and the non-text key information is the non-text modality extraction information of the sample file. The text information and the semantic description have corresponding attribute information. The attribute information is used to characterize the position and / or structure of the non-text key information corresponding to the text information or the semantic description in the sample file. The multimodal model is trained under supervised fine-tuning based on the multiple fine-tuning samples; The file parsing model is obtained from the supervised fine-tuned multimodal model.

[0005] Secondly, embodiments of the present invention aim to provide a file parsing model training apparatus, the apparatus comprising: The initial model acquisition unit is used to acquire the initial multimodal model; A sample acquisition unit is used to acquire multiple fine-tuning samples, wherein the fine-tuning samples include sample files and corresponding sample parsing data. The sample parsing data includes semantic descriptions of text information and non-text key information. The text information is the text modality extraction information of the sample file, and the non-text key information is the non-text modality extraction information of the sample file. The text information and the semantic description have corresponding attribute information, and the attribute information is used to characterize the position and / or structure of the non-text key information corresponding to the text information or the semantic description in the sample file. The training unit is used to perform supervised fine-tuning training of the multimodal model based on the plurality of fine-tuning samples; The parsing model acquisition unit is used to acquire the file parsing model based on the supervised fine-tuned multimodal model.

[0006] Thirdly, embodiments of the present invention aim to provide a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the method described in the first aspect.

[0007] Fourthly, embodiments of the present invention aim to provide an electronic device, the device comprising: Memory is used to store one or more computer program instructions; A processor, wherein the one or more computer program instructions are executed by the processor to implement the method as described in the first aspect.

[0008] Fifthly, embodiments of the present invention aim to provide a computer program product that, when run on a computer, causes the computer to perform the method described in the first aspect.

[0009] This invention, in its embodiments, acquires multiple fine-tuning samples after obtaining an initial multimodal model. Supervised fine-tuning training is then performed on the multimodal model based on these samples, and a file parsing model is obtained from the supervised fine-tuned multimodal model. The fine-tuning samples include sample files and corresponding sample parsing data. The sample parsing data includes textual information and semantic descriptions of non-textual key information. The textual information is the extracted textual modality information from the sample file, and the non-textual key information is the extracted non-textual modality information from the sample file. The textual information and semantic descriptions have corresponding attribute information, which characterizes the position and / or structure of the non-textual key information corresponding to the textual information or semantic description within the sample file. Therefore, this invention enables the trained file parsing model to balance fine-grained parsing capabilities and semantic description capabilities, thereby meeting the file parsing requirements of artificial intelligence applications. Attached Figure Description

[0010] The above and other objects, features and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which: Figure 1 This is a flowchart of the file parsing model training method according to an embodiment of the present invention; Figure 2 This is a flowchart of the fine-tuning sample acquisition method according to an embodiment of the present invention; Figure 3 This is a flowchart of the information extraction method according to an embodiment of the present invention; Figure 4 This is a schematic diagram of a composite file according to an embodiment of the present invention; Figure 5 This is a flowchart of another information extraction method according to an embodiment of the present invention; Figure 6 This is a schematic diagram illustrating the information extraction process of a video file according to an embodiment of the present invention; Figure 7 This is a flowchart of the model invocation method according to an embodiment of the present invention; Figure 8 This is a schematic diagram of sample analysis data in an embodiment of the present invention; Figure 9 This is a flowchart of the preference alignment training method according to an embodiment of the present invention; Figure 10 This is a flowchart of the knowledge enhancement training method according to an embodiment of the present invention; Figure 11 A flowchart illustrating a method for providing file parsing services according to an embodiment of the present invention; Figure 12 This is a schematic diagram of the model training process according to an embodiment of the present invention; Figure 13 This is a schematic diagram of a file parsing model training device according to an embodiment of the present invention; Figure 14 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0011] The present application is described below based on embodiments, but it is not limited to these embodiments. In the detailed description of the present application below, certain specific details are described in detail. Those skilled in the art can fully understand the present application without these details. To avoid obscuring the substance of the present application, well-known methods, processes, flows, elements, and circuits are not described in detail.

[0012] Furthermore, those skilled in the art should understand that the accompanying drawings provided herein are for illustrative purposes only and are not necessarily drawn to scale.

[0013] Unless the context explicitly requires it, words such as "including" or "contains" throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, meaning "including but not limited to".

[0014] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0015] The solutions described in this specification and embodiments, if involving the processing of personal information, will be processed only on the premise of having a legal basis (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be processed within the scope stipulated or agreed upon. A user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.

[0016] Figure 1 This is a flowchart illustrating the file parsing model training method according to an embodiment of the present invention. It is intended to be noted that... Figure 1 The file parsing model training method shown can be implemented by a general-purpose data processing device (hereinafter referred to as the processing device). By executing... Figure 1 The document parsing model training method shown herein allows the processing device to train and acquire a document parsing model that balances fine-grained parsing capabilities and semantic description capabilities, thereby meeting the document parsing needs of artificial intelligence applications. Optionally, the processing device can be a terminal device (e.g., a desktop computer, laptop computer, smartphone, smart speaker, smart wearable device, tablet computer, or in-vehicle terminal, etc.) or a server (a server can be a single computer, a cluster of multiple computers, or a cloud server that can flexibly adjust computing resources through cloud technology); this application does not impose any limitations on this. Figure 1 As shown, the file parsing model training method may specifically include the following steps: Step S100: Obtain the initial multimodal model.

[0017] Specifically, the processing device can acquire an initial multimodal model. It should be noted that, in this embodiment of the invention, the multimodal model used as the training object can be any existing multimodal model, and this application does not impose any limitations on it. Indicatively, the multimodal model may include Qwen3-Omni, Ming-Omni, and Uni-MoE-2.0-Omni, etc.

[0018] Step S200: Obtain multiple fine-tuning samples.

[0019] Specifically, after the multimodal model, the processing device can acquire multiple fine-tuning samples. These fine-tuning samples include sample files and corresponding sample parsing data. The sample parsing data includes semantic descriptions of textual information and non-textual key information. The textual information is the textual modality extraction information from the sample files, and the non-textual key information is the non-textual modality extraction information from the sample files. The textual information and semantic descriptions have corresponding attribute information, which is used to characterize the position and / or structure of the non-textual key information corresponding to the textual information or semantic description in the sample files.

[0020] Figure 2 This is a flowchart of a fine-tuning sample acquisition method according to an embodiment of the present invention. It is intended to illustrate that by executing... Figure 2 The fine-tuning sample acquisition method shown allows the processing device to acquire multiple fine-tuning samples that meet the requirements, that is, to implement the above step S200. Figure 2 As shown, the fine-tuning sample acquisition method may specifically include the following steps: Step S210: Obtain multiple sample files.

[0021] Specifically, the processing device can acquire multiple sample files. It should be noted that, in this embodiment of the invention, sample files can include files of various formats. For example, composite files such as PDFs (Portable Document Format), scanned documents, or reports. Other examples include video files such as recorded courses, online open courses, and corporate training materials. This application does not limit the format of the sample files.

[0022] Step S220: For each of the sample files, information is extracted from the sample files to obtain the text information and the non-text key information.

[0023] Specifically, after acquiring multiple sample files, the processing device can extract information from each sample file to obtain textual and non-textual key information.

[0024] Optionally, the processing device can employ different methods to extract information from sample files of different formats, obtaining both textual and non-textual key information. Furthermore, the content of the non-textual key information extracted by the processing device can differ depending on the format of the sample file. For example, for composite files, the non-textual key information extracted by the processing device may include visual information. As another example, for video files, the non-textual key information extracted by the processing device may include both visual and auditory information.

[0025] Figure 3This is a flowchart of an information extraction method according to an embodiment of the present invention. It is intended to illustrate that when the sample file includes a composite file, the processing device can perform actions such as... Figure 3 The information extraction method shown herein extracts information from a composite document to obtain textual information and the required non-textual key information, thus achieving step S220 above. Figure 3 As shown, the information extraction method may specifically include the following steps: Step S221: Extract text information from the composite file to obtain the text information.

[0026] Specifically, the processing device can extract text information from the composite document to obtain text information. It should be noted that the composite document can exist as a file itself, or as an image or other form; this application does not impose any restrictions on this.

[0027] Optionally, a complete composite document typically includes multiple text content sections belonging to different text structures (e.g., title, subtitle, header / footer, table of contents, introduction, body text, clauses, and figure / table descriptions). In step S210, as one implementation, when extracting text information from the composite document, the processing device can first identify the text recognition region where each text content section is located, and then extract text information from each text recognition region separately to obtain the corresponding text information. Further optionally, the text information extraction method used by the processing device may include Optical Character Recognition (OCR) and embedded text reading, etc., and this application does not limit this.

[0028] Step S222: Extract visual information from the composite file to obtain the visual information.

[0029] Specifically, the processing device can extract visual information from the composite document to obtain visual information. It should be noted that, in this embodiment of the invention, the visual information extracted by the processing device from the sample file may specifically include illustrations and / or diagrams in the sample file.

[0030] Figure 4 This is a schematic diagram of a composite file according to an embodiment of the present invention. Figure 4As shown, for the composite document 41, when extracting text information, the processing device can first identify the text recognition regions 411, 412, and 413 where each text content part is located, and then extract text information from the text recognition regions 411, 412, and 413 respectively to obtain the corresponding text information. Furthermore, the processing device can extract visual information from the composite document 41 to obtain the illustration 414. It should be noted that, in order to determine the semantic description attributes of text information and non-text key information in subsequent steps, the position and / or structure of text information and non-text key information in the sample file can be determined synchronously by the processing device during the information extraction process; this application does not specifically limit this.

[0031] Figure 5 This is a flowchart illustrating another information extraction method according to an embodiment of the present invention. It is intended to illustrate that when the sample file includes a video file, the processing device can perform actions such as... Figure 5 The information extraction method shown extracts information from the video file to obtain text information and the required non-text key information, that is, to achieve the above step S220. Figure 5 As shown, the information extraction method may specifically include the following steps: Step S221': Extract keyframes from the video file to obtain at least one keyframe.

[0032] Specifically, a keyframe can refer to a video frame that contains a large amount of information or is representative of the video file. Different keyframes can reflect the core content of the video file at different video segments. In this step, the processing device can extract keyframes from the video file to obtain at least one keyframe.

[0033] Optionally, in step S221', the keyframe extraction method used by the processing device may include visual transformation detection based on adjacent video frames, motion detection based on target objects in the video file, and / or end-to-end detection methods based on deep learning. Furthermore, as an implementation method, to improve the keyframe extraction effect, the processing device may select an appropriate method to extract keyframes from the video file based on the video type. This application does not limit the specific keyframe extraction method.

[0034] Step S222': Extract text information from each of the keyframes to obtain the text information.

[0035] Specifically, after obtaining the keyframes, the processing device can extract text information from each keyframe to obtain the text information. It should be noted that the processing device's extraction of text information from keyframes is essentially equivalent to the processing device's extraction of text information from composite documents; the specifics can be found in the description above and will not be repeated here.

[0036] Step S223': Visual information is extracted from each of the keyframes to obtain the visual information.

[0037] Specifically, after obtaining the keyframes, the processing device can extract visual information from each keyframe to obtain visual information. It should be noted that the processing device's extraction of visual information from keyframes is essentially equivalent to the processing device's extraction of visual information from composite files; the specifics can be found in the description above and will not be repeated here.

[0038] Step S224': Extract audio information from the video time segment corresponding to each key frame to obtain the auditory information.

[0039] Specifically, after obtaining the keyframes, the processing device can extract audio information from the video segment corresponding to each keyframe to obtain auditory information.

[0040] Optionally, in step S224', the auditory information extraction method used by the processing device may include ASR (Automatic Speech Recognition), and this application does not limit this. Furthermore, as one implementation, the auditory information extracted by the processing device may exist in text form.

[0041] Figure 6 This is a schematic diagram illustrating the information extraction process of a video file according to an embodiment of the present invention. It should be noted that this is for ease of demonstration. Figure 6 This only includes the information extraction process for a single keyframe. For example... Figure 6 As shown, by extracting keyframes from a video file, the processing device can obtain keyframe T. Then, by analyzing the video segment corresponding to keyframe T (…),… Figure 6 The shadowed area of ​​the video file shown is used to represent the video segment corresponding to keyframe T. By extracting audio information from keyframe T, the processing device can obtain auditory information 62. By extracting text information from keyframe T, the processing device can obtain text information 63. By extracting visual information from keyframe T, the processing device can obtain visual information 64.

[0042] Step S230: Determine the semantic description of each of the non-text key information based on the text information and the non-text key information.

[0043] Specifically, after determining the textual and non-textual key information, the processing device can determine the semantic description of each non-textual key information based on the textual and non-textual key information. It is worth noting that by determining the semantic description of each non-textual key information based on the textual and non-textual key information, embodiments of the present invention can generate semantic descriptions of non-textual key information with contextual meaning. Furthermore, by using sample files constructed using these semantic descriptions to perform supervised fine-tuning training of the multimodal model, embodiments of the present invention can ensure that the trained file parsing model possesses strong semantic description capabilities.

[0044] Optionally, in step S230, as one implementation method, for each non-textual key information, the processing device can call a large language model based on the textual information and the non-textual key information to generate a semantic description of the non-textual key information.

[0045] Figure 7 This is a flowchart of a model invocation method according to an embodiment of the present invention. It is intended to illustrate that for any non-textual key information, by executing... Figure 7 The model invocation method shown allows the processing device to invoke a large language model based on textual and non-textual key information to generate a semantic description of the non-textual key information. For example... Figure 7 As shown, the model invocation method may specifically include the following steps: Step S231: Construct semantic description prompts based on the text information and the non-text key information.

[0046] Specifically, the processing device can construct semantic description prompts based on textual and non-textual key information. These semantic description prompts can be a textual description used to guide the large language model in generating semantic descriptions of non-textual key information, and then outputting the generated semantic descriptions in the desired form.

[0047] Optionally, in step S231, as one implementation, the semantic description prompt can be constructed by the processing device by filling textual information and non-textual key information into the corresponding semantic description generation prompt template. The semantic description generation prompt template can be a statement template prepared in advance by relevant personnel. Optionally, the semantic description generation prompt template can include at least a semantic description generation instruction and a semantic description generation requirement. The semantic description generation instruction can be used to specify the instructions required by the large language model to complete the semantic description generation task. For example, the semantic description generation instruction could be "Please generate a semantic description of the non-textual key information based on the input textual information and non-textual key information." The semantic description generation requirement can be used to assist the large language model in generating semantic descriptions and instruct it to output the generated results in a corresponding format. For example, the semantic description generation requirement could be: "If the non-textual key information is auditory, please convert the audio content into a contextually meaningful natural language description. For related objects involved in the auditory information, please provide the intent or purpose of the related objects. For event narrations or process descriptions involved in the audio content, please outline the chronological order to help understand the development of the events. If the non-textual key information is visual, please convert the visual content into a contextually meaningful natural language description. For charts involved in the visual information, please interpret the relationships, trends, and proportions between the data, and provide the intended purpose of the chart. For objects involved in the visual information, please describe each object and the relationships between them."

[0048] For example, the semantic description prompts constructed by the processing device can be: [Please generate a semantic description of the non-textual key information based on the input text information and non-textual key information. Wherein, if the non-textual key information is auditory information, please convert the speech content into a natural language description with contextual meaning. For the speaker involved in the auditory information, please give the speaker's intention or purpose; for the event narration or process description involved in the speech content, please sort out the time sequence to help understand the development of the event. If the non-textual key information is visual information, please convert the visual content into a natural language description with contextual meaning. For the chart involved in the visual information, please interpret the relationships, trends, and proportions between the data in the chart, and give the display intention of the chart in combination with the contextual information; for the display objects involved in the visual information, please specifically describe each display object and explain the relationship between each display object]. It should be noted that the content included in the semantic description generation prompt template and the semantic description prompts constructed based on the template given above are merely illustrative. In actual applications, the content included in the semantic description generation prompt template and the semantic description prompts constructed based on the template are not limited to this.

[0049] Step S232: Input the semantic description prompts into the large language model to generate a semantic description of the non-textual key information based on the large language model.

[0050] Specifically, after constructing semantic description prompts, the processing device can input the semantic description prompts into a large language model to generate semantic descriptions of non-textual key information based on the large language model.

[0051] Optionally, in this embodiment of the invention, the large language model used for invocation by the processing device can be any existing large language model, such as the Deepseek large language model, the Qwen large language model, the Wenxin Yiyan large language model, or any other large language model. This application does not impose any limitations on this. Furthermore, in this embodiment of the invention, the large language model used for invocation by the processing device can be deployed locally or on an online server. This application does not impose any limitations on this. When the large language model is deployed locally, the processing device can directly invoke the large language model locally. When the large language model is deployed on an online server, the processing device can directly interact with the online server through an API (Application Programming Interface) to invoke the large language model.

[0052] Step S240: For each of the sample files, generate corresponding sample parsing data based on the text information, the semantic description, and the attribute information.

[0053] Specifically, after determining the semantic description of each non-textual key information, for each sample file, the processing device can generate corresponding sample parsing data based on the text information, semantic description, and attribute information.

[0054] Optionally, as one implementation, to characterize the position and structure of text information in a composite file, the attribute information for text information may include the structural type information (representing the specific structural type) to which the text information belongs in the composite file and the vertical coordinates of the text recognition region where the text information is located in the composite file. Similarly, to characterize the position and structure of visual information in a composite file, the attribute information for visual information may include the structural type information to which the visual information belongs in the composite file and the vertical coordinates of the text recognition region where the visual information is located in the composite file.

[0055] Optionally, as one implementation, to represent the position of text information in a video file, the attribute information may include the extraction timestamp of the keyframe to which the text information belongs in the video file and the vertical coordinates of the text recognition region where the text information is located in that keyframe. To represent the position of auditory information in a video file, the attribute information may include the extraction timestamp of the keyframe to which the auditory information belongs in the video file, and / or, the attribute information may include video segment information corresponding to the keyframe to which the auditory information belongs. And, to represent the position of visual information in a video file, the attribute information may include the extraction timestamp of the keyframe to which the visual information belongs in the video file and the vertical coordinates of the visual information.

[0056] It should be noted that the attribute information representation methods given above are merely illustrative. In actual applications, attribute information can also be represented in other ways by relevant personnel to represent the position and / or structure of the non-textual key information corresponding to the text information or semantic description in the sample file.

[0057] Optionally, in step S240, as one implementation, the processing device can organize the text information and semantic description according to the attribute information to generate corresponding sample parsing data in a preset format. The preset format can be pre-set by relevant personnel, and this application does not impose any restrictions on it.

[0058] Figure 8 This is a schematic diagram of sample parsing data according to an embodiment of the present invention. It is intended to illustrate that... Figure 8 The sample parsing data shown can be based on Figure 4 The parsed data generated from the composite file 41 shown. For example... Figure 8As shown, the processing device can organize and process the text information and semantic descriptions in the composite file 41 according to the order in which the text information and non-text key information appear, thereby determining the sample parsing data 81. The sample parsing data 81 can contain four segments. The first segment can represent the location and specific content of the text information "ABC team vs. EFG team" in the composite file 41. The second segment can represent the location and semantic description of the image 414 in the composite file 41. The third segment can represent the location and specific content of the text information "Persist to the end" in the composite file 41. The fourth segment can represent the location and specific content of the text information "ABC team forward XXX, in a match held on Sunday night at XXX, fiercely battled with EFG team player XXX for a puck scattered on the ice. For details, please visit XXX.COM." in the composite file 41. It should be noted that the composite file and sample parsing data given in this embodiment are merely illustrative; in actual applications, the content included in the composite file and sample parsing data is not limited to this.

[0059] Step S300: Supervised fine-tuning training of the multimodal model is performed based on the multiple fine-tuning samples.

[0060] Specifically, after acquiring multiple fine-tuning samples, the processing device can use these samples to perform supervised fine-tuning training on the multimodal model. It is important to note that supervised fine-tuning can be a method for fine-tuning a pre-trained model. Supervised fine-tuning aims to use specific sample datasets to train the pre-trained model more precisely, enabling it to have stronger task adaptability and thus exhibit higher accuracy and controllability when facing specific language processing tasks. In this embodiment of the invention, by performing supervised fine-tuning training on the multimodal model, the multimodal model can possess semantic completion capabilities for non-textual elements, context-consistent chart / illustration description generation capabilities, keyframe extraction capabilities, and structured summarization capabilities when facing morphological document parsing tasks.

[0061] It should be understood that in step S300, the model training method used by the data processing device can be Supervised Fine-Tuning (SFT). Supervised Fine-Tuning is a deep learning technique that can be used to further optimize model performance based on a pre-trained model using a dataset for a specific task. In this embodiment of the invention, Supervised Fine-Tuning can use multiple fine-tuning samples as a dataset to further optimize the performance of a large language model.

[0062] Step S400: Obtain the file parsing model based on the supervised fine-tuned multimodal model.

[0063] Specifically, after supervised fine-tuning training of the multimodal model, the processing device can obtain a file parsing model based on the supervised fine-tuned multimodal model.

[0064] Optionally, in step S400, as one implementation, after supervised fine-tuning training of the multimodal model, the processing device can further perform preference alignment training on the supervised fine-tuned multimodal model to obtain the final file parsing model. Preference alignment is a technique or method aimed at aligning the output of an artificial intelligence model with human preferences. Its core objective is to improve the model's generated content by training and optimization to better meet human expectations and needs, thereby enhancing the model's practical application and user experience. In this embodiment of the invention, by performing preference alignment training on the multimodal model, the processing device can optimize instruction following and reduce illusions and inconsistent outputs, thereby improving the usability and traceability of the model-generated content.

[0065] Figure 9 This is a flowchart of a preference alignment training method according to an embodiment of the present invention. It is intended to illustrate that by executing... Figure 9 The preference alignment training method shown allows the processing device to perform preference alignment training on a multimodal model, i.e., to implement step S400 above. Figure 9 As shown, the preference alignment training method may specifically include the following steps: Step S410: Obtain preference sample pairs.

[0066] Specifically, the processing device can acquire preference sample pairs. Here, preference samples can be understood as samples used for preference alignment training of the multimodal model.

[0067] Optionally, as a configuration method, to optimize instruction following and reduce phantom and inconsistent output, the preference sample can be set to include a first preference sample pair, and / or, the preference sample pair can be set to include a second preference sample pair. The first preference sample pair can include first positive parsing data and corresponding first negative parsing data. The first positive parsing data can refer to parsing data that meets the parsing requirements (i.e., the parsing content is consistent with the parsing requirements), and the first negative parsing data can refer to parsing data that does not meet the parsing requirements (i.e., the parsing content deviates from the parsing requirements). The second preference sample pair can include second positive parsing data and corresponding second negative parsing data. The second positive parsing data can refer to parsing data that does not contain phantom data (i.e., all included parsing content can be found in the given file), and the second negative parsing data can refer to parsing data that contains phantom data (i.e., some parsing content cannot be found in the given file).

[0068] Optionally, both the first and second preference sample pairs can have corresponding sample files and sample file parsing prompts. The sample file parsing prompts can be textual descriptions used as training samples for the model, guiding the multimodal model to parse the sample files and output the parsing results in the desired format. Furthermore, as an implementation method, the negative parsing data in the preference sample pairs (i.e., including the first and second negative parsing data) can be generated by the processing device invoking the multimodal training to parse the sample files based on the file parsing prompts. The positive parsing data in the preference sample pairs (i.e., including the first and second positive parsing data) can be determined by relevant personnel through manual correction of the corresponding negative parsing data.

[0069] Step S420: Perform preference alignment training on the supervised fine-tuned multimodal model based on the preference samples to obtain the file parsing model.

[0070] Specifically, after obtaining the preference samples, this embodiment can perform preference alignment training on the supervised fine-tuned multimodal model based on the preference samples. The document parsing model is the multimodal model that has undergone preference alignment training.

[0071] Optionally, in step S420, the model training method used in this embodiment can be Direct Preference Optimization (DPO). DPO is a model training method for aligning model preferences, aiming to make the model output more consistent with human preferences. In this embodiment, DPO uses positive parsing data from multiple preference samples as parsing data preferred by humans, and negative parsing data as parsing data not preferred by humans, thereby training a multimodal model. The trained multimodal model will be more inclined to generate parsing data preferred by humans, that is, generating parsing data consistent with the demand instructions and free from illusionary data. Therefore, by training the multimodal model with preference alignment, this embodiment can optimize instruction following and reduce illusionary and inconsistent outputs.

[0072] Optionally, the processing device can also perform knowledge-enhanced training on the multimodal model before performing supervised fine-tuning training on the multimodal model based on multiple fine-tuning samples.

[0073] Figure 10 This is a flowchart illustrating a knowledge enhancement training method according to an embodiment of the present invention. It is intended to explain that by executing... Figure 10 The knowledge-enhanced training method shown in this embodiment can be used to perform knowledge-enhanced training on multimodal models. For example... Figure 10 As shown, the knowledge enhancement training method may specifically include the following steps: Step S110: Obtain multiple image knowledge question-and-answer pairs.

[0074] Specifically, to achieve knowledge-enhanced training, the processing device can first acquire multiple image-based question-and-answer pairs. These image-based question-and-answer pairs can include sample images and the corresponding question-and-answer pairs.

[0075] It should be noted that, in the embodiments of this invention, the content involved in the image knowledge question-answering pairs can be set to cover multiple fields such as celebrities and scenic spots, IP images, and animals and plants, and this application does not impose any limitations on this. Furthermore, as one implementation method, image knowledge question-answering can be obtained by relevant personnel manually annotating sample images from different fields, or image knowledge question-answering can also be generated based on sample images from different fields using a corresponding question-answering pair generation model, and this application does not impose any limitations on this.

[0076] Step S120: Perform knowledge enhancement training on the multimodal model based on the multiple image knowledge question-answering pairs.

[0077] Specifically, after acquiring image knowledge question-answer pairs, this embodiment can perform knowledge-enhanced training on the multimodal model based on multiple image knowledge question-answer pairs. Therefore, by performing knowledge-enhanced training on the multimodal model, this embodiment can enable the multimodal model to acquire stronger entity recognition and background knowledge representation capabilities.

[0078] Optionally, after obtaining the file parsing model based on the supervised fine-tuning of the multimodal model, the processing device or related devices can provide file parsing services to the outside world based on the file parsing model.

[0079] Figure 11 This is a flowchart illustrating a method for providing a file parsing service according to an embodiment of the present invention. It is intended to illustrate that by executing... Figure 11 The file parsing service method shown in this embodiment can provide relevant file parsing services based on the target parsing data. For example... Figure 11 As shown, the method for providing the file parsing service may specifically include the following steps: Step S510: Receive the file to be parsed.

[0080] Specifically, the processing device can receive a file to be parsed. This file can be any document to be parsed. It should be noted that, similar to the sample file, the file to be parsed can include files of various formats. For example, it can be a composite file such as a PDF (Portable Document Format), a scanned copy, or a report. Another example is video files such as recorded courses, online open courses, and corporate training materials. This application does not limit the format of the file to be parsed. Furthermore, when the file to be parsed is a composite file, it can exist as a file in its original form, or as an image or other form; this application does not impose any restrictions on this.

[0081] Step S520: Using the file to be parsed as input, obtain the target parsing data corresponding to the file to be parsed based on the file parsing model.

[0082] Specifically, the processing device can take the file to be parsed as input and obtain the target parsing data corresponding to the file to be parsed based on the file parsing model.

[0083] Optionally, in step S520, as one implementation, the processing device can input the file to be parsed and the corresponding parsing prompt words into the file parsing model to obtain the target parsing data corresponding to the file to be parsed based on the file parsing model. The target file parsing prompt words can be a text description used to guide the file parsing model to generate the parsing content required by the user and output the parsing content in the form required by the user.

[0084] As an illustration, file parsing prompts could be: "Please describe this video in detail," "Please identify all text (text, code, tables, formulas, etc.) appearing in the video in chronological order and output them in JSON format," "Describe the video content and explain its camera movement characteristics," and "Please perform in-depth analysis of the input course video and output a professional and detailed course description." It should be noted that the natural language requirement statements given above are merely illustrative; in actual application, this application does not impose any restrictions on the specific content included in the natural language requirement statements.

[0085] This invention, in its embodiments, acquires multiple fine-tuning samples after obtaining an initial multimodal model. Supervised fine-tuning training is then performed on the multimodal model based on these samples, and a file parsing model is obtained from the supervised fine-tuned multimodal model. The fine-tuning samples include sample files and corresponding sample parsing data. The sample parsing data includes textual information and semantic descriptions of non-textual key information. The textual information is the extracted textual modality information from the sample file, and the non-textual key information is the extracted non-textual modality information from the sample file. The textual information and semantic descriptions have corresponding attribute information, which characterizes the position and / or structure of the non-textual key information corresponding to the textual information or semantic description within the sample file. Therefore, this invention enables the trained file parsing model to balance fine-grained parsing capabilities and semantic description capabilities, thereby meeting the file parsing requirements of artificial intelligence applications.

[0086] Figure 12 This is a schematic diagram illustrating the model training process according to an embodiment of the present invention. Figure 12 As shown, to train the text parsing model, the processing device can first perform knowledge-enhanced training on the initial multimodal model 123 using multiple image knowledge question-answering pairs 122. The image knowledge question-answering pairs can be configured to include sample images and the corresponding knowledge question-answering pairs. Therefore, by performing knowledge-enhanced training on the multimodal model, this embodiment enables the multimodal model to acquire stronger entity recognition and background knowledge representation capabilities.

[0087] Furthermore, after using multiple image knowledge question answering pairs 122 to perform knowledge-enhanced training on the initial multimodal model 123, the processing device can use multiple fine-tuning samples 124 to perform supervised fine-tuning training on the knowledge-enhanced multimodal model 125. The fine-tuning samples can be configured to include sample files and corresponding sample parsing data. The sample parsing data includes semantic descriptions of textual information and non-textual key information. The textual information is the textual modality extraction information of the sample file, and the non-textual key information is the non-textual modality extraction information of the sample file. The textual information and semantic descriptions have corresponding attribute information, which is used to characterize the position and / or structure of the non-textual key information corresponding to the textual information or semantic description in the sample file. Therefore, by performing supervised fine-tuning training on the multimodal model, this embodiment of the invention enables the multimodal model to possess the ability to semantically complete non-textual elements, generate context-consistent chart / illustration descriptions, extract keyframes, and provide structured summaries when facing morphological file parsing tasks.

[0088] Furthermore, after supervised fine-tuning training of the knowledge-enhanced multimodal model 125 using multiple fine-tuning samples 124, the processing device can perform preference alignment training on the supervised fine-tuned multimodal model 127 using multiple preference sample pairs 126 to obtain the final file parsing model 121. The preference samples can be configured to include a first preference sample pair, and / or, the preference sample pair can be configured to include a second preference sample pair. The first preference sample pair can include first positive parsing data and corresponding first negative parsing data. The first positive parsing data can refer to parsing data that meets the parsing requirements, and the first negative parsing data can refer to parsing data that does not meet the parsing requirements. The second preference sample pair can include second positive parsing data and corresponding second negative parsing data. The second positive parsing data can refer to parsing data without hallucinatory data, and the second negative parsing data can refer to parsing data with hallucinatory data. Therefore, by performing preference alignment training on the multimodal model, this embodiment of the invention can optimize instruction following and reduce hallucination and inconsistent output.

[0089] Figure 13 This is a schematic diagram of a file parsing model training device according to an embodiment of the present invention. Figure 13 As shown, the file parsing model training device of this embodiment includes an initial model acquisition unit 131, a sample acquisition unit 132, a training unit 133, and a parsing model acquisition unit 134.

[0090] Specifically, the initial model acquisition unit 131 is used to acquire an initial multimodal model.

[0091] The sample acquisition unit 132 is used to acquire multiple fine-tuning samples, wherein the fine-tuning samples include sample files and corresponding sample parsing data. The sample parsing data includes semantic descriptions of text information and non-text key information. The text information is the text modality extraction information of the sample file, and the non-text key information is the non-text modality extraction information of the sample file. The text information and the semantic description have corresponding attribute information. The attribute information is used to characterize the position and / or structure of the non-text key information corresponding to the text information or the semantic description in the sample file.

[0092] The training unit 133 is used to perform supervised fine-tuning training of the multimodal model based on the plurality of fine-tuning samples.

[0093] The parsing model acquisition unit 133 is used to acquire a file parsing model based on the supervised fine-tuned multimodal model.

[0094] This invention, in its embodiments, acquires multiple fine-tuning samples after obtaining an initial multimodal model. Supervised fine-tuning training is then performed on the multimodal model based on these samples, and a file parsing model is obtained from the supervised fine-tuned multimodal model. The fine-tuning samples include sample files and corresponding sample parsing data. The sample parsing data includes textual information and semantic descriptions of non-textual key information. The textual information is the extracted textual modality information from the sample file, and the non-textual key information is the extracted non-textual modality information from the sample file. The textual information and semantic descriptions have corresponding attribute information, which characterizes the position and / or structure of the non-textual key information corresponding to the textual information or semantic description within the sample file. Therefore, this invention enables the trained file parsing model to balance fine-grained parsing capabilities and semantic description capabilities, thereby meeting the file parsing requirements of artificial intelligence applications.

[0095] Figure 14 This is a schematic diagram of an electronic device according to an embodiment of the present invention. It is intended to be noted that this electronic device can be the data processing device described in the above embodiments. For example... Figure 14 As shown, the electronic device 14 includes at least one processor 141; a memory 142 communicatively connected to at least one processor 141; and a communication component 143 communicatively connected to a scanning device, wherein the communication component 143 receives and transmits data under the control of the processor 141; wherein the memory 142 stores instructions executable by at least one processor 141, the instructions being executed by at least one processor 141 to implement the above-described file parsing model training method.

[0096] Specifically, the electronic device includes: one or more processors 141 and a memory 142. Figure 14 Taking a processor 141 as an example, the processor 141 and the memory 142 can be connected via a bus or other means. Figure 14 Taking a bus connection as an example, memory 142, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Processor 141 executes various functional applications and data processing of the device by running the non-volatile software programs, instructions, and modules stored in memory 142, thereby implementing the above-mentioned file parsing model training method.

[0097] Memory 142 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store an option list, etc. Furthermore, memory 142 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 142 may optionally include memory remotely located relative to processor 141, and these remote memories may be connected to external devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0098] One or more modules are stored in memory 142 and, when executed by one or more processors 141, execute the file parsing model training method in any of the above method embodiments.

[0099] The above-mentioned products can perform the methods provided in the embodiments of this application, and have the corresponding functional modules and beneficial effects of performing the methods. For technical details not described in detail in this embodiment, please refer to the methods provided in the embodiments of this application.

[0100] This invention, in its embodiments, acquires multiple fine-tuning samples after obtaining an initial multimodal model. Supervised fine-tuning training is then performed on the multimodal model based on these samples, and a file parsing model is obtained from the supervised fine-tuned multimodal model. The fine-tuning samples include sample files and corresponding sample parsing data. The sample parsing data includes textual information and semantic descriptions of non-textual key information. The textual information is the extracted textual modality information from the sample file, and the non-textual key information is the extracted non-textual modality information from the sample file. The textual information and semantic descriptions have corresponding attribute information, which characterizes the position and / or structure of the non-textual key information corresponding to the textual information or semantic description within the sample file. Therefore, this invention enables the trained file parsing model to balance fine-grained parsing capabilities and semantic description capabilities, thereby meeting the file parsing requirements of artificial intelligence applications.

[0101] Another embodiment of the present invention relates to a non-volatile storage medium for storing a computer-readable program for use by a computer to execute some or all of the above-described method embodiments.

[0102] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0103] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for training a file parsing model, characterized in that, The method includes: Obtain the initial multimodal model; Multiple fine-tuning samples are obtained, wherein the fine-tuning samples include sample files and corresponding sample parsing data. The sample parsing data includes semantic descriptions of text information and non-text key information. The text information is the text modality extraction information of the sample file, and the non-text key information is the non-text modality extraction information of the sample file. The text information and the semantic description have corresponding attribute information. The attribute information is used to characterize the position and / or structure of the non-text key information corresponding to the text information or the semantic description in the sample file. The multimodal model is trained under supervised fine-tuning based on the multiple fine-tuning samples; The file parsing model is obtained from the supervised fine-tuned multimodal model.

2. The method according to claim 1, characterized in that, The acquisition of multiple fine-tuning samples includes: Obtain multiple of the aforementioned sample files; For each of the sample files, information is extracted from the sample files to obtain the text information and the non-text key information; Determine the semantic description of each non-text key information based on the text information and the non-text key information; For each of the sample files, corresponding sample parsing data is generated based on the text information, the semantic description, and the attribute information.

3. The method according to claim 2, characterized in that, The sample file includes a composite file, and the non-textual key information includes visual information: The step of extracting information from the sample file to obtain the text information and the non-text key information includes: The text information is extracted from the composite file to obtain the text information; Visual information is extracted from the composite file to obtain the visual information.

4. The method according to claim 2, characterized in that, The sample files include video files, and the non-textual key information includes visual and auditory information; The step of extracting information from the sample file to obtain the text information and the non-text key information includes: Keyframes are extracted from the video file to obtain at least one keyframe; Text information is extracted from each of the keyframes to obtain the text information; Visual information is extracted from each of the keyframes to obtain the visual information; Speech information is extracted from the video segments corresponding to each keyframe to obtain the auditory information.

5. The method according to claim 3 or 4, characterized in that, The visual information includes illustrations and / or charts.

6. The method according to claim 2, characterized in that, The step of determining the semantic description of each non-text key information based on the text information and the non-text key information includes: For each of the aforementioned non-textual key information, a semantic description of the non-textual key information is generated by calling a large language model based on the textual information and the non-textual key information.

7. The method according to claim 1, characterized in that, The process of obtaining the file parsing model based on the supervised fine-tuned multimodal model includes: Obtain preference sample pairs; The supervised fine-tuned multimodal model is trained with preference alignment based on the preference samples to obtain the file parsing model.

8. The method according to claim 7, characterized in that, The preference sample pair includes a first preference sample pair, and / or the preference sample pair includes a second preference sample pair, wherein the first preference sample pair includes first positive parsing data and corresponding first negative parsing data, the first positive parsing data being parsing data that meets the parsing requirements, and the first negative parsing data being parsing data that does not meet the parsing requirements; the second preference sample pair includes second positive parsing data and corresponding second negative parsing data, the second positive parsing data being parsing data that does not contain hallucination data, and the second negative parsing data being parsing data that contains hallucination data.

9. The method according to claim 1, characterized in that, Before performing supervised fine-tuning training of the multimodal model based on the plurality of fine-tuning samples, the method further includes: Multiple image knowledge question-answer pairs are obtained, wherein the image knowledge question-answer pairs include sample images and knowledge question-answer pairs corresponding to the sample images; The multimodal model is trained with knowledge enhancement based on the multiple image knowledge question-answering pairs.

10. The method according to claim 1, characterized in that, After obtaining the file parsing model based on the supervised fine-tuned multimodal model, the method further includes: Receive the file to be parsed; Using the file to be parsed as input, target parsing data corresponding to the file to be parsed is obtained based on the file parsing model.

11. A file parsing model training device, characterized in that, The device includes: The initial model acquisition unit is used to acquire the initial multimodal model; A sample acquisition unit is used to acquire multiple fine-tuning samples, wherein the fine-tuning samples include sample files and corresponding sample parsing data. The sample parsing data includes semantic descriptions of text information and non-text key information. The text information is the text modality extraction information of the sample file, and the non-text key information is the non-text modality extraction information of the sample file. The text information and the semantic description have corresponding attribute information, and the attribute information is used to characterize the position and / or structure of the non-text key information corresponding to the text information or the semantic description in the sample file. The training unit is used to perform supervised fine-tuning training of the multimodal model based on the plurality of fine-tuning samples; The parsing model acquisition unit is used to acquire the file parsing model based on the supervised fine-tuned multimodal model.

12. A computer-readable storage medium storing computer program instructions thereon, characterized in that, The computer program instructions, when executed by a processor, implement the method as described in any one of claims 1-10.

13. An electronic device, characterized in that, The device includes: Memory is used to store one or more computer program instructions; A processor, wherein the one or more computer program instructions are executed by the processor to implement the method as described in any one of claims 1-10.

14. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1-10.

Images