Model training method and system, data processing method and system and storage medium

Abstract

The invention discloses a model training method and system, a data processing method and system and a storage medium, and relates to the field of large model technologies and data processing. The method comprises the following steps: pre-training an initial language model by adopting upstream task data to obtain a pre-trained model; performing model fine tuning on the pre-training model based on the target prompt and the target training text to obtain a target language model; wherein the target prompt is used for guiding the pre-training model to execute an information extraction task according to a structured format output by the predefined model, the target training text is obtained by converting the multi-modal training data, and the target language model is used for performing information extraction on the multi-modal application data to obtain a target extraction result. According to the method and the device, the technical problems of low information extraction efficiency and poor accuracy when the document extraction task is processed in related technologies are solved.

Description

Technical Field

[0001] The present application relates to large model technology and data processing fields, and specifically, to a model training method, a data processing method, a system and a storage medium. Background Art

[0002] Document information extraction technology is the key to mining the value of text data. With the rise of deep learning in recent years, especially the widespread application of pre-trained models, its importance has increased day by day. Pre-trained models have significantly improved the efficiency and accuracy of natural language processing tasks through self-training learning of large-scale texts, and have become a core technology in the field of information extraction. However, despite the obvious advantages of pre-trained models, their huge number of parameters and computing requirements have become a bottleneck for large-scale applications. When dealing with high-throughput document information extraction tasks, related technologies often face the challenges of high resource consumption and low information extraction efficiency. In addition, since pre-trained models lack an in-depth understanding of entities, relationships, and event patterns in specific fields, especially when dealing with multimodal documents, they need to process multiple data formats at the same time, which increases the complexity of information extraction, resulting in low information extraction efficiency and reduced accuracy of extraction results.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a model training method, a data processing method, a system and a storage medium to at least solve the technical problems of low information extraction efficiency and poor accuracy in related technologies when processing document extraction tasks.

[0005] According to one aspect of an embodiment of the present application, a model training method is provided, comprising: pre-training an initial language model using upstream task data to obtain a pre-trained model; fine-tuning the pre-trained model based on a target prompt and a target training text to obtain a target language model; wherein the target prompt is used to guide the pre-trained model to perform an information extraction task in accordance with a structured format output by a predefined model, the target training text is converted from multimodal training data, and the target language model is used to extract information from multimodal application data to obtain a target extraction result.

[0006] According to another aspect of an embodiment of the present application, a data processing method is also provided, including: obtaining multimodal application data; using a target language model to extract information from the multimodal application data to obtain a target extraction result; wherein the target language model is generated according to the model training method of any one of the embodiments of the present application.

[0007] According to another aspect of the embodiments of the present application, a data processing method is also provided, including: obtaining multimodal legal consultation data; using a target language model to extract information from the multimodal legal consultation data to obtain legal consultation answer results; wherein the target language model is generated according to the model training method of any one of the embodiments of the present application.

[0008] According to another aspect of an embodiment of the present application, a data processing method is also provided, including: obtaining a data processing request through a first application programming interface, wherein the request data carried in the data processing request includes: multimodal application data; returning a data processing response through a second application programming interface, wherein the response data carried in the data processing response includes: a target extraction result, the target extraction result is obtained by extracting information from the multimodal application data using a target language model, and the target language model is generated according to the model training method of any one of the embodiments of the present application.

[0009] According to another aspect of an embodiment of the present application, a data processing method is also provided, including: obtaining a currently input data processing dialogue request, wherein the request data carried in the data processing dialogue request includes: multimodal application data; in response to the data processing dialogue request, returning a data processing dialogue reply, wherein the information carried in the data processing dialogue reply includes: a target extraction result, the target extraction result is obtained by extracting information from the multimodal application data using a target language model, and the target language model is generated according to the model training method of any one of the embodiments of the present application; and displaying the target extraction result in a graphical user interface.

[0010] According to another aspect of an embodiment of the present application, a data processing method is also provided, including: in response to an input instruction acting on an operation interface, displaying multimodal application data on the operation interface; in response to a processing instruction acting on the operation interface, displaying a target extraction result on the operation interface; wherein the target extraction result is obtained by extracting information from the multimodal application data using a target language model, and the target language model is generated according to the model training method of any one of the embodiments of the present application.

[0011] According to another aspect of an embodiment of the present application, a data processing system is also provided, including: a client, used to send multimodal application data; a server, connected to the client, used to extract information from the multimodal application data using a target language model to obtain a target extraction result; the client is also used to output the target extraction result; wherein the target language model is generated according to the model training method of any one of the embodiments of the present application.

[0012] According to another aspect of the embodiments of the present application, a computing device is further provided, including: a memory storing an executable program; and a processor for running the program, wherein the method in each embodiment of the present application is executed when the program is running.

[0013] According to another aspect of the embodiments of the present application, an electronic device is also provided, including: a memory storing an executable program; a processor connected to the memory via a bus and used to run the program, wherein the method of each embodiment of the present application is executed when the program is running.

[0014] According to another aspect of an embodiment of the present application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the methods in each embodiment of the present application.

[0015] According to another aspect of the embodiments of the present application, a computer program product is also provided, including a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present application is implemented.

[0016] According to another aspect of an embodiment of the present application, a computer program product is also provided, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present application is implemented.

[0017] According to another aspect of the embodiments of the present application, a computer program is also provided, and when the computer program is executed by a processor, the methods in the various embodiments of the present application are implemented.

[0018] In an embodiment of the present application, the initial language model is pre-trained by using upstream task data to obtain a pre-trained model, and then the pre-trained model is fine-tuned based on the target prompt and the target training text to obtain a target language model. The target language model is used to extract information from multimodal application data to obtain a target extraction result, thereby utilizing the language understanding ability and multimodal data processing ability of the pre-trained model, combined with the target prompt to guide the pre-trained model to perform the information extraction task in accordance with the structured format output by the predefined model, so that the key information in the multimodal application data can be quickly located and extracted, and the speed of information extraction can be effectively improved. In addition, the target prompt can guide the pre-trained model to output information in a predefined structured format, which not only helps to reduce the processing complexity caused by unstructured output, but also through the learning of the target training text in the fine-tuning stage, the target language model can more accurately understand the content of the multimodal training data, improve the accuracy of the extracted information, reduce errors and omissions, and thus solve the technical problems of low information extraction efficiency and poor accuracy in the related technology when processing document extraction tasks.

[0019] It is easy to notice that the above general description and the following detailed description are only for the purpose of exemplifying and explaining the present application, and do not constitute a limitation of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0021] Figure 1 is a schematic diagram of an application scenario of a data processing method according to an embodiment of the present application;

[0022] Figure 2 is a flow chart of a model training method according to an embodiment of the present application;

[0023] Figure 3 is a schematic diagram of a model training method according to an embodiment of the present application;

[0024] Figure 4 is a flow chart of a data processing method according to an embodiment of the present application;

[0025] Figure 5 is a flow chart of another data processing method according to an embodiment of the present application;

[0026] Figure 6 is a flow chart of another data processing method according to an embodiment of the present application;

[0027] Figure 7 is a flow chart of another data processing method according to an embodiment of the present application;

[0028] Figure 8 is a flow chart of another data processing method according to an embodiment of the present application;

[0029] Fig. 9 is a structural block diagram of a model training device according to an embodiment of the present application;

[0030] Fig.10 is a structural block diagram of a data processing device according to an embodiment of the present application;

[0031] Fig.11 is a structural block diagram of another data processing device according to an embodiment of the present application;

[0032] Fig.12 is a structural block diagram of another data processing device according to an embodiment of the present application;

[0033] Fig.13is a structural block diagram of another data processing device according to an embodiment of the present application;

[0034] Fig.14 is a structural block diagram of another data processing device according to an embodiment of the present application;

[0035] Fig.15 is a structural block diagram of a computing device according to an embodiment of the present application;

[0036] Fig.16 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0038] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0039] The technical solution provided in this application is mainly implemented using large-scale model technology. The large model here refers to a deep learning model with large-scale model parameters, which can usually contain hundreds of millions, tens of billions, hundreds of billions, trillions or even more than 10 trillion model parameters. The large model can also be called a foundation model / foundation model. The large model is pre-trained with large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as large-scale language model (LLM), multi-modal pre-training model, etc.

[0040] It should be noted that when the large model is actually applied, the pre-trained model can be fine-tuned through a small number of samples so that the large model can be applied to different tasks. For example, the large model can be widely used in natural language processing (NLP), computer vision, speech processing and other fields, and can be specifically applied to computer vision tasks such as visual question answering (VQA), image caption (IC), image generation, etc. It can also be widely used in natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. Therefore, the main application scenarios of the large model include but are not limited to digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc. In the embodiment of the present application, data processing through the target language model in the information extraction scenario is used as an example for explanation.

[0041] According to an embodiment of the present application, a data processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0042] Considering the huge number of model parameters of the large model and the limited computing resources of the mobile terminal, the above method provided in the embodiment of the present application can be applied to Figure 1 The application scenarios shown are not limited to these. Figure 1 In the application scenario shown, the large model is deployed in the server 10, and the server 10 can be connected to one or more client devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client device 20 here may include but is not limited to: a smart phone, a tablet computer, a laptop computer, a PDA, a personal computer, a smart home device, a vehicle-mounted device, etc. The client device 20 can interact with the user through a graphical user interface to implement the call of the large model, thereby implementing the method provided in the embodiment of the present application.

[0043] In an embodiment of the present application, a system composed of a client device and a server can perform the following steps: the client device is connected to the server, the client device sends multimodal application data to the server, the server uses a target language model to extract information from the multimodal application data to obtain a target extraction result, and the target language model is generated according to any model training method in the embodiment of the present application. The client device can also display the target extraction result in a graphical user interface.

[0044] It should be noted that, with the rapid development of high-performance computing units, in other application scenarios, the above method provided in the embodiment of the present application can also be applied to the model all-in-one machine. In an optional embodiment, a plurality of models are built into the model all-in-one machine, and the user can choose to adjust a model as needed to obtain the user's own model, so that the high-performance computing unit built into the model all-in-one machine can directly call the adjusted model to execute the above method provided in the embodiment of the present application. In another optional embodiment, a trained model is built into the large model all-in-one machine, so that the high-performance computing unit built into the model all-in-one machine can directly call the model to execute the above method provided in the embodiment of the present application.

[0045] Furthermore, when users need to train their own models, they can also upload their own data sets through the client, which are sent by the client to the server, so that the server can adjust the pre-trained model with the data set to obtain the user's own model and then deploy it to the production environment. In order to facilitate users' needs for model adjustment, the server can provide complete adjustment tools, development frameworks and processes, and can support multiple adjustment strategies, so that the adjusted model can better adapt to applications in different fields and achieve a high degree of customization.

[0046] Under the above operating environment, this application provides Figure 2 The model training method shown. Figure 2 is a flow chart of a model training method according to an embodiment of the present application. Figure 2 As shown, the method may include the following steps:

[0047] Step S21, pre-training the initial language model using upstream task data to obtain a pre-trained model;

[0048] Step S22, fine-tuning the pre-trained model based on the target prompt and the target training text to obtain a target language model; wherein the target prompt is used to guide the pre-trained model to perform information extraction tasks according to the structured format of the predefined model output, the target training text is converted from the multimodal training data, and the target language model is used to extract information from the multimodal application data to obtain the target extraction result.

[0049] The above-mentioned upstream task data can be a large amount of unstructured or semi-structured data for pre-training the initial language model, aiming to enable the initial language model to learn the general representation and structure of the language. There are various ways to obtain upstream task data, depending on the type, field and specific requirements of the required data. Specific acquisition methods include but are not limited to: Internet text, book documents, preset corpus and internal database. Upstream task data can come from different application scenarios, in which information extraction tasks need to be performed. Information extraction is a technology that automatically extracts structured information from unstructured or semi-structured text, aiming to convert key information in the text into a data format that can be understood and processed by computers. Information extraction technology can be applied in multiple fields, by automatically extracting specific types of information from unstructured text and converting it into structured data to support subsequent task processing, analytical decision-making and knowledge management.

[0050] For example, in the legal field, judicial documents usually contain various details of the case and the court's ruling. Judicial document structuring refers to the automatic extraction of key information from these complex documents, such as case number, party information, cause of action, litigation request, judgment result, case facts, legal basis, etc. The above key information usually exists in unstructured text, and through information extraction, it can be converted into a structured data format, which is convenient for subsequent legal analysis, case retrieval and knowledge management.

[0051] For example, in the field of bidding, information extraction technology can be used to extract relevant information of successful bidders from documents such as tender notices and public notices of successful bid results, such as company name, bid amount, successful bid ranking, project manager name and qualification certificate number, thereby enabling efficient and intelligent management of the bidding process.

[0052] For example, in the analysis of traffic accidents and legal events, road driving event extraction involves identifying details of road driving events from accident reports, news reports or court documents, including the time and location of the event, information about the vehicles involved, driver information, event description, legal consequences, etc. The information extraction results have practical application value for traffic accident analysis, driving behavior assessment and accident prevention.

[0053] For example, in the legal and regulatory fields, information can be extracted from relevant reports, news or investigation documents to obtain detailed information on illegal activities, including the type of illegal incident, participants, the amount involved, the location of the incident, the time of the incident, the legal consequences, etc. The above information extraction results are very important for monitoring illegal activities and enforcing the law.

[0054] The above initial language model is the starting point of pre-training. It can usually be a randomly initialized model or a basic model trained based on some simpler tasks. The model parameters of the initial language model have not been trained on any specific task before pre-training. The initial language model can automatically learn the statistical characteristics of language from large-scale unlabeled texts, such as the frequency of vocabulary use and the association between words. The above characteristics constitute the basis for the initial language model to understand language.

[0055] The above pre-trained model is a model optimized through the pre-training process. The pre-trained model can learn rich language representation and understanding capabilities through unsupervised training on a large amount of text data. The pre-trained model can show excellent performance in a variety of natural language understanding tasks. Since the pre-trained model has learned the common features of the language during the pre-training stage, these features can be transferred to downstream tasks such as entity recognition, sentiment analysis, text classification, machine translation, etc., without having to start training from scratch.

[0056] For example, in the continuous pre-training (CPT) process, the upstream task data can be pre-processed first, such as cleaning and formatting the collected text data, such as removing hypertext markup language tags, word segmentation, removing stop words, converting to model input format, etc. Further, it is necessary to design pre-training tasks, which can usually be unsupervised types, including but not limited to: Masked Language Model (MLM) tasks, Next Sentence Prediction (NSP) tasks, and Causal Language Model (CLM) tasks. Specifically, in the MLM task, a part of the words in the input text needs to be randomly masked, and the initial language model needs to predict the masked words; in the NSP task, given two sentences, the initial language model needs to determine whether the second sentence is the next sentence of the first sentence; in the CLM task, sequence-based prediction can predict the next word in the text. Using the parameters of the initial language model, the upstream task data is input for training. The initial language model is repeatedly adjusted through pre-training tasks to learn parameters that can effectively predict or generate text, and finally a pre-trained model is obtained.

[0057] After obtaining the pre-trained model, the pre-trained model is fine-tuned based on the target prompt and the target training text to obtain the target language model. The above target prompt is a special form of input used to guide the pre-trained model to output information in a specific structure or format. In information extraction tasks, the target prompt can contain instructions about the type of information and format requirements that the pre-trained model should recognize and extract, such as requiring the pre-trained model to output entity information, relationship or event details in a lightweight data exchange format (JSON). The design of the target prompt is crucial to guiding the model to generate structured output, and usually needs to match the structured format of the predefined model output.

[0058] The target training text can be a data set for fine-tuning the pre-trained model. The target training text contains annotated data in a specific task field, such as entity extraction, relationship extraction, event extraction, etc. The target training text is carefully designed and annotated to reflect the language pattern and information structure of the target field, and is used to guide the model to learn how to accurately perform information extraction tasks. The target training text focuses more on specific application scenarios, such as case details in the judicial field, winning bid information in bidding, etc.

[0059] The target training text is converted from multimodal training data, which can be a dataset containing multiple data types, such as text, images, tables, etc. For example, in scenarios such as law and bidding, documents usually contain text as well as non-text components such as tables and images (such as seals and signatures), so multimodal data processing is a key challenge. Multimodal training data is used to train the model to understand and extract information integrated in different data types.

[0060] The target language model is a fine-tuned model that can perform downstream tasks such as information extraction more accurately and efficiently by learning specific tasks in the target training text based on pre-training. The target language model not only retains the general language understanding ability of the pre-trained model, but also enhances the mastery of specific domain knowledge and the ability to perform specific tasks through fine-tuning.

[0061] The above multimodal application data is a data set that contains multiple data types encountered in actual applications. The specific data structure and content can be similar to multimodal training data, but contain the complexity and diversity in actual scenarios. Multimodal application data is used to test and evaluate the performance of the target language model in real-world tasks, such as extracting case information from legal documents, identifying winning candidate information from bidding documents, etc.

[0062] Based on the target prompt and target training text, the pre-trained model is fine-tuned (Supervised Fine-Tuning, SFT), and the target language model obtained can better understand multimodal application data and output information extraction results in a predefined structured format, thereby improving the accuracy and efficiency of information extraction. For example, the target language model can process text and non-text information at the same time and generate structured output results, which is particularly effective when processing complex and diversely structured texts such as judgment documents, basic case information, evidence information, and information on successful bidders.

[0063] Based on the above steps S21 to S22, the initial language model is pre-trained by using the upstream task data to obtain a pre-trained model, and then the pre-trained model is fine-tuned based on the target prompt and the target training text to obtain a target language model. The target language model is used to extract information from the multimodal application data to obtain the target extraction result, thereby utilizing the language understanding ability and multimodal data processing ability of the pre-trained model, combined with the target prompt, to guide the pre-trained model to perform the information extraction task in accordance with the structured format output by the predefined model, so that the key information in the multimodal application data can be quickly located and extracted, and the speed of information extraction can be effectively improved. In addition, the target prompt can guide the pre-trained model to output information in a predefined structured format, which not only helps to reduce the processing complexity caused by unstructured output, but also through the learning of the target training text in the fine-tuning stage, the target language model can more accurately understand the content of the multimodal training data, improve the accuracy of the extracted information, reduce errors and omissions, and thus solve the technical problems of low information extraction efficiency and poor accuracy in the related technology when processing document extraction tasks.

[0064] The model training method in the embodiment of the present application is further explained below.

[0065] In an optional embodiment, the model training method in the embodiment of the present application further includes:

[0066] Optical character recognition is performed on the multimodal training data to obtain a recognition result; multi-line text detection is performed on the recognition result to obtain a detection result; intelligent document processing is performed on the detection result to obtain a target training text, wherein the target training text is written in a preset level markup language format.

[0067] In the process of Optical Character Recognition (OCR) on multimodal training data, the recognition goal is to convert the text in the image into an editable and searchable text format. The OCR application or algorithm can analyze the input image, which can be a scan or photo of a document, and identify the text area in it. Then, a machine learning algorithm can be used to convert the characters in the image into text. The above conversion process can involve steps such as character segmentation, feature extraction, and character recognition. The output recognition result is that the text in the image is displayed in a plain text format, which provides a basis for subsequent processing.

[0068] Based on the text recognized by OCR, the layout and structure of the text are further detected and understood, and the text content of different lines is identified to obtain the detection results. Specifically, multi-line text detection usually analyzes the text output by OCR, identifies the boundaries of text lines, and the correlation between text lines. Multi-line text detection is especially important for processing documents containing structures such as tables, lists, and titles, which helps to correctly group and sort the text content. In actual operation, computer vision technology or natural language processing technology can be used to identify and parse the layout information of the text, thereby further enhancing the model's ability to understand complex layout documents.

[0069] After obtaining the detection results, the detection results are subjected to intelligent document processing (IDP) to obtain the target training text. The target training text is written in a preset markup language format (Markdown), thereby converting the text extracted from the image and document into structured data for subsequent model understanding and processing. Intelligent document processing converts the detection results obtained from multi-line text detection into Markdown format based on the type and content of the document. IDP technology usually includes information extraction, table understanding, identification and structuring of key information, etc. By identifying entities, relationships, and events in the document, this information is then output in a structured format for subsequent analysis and application. During intelligent document processing, IDP can consider the context and semantics of the document to improve the accuracy and completeness of the conversion.

[0070] Based on the above optional embodiments, optical character recognition is performed on the multimodal training data to obtain a recognition result, and then multi-line text detection is performed on the recognition result to obtain a detection result, and finally intelligent document processing is performed on the detection result to obtain a target training text. In this way, the multimodal training data can be converted into a structured and standardized text format, which can better adapt to the model training requirements, thereby improving the performance and efficiency of the pre-trained model in information extraction tasks.

[0071] In an optional embodiment, the model training method in the embodiment of the present application further includes:

[0072] The target prompt is generated based on a preset structural paradigm and a preset prompt paradigm, wherein the preset structural paradigm is used to define the structured format of the model output, and the preset prompt paradigm is used to guide the pre-trained model to perform information extraction tasks.

[0073] The step of generating target prompts based on the preset structural paradigm and the preset prompt paradigm is the key to guiding the model to output structured information in the information extraction task. The above preset structure (Schema) paradigm is to define the structured format of the model output. In the information extraction task, the preset structural paradigm usually means that the information that the model needs to output should follow a certain structure, such as JSON, eXtensible Markup Language (XML) or other types of data formats, so as to facilitate further processing by downstream applications. The preset structural paradigm can specify the output fields, field types, relationships between fields, etc. in detail to ensure that the information output by the model is structured and consistent. For example, for bidding information extraction, the preset structural paradigm can include fields such as "project name", "bidding unit", "name of the candidate unit", and specify the type and output format of each field.

[0074] The above preset prompt pattern can be an instruction template used to guide the pre-trained model to perform information extraction tasks. The preset prompt pattern combines the definition of the structural pattern with specific task instructions to tell the model what the specific information extraction task is and how to output the results. The preset prompt pattern should clearly explain the task and provide enough context and examples so that the model can understand and execute it. For example, in an information extraction task, the preset prompt pattern can be: "Please extract information from the following fields from the provided text: project name, bidding unit, and output in the specified format." The preset prompt pattern can also contain one or more complete examples to help the model understand the specific requirements and data format of the task.

[0075] In practical applications, target prompts are generated based on preset structural paradigms and preset prompt paradigms. The above process usually involves specifying the requirements for structured information into prompt information to guide the model on how to process input data and in what format to output information. For example, for a task to extract bidding information, the target prompt can be: "Task Description: This is an information extraction task. The goal is to extract key information from the provided text content and output this information in JSON format. If the field content is not clear, the output of the field should be null. Content to be extracted: {content}. Extracted fields: {entity}. Request: Please extract the required information from the provided text based on the above information and one or more complete examples. Make sure the output result is in detailed JSON format."

[0076] Based on the above optional embodiments, by generating target prompts based on preset structural paradigms and preset prompting paradigms, the preset structural paradigms and format requirements can be followed when performing information extraction, thereby generating unified and structured information outputs, which is convenient for subsequent data processing and analysis. This is particularly effective when processing diverse information extraction tasks, and can ensure that the model's understanding and execution capabilities match the specific task requirements, while improving the accuracy and consistency of information extraction.

[0077] In an optional embodiment, generating a target prompt based on a preset structural paradigm and a preset prompt paradigm includes:

[0078] A preset structural paradigm is used to define the structured output format corresponding to the target extraction object in the target training text; a preset prompt paradigm is used to define the extraction field corresponding to the structured output format; and a target prompt is generated based on the structured output format and the extraction field.

[0079] The above-mentioned target extraction object is the key information in the target training text. The structured output format corresponding to the target extraction object in the target training text is defined by a preset structural paradigm, which can determine the organizational structure and data type of the model output data, including but not limited to JSON, XML, table or custom data structure. For example, for a bidding document, the structured output format may include the following fields: "Project Name", "Bidding Unit Name", "Successful Bidder Information", etc. The value of each field needs to meet specific format requirements. For example, the "Successful Bidder Information" field may be a list containing sub-fields such as "Package Name", "Unit Name", "Ranking", and "Bidding Price", and its data type is a string or a numeric value. The key to defining a structured output format is to identify all target extraction objects and design a clear and consistent representation for them so that the model can output in a unified format.

[0080] When using a preset prompt pattern to define the extraction fields corresponding to the structured output format, the preset prompt pattern can usually include but is not limited to: task description, sample text, expected field list and output format requirements. For example, for a bidding document, the preset prompt pattern can be defined as follows: "Task Description: This is an information extraction task. The goal is to extract key information from the provided text content and output it in JSON format. Content to be extracted: {content}. Please extract the following fields: project name, bidding unit name, winning candidate information. Request: Please extract the required information from the provided text based on the above information. Please output all winning candidate information. The returned fields include package name, unit name, ranking, bid price, etc., and ensure that the JSON format is correct."

[0081] Based on the pre-defined structured output format and extraction fields, generating target hints is to concretize the above definition into instructions that the model can understand and execute. The target hint can combine the structured output format and the information of specific fields to form a complete instruction to guide the model on how to process the input data to generate the required structured output. For example, if the input of the model is a bidding document containing multiple winning candidates, the target hint will tell the model: "Please extract information from the following text and output it in the following JSON format: [{"Package Name": "Package 1", "Unit Name": "Unit A", "Ranking": "First", "Bidding Price": "Price A"}, {……}……]". The above target hints detail what the input data is, which fields to extract, and how to output this information in a structured manner.

[0082] Based on the above optional embodiments, by adopting a preset structural paradigm to define the structured output format corresponding to the target extraction object in the target training text, and then adopting a preset prompting paradigm to define the extraction field corresponding to the structured output format, and finally generating a target prompt based on the structured output format and the extraction field, the execution and output format of the information extraction task are strictly controlled, ensuring the accuracy and consistency of information extracted from complex or unstructured texts. The above process is particularly suitable for scenarios that require structured output, such as internal enterprise information integration, legal document analysis, bidding document parsing, etc. Through the preset structural paradigm and preset prompting paradigm, the model can be guided to output results in a specific format, which is convenient for subsequent data processing and application, thereby further improving the efficiency and accuracy of information extraction.

[0083] In an optional embodiment, defining the structured output format corresponding to the target extraction object in the target training text using a preset structural paradigm includes:

[0084] A preset structural paradigm is used to define the extraction object type and / or extraction object grouping relationship of the target extraction object in the target training text, and a structured output format corresponding to the target extraction object is obtained, wherein the extraction object type includes at least one of the following: entity extraction object, event extraction object.

[0085] The above-mentioned extraction object types can be information categories that need to be identified and extracted in information extraction tasks, mainly including entity extraction objects and event extraction objects. Entity extraction objects usually involve identifying entities with specific meanings from texts, such as names of people, organization names, dates, places, etc.; while event extraction objects involve identifying events described in texts, including the types of events, such as transactions, conflict events, etc., as well as details such as participants, time, and location of the events. For example, in legal document information extraction, entity extraction objects can include "plaintiff", "defendant", "court name", etc., and event extraction objects can include "trial starts", "judgment results", etc.

[0086] The above extraction object grouping relationship indicates how to group related target extraction objects together in the information extraction process to reflect the logical or structural association of the target extraction objects in the original text. For example, in the information extraction of bidding documents, each "candidate information" can include sub-entities such as "package name", "unit name", "ranking", and "bid price". These sub-entities should be organized together to form a structured candidate information group. When defining grouping relationships, nested structures, such as dictionaries in lists, can be used to represent multiple groups of related entities or events. For example, a "candidate information" group can be represented as [{"package name": "package one", "unit name": "A Equipment Co., Ltd.", "ranking": "first", "bid price": "32 million yuan"}, {……}].

[0087] The structured output format corresponding to the target extraction object can specify how the information extraction results are presented in a structured manner. It usually involves using JSON, XML, or a custom table structure to organize the target extraction objects and their corresponding grouping relationships. The design of the structured output format needs to take into account how downstream applications use the target extraction results. For example, if the target extraction results will be used to populate a database, a structured output format of JSON is more appropriate; if the information will be used in a web application, XML may be easier to parse and use. When defining a structured output format, you need to ensure that all extraction object types and grouping relationships are correctly represented, and that the output format can be easily parsed and used by downstream applications or systems.

[0088] Based on the above optional embodiments, by adopting a preset structural paradigm to define the extraction object type and / or extraction object grouping relationship of the target extraction object in the target training text, a structured output format corresponding to the target extraction object is obtained, thereby improving the practicality and flexibility of information extraction, so that the extracted information can be directly used in various data processing and application systems, further improving the efficiency of information extraction.

[0089] In an optional embodiment, the extraction fields corresponding to the structured output format defined by the preset prompting pattern include:

[0090] At least one field level corresponding to the structured output format and main extraction object information and sub-extraction object information under at least one field level are defined by using a preset prompting pattern to obtain an extraction field.

[0091] In information extraction tasks, since data often has a hierarchical structure, the field hierarchy is used to describe the hierarchical relationship in the data. For example, when processing bidding documents, it is necessary to extract "successful bidder information", which contains multiple subfields, such as "package name", "unit name", "ranking", "bid price", etc. These subfields constitute the hierarchical structure of the main field "successful bidder information". The purpose of defining the field hierarchy is to obtain the organization of information so that the model can output information according to the preset hierarchical structure.

[0092] The above-mentioned main extraction object information can be the main entity or event that needs to be identified and extracted in the information extraction task, while the sub-extraction object information is the specific details in the main extraction object information. For example, when the main extraction object information is "successful bid candidate information", the "unit name", "ranking", etc. of each candidate are sub-extraction object information. When defining at least one field level and the main extraction object information and sub-extraction object information under at least one field level, it is necessary to ensure that they match the preset structural paradigm and field level, that is, clearly indicate in the prompt which are the main extraction objects and which are the sub-extraction objects, as well as the position of the target extraction object in the structured output.

[0093] Based on the above optional embodiments, by adopting a preset prompt pattern to define at least one field level corresponding to the structured output format and the main extraction object information and sub-extraction object information under at least one field level, the extraction field is obtained, thereby significantly improving the efficiency, accuracy and structured output quality of the information extraction task.

[0094] In an optional embodiment, the model training method in the embodiment of the present application further includes: in response to the presence of unclear field content in the extracted field, setting the field content to empty.

[0095] Specifically, in the information extraction stage, the target language model can try to identify and extract all predefined field contents, which usually involves natural language processing techniques such as entity recognition, relationship extraction, event detection, etc., to automatically locate and extract key information in the text.

[0096] When the target language model has finished processing the text, the next step is to check whether each field has a corresponding value identified and extracted. If a field is not explicitly mentioned in the input text or no relevant information can be found, the field can be marked as "unclear". For fields marked as "unclear", their field contents can be automatically set to empty values ​​or null. The purpose of setting empty values ​​is to ensure that the output structured data is complete on all predefined fields, even if some fields do not actually appear in the text. This allows the output data to maintain the expected structure and format, which is convenient for subsequent data processing and analysis, without causing data parsing errors or application failures due to missing fields.

[0097] In specific implementations, the representation of null values ​​varies. In some cases, null or an empty string can be used to represent null values. In some scenarios, you also need to choose how to fill in unclear fields based on specific application requirements, such as default values ​​or logically inferred values ​​from other fields.

[0098] Based on the above optional embodiment, in response to the existence of unclear field content in the extracted field, the field content is set to empty, thereby providing a more stable and reliable structured output, thereby further improving the accuracy of the information extraction result.

[0099] In an optional embodiment, in step S22, fine-tuning the pre-trained model based on the target prompt and the target training text to obtain the target language model includes:

[0100] The pre-trained model is fine-tuned based on the target prompt and the target training text to obtain a fine-tuned model; the fine-tuned model is subjected to inference acceleration processing to generate a target language model.

[0101] Specifically, the pre-trained model is used as a baseline, and the target training text and target prompts are used to fine-tune the model. The purpose of fine-tuning is to adapt the pre-trained model to the data of a specific task or field, thereby improving the performance of the pre-trained model on this data. During the fine-tuning process, the pre-trained model can learn how to parse the target training text according to the target prompt and output the information of a specific field in a structured format.

[0102] The fine-tuned model has improved accuracy, but its reasoning speed and efficiency are still limited by the complexity of the model. In order to achieve better performance in practical applications, the fine-tuned model needs to be accelerated for reasoning, which includes but is not limited to: quantization, pruning, knowledge distillation, and batch reasoning. Quantization is the process of converting the weights of the model from high precision (such as 32-bit floating point numbers) to low precision (such as 8-bit integers) to reduce computing and storage requirements. Pruning can remove unimportant connections or layers in the model to reduce the number of parameters of the model, thereby accelerating the reasoning process. Knowledge distillation can create a smaller and faster model by transferring the knowledge of a large model to a small model while maintaining high prediction performance. Batch reasoning is to accelerate the reasoning process of the model by packaging multiple input instances into a batch and utilizing the parallel processing capabilities of modern computing hardware.

[0103] After fine-tuning and inference acceleration processing, the target language model is finally obtained. While maintaining high prediction performance, the target language model has faster inference speed and lower hardware resource requirements, making it suitable for actual deployment and large-scale applications.

[0104] Based on the above optional embodiments, the pre-trained model is fine-tuned based on the target prompt and the target training text to obtain a fine-tuned model, and then the fine-tuned model is subjected to inference acceleration processing to generate a target language model. The model is enabled to learn knowledge in a specific field through fine-tuning, and the operating efficiency of the model is improved through inference acceleration processing. The target language model finally generated can provide a better user experience in actual deployment, and further improve the efficiency and accuracy of information extraction.

[0105] In an optional embodiment, performing reasoning acceleration processing on the fine-tuned model to generate a target language model includes:

[0106] The fine-tuned model is quantized to obtain a quantized model; batch inference is performed on the quantized model to generate a target language model.

[0107] Specifically, quantization is a model compression technology that aims to reduce the computational and storage requirements of the model, especially to reduce the accuracy of the model weights, thereby reducing the memory usage and computing resources at runtime. Exemplarily, during the quantization process, the weights of the model are converted from high precision to low precision, which can be achieved by linear quantization (mapping the weights to low precision values) or nonlinear quantization (such as quantization using custom functions). After the quantization process, the quantization parameters need to be adjusted through a calibration process to ensure that the model performance does not drop significantly due to quantization. Calibration usually uses a small number of representative data sets, through which forward propagation is performed to adjust the quantization parameters to find a suitable quantization point. In some cases, quantization and calibration alone may not be enough to maintain model performance. Therefore, after quantization, the model can be additionally fine-tuned using a representative data set to adapt to the changes brought about by quantization. This process is usually called quantization-aware fine-tuning. By fine-tuning in a quantized state, the model can learn how to perform tasks at lower precision, thereby reducing resource requirements while maintaining performance. After the above quantization process is completed, the quantized model will become more lightweight and more suitable for deployment on resource-constrained devices.

[0108] Batch reasoning can package multiple input instances into a batch, and then use the quantized model to process these instances simultaneously to improve the efficiency of reasoning. Batch reasoning takes advantage of the parallel processing capabilities of modern computing hardware. For example, in the batch reasoning process, multiple input texts are first collected and combined into a batch. Batch processing can optimize data transmission and memory usage, effectively reducing the number of data loading and preprocessing. The quantized model is then used to reason about the entire batch instead of processing it one by one, thereby making full use of the parallel computing power of computing resources and reducing reasoning time. After the batch reasoning is completed, the reasoning results can also be post-processed to split them into the output of a single instance for further processing or use. With batch reasoning, the system can maintain a high throughput even when processing large amounts of data.

[0109] The target language model is obtained after quantization processing and batch inference. While maintaining high prediction performance, the target language model has lower computing and storage requirements, faster inference speed, and is more suitable for deployment in practical applications, especially in resource-constrained environments such as mobile devices or edge servers.

[0110] Based on the above optional embodiments, the fine-tuned model is quantized to obtain a quantized model, and then batch inference is performed on the quantized model to generate a target language model. The size and operating resources of the model are significantly reduced through quantization, and batch inference accelerates the reasoning process of the model, improves the overall processing capability of the system, and further improves the efficiency of information extraction.

[0111] Figure 3 is a schematic diagram of a model training method according to an embodiment of the present application, such as Figure 3 As shown in the figure, taking the legal field as an example, the upstream task data can be obtained through channels such as judgment text structuring, case basic information extraction and evidence information extraction. After obtaining the upstream task data, the initial language model is pre-trained using the upstream task data to obtain a pre-trained model. The pre-trained model is fine-tuned based on the target prompt and the target training text to obtain a fine-tuned model, and then the fine-tuned model is quantized to obtain a quantized model. Finally, the quantized model is batch inferenced to generate a target language model. The target training text can be obtained through the following process: first, optical character recognition is performed on the multimodal training data to obtain the recognition result, and then multi-line text detection is performed on the recognition result to obtain the detection result. Finally, intelligent document processing is performed on the detection result to obtain the target training text. After obtaining the target language model, the target language model is used to extract information from the multimodal application data to obtain the target extraction result.

[0112] Figure 4 is a flow chart of a data processing method according to an embodiment of the present application, such as Figure 4 As shown, the method may include the following steps:

[0113] Step S41, obtaining multimodal application data;

[0114] Step S42, using the target language model to extract information from the multimodal application data to obtain a target extraction result; wherein the target language model is generated according to the model training method of any one of the embodiments of the present application.

[0115] Based on the above steps S41 to S42, by acquiring multimodal application data, and then using the target language model to extract information from the multimodal application data, a target extraction result is obtained. In the training process of the target language model, the initial language model is pre-trained by using the upstream task data to obtain a pre-trained model, and then the pre-trained model is fine-tuned based on the target prompt and the target training text to obtain the target language model. The target language model is used to extract information from the multimodal application data to obtain the target extraction result, thereby utilizing the language understanding ability and multimodal data processing ability of the pre-trained model, combined with the target prompt, to guide the pre-trained model to perform the information extraction task in the structured format output by the predefined model, so that the key information in the multimodal application data can be quickly located and extracted, effectively improving the speed of information extraction. In addition, target prompts can guide the pre-trained model to output information in a predefined structured format, which not only helps to reduce the processing complexity caused by unstructured output, but also through learning the target training text in the fine-tuning stage, the target language model can more accurately understand the content of the multimodal training data, improve the accuracy of extracted information, reduce errors and omissions, and thus solve the technical problems of low information extraction efficiency and poor accuracy in related technologies when processing document extraction tasks.

[0116] Figure 5 is a flow chart of another data processing method according to an embodiment of the present application, such as Figure 5 As shown, the method may include the following steps:

[0117] Step S51, obtaining multimodal legal consultation data;

[0118] Step S52, using a target language model to extract information from the multimodal legal consultation data to obtain a legal consultation answer result; wherein the target language model is generated according to any one of the model training methods in the embodiments of the present application.

[0119] Based on the above steps S51 to S52, by acquiring multimodal legal consultation data, and then using the target language model to extract information from the multimodal legal consultation data, the legal consultation answer result is obtained. In the training process of the target language model, the initial language model is pre-trained by using the upstream task data to obtain a pre-trained model, and then the pre-trained model is fine-tuned based on the target prompt and the target training text to obtain the target language model. The target language model is used to extract information from the multimodal legal consultation data to obtain the legal consultation answer result, thereby utilizing the language understanding ability and multimodal data processing ability of the pre-trained model, combined with the target prompt, and guiding the pre-trained model to perform the information extraction task according to the structured format output by the predefined model, so that the key information in the multimodal legal consultation data can be quickly located and extracted, effectively improving the speed of information extraction. In addition, target prompts can guide the pre-trained model to output information in a predefined structured format, which not only helps to reduce the processing complexity caused by unstructured output, but also through learning the target training text in the fine-tuning stage, the target language model can more accurately understand the content of the multimodal training data, improve the accuracy of extracted information, reduce errors and omissions, and thus solve the technical problems of low information extraction efficiency and poor accuracy in related technologies when processing document extraction tasks.

[0120] Figure 6 is a flow chart of another data processing method according to an embodiment of the present application, such as Figure 6 As shown, the method may include the following steps:

[0121] Step S61, obtaining a data processing request through a first application programming interface, wherein the request data carried in the data processing request includes: multimodal application data;

[0122] Step S62, returning a data processing response through a second application programming interface, wherein the response data carried in the data processing response includes: a target extraction result, the target extraction result is obtained by extracting information from the multimodal application data using a target language model, and the target language model is generated according to the model training method of any one of the embodiments of the present application.

[0123] Based on the above steps S61 to S62, a data processing request is obtained through the first application programming interface, and the request data carried in the data processing request includes: multimodal application data, and then a data processing response is returned through the second application programming interface, and the response data carried in the data processing response includes: target extraction results, and the target extraction results are obtained after information extraction from the multimodal application data using the target language model. In the training process of the target language model, the initial language model is pre-trained by using the upstream task data to obtain a pre-trained model, and then the pre-trained model is fine-tuned based on the target prompt and the target training text to obtain the target language model. The target language model is used to extract information from the multimodal application data to obtain the target extraction result, thereby utilizing the language understanding ability and multimodal data processing ability of the pre-trained model, combined with the target prompt, to guide the pre-trained model to perform the information extraction task according to the structured format output by the predefined model, so that the key information in the multimodal application data can be quickly located and extracted, effectively improving the speed of information extraction. In addition, target prompts can guide the pre-trained model to output information in a predefined structured format, which not only helps to reduce the processing complexity caused by unstructured output, but also through learning the target training text in the fine-tuning stage, the target language model can more accurately understand the content of the multimodal training data, improve the accuracy of extracted information, reduce errors and omissions, and thus solve the technical problems of low information extraction efficiency and poor accuracy in related technologies when processing document extraction tasks.

[0124] Figure 7 is a flow chart of another data processing method according to an embodiment of the present application, such as Figure 7 As shown, the method may include the following steps:

[0125] Step S71, obtaining a currently input data processing dialogue request, wherein the request data carried in the data processing dialogue request includes: multimodal application data;

[0126] Step S72, in response to the data processing dialogue request, returning a data processing dialogue reply, wherein the information carried in the data processing dialogue reply includes: a target extraction result, the target extraction result is obtained by extracting information from the multimodal application data using a target language model, and the target language model is generated according to any one of the model training methods in the embodiments of the present application;

[0127] Step S73, displaying the target extraction result in the graphical user interface.

[0128] Based on the above steps S71 to S73, by obtaining the currently input data processing dialogue request, the request data carried in the data processing dialogue request includes: multimodal application data, and then responding to the data processing dialogue request, returning the data processing dialogue reply, the information carried in the data processing dialogue reply includes: target extraction result, the target extraction result is obtained by extracting information from the multimodal application data using the target language model, and finally the target extraction result is displayed in the graphical user interface. In the training process of the target language model, the initial language model is pre-trained by using the upstream task data to obtain a pre-trained model, and then the pre-trained model is fine-tuned based on the target prompt and the target training text to obtain the target language model. The target language model is used to extract information from the multimodal application data to obtain the target extraction result, thereby using the language understanding ability and multimodal data processing ability of the pre-trained model, combined with the target prompt, the pre-trained model is guided to perform the information extraction task according to the structured format output by the predefined model, so that the key information in the multimodal application data can be quickly located and extracted, and the speed of information extraction is effectively improved. In addition, target prompts can guide the pre-trained model to output information in a predefined structured format, which not only helps to reduce the processing complexity caused by unstructured output, but also through learning the target training text in the fine-tuning stage, the target language model can more accurately understand the content of the multimodal training data, improve the accuracy of extracted information, reduce errors and omissions, and thus solve the technical problems of low information extraction efficiency and poor accuracy in related technologies when processing document extraction tasks.

[0129] Figure 8 is a flow chart of another data processing method according to an embodiment of the present application, such as Figure 8 As shown, the method may include the following steps:

[0130] Step S81, in response to an input instruction acting on the operation interface, displaying multimodal application data on the operation interface;

[0131] Step S82, in response to the processing instruction acting on the operation interface, the target extraction result is displayed on the operation interface; wherein the target extraction result is obtained by extracting information from the multimodal application data using a target language model, and the target language model is generated according to the model training method of any one of the embodiments of the present application.

[0132] Based on the above steps S81 to S82, in response to the input instruction acting on the operation interface, the multimodal application data is displayed on the operation interface, and then in response to the processing instruction acting on the operation interface, the target extraction result is displayed on the operation interface. In the training process of the target language model, the initial language model is pre-trained by using the upstream task data to obtain a pre-trained model, and then the pre-trained model is fine-tuned based on the target prompt and the target training text to obtain the target language model. The target language model is used to extract information from the multimodal application data to obtain the target extraction result, thereby utilizing the language understanding ability and multimodal data processing ability of the pre-trained model, combined with the target prompt, to guide the pre-trained model to perform the information extraction task according to the structured format output by the predefined model, so that the key information in the multimodal application data can be quickly located and extracted, effectively improving the speed of information extraction. In addition, target prompts can guide the pre-trained model to output information in a predefined structured format, which not only helps to reduce the processing complexity caused by unstructured output, but also through learning the target training text in the fine-tuning stage, the target language model can more accurately understand the content of the multimodal training data, improve the accuracy of extracted information, reduce errors and omissions, and thus solve the technical problems of low information extraction efficiency and poor accuracy in related technologies when processing document extraction tasks.

[0133] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0134] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0135] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0136] According to an embodiment of the present application, a model training device for implementing the above-mentioned model training method is also provided. Fig. 9 is a structural block diagram of a model training device according to an embodiment of the present application, such as Fig. 9 As shown, the device comprises:

[0137] A pre-training module 901 is used to pre-train the initial language model using upstream task data to obtain a pre-trained model;

[0138] The fine-tuning module 902 is used to fine-tune the pre-trained model based on the target prompt and the target training text to obtain the target language model; wherein the target prompt is used to guide the pre-trained model to perform information extraction tasks according to the structured format of the predefined model output, the target training text is converted from the multimodal training data, and the target language model is used to extract information from the multimodal application data to obtain the target extraction result.

[0139] Optionally, the model training device in the embodiment of the present application also includes: a recognition module 903, which is used to perform optical character recognition on multimodal training data to obtain a recognition result; a detection module 904, which is used to perform multi-line text detection on the recognition result to obtain a detection result; a processing module 905, which is used to perform intelligent document processing on the detection result to obtain a target training text, wherein the target training text is written in a preset level markup language format.

[0140] Optionally, the model training device in the embodiment of the present application also includes: a generation module 906, which is used to generate target prompts based on a preset structural paradigm and a preset prompt paradigm, wherein the preset structural paradigm is used to define the structured format of the model output, and the preset prompt paradigm is used to guide the pre-trained model to perform information extraction tasks.

[0141] Optionally, the generation module 906 is also used to: use a preset structural paradigm to define a structured output format corresponding to a target extraction object in a target training text; use a preset prompt paradigm to define an extraction field corresponding to the structured output format; and generate a target prompt based on the structured output format and the extraction field.

[0142] Optionally, the generation module 906 is also used to: use a preset structural paradigm to define the extraction object type and / or extraction object grouping relationship of the target extraction object in the target training text, and obtain a structured output format corresponding to the target extraction object, wherein the extraction object type includes at least one of the following: entity extraction object, event extraction object.

[0143] Optionally, the generating module 906 is further used to: define at least one field level corresponding to the structured output format and main extraction object information and sub-extraction object information under at least one field level by using a preset prompting pattern to obtain an extraction field.

[0144] Optionally, the model training device in the embodiment of the present application also includes: a setting module 907, which is used to set the field content to empty in response to the presence of unclear field content in the extracted field.

[0145] Optionally, the fine-tuning module 902 is also used to: fine-tune the pre-trained model based on the target prompt and the target training text to obtain a fine-tuned model; perform inference acceleration processing on the fine-tuned model to generate a target language model.

[0146] Optionally, the fine-tuning module 902 is further used to: quantize the fine-tuned model to obtain a quantized model; and perform batch inference on the quantized model to generate a target language model.

[0147] It should be noted that the above-mentioned pre-training module 901 and fine-tuning module 902 correspond to steps S21 to S22 in the above-mentioned embodiment, and the two modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned embodiment. It should be noted that the above-mentioned modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above-mentioned modules can also be run in the server provided in the above-mentioned embodiment as part of the device.

[0148] Fig.10 is a structural block diagram of a data processing device according to an embodiment of the present application, such as Fig.10 As shown, the device comprises:

[0149] An acquisition module 1001 is used to acquire multimodal application data;

[0150] The processing module 1002 is used to extract information from the multimodal application data using a target language model to obtain a target extraction result; wherein the target language model is generated according to any one of the model training methods in the embodiments of the present application.

[0151] It should be noted that the acquisition module 1001 and the processing module 1002 correspond to steps S41 to S42 in the above embodiment, and the examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be run in the server provided in the above embodiment as part of the device.

[0152] Fig.11 is a structural block diagram of another data processing device according to an embodiment of the present application, such as Fig.11 As shown, the device comprises:

[0153] An acquisition module 1101 is used to acquire multimodal legal consultation data;

[0154] The processing module 1102 is used to extract information from the multimodal legal consultation data using a target language model to obtain a legal consultation answer result; wherein the target language model is generated according to any one of the model training methods in the embodiments of the present application.

[0155] It should be noted that the acquisition module 1101 and the processing 1102 correspond to steps S51 to S52 in the above embodiment, and the examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be run in the server provided in the above embodiment as part of the device.

[0156] Fig.12 is a structural block diagram of another data processing device according to an embodiment of the present application, such as Fig.12 As shown, the device comprises:

[0157] The acquisition module 1201 is used to acquire a data processing request through a first application programming interface, wherein the request data carried in the data processing request includes: multimodal application data;

[0158] The return module 1202 is used to return a data processing response through a second application programming interface, wherein the response data carried in the data processing response includes: a target extraction result, which is obtained by extracting information from multimodal application data using a target language model, and the target language model is generated according to the model training method of any one of the embodiments of the present application.

[0159] It should be noted that the acquisition module 1201 and the return module 1202 correspond to steps S61 to S62 in the above embodiment, and the examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be run in the server provided in the above embodiment as part of the device.

[0160] Fig.13 is a structural block diagram of another data processing device according to an embodiment of the present application, such as Fig.13 As shown, the device comprises:

[0161] The acquisition module 1301 is used to acquire the currently input data processing dialogue request, wherein the request data carried in the data processing dialogue request includes: multimodal application data;

[0162] A return module 1302 is used to respond to the data processing dialogue request and return a data processing dialogue reply, wherein the information carried in the data processing dialogue reply includes: a target extraction result, which is obtained by extracting information from the multimodal application data using a target language model, and the target language model is generated according to any one of the model training methods in the embodiments of the present application;

[0163] The display module 1303 is used to display the target extraction result in the graphical user interface.

[0164] It should be noted that the acquisition module 1301, the return module 1302 and the display module 1303 correspond to steps S71 to S73 in the above embodiment, and the three modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be part of the device and can run in the server provided in the above embodiment.

[0165] Fig.14 is a structural block diagram of another data processing device according to an embodiment of the present application, such as Fig.14 As shown, the device comprises:

[0166] The first display module 1401 is used to respond to input instructions acting on the operation interface and display multimodal application data on the operation interface;

[0167] The second display module 1402 is used to respond to the processing instructions acting on the operation interface and display the target extraction result on the operation interface; wherein the target extraction result is obtained after information extraction of multimodal application data using a target language model, and the target language model is generated according to the model training method of any one of the embodiments of the present application.

[0168] It should be noted that the first display module 1401 and the second display module 1402 correspond to steps S81 to S82 in the above embodiment, and the examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be run in the server provided in the above embodiment as part of the device.

[0169] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in the above embodiments, as well as the application scenario and implementation process, but is not limited to the scheme provided in the above embodiments.

[0170] An embodiment of the present application may provide a computing device. Fig.15 is a structural block diagram of a computing device according to an embodiment of the present application. Fig.15 As shown, the computing device may include: one or more (only one is shown in the figure) processors 152, a memory 154, a storage controller, and a peripheral interface.

[0171] The above-mentioned computing device can be understood as an integrated intelligent terminal, including but not limited to a server, a desktop computer, a PC (Personal Computer), a model all-in-one machine, etc., and the computing device can be pre-installed with the model described in the above-mentioned embodiments of the present application.

[0172] Specifically, the computing device can pre-set multiple types of models, including but not limited to models in the fields of natural language processing, visual processing, speech processing, code processing, multimodal task processing, etc., so as to provide a variety of model choices. In different product forms, the computing device can support one or more model usage methods, including but not limited to model training, model calling, model fine-tuning, model deployment, model reasoning and application, etc. In some product forms, the computing device also supports model management, including but not limited to multi-type model management (supporting the management of multiple types of models such as discriminant and generative models), model version control (supporting the control of different model versions), model evaluation (based on model evaluation tools, evaluating the performance and effect of the model), etc. In other product forms, the computing device can also create applications based on models, provide API calling capabilities, and can call models to the created applications through API interfaces, while providing application management tools to achieve management and monitoring of applications.

[0173] Furthermore, the computing device can also include data management (supporting the creation and management of model tuning data sets), training center (providing rich training resources to help users learn and master AI technology), and basic management and control capabilities (providing enterprise-level basic management and control capabilities to ensure the security and efficient operation of the system). Through the above functions, a comprehensive, integrated AI development, training, deployment and application device is provided.

[0174] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implementing the methods in the above embodiments. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0175] The processor may call the executable program stored in the memory through the transmission device to execute the method described in any one of the above embodiments.

[0176] An embodiment of the present application may provide an electronic device. Fig.16 is a structural block diagram of an electronic device according to an embodiment of the present application. Fig.16 As shown, the electronic device may include: an input-output device 162 ; a memory 164 and a processor 166 , wherein the processor 166 is connected to the input-output device 162 and the memory 164 via a bus 168 .

[0177] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implementing the methods in the above embodiments. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0178] The processor may call the executable program stored in the memory through the transmission device to execute the method described in any one of the above embodiments.

[0179] It is understood by those skilled in the art that the structure shown in the figure is for illustration only, and the computing device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, and other terminal devices. The figure does not limit the structure of the above computing device. For example, the computing device may also include more or fewer components (such as a network interface, a display device, etc.) than those shown in the figure, or have a configuration different from that shown in the figure.

[0180] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0181] The embodiment of the present application further provides a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the method provided in the above embodiment.

[0182] Optionally, in this embodiment, the above storage medium may be located in a computing device.

[0183] Optionally, in this embodiment, the computer-readable storage medium is configured to store an executable program, and when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the method described in any one of the above embodiments.

[0184] The embodiment of the present application further provides a computer program product. Optionally, in this embodiment, the computer program product may include a computer program, and the computer program implements the method provided in the embodiment when executed by a processor.

[0185] The embodiments of the present application also provide a computer program product. Optionally, the computer program product may include a non-volatile computer-readable storage medium, which may be used to store a computer program, and when the computer program is executed by a processor, the method provided in the embodiments is implemented.

[0186] The embodiment of the present application further provides a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, the method provided in the above embodiment is implemented.

[0187] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0188] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0189] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0190] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0191] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc., which can store program code.

[0192] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A model training method, characterized in that: include: Use upstream task data to pre-train the initial language model to obtain a pre-trained model; Fine-tune the pre-trained model based on the target prompt and the target training text to obtain a target language model; Among them, the target prompt is used to guide the pre-trained model to perform information extraction tasks according to the structured format output by the predefined model, the target training text is converted from multimodal training data, and the target language model is used to extract information from multimodal application data to obtain target extraction results.

2. The model training method according to claim 1, characterized in that: The model training method also includes: Performing optical character recognition on the multimodal training data to obtain a recognition result; Performing multi-line text detection on the recognition result to obtain a detection result; Intelligent document processing is performed on the detection result to obtain the target training text, wherein the target training text is written in a preset level markup language format.

3. The model training method according to claim 1, characterized in that: The model training method also includes: The target prompt is generated based on a preset structural paradigm and a preset prompt paradigm, wherein the preset structural paradigm is used to define a structured format of a model output, and the preset prompt paradigm is used to guide the pre-trained model to perform an information extraction task.

4. The model training method according to claim 3, characterized in that: Generating the target prompt based on the preset structural paradigm and the preset prompt paradigm includes: The preset structural paradigm is used to define the structured output format corresponding to the target extraction object in the target training text; Using the preset prompting pattern to define the extraction fields corresponding to the structured output format; The target prompt is generated based on the structured output format and the extracted fields.

5. The model training method according to claim 4, characterized in that: The structured output format corresponding to the target extraction object in the target training text is defined by using the preset structural paradigm, including: The preset structural paradigm is used to define the extraction object type and / or extraction object grouping relationship of the target extraction object in the target training text to obtain the structured output format corresponding to the target extraction object, wherein the extraction object type includes at least one of the following: entity extraction object, event extraction object.

6. The model training method according to claim 4, characterized in that: Defining the extraction fields corresponding to the structured output format using the preset prompting format includes: The preset prompting pattern is used to define at least one field level corresponding to the structured output format and the main extraction object information and the sub-extraction object information under the at least one field level to obtain the extraction field.

7. The model training method according to claim 4, characterized in that: The model training method also includes: In response to the presence of unclear field content in the extracted field, the field content is set to empty.

8. The model training method according to claim 1, characterized in that: Fine-tuning the pre-trained model based on the target prompt and the target training text to obtain the target language model includes: Fine-tune the pre-trained model based on the target prompt and the target training text to obtain a fine-tuned model; Performing inference acceleration processing on the fine-tuned model to generate the target language model.

9. The model training method according to claim 8, characterized in that: Performing reasoning acceleration processing on the fine-tuned model to generate the target language model includes: Performing quantization processing on the fine-tuned model to obtain a quantized model; Batch inference is performed on the quantized model to generate the target language model.

10. A data processing method, characterized in that: include: Get multimodal application data; Using a target language model to extract information from the multimodal application data to obtain a target extraction result; Wherein, the target language model is generated according to the model training method described in any one of claims 1 to 9.

11. A data processing method, characterized in that: include: Access to multimodal legal consultation data; Using a target language model to extract information from the multimodal legal consultation data to obtain a legal consultation answer result; Wherein, the target language model is generated according to the model training method described in any one of claims 1 to 9.

12. A data processing method, characterized in that: include: Obtaining a data processing request through a first application programming interface, wherein the request data carried in the data processing request includes: multimodal application data; A data processing response is returned through a second application programming interface, wherein the response data carried in the data processing response includes: a target extraction result, the target extraction result is obtained by extracting information from the multimodal application data using a target language model, and the target language model is generated according to the model training method described in any one of claims 1 to 9.

13. A data processing method, characterized in that: include: Acquire a currently input data processing dialogue request, wherein the request data carried in the data processing dialogue request includes: multimodal application data; In response to the data processing dialogue request, a data processing dialogue reply is returned, wherein the information carried in the data processing dialogue reply includes: a target extraction result, the target extraction result is obtained by extracting information from the multimodal application data using a target language model, and the target language model is generated according to the model training method according to any one of claims 1 to 9; The target extraction result is displayed in a graphical user interface.

14. A data processing method, characterized in that: include: In response to an input instruction acting on the operation interface, displaying multimodal application data on the operation interface; In response to a processing instruction acting on the operation interface, displaying a target extraction result on the operation interface; The target extraction result is obtained by extracting information from the multimodal application data using a target language model, and the target language model is generated according to the model training method described in any one of claims 1 to 9.

15. A data processing system, characterized in that: include: Client, used to send multimodal application data; A server, connected to the client, configured to extract information from the multimodal application data using a target language model to obtain a target extraction result; The client is also used to output the target extraction result; Wherein, the target language model is generated according to the model training method described in any one of claims 1 to 9.

16. An electronic device, characterized in that: include: A memory storing an executable program; A processor, used to run the program, wherein when the program is run, the model training method described in any one of claims 1 to 9 or the data processing method described in any one of claims 10 to 14 is executed.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein, when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the model training method described in any one of claims 1 to 9 or the data processing method described in any one of claims 10 to 14.

18. A computer program product, characterized in that It includes a computer program which, when executed by a processor, implements the model training method described in any one of claims 1 to 9 or the data processing method described in any one of claims 10 to 14.

Images