Model training method, data processing method, electronic device and storage medium

Abstract

The present application discloses a model training method, a data processing method, an electronic device and a storage medium, and relates to the fields of large model technology and computer technology. The method comprises: pre-training the original language processing model with the original data set to obtain a first language processing model; fine-tuning the first language processing model with the extended data set to obtain a second language processing model; and improving the preference of the second language processing model with the preference data set to obtain a target language processing model. The present application solves the technical problems in the related art that the model has poor compliance with diverse instructions and low model processing performance.

Description

Technical Field

[0001] The present application relates to the fields of large model technology and computer technology, and specifically, to a model training method, a data processing method, an electronic device and a storage medium. Background Art

[0002] In products based on large models, since each service provider has unique task processing requirements, it is necessary to perform customized effect tuning through supervised fine-tuning (SFT). However, this method faces difficulties such as difficulty in obtaining data, high training costs, slow iteration speed, and excessive human investment in algorithms. As a result, it is impossible to effectively customize effect tuning in large-scale applications, which results in the model being unable to adapt well to different task processing requirements and poor compliance with diverse instructions. In order to improve the model's compliance with diverse instructions in related technologies, the instructions can be evolved in diversity or complexity, and instruction data that can be verified by code can be added. However, there are still limitations when facing instructions with more constraints, which results in errors in the model when processing complex instructions or failure to fully follow the instructions.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a model training method, a data processing method, an electronic device and a storage medium to at least solve the technical problems in the related art that the model has poor compliance with diverse instructions and low model processing performance.

[0005] According to one aspect of an embodiment of the present application, a model training method is provided, comprising: pre-training an original language processing model using an original data set to obtain a first language processing model, wherein the original data set comprises: a plurality of original data pairs, and original prompt texts in the plurality of original data pairs contain original constraints; fine-tuning the first language processing model using an extended data set to obtain a second language processing model, wherein the extended data set comprises: a plurality of extended data pairs, and extended prompt texts in the plurality of extended data pairs contain extended constraints, and the extended constraints are obtained by expanding the original constraints; performing preference improvement on the second language processing model using a preference data set to obtain a target language processing model, wherein the preference data set comprises: a plurality of preference data pairs, and the plurality of preference data pairs are extended data pairs in which the degree of compliance of extended reply texts with extended constraints in the plurality of extended data pairs meets preset verification conditions, and the target language processing model is used to perform natural language understanding on the questions to be queried in the target application scenario to generate target replies.

[0006] According to another aspect of an embodiment of the present application, a data processing method is also provided, including: obtaining a query request, wherein the data carried in the query request includes: a question to be queried in a target application scenario; in response to the query request, using a target language processing model to perform natural language understanding on the question to be queried, and generating a target reply, wherein the target language processing model is trained using the model training method of any one of the embodiments of the present application.

[0007] According to another aspect of an embodiment of the present application, a data processing method is also provided, including: obtaining a query request, wherein the data carried in the query request includes: an intelligent customer service question to be queried in an intelligent customer service scenario; in response to the query request, using a target language processing model to perform natural language understanding on the intelligent customer service question to be queried, and generating a target intelligent customer service reply, wherein the target language processing model is trained using the model training method of any one of the embodiments of the present application.

[0008] According to another aspect of an embodiment of the present application, a data processing method is also provided, including: obtaining a data processing request through a first application programming interface, wherein the request data carried in the data processing request includes: a question to be queried in a target application scenario; returning a data processing response through a second application programming interface, wherein the response data carried in the data processing response includes: a target reply, wherein the target reply is generated after natural language understanding of the question to be queried using a target language processing model, and the target language processing model is trained using the model training method of any one of the embodiments of the present application.

[0009] According to another aspect of an embodiment of the present application, a data processing method is also provided, including: obtaining a currently input data processing dialogue request, wherein the request data carried in the data processing dialogue request includes: a question to be queried in a target application scenario; in response to the data processing dialogue request, returning a data processing dialogue reply, wherein the information carried in the data processing dialogue reply includes: a target reply, the target reply is generated after natural language understanding of the question to be queried using a target language processing model, and the target language processing model is trained using the model training method of any one of the embodiments of the present application; and displaying the target reply in a graphical user interface.

[0010] According to another aspect of the embodiments of the present application, an electronic device is further provided, including: a memory storing an executable program; and a processor for running the program, wherein the method of each embodiment of the present application is executed when the program is running.

[0011] According to another aspect of an embodiment of the present application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the methods in each embodiment of the present application.

[0012] According to another aspect of the embodiments of the present application, a computer program product is also provided, including a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present application is implemented.

[0013] According to another aspect of an embodiment of the present application, a computer program product is also provided, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present application is implemented.

[0014] According to another aspect of the embodiments of the present application, a computer program is also provided, and when the computer program is executed by a processor, the methods in the various embodiments of the present application are implemented.

[0015] In the embodiment of the present application, the original language processing model is pre-trained by the original data set so that it can learn the information and constraints in the original data pair, thereby establishing a basic language processing capability. Then the pre-trained language processing model is fine-tuned by the extended data set, and the extended constraints are introduced so that the model can better adapt to the problems in the target application scenario. Finally, the fine-tuned language processing model is improved by the preference data set to further improve the model performance so that it can better understand and reply to the user's query questions. Thus, through the methods of pre-training, fine-tuning and preference improvement, a target language processing model that performs better in the target application scenario can be obtained, which can more accurately understand the user's query questions and generate more appropriate replies, thereby improving the practicality and reliability of the model. Through continuous optimization and iteration, the performance of the model can be further improved so that it can better adapt to different application scenarios and needs, thereby achieving the technical effect of improving the model's compliance effect on diversity instructions and the model processing performance, thereby solving the technical problems of poor compliance effect of the model on diversity instructions and low model processing performance in the related technology.

[0016] It is easy to notice that the above general description and the following detailed description are only for the purpose of exemplifying and explaining the present application, and do not constitute a limitation of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 is a schematic diagram of an application scenario of a model training method according to an embodiment of the present application;

[0019] Figure 2is a flow chart of a model training method according to an embodiment of the present application;

[0020] Figure 3 is a schematic diagram of a model training method according to an embodiment of the present application;

[0021] Figure 4 is a flow chart of a data processing method according to an embodiment of the present application;

[0022] Figure 5 is a flow chart of another data processing method according to an embodiment of the present application;

[0023] Figure 6 is a flow chart of another data processing method according to an embodiment of the present application;

[0024] Figure 7 is a flow chart of another data processing method according to an embodiment of the present application;

[0025] Figure 8 is a structural block diagram of a model training device according to an embodiment of the present application;

[0026] Fig. 9 is a structural block diagram of a data processing device according to an embodiment of the present application;

[0027] Fig.10 is a flow chart of another data processing method according to an embodiment of the present application;

[0028] Fig.11 is a flow chart of another data processing method according to an embodiment of the present application;

[0029] Fig.12 is a flow chart of another data processing method according to an embodiment of the present application;

[0030] Fig.13 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] The technical solution provided in this application is mainly implemented using large-scale model technology. The large model here refers to a deep learning model with large-scale model parameters, which can usually contain hundreds of millions, tens of billions, hundreds of billions, trillions or even more than 10 trillion model parameters. The large model can also be called a foundation model / foundation model. The large model is pre-trained with large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as large-scale language model (LLM), multi-modal pre-training model, etc.

[0034] It should be noted that when the large model is actually applied, the pre-trained model can be fine-tuned through a small number of samples so that the large model can be applied to different tasks. For example, the large model can be widely used in natural language processing (NLP), computer vision, speech processing and other fields, and can be specifically applied to computer vision tasks such as visual question answering (VQA), image caption (IC), image generation, etc. It can also be widely used in natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. Therefore, the main application scenarios of the large model include but are not limited to digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc. In the embodiment of the present application, data processing through the target language processing model in the language processing scenario is used as an example for explanation.

[0035] According to an embodiment of the present application, a model training method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0036] Considering the huge amount of model parameters of the large model and the limited computing resources of the mobile terminal, the above model training method provided in the embodiment of the present application can be applied to Figure 1 The application scenarios shown are not limited to these. Figure 1 In the application scenario shown, the large model is deployed in the server 10, and the server 10 can be connected to one or more client devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client device 20 here may include but is not limited to: a smart phone, a tablet computer, a laptop computer, a PDA, a personal computer, a smart home device, a vehicle-mounted device, etc. The client device 20 can interact with the user through a graphical user interface to implement the call of the large model, thereby implementing the method provided in the embodiment of the present application.

[0037] In an embodiment of the present application, a system composed of a client device and a server can perform the following steps: the server obtains a query request via the client, wherein the data carried in the query request includes: the question to be queried in the target application scenario. Furthermore, in response to the query request, the server uses the target language processing model to perform natural language understanding on the question to be queried and generates a target reply. The original language processing model is pre-trained in the server using the original data set to obtain a first language processing model, the first language processing model is fine-tuned using the extended data set to obtain a second language processing model, and the second language processing model is improved using the preference data set to obtain a target language processing model. The server returns the target reply to the client and displays the target reply in the client's graphical user interface. It should be noted that, when the operating resources of the client device can meet the deployment and operating conditions of the large model, the embodiment of the present application can be performed in the client device.

[0038] Under the above operating environment, this application provides Figure 2 The model training method shown. Figure 2 is a flow chart of a model training method according to an embodiment of the present application. Figure 2 As shown, the method may include the following steps:

[0039] Step S21, pre-training an original language processing model using an original data set to obtain a first language processing model, wherein the original data set includes: a plurality of original data pairs, and original prompt texts in the plurality of original data pairs include original constraints;

[0040] Step S22, fine-tuning the first language processing model using an extended data set to obtain a second language processing model, wherein the extended data set includes: a plurality of extended data pairs, the extended prompt texts in the plurality of extended data pairs include extended constraints, and the extended constraints are obtained by expanding the original constraints;

[0041] Step S23, using the preference data set to perform preference improvement on the second language processing model to obtain a target language processing model, wherein the preference data set includes: multiple preference data pairs, the multiple preference data pairs are extended data pairs in which the degree of compliance of the extended reply text with the extended constraints in the multiple extended data pairs meets the preset verification conditions, and the target language processing model is used to perform natural language understanding on the query questions in the target application scenario to generate target replies.

[0042] The above-mentioned original data set includes multiple original data pairs. The original data pairs are prompt-response tasks without extended constraints. The original data pairs contain the original prompt text and the original response text corresponding to the original prompt text. The original prompt text contains multiple original constraints. The original data pairs can be obtained from public data on the Internet. The original constraints can represent restrictions or requirements related to the conversation, which are used to guide the process of generating responses. The original constraints can include but are not limited to restrictions on the topic of the conversation, tone, content length, response format, etc., to help the original language processing model better understand the user's intention and generate appropriate responses. By using the original data pairs in the original data set to pre-train the original language processing model, the original language processing model's ability to understand and respond to user needs can be improved, thereby obtaining a first language processing model.

[0043] The above-mentioned original language processing model can be, but is not limited to, based on models such as Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM), or Transformer, and can be used for natural language processing tasks such as text classification, named entity recognition, and sentiment analysis.

[0044] The first language processing model is a model obtained by pre-training the original language processing model on the original data set. The first language processing model can better understand and process the original prompt text containing the original constraints, thereby improving the performance and generalization ability of the model on related tasks. Through pre-training, the original language processing model can learn more about language structure and semantics, so as to perform better in downstream tasks.

[0045] The above-mentioned extended data set is complex constraint data, which includes multiple extended data pairs. The extended data pairs include extended prompt text and extended reply text corresponding to the extended prompt text. The extended prompt text contains multiple extended constraints. The extended constraints are obtained by expanding the original constraints. They usually involve more conditions, details or restrictions to better ensure the accuracy, completeness and consistency of the data. Extended constraints can be customized according to specific needs or situations to meet the needs of data processing and analysis. The existence of extended constraints makes the first language processing model more comprehensive and accurate, and can better reflect the actual situation and user needs. When processing extended data sets, special attention should be paid to the processing of extended constraints to ensure that the model can correctly understand and apply the extended constraints, so as to better complete the task.

[0046] Exemplarily, when the original constraints are expanded to obtain the extended constraints, it is possible but not limited to replacing synonyms, adding qualifiers or modifiers, introducing related concepts or domain knowledge, adding conditions or assumptions, introducing antonyms or opposing viewpoints, etc. The original prompt text contains an average of 2 to 3 original constraints, and the expanded prompt text contains an average of 10 extended constraints after expansion, thereby obtaining richer and more diverse constraint information, which can be used to fine-tune the first language processing model and obtain a second language processing model suitable for the extended data set.

[0047] Specifically, replace the keywords in the original constraints with their synonyms to enrich the expression of the constraints. Add qualifiers or modifiers to the original constraints to make the constraints more specific and detailed. Introduce concepts or domain knowledge related to the original constraints in the extended prompt text to expand the scope and depth of the constraints. Add conditions or assumptions to the extended prompt text to further limit or relax the original constraints. Introduce viewpoints or antonyms that are opposite to the original constraints in the extended prompt text to provide diversified constraint information. The expansion operation of the original constraints can improve the generalization ability and adaptability of the first language processing model, so that it performs better when processing diversified data.

[0048] The first language processing model is fine-tuned using an extended dataset to obtain a second language processing model. The fine-tuning process is based on an already trained model, trained using a new dataset, and adjusting the model's parameters to adapt to the new dataset. In the field of natural language processing, a model that has been pre-trained on a large-scale dataset is used, and then further trained on a dataset in a specific field through fine-tuning to improve the model's performance on specific query tasks. Through fine-tuning, the model can better adapt to the new dataset and improve its generalization ability and performance.

[0049] Exemplarily, in the process of fine-tuning the first language processing model using the extended data set, first, multiple extended data pairs in the extended data set are input into the first language processing model for SFT training, and the extended prompt text and extended constraints are used as additional input information. In the fine-tuning process, the learning of the first language processing model to the extended constraints can be strengthened by increasing the weight of the extended data pairs or adjusting the learning rate. Finally, the performance of the fine-tuned model can be evaluated by a validation set or a test set, and necessary adjustments and optimizations can be made. Thus, through the above steps, the first language processing model can be effectively fine-tuned so that it can better understand and process the data containing the extended constraints, thereby obtaining the second language processing model.

[0050] The above-mentioned preference data set is a preference data set, and the preference data set includes multiple preference data pairs, and the multiple preference data pairs are extended data pairs in which the degree of compliance of the extended reply text to the extended constraints in the multiple extended data pairs meets the preset verification conditions. For example, after extracting 10 extended constraints, they are split into 10 verification prompts and fine-grained verification is performed through LLM. The verification prompt can be "Please ask whether the following prompt complies with the constraint-output in json format". When the verification result of the extended constraint is verification passed, it is determined that the extended reply text complies with the extended constraint. The more extended constraints the extended reply text complies with, the higher the ranking of the extended reply text, that is, the higher the degree of compliance of the extended reply text to the extended constraints. If the number of extended constraints followed by the extended reply text in the extended data pair reaches the preset threshold, it is determined that the extended data pair meets the preset verification condition, and the screening can be used as a preference data pair.

[0051] The preference data set is used to improve the preferences of the second language processing model, thereby using the preference data pairs for more fine-grained training, so that the second language processing model can further learn some subtle changes and further enhance the model's fine-grained instruction following ability. The preference improvement process is to train and optimize the second language processing model using the preference data set to improve the performance and effect of the model in specific application scenarios. By using preference data pairs, the second language processing model can learn more constraints and rules to better understand and generate target responses. Preference improvement can help the model understand the user's intentions more accurately, improve the accuracy and naturalness of the response, and thus enhance the overall user experience. Through continuous optimization and improvement, the target language processing model can continuously adapt to different application scenarios and user needs to achieve better performance and results.

[0052] The model training difficulty corresponding to the above-mentioned pre-training process, fine-tuning process and preference improvement process increases from easy to difficult. Pre-training the original language processing model through the original data set, that is, training on simple tasks without restrictions, allows the original language processing model to learn the input and output patterns of simple tasks. Fine-tuning the pre-trained language processing model through the extended data set can be carried out using complex constraint data for training, so that the first language processing model can learn to handle complex instructions with multiple constraints. Preference improvement of the fine-tuned language processing model through the preference data set can further enhance the model's fine-grained instruction-following ability, so that the target language processing model generates the target response for the natural language understanding of the question to be checked more in line with user needs.

[0053] The above-mentioned target application scenarios include, but are not limited to, intelligent customer service systems, legal consulting systems, medical information query systems, educational assistance systems, enterprise knowledge management, financial risk assessment, intelligent writing assistance, multilingual translation systems, smart home dialogue systems, autonomous driving voice navigation and other scenarios. The target language processing model usually includes two main components: natural language understanding and natural language generation, which are used to convert the natural language text input by the user into a form that can be understood by the computer and generate corresponding natural language replies. It should be noted that the description of the target application scenario in the embodiments of the present application is only an example, but does not constitute a specific limitation.

[0054] In customer service scenarios, the target language processing model needs to follow specific polite language, company policies, product information and other constraints to provide a professional and consistent customer service experience. For example, the intelligent customer service of e-commerce websites, the automatic answering system of banks, and the ticket consultation of airlines. In legal consultation, the target language processing model needs to accurately understand and apply legal terms, follow the format and style of legal documents, and ensure the legal compliance and professionalism of the response. In the medical field, the model needs to follow medical knowledge, medical terminology and privacy protection rules to accurately provide medical information and guidance while protecting patient privacy. In educational scenarios, the target language processing model needs to understand constraints such as the syllabus and textbook content, and provide answers that meet the teaching objectives and levels, such as online tutoring, intelligent question banks, educational games, etc. For queries on the internal knowledge base of an enterprise, the target language processing model needs to follow constraints such as the internal terminology, processes, and policies of the enterprise to provide accurate enterprise information and knowledge. In financial risk control scenarios, the target language processing model needs to understand and follow constraints such as financial regulations and risk control strategies to assess risks and provide compliance advice. In writing assistance, the target language processing model needs to follow specific writing style, format, and subject constraints to help users create articles or reports that meet the requirements. In translation scenarios, the target language processing model needs to understand and follow the grammar, culture, and professional terminology of the target language to provide accurate translation results. In smart home environments, the target language processing model needs to understand and follow security, privacy, and device operation rules to provide safe smart home control and information query services. In autonomous vehicles, the target language processing model needs to follow constraints such as traffic rules, safety instructions, and operating procedures to provide accurate voice navigation and vehicle control instructions.

[0055] Through the model training method in the embodiment of the present application, the performance of the language processing model in the above-mentioned fields and scenarios can be significantly improved, making the model's responses more in line with specific needs, and improving user experience and service efficiency.

[0056] Based on the above steps S21 to S23, the original language processing model is pre-trained through the original data set so that it can learn the information and constraints in the original data pair, thereby establishing basic language processing capabilities. Then the pre-trained language processing model is fine-tuned through the extended data set, and the extended constraints are introduced so that the model can better adapt to the problems in the target application scenario. Finally, the fine-tuned language processing model is improved by preference data sets to further improve the model performance so that it can better understand and reply to the user's query questions. Thus, through pre-training, fine-tuning and preference improvement, a target language processing model that performs better in the target application scenario can be obtained, which can more accurately understand the user's query questions and generate more appropriate replies, thereby improving the practicality and reliability of the model. Through continuous optimization and iteration, the performance of the model can be further improved so that it can better adapt to different application scenarios and needs, thereby achieving the technical effect of improving the model's compliance effect on diversity instructions and the model processing performance, thereby solving the technical problems of poor compliance effect of the model on diversity instructions and low model processing performance in the related technology.

[0057] The model training method in the embodiment of the present application is further introduced below.

[0058] In an optional embodiment, the model training method in the embodiment of the present application further includes:

[0059] Based on the preset constraint library, the original constraints contained in the original prompt text are expanded to obtain the extended prompt text; the extended reply text corresponding to the extended prompt text is obtained by using the third language processing model; and an extended data set is generated based on the extended prompt text and the extended reply text.

[0060] Exemplarily, the preset constraint class library includes multiple constraint classes. In the process of expanding the original constraints contained in the original prompt text based on the preset constraint class library, each constraint class in the preset constraint class library can be traversed, and then the original constraints contained in the original prompt text are matched and replaced, thereby obtaining the extended prompt text. The extended prompt text includes multiple extended constraints, and the extended constraints are fine-grained rules.

[0061] Specifically, first traverse each constraint class in the preset constraint class library, obtain the constraint rules therein, and then traverse each sentence or phrase in the original prompt text to check whether it contains the constraint rules in the preset constraint class library. If the original prompt text contains one or more constraint rules, replace or expand the constraints in the original prompt text according to the replacement rules of the constraint rules, and finally save the replaced or expanded prompt text as the extended prompt text. This can increase the richness and diversity of the prompt text, thereby improving user experience and understanding.

[0062] For example, the original prompt text is "Please introduce city A to me. The answer must be accurate." The extended prompt text after expansion is "Please introduce city A to me. The generated answer format needs to be a lightweight data exchange format (JSON). The character length of the data field in the JSON object must not exceed 1,000 characters. The generated answer must be an objective implementation example and cannot contain any subjective opinions or guesses. All characters need to be variable-length characters (UTF-8) encoded to ensure support for multi-language answers. The answer must not contain any personal privacy information, copyright-restricted content, or illegal information. The numbers and place names involved in the answer need to be specially marked in bold..."

[0063] After obtaining the extended prompt text, a third language processing model is used to obtain the extended reply text corresponding to the extended prompt text. Specifically, the third language processing model can be a pre-trained language model, such as GPT-3 or Bidirectional Encoder Representations from Transformers (BERT). The extended prompt text is input into the third language model, so as to perform rule mining on the extended prompt text and extract fine-grained rules in the extended prompt text. Through the generation capability of the third language processing model, a more detailed and rich extended reply text is generated according to the fine-grained rules to respond to the information or questions provided by the extended prompt text, thereby providing more comprehensive and in-depth information.

[0064] An extended data pair can be determined based on the extended prompt text and the corresponding extended reply text, and an extended data set can be generated using multiple extended data pairs to fine-tune the first language processing model, thereby obtaining a second language processing model.

[0065] Based on the above optional embodiment, by expanding the original constraints contained in the original prompt text based on the preset constraint library, an extended prompt text is obtained, so that the third language processing model can understand the user's intention more accurately. At the same time, by obtaining the extended reply text corresponding to the extended prompt text, a richer and more comprehensive data set can be constructed, thereby improving the performance and generalization ability of the machine learning model.

[0066] In an optional embodiment, the model training method in the embodiment of the present application further includes:

[0067] Clustering is performed on multiple known constraints to obtain multiple constraint classes, wherein each constraint class in the multiple constraint classes contains multiple constraint subclasses, and the multiple constraint classes include at least some of the following classes: content constraint class, scenario constraint class, style constraint class, format constraint class, and example constraint class; a preset constraint class library is constructed based on the multiple constraint classes.

[0068] The above-mentioned known constraints are a set of conditions or rules that are clearly known when processing data or problems. For example, when clustering multiple known constraints, constraints with high similarity can be classified into the same category to better understand the structure of the problem, to better understand the characteristics of the problem, and to improve the efficiency and accuracy of question responses.

[0069] Each of the above-mentioned multiple constraint classes includes multiple constraint subclasses, and the multiple constraint classes include at least some of the following classes: content constraint class, situation constraint class, style constraint class, format constraint class, and example constraint class.

[0070] Content constraints include, but are not limited to, topic constraints, exclusion constraints, inclusion constraints, value constraints, privacy constraints, and numerical constraints. Specifically, topic constraints require that the generated content should revolve around a specific topic or field, exclusion constraints explicitly point out the information or content that should not be included in the generated content, inclusion constraints explicitly require specific information or content that should be included in the generated content, and value constraints require that the generated content does not contain information that violates values, such as security, false information, bias, and other information. Privacy constraints require that the generated content does not contain details that may violate privacy, such as personal data or sensitive information. Numerical constraints limit the length and number of words, sentences, and paragraphs of the generated content or use numerical precision constraint instructions to ensure that they meet accuracy requirements.

[0071] Situational constraints include but are not limited to: role-playing constraints, audience target constraints, prior condition constraints, process background information constraints, table background information constraints, and text background information constraints. Specifically, role-playing constraints require that relevant information be generated based on specific roles or situational contexts. Audience target constraints require that the generated content is targeted at a specific audience group, which will affect the terminology used or the level of detail provided and the complexity of the generated content. Prior condition constraints require that when a certain intention is met, a certain process must be followed to perform a certain operation, output a certain specific content, etc. Process background information constraints assist in the generation of answers by adding process information in natural language form, such as some business processes and service processes. Table background information constraints refer to the background information existing in a table form, providing a series of table background information in lightweight markup language (Markdown) format to assist in the generation of answers. Text background information constraints refer to the background information existing in a text form, providing a series of text background information to assist in the generation of answers.

[0072] Style constraints include, but are not limited to: tone style constraints, emotion constraints, language feature constraints, and multilingual constraints. Specifically, tone style constraints require that the generated content adopt a specific tone and style, such as rigorous and formal, polite, professional and academic, concise, etc. Emotional and emotional constraints require that the generated content express specific emotions or emotions, such as ensuring that the content is positive, inspiring, or sympathetic. Language feature constraints use specific language features, such as the use of metaphors, personification and other rhetorical devices, or specific vocabulary and sentence structures. Multilingual constraints require generation in a certain language, or switching between languages ​​according to complex patterns.

[0073] Format constraints include but are not limited to: output format constraints, text style constraints, grammatical structure constraints, citation and reference constraints, numbering and list constraints, hierarchical structure constraints, and template constraints. Specifically, output format constraints require specific data formats, such as tables, JSON, Markdown, Hypertext Markup Language (HTML), etc. Text style constraints require the use of specified fonts and font sizes, or the use of special emoticons to ensure the readability of text on different devices and platforms. Grammatical structure constraints require that the generated content strictly follow specific grammatical structures, such as subject-column-table, subject-predicate, etc. Citation and reference constraints require that the generated content contain citations and references to sources, provide reliable sources and literature support, and require the use of specific citation formats or reference styles. Numbering and list constraints require that the generated content use numbered lists or unordered lists to organize information. Hierarchical structure constraints require that the generated content be organized according to a specific hierarchical structure, such as using titles and subtitles. Template constraints require that the generated content be generated according to a certain typesetting and layout, such as the way the text is aligned, the indentation style of the paragraph, etc., and also must comply with a certain structural template, such as total-specific, total-specific-total, etc.

[0074] Example constraints include but are not limited to: positive example constraints and negative example constraints. Among them, positive example constraints are used to provide examples that meet the requirements and require the template to be generated according to the examples, while negative example constraints are used to provide examples that do not meet the requirements and require the model to avoid generating answers of this type of example when generating.

[0075] Exemplarily, when constructing a preset constraint library, it is also necessary to consider the association between different constraint categories. For example, there may be a correlation between the content constraint class and the style constraint class, and a corresponding association needs to be established in the preset constraint library. Each constraint category is analyzed to determine its properties and characteristics, including the specific requirements of the constraint, the scope of application, the implementation method, etc. Furthermore, according to the properties and characteristics of the constraint category, corresponding models and rules are established for automatic detection and verification of the compliance of the constraints. After the preset constraint library is established, verification and improvement work needs to be carried out to ensure that the constraints in the library fully cover various situations and can be accurately and effectively applied to actual constraint detection and management.

[0076] Based on the above optional embodiments, building a preset constraint library can help users manage and apply constraints more effectively, and improve the reuse rate and accuracy of constraints. By clustering known constraints, constraints can be organized and classified more clearly, making the constraint library easier to find and use. At the same time, classifying constraints into different categories can help users better understand the meaning and function of constraints, and improve the efficiency and accuracy of users' use of constraints. In addition, a preset constraint library built based on multiple constraint classes can also provide users with a richer selection of constraints to meet the constraint design requirements under different scenarios and needs.

[0077] In an optional embodiment, based on a preset constraint library, the original constraints contained in the original prompt text are expanded to obtain an extended prompt text including:

[0078] A constraint query is performed on a preset constraint library to obtain a query result, wherein the query result is used to determine a constraint category to be used; and the original constraint included in the original prompt text is expanded according to the query result to obtain an extended prompt text.

[0079] For example, when querying the preset constraint library, you can enter a keyword or select a specific constraint category to query, and the query results can list the constraint information related to the input keyword or the selected category. After obtaining the query results, you can determine the constraint category to be used based on the constraint information of the query results, and check the description and examples of each constraint category to determine which category is most suitable for the current scenario.

[0080] Furthermore, based on the query results, the original constraints contained in the original prompt text may be expanded, for example, by adding more specific restrictions, providing more detailed guidance, or providing additional prompts.

[0081] Based on the above optional embodiment, by querying the preset constraint library, automatic identification and matching of constraints can be achieved, and the constraint category to be used is determined according to the query results, which can ensure accurate matching and identification of the constraints in the original prompt text, thereby improving the accuracy and efficiency of matching. Expanding the original constraints according to the query results can effectively expand the expression and semantic content of the constraints, and enrich the information content and accuracy of the constraints in the prompt text.

[0082] In an optional embodiment, using the third language processing model to obtain the extended reply text corresponding to the extended prompt text includes:

[0083] A third language processing model is used to extract extension constraints from the extension prompt text; and a third language processing model is used to perform natural language understanding on the extension prompt text and at least one of the extension constraints to obtain an extended reply text.

[0084] Specifically, when the third language processing model is used to extract the extension constraint from the extension prompt text, the third language processing model can be used to perform keyword extraction and semantic analysis. First, the third language processing model can be used to identify the keywords and phrases in the extension prompt text, and then the relationship between the keywords and phrases can be determined based on the semantic analysis, thereby extracting the extension constraint. For example, keywords such as action, time, and place in the extension prompt text can be identified, and then the constraint conditions can be inferred based on the relationship between the keywords.

[0085] For the natural language understanding of the text description of the extended prompt text and the extended constraints, a third language processing model can be used for semantic understanding and reasoning. First, the text description and the extended constraints are input into the third language processing model, which will perform semantic understanding and identify the information and constraints in the text description. Further, reasoning is performed based on the constraints to generate an extended reply text. For example, if the constraint is a time constraint, the third language processing model can generate a corresponding extended reply text based on the time relationship.

[0086] Based on the above optional embodiment, by using the third language processing model to extract the extension constraints from the extension prompt text, the constraints in the text can be more accurately identified and understood, thereby helping the model to better understand the user's needs. At the same time, by performing natural language understanding on the text description of the extension prompt text and the extension constraints, a more accurate and reasonable extension reply text can be generated, thereby improving the model performance.

[0087] In an optional embodiment, the model training method in the embodiment of the present application further includes:

[0088] Based on the extended reply text, constraint verification is performed on at least one constraint to obtain a verification result, wherein the verification result is used to evaluate the degree of compliance of the extended reply text with the at least one constraint; based on the verification result, the extended data set is screened to obtain a preference data set.

[0089] Specifically, when fine-grained verification is performed on at least one constraint based on the extended reply text, an automated tool or programming language may be selected to implement the verification. First, the constraint rules need to be converted into a computer-recognizable form, and then the extended reply text is input into the verification tool for verification. The verification result may show whether the extended reply text complies with the constraint rules, and the specific location or content that does not comply with the rules.

[0090] After obtaining the verification result, if the extended reply text meets at least one constraint, it can be added to the preference data set; if the extended reply text does not meet at least one constraint, it can be discarded or further processed, such as manual modification or automatic repair. The final preference data set will be a collection of extended reply texts that meet the constraint rules.

[0091] Based on the above optional embodiments, through the constraint verification and screening process, the degree of compliance of the extended reply text with the constraints can be evaluated, and the data that meets the constraints can be screened out from the extended data set to obtain the preference data set, thereby improving the quality and accuracy of the data, ensuring the integrity and consistency of the data, and thus improving the effectiveness and reliability of data analysis and application.

[0092] In an optional embodiment, the extended data set is screened according to the verification result to obtain the preferred data set including:

[0093] According to the verification result, multiple extended data pairs included in the extended data set are sorted to obtain a sorting result, wherein the multiple extended data pairs are sorted from high to low according to the compliance degree in the sorting result; based on the sorting result, the preferred data set is screened from the extended data set to obtain.

[0094] Specifically, multiple extended data pairs included in the extended data set are sorted according to the verification results, and are sorted from high to low according to the degree of compliance. The degree of compliance can be evaluated based on factors such as the credibility and relevance of the verification results, and the extended data pairs are arranged in sequence. For example, when the number of constraints followed by the extended reply text is more, the higher the degree of compliance is determined, and the higher the ranking in the sorting result is; when the number of constraints followed by the extended reply text is less, the lower the degree of compliance is determined, and the lower the ranking in the sorting result is.

[0095] Furthermore, based on the sorting result, the preferred data set is obtained by screening from the extended data set. The screening method can be selected according to specific needs. For example, the first few extended data pairs with the highest compliance degree can be selected to form the preferred data set, or a threshold can be set to only select extended data pairs with a compliance degree exceeding the threshold. In the screening process, data pairs can be selected one by one according to the sorting result until the screening condition is met.

[0096] Based on the above optional embodiment, by sorting the extended data pairs in the extended data set, the extended data pairs with a higher degree of compliance can be found, so as to more accurately screen out data with a stronger correlation to form a preference data set, which is used to improve the preference of the second language processing model, thereby further improving the model's compliance with diverse instructions.

[0097] In an optional embodiment, in step S23, the preference data set is used to perform preference improvement on the second language processing model to obtain the target language processing model, including: using a direct preference optimization method to control the second language processing model to perform preference learning on the preference data set to obtain the target language processing model.

[0098] Specifically, the direct preference optimization (DPO) method is used to control the second language processing model to perform preference learning on the preference data set to obtain the target language processing model. The DPO method is a method for optimizing the second language processing model by directly learning the preferences of the target language processing model.

[0099] For example, in the process of controlling the second language processing model to perform preference learning on the preference data set, the DPO method will first define an objective function, which can measure the performance of the second language processing model when processing the preference data set. Then, according to the definition of the objective function, the second language processing model is adjusted to continuously improve its performance when processing the preference data set. In this process, it can be achieved through optimization algorithms such as gradient descent, and the parameters of the model are continuously adjusted to gradually approach the target solution. The DPO method can adjust the preference of the second language processing model according to the definition of the objective function, so that it is more in line with the characteristics of the preference data set, thereby improving the performance of the model when processing the preference data set.

[0100] Based on the above optional embodiments, using a direct preference optimization method to control the second language processing model to perform preference learning on the preference data set can improve the performance and accuracy of the target language processing model, and help the second language processing model better understand the characteristics and rules in the preference data set, so as to better adapt to the processing requirements of the target language processing model, thereby effectively improving the performance of the target language processing model on various tasks and improving its effectiveness and efficiency in practical applications.

[0101] In practical applications, the fine-grained rule-following ability of large models can help the model better understand and process complex language, images, audio and other data, thereby improving the performance and accuracy of the model. In an embodiment of the present application, the original language processing model is pre-trained through the original data set so that it can learn the information and constraints in the original data pair, thereby establishing basic language processing capabilities. Then the pre-trained language processing model is fine-tuned through the extended data set, and extended constraints are introduced so that the model can better adapt to the problems in the target application scenario. Finally, the preference data set is used to improve the preference of the fine-tuned language processing model, further improving the model performance so that it can better understand and respond to user queries. By constructing millions of multi-constraint instruction data sets for SFT training, the model is allowed to learn to improve its compliance with diverse instructions; finally, preference data is used for human feedback reinforcement learning (RLHF) training, so that the target language processing model can follow the output constraints in a more fine-grained manner under multiple constraints.

[0102] Figure 3 is a schematic diagram of a model training method according to an embodiment of the present application, such as Figure 3 As shown, the original language processing model is pre-trained through the original data set so that it can learn the information and constraints in the original data pair, thereby establishing basic language processing capabilities. Then, the pre-trained language processing model is fine-tuned through the extended data set, and the extended constraints are introduced so that the model can better adapt to the problems in the target application scenario. Finally, the fine-tuned language processing model is improved by preference through the preference data set to further improve the model performance so that it can better understand and respond to user queries. Thus, through pre-training, fine-tuning and preference improvement, a target language processing model that performs better in the target application scenario can be obtained, which can more accurately understand the user's query and generate more appropriate responses, thereby improving the practicality and reliability of the model. Through continuous optimization and iteration, the performance of the model can be further improved so that it can better adapt to different application scenarios and needs, thereby achieving the technical effect of improving the model's compliance effect on diverse instructions and the model processing performance, thereby solving the technical problems of poor compliance effect of the model on diverse instructions and low model processing performance in related technologies.

[0103] Figure 4 is a flow chart of a data processing method according to an embodiment of the present application, such as Figure 4 As shown, the method may include the following steps:

[0104] Step S41, obtaining a query request, wherein the data carried in the query request includes: questions to be queried in the target application scenario;

[0105] Step S42, in response to the query request, using the target language processing model to perform natural language understanding on the query question and generate a target response, wherein the target language processing model is trained using the model training method of any one of the embodiments of the present application.

[0106] Based on the above steps S41 to S42, by obtaining the query questions in the target application scenario, and then using the target language processing model to perform natural language understanding on the query questions, and generating target replies, it is possible to accurately understand the user's intentions and generate accurate replies. Through pre-training, fine-tuning and preference improvement, a target language processing model that performs better in the target application scenario can be obtained, which can more accurately understand the user's query questions and generate more appropriate replies, thereby improving the practicality and reliability of the model. Through continuous optimization and iteration, the performance of the model can be further improved to better adapt to different application scenarios and needs, thereby achieving the technical effect of improving the model's compliance with diverse instructions and the model's processing performance, thereby solving the technical problems of poor model compliance with diverse instructions and low model processing performance in related technologies.

[0107] Figure 5 is a flow chart of another data processing method according to an embodiment of the present application, such as Figure 5 As shown, the method may include the following steps:

[0108] Step S51, obtaining a query request, wherein the data carried in the query request includes: intelligent customer service questions to be queried in the intelligent customer service scenario;

[0109] Step S52, in response to the query request, using the target language processing model to perform natural language understanding on the intelligent customer service question to be queried, and generating a target intelligent customer service reply, wherein the target language processing model is trained using the model training method of any one of the embodiments of the present application.

[0110] Based on the above steps S51 to S52, by obtaining the intelligent customer service questions to be queried in the intelligent customer service scenario, and then using the target language processing model to perform natural language understanding on the queried intelligent customer service questions, it is possible to accurately understand the user's intentions and generate accurate target intelligent customer service responses. Through pre-training, fine-tuning and preference improvement, a target language processing model that performs better in the target application scenario can be obtained, which can more accurately understand the user's query questions and generate more appropriate responses, thereby improving the practicality and reliability of the model. Through continuous optimization and iteration, the performance of the model can be further improved to better adapt to different application scenarios and needs, thereby achieving the technical effect of improving the model's compliance with diverse instructions and the model's processing performance, thereby solving the technical problems of poor model compliance with diverse instructions and low model processing performance in related technologies.

[0111] Figure 6 is a flow chart of another data processing method according to an embodiment of the present application, such as Figure 6 As shown, the method may include the following steps:

[0112] Step S61, obtaining a data processing request through a first application programming interface, wherein the request data carried in the data processing request includes: a question to be queried in a target application scenario;

[0113] Step S62, returning a data processing response through a second application programming interface, wherein the response data carried in the data processing response includes: a target reply, the target reply is generated by performing natural language understanding on the query question using a target language processing model, and the target language processing model is trained using the model training method of any one of the embodiments of the present application.

[0114] Based on the above steps S61 to S62, a data processing request is obtained through the first application programming interface, wherein the request data carried in the data processing request includes: the question to be queried in the target application scenario, and then a data processing response is returned through the second application programming interface, wherein the response data carried in the data processing response includes: a target reply, and the target reply is generated after natural language understanding of the query question using the target language processing model. Through pre-training, fine-tuning and preference improvement, a target language processing model that performs better in the target application scenario can be obtained, which can more accurately understand the user's query question and generate more appropriate replies, thereby improving the practicality and reliability of the model. Through continuous optimization and iteration, the performance of the model can be further improved, so that it can better adapt to different application scenarios and needs, thereby achieving the technical effect of improving the model's compliance with diverse instructions and the model processing performance, thereby solving the technical problems of poor compliance of the model with diverse instructions and low model processing performance in related technologies.

[0115] Figure 7 is a flow chart of another data processing method according to an embodiment of the present application, such as Figure 7 As shown, the method may include the following steps:

[0116] Step S71, obtaining the currently input data processing dialogue request, wherein the request data carried in the data processing dialogue request includes: the question to be queried in the target application scenario;

[0117] Step S72, in response to the data processing dialogue request, returning a data processing dialogue reply, wherein the information carried in the data processing dialogue reply includes: a target reply, the target reply is generated by using a target language processing model to perform natural language understanding on the query question, and the target language processing model is trained using any one of the model training methods in the embodiments of the present application;

[0118] Step S73, displaying the target response in the graphical user interface.

[0119] Based on the above steps S71 to S73, by obtaining the currently input data processing dialogue request, and then responding to the data processing dialogue request, returning the data processing dialogue reply, and finally displaying the target reply in the graphical user interface. Through pre-training, fine-tuning and preference improvement, a target language processing model that performs better in the target application scenario can be obtained, which can more accurately understand the user's query questions and generate more appropriate replies, thereby improving the practicality and reliability of the model. Through continuous optimization and iteration, the performance of the model can be further improved to better adapt to different application scenarios and needs, thereby achieving the technical effect of improving the model's compliance with diverse instructions and the model's processing performance, thereby solving the technical problems of poor model compliance with diverse instructions and low model processing performance in related technologies.

[0120] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0121] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0122] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0123] According to an embodiment of the present application, a model training device for implementing the above-mentioned model training method is also provided. Figure 8 is a structural block diagram of a model training device according to an embodiment of the present application, such as Figure 8 As shown, the device comprises:

[0124] A pre-training module 801 is used to pre-train an original language processing model using an original data set to obtain a first language processing model, wherein the original data set includes: a plurality of original data pairs, and original prompt texts in the plurality of original data pairs include original constraints;

[0125] A fine-tuning module 802 is used to fine-tune the first language processing model using an extended data set to obtain a second language processing model, wherein the extended data set includes: a plurality of extended data pairs, the extended prompt texts in the plurality of extended data pairs include extended constraints, and the extended constraints are obtained by expanding the original constraints;

[0126] Processing module 803 is used to use the preference data set to perform preference improvement on the second language processing model to obtain a target language processing model, wherein the preference data set includes: multiple preference data pairs, the multiple preference data pairs are extended data pairs in which the degree of compliance of the extended reply text with the extended constraints in the multiple extended data pairs meets the preset verification conditions, and the target language processing model is used to perform natural language understanding on the query questions in the target application scenario to generate the target reply.

[0127] Optionally, the model training device also includes: an expansion module 804, which is used to expand the original constraints contained in the original prompt text based on a preset constraint library to obtain an extended prompt text; an acquisition module 805, which is used to use a third language processing model to obtain an extended reply text corresponding to the extended prompt text; and a generation module 806, which is used to generate an extended data set based on the extended prompt text and the extended reply text.

[0128] Optionally, the model training device also includes: a clustering module 806, which is used to cluster multiple known constraints to obtain multiple constraint classes, wherein each constraint class in the multiple constraint classes respectively contains multiple constraint subclasses, and the multiple constraint classes include at least some of the following classes: content constraint class, scenario constraint class, style constraint class, format constraint class, example constraint class; a construction module 807, which is used to construct a preset constraint class library based on multiple constraint classes.

[0129] Optionally, the expansion module 804 is also used to: perform a constraint query on a preset constraint library to obtain a query result, wherein the query result is used to determine the constraint category to be used; and expand the original constraint contained in the original prompt text according to the query result to obtain an extended prompt text.

[0130] Optionally, the acquisition module 805 is further used to: extract the extension constraint from the extension prompt text using a third language processing model; and perform natural language understanding on the extension prompt text and at least one of the extension constraints using the third language processing model to obtain an extended reply text.

[0131] Optionally, the model training device also includes: a verification module 808, which is used to perform constraint verification on at least one constraint based on the extended reply text to obtain a verification result, wherein the verification result is used to evaluate the degree of compliance of the extended reply text with at least one constraint; a screening module 809, which is used to screen the extended data set based on the verification result to obtain a preference data set.

[0132] Optionally, the screening module 809 is further used to: sort the multiple extended data pairs contained in the extended data set according to the verification result to obtain a sorting result, wherein the multiple extended data pairs are sorted from high to low according to the compliance degree in the sorting result; and screen the extended data set based on the sorting result to obtain the preferred data set.

[0133] Optionally, the processing module 803 is further used to: adopt a direct preference optimization method to control the second language processing model to perform preference learning on the preference data set to obtain a target language processing model.

[0134] It should be noted that the above-mentioned pre-training module 801, fine-tuning module 802 and processing module 803 correspond to steps S21 to S23 in the above-mentioned embodiment, and the three modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned embodiment. It should be noted that the above-mentioned modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above-mentioned modules can also be run in the server provided in the above-mentioned embodiment as part of the device.

[0135] Fig. 9 is a structural block diagram of a data processing device according to an embodiment of the present application, such as Fig. 9 As shown, the method may include the following steps:

[0136] The acquisition module 901 is used to acquire a query request, wherein the data carried in the query request includes: a question to be queried in a target application scenario;

[0137] The generation module 902 is used to respond to the query request, use the target language processing model to perform natural language understanding on the query question, and generate a target reply, wherein the target language processing model is trained using the model training method of any one of the embodiments of the present application.

[0138] It should be noted that the acquisition module 901 and the generation module 902 correspond to steps S41 to S42 in the above embodiment, and the examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be run in the server provided in the above embodiment as part of the device.

[0139] Fig.10 is a flow chart of another data processing method according to an embodiment of the present application, such as Fig.10 As shown, the method may include the following steps:

[0140] The acquisition module 1001 is used to acquire a query request, wherein the data carried in the query request includes: intelligent customer service questions to be queried in the intelligent customer service scenario;

[0141] The generation module 1002 is used to respond to the query request, use the target language processing model to perform natural language understanding on the intelligent customer service question to be queried, and generate a target intelligent customer service reply, wherein the target language processing model is trained using the model training method of any one of the embodiments of the present application.

[0142] It should be noted that the acquisition module 1001 and the generation module 1002 correspond to steps S51 to S52 in the above embodiment, and the examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be run in the server provided in the above embodiment as part of the device.

[0143] Fig.11 is a flow chart of another data processing method according to an embodiment of the present application, such as Fig.11 As shown, the method may include the following steps:

[0144] The acquisition module 1101 is used to acquire a data processing request through a first application programming interface, wherein the request data carried in the data processing request includes: a question to be queried in a target application scenario;

[0145] The return module 1102 is used to return a data processing response through a second application programming interface, wherein the response data carried in the data processing response includes: a target reply, which is generated by performing natural language understanding on the query question using a target language processing model, and the target language processing model is trained using the model training method of any one of the embodiments of the present application.

[0146] It should be noted that the acquisition module 1101 and the return module 1102 correspond to steps S61 to S62 in the above embodiment, and the examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be run in the server 10 provided in the above embodiment as part of the device.

[0147] Fig.12 is a flow chart of another data processing method according to an embodiment of the present application, such as Fig.12 As shown, the method may include the following steps:

[0148] The acquisition module 1201 is used to acquire the currently input data processing dialogue request, wherein the request data carried in the data processing dialogue request includes: the question to be queried in the target application scenario;

[0149] A return module 1202 is used to respond to the data processing dialogue request and return a data processing dialogue reply, wherein the information carried in the data processing dialogue reply includes: a target reply, which is generated by using a target language processing model to perform natural language understanding on the query question, and the target language processing model is trained using any one of the model training methods in the embodiments of the present application;

[0150] The display module 1203 is used to display the target response in the graphical user interface.

[0151] It should be noted that the acquisition module 1201, the return module 1202 and the display module 1203 correspond to steps S71 to S73 in the above embodiment, and the three modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be part of the device and can run in the server provided in the above embodiment.

[0152] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in the above embodiments, as well as the application scenario and implementation process, but is not limited to the scheme provided in the above embodiments.

[0153] The embodiment of the present application may provide an electronic device, which may be any electronic device in a group of electronic devices. Optionally, in this embodiment, the electronic device may also be replaced by a terminal device such as a mobile terminal.

[0154] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0155] In this embodiment, the computer terminal can execute the program code in the method.

[0156] Optionally, Fig.13 1 is a block diagram of an electronic device according to an embodiment of the present application. As shown in the figure, the electronic device may include: one or more (only one is shown in the figure) processors 132, a memory 134, a storage controller, and a peripheral interface, wherein the peripheral interface is connected to a radio frequency module, an audio module, and a display.

[0157] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implementing the methods in the above embodiments. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0158] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: pre-training the original language processing model with the original data set to obtain a first language processing model, wherein the original data set includes: multiple original data pairs, and the original prompt texts in the multiple original data pairs contain original constraints; fine-tuning the first language processing model with the extended data set to obtain a second language processing model, wherein the extended data set includes: multiple extended data pairs, and the extended prompt texts in the multiple extended data pairs contain extended constraints, and the extended constraints are obtained by expanding the original constraints; using the preference data set to improve the preferences of the second language processing model to obtain a target language processing model, wherein the preference data set includes: multiple preference data pairs, and the multiple preference data pairs are extended data pairs in which the degree of compliance of the extended reply texts with the extended constraints in the multiple extended data pairs meets the preset verification conditions, and the target language processing model is used to perform natural language understanding on the questions to be queried in the target application scenario to generate target replies.

[0159] Optionally, the processor may also execute the program code of the following steps: based on a preset constraint library, expanding the original constraints contained in the original prompt text to obtain an extended prompt text; using a third language processing model to obtain an extended reply text corresponding to the extended prompt text; and generating an extended data set based on the extended prompt text and the extended reply text.

[0160] Optionally, the processor may also execute program code of the following steps: clustering multiple known constraints to obtain multiple constraint classes, wherein each constraint class in the multiple constraint classes respectively contains multiple constraint subclasses, and the multiple constraint classes include at least some of the following classes: content constraint class, scenario constraint class, style constraint class, format constraint class, example constraint class; and constructing a preset constraint class library based on the multiple constraint classes.

[0161] Optionally, the processor may also execute the program code of the following steps: performing a constraint query on a preset constraint library to obtain a query result, wherein the query result is used to determine a constraint category to be used; and expanding the original constraint contained in the original prompt text based on the query result to obtain an extended prompt text.

[0162] Optionally, the processor may also execute program code of the following steps: extracting extension constraints from the extension prompt text using a third language processing model; performing natural language understanding on the extension prompt text and at least one of the extension constraints using the third language processing model to obtain an extended reply text.

[0163] Optionally, the processor may also execute the program code of the following steps: performing constraint verification on at least one constraint based on the extended reply text to obtain a verification result, wherein the verification result is used to evaluate the degree of compliance of the extended reply text with at least one constraint; and screening the extended data set based on the verification result to obtain a preference data set.

[0164] Optionally, the processor may also execute program code of the following steps: sorting multiple extended data pairs contained in the extended data set according to the verification result to obtain a sorting result, wherein the multiple extended data pairs are sorted from high to low according to the degree of compliance in the sorting result; and filtering the extended data set to obtain a preferred data set based on the sorting result.

[0165] Optionally, the processor may also execute program code of the following steps: using a direct preference optimization method to control the second language processing model to perform preference learning on the preference data set to obtain a target language processing model.

[0166] Optionally, the processor may also execute the program code of the following steps: obtaining a query request, wherein the data carried in the query request includes: questions to be queried in a target application scenario; in response to the query request, using a target language processing model to perform natural language understanding on the questions to be queried and generate a target reply, wherein the target language processing model is trained using the model training method of any one of the embodiments of the present application.

[0167] Optionally, the processor may also execute the program code of the following steps: obtaining a query request, wherein the data carried in the query request includes: intelligent customer service questions to be queried in the intelligent customer service scenario; in response to the query request, using a target language processing model to perform natural language understanding on the intelligent customer service questions to be queried, and generating a target intelligent customer service reply, wherein the target language processing model is trained using the model training method of any one of the embodiments of the present application.

[0168] Optionally, the processor may also execute the program code of the following steps: obtaining a data processing request through a first application programming interface, wherein the request data carried in the data processing request includes: the question to be queried in the target application scenario; returning a data processing response through a second application programming interface, wherein the response data carried in the data processing response includes: a target reply, wherein the target reply is generated after natural language understanding of the question to be queried using a target language processing model, and the target language processing model is trained using the model training method of any one of the embodiments of the present application.

[0169] Optionally, the processor may also execute program code for the following steps: obtaining a currently input data processing dialogue request, wherein the request data carried in the data processing dialogue request includes: the question to be queried in the target application scenario; returning a data processing dialogue reply in response to the data processing dialogue request, wherein the information carried in the data processing dialogue reply includes: a target reply, the target reply is generated by performing natural language understanding on the question to be queried using a target language processing model, and the target language processing model is trained using the model training method of any one of the embodiments of the present application; and displaying the target reply in a graphical user interface.

[0170] According to the embodiment of the present application, the original language processing model is pre-trained by the original data set, so that it can learn the information and constraints in the original data pair, thereby establishing a basic language processing capability. Then, the pre-trained language processing model is fine-tuned by the extended data set, and the extended constraints are introduced so that the model can better adapt to the problems in the target application scenario. Finally, the fine-tuned language processing model is improved by the preference data set, and the model performance is further improved so that it can better understand and reply to the user's query questions. Thus, through the methods of pre-training, fine-tuning and preference improvement, a target language processing model that performs better in the target application scenario can be obtained, which can more accurately understand the user's query questions and generate more appropriate replies, thereby improving the practicality and reliability of the model. Through continuous optimization and iteration, the performance of the model can be further improved, so that it can better adapt to different application scenarios and needs, thereby achieving the technical effect of improving the model's compliance effect on diversity instructions and the model processing performance, thereby solving the technical problems of poor compliance effect of the model on diversity instructions and low model processing performance in the related technology.

[0171] It is understood by those skilled in the art that the structure shown in the figure is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, and other terminal devices. The figure does not limit the structure of the above-mentioned electronic device. For example, the electronic device may also include more or fewer components (such as a network interface, a display device, etc.) than those shown in the figure, or have a configuration different from that shown in the figure.

[0172] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0173] The embodiment of the present application further provides a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the method provided in the above embodiment.

[0174] Optionally, in this embodiment, the above storage medium may be located in any electronic device in a group of electronic devices in a computer network, or in any mobile terminal in a group of mobile terminals.

[0175] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: pre-training an original language processing model using an original data set to obtain a first language processing model, wherein the original data set includes: multiple original data pairs, and the original prompt texts in the multiple original data pairs contain original constraints; fine-tuning the first language processing model using an extended data set to obtain a second language processing model, wherein the extended data set includes: multiple extended data pairs, and the extended prompt texts in the multiple extended data pairs contain extended constraints, and the extended constraints are obtained by expanding the original constraints; performing preference improvement on the second language processing model using a preference data set to obtain a target language processing model, wherein the preference data set includes: multiple preference data pairs, and the multiple preference data pairs are extended data pairs in which the degree of compliance of the extended reply texts in the multiple extended data pairs with the extended constraints meets preset verification conditions, and the target language processing model is used to perform natural language understanding on the questions to be queried in the target application scenario to generate target replies.

[0176] Optionally, the computer-readable storage medium is also configured to store program code for executing the following steps: based on a preset constraint library, expanding the original constraints contained in the original prompt text to obtain an extended prompt text; using a third language processing model to obtain an extended reply text corresponding to the extended prompt text; and generating an extended data set based on the extended prompt text and the extended reply text.

[0177] Optionally, the computer-readable storage medium is also configured to store program code for executing the following steps: clustering multiple known constraints to obtain multiple constraint classes, wherein each constraint class in the multiple constraint classes respectively contains multiple constraint subclasses, and the multiple constraint classes include at least some of the following classes: content constraint class, scenario constraint class, style constraint class, format constraint class, example constraint class; and constructing a preset constraint class library based on the multiple constraint classes.

[0178] Optionally, the computer-readable storage medium is also configured to store program code for executing the following steps: performing a constraint query on a preset constraint library to obtain a query result, wherein the query result is used to determine a constraint category to be used; and expanding the original constraints contained in the original prompt text based on the query result to obtain an extended prompt text.

[0179] Optionally, the computer-readable storage medium is also configured to store program code for executing the following steps: extracting extended constraints from the extended prompt text using a third language processing model; performing natural language understanding on the extended prompt text and at least one of the extended constraints using the third language processing model to obtain an extended reply text.

[0180] Optionally, the computer-readable storage medium is also configured to store program code for executing the following steps: performing constraint verification on at least one constraint based on the extended reply text to obtain a verification result, wherein the verification result is used to evaluate the degree of compliance of the extended reply text with the at least one constraint; and screening the extended data set based on the verification result to obtain a preference data set.

[0181] Optionally, the computer-readable storage medium is also configured to store program code for executing the following steps: sorting multiple extended data pairs contained in the extended data set according to the verification result to obtain a sorting result, wherein the multiple extended data pairs are sorted from high to low according to the degree of compliance in the sorting result; and filtering the extended data set based on the sorting result to obtain a preference data set.

[0182] Optionally, the computer-readable storage medium is further configured to store program code for executing the following steps: using a direct preference optimization method, controlling the second language processing model to perform preference learning on the preference data set to obtain a target language processing model.

[0183] Optionally, the computer-readable storage medium is also configured to store program code for executing the following steps: obtaining a query request, wherein the data carried in the query request includes: the question to be queried in the target application scenario; in response to the query request, using a target language processing model to perform natural language understanding on the question to be queried and generate a target reply, wherein the target language processing model is trained using the model training method of any one of the embodiments of the present application.

[0184] Optionally, the computer-readable storage medium is also configured to store program code for executing the following steps: obtaining a query request, wherein the data carried in the query request includes: intelligent customer service questions to be queried in the intelligent customer service scenario; in response to the query request, using a target language processing model to perform natural language understanding on the intelligent customer service questions to be queried, and generating a target intelligent customer service reply, wherein the target language processing model is trained using the model training method of any one of the embodiments of the present application.

[0185] Optionally, the computer-readable storage medium is also configured to store program code for performing the following steps: obtaining a data processing request through a first application programming interface, wherein the request data carried in the data processing request includes: the question to be queried in the target application scenario; returning a data processing response through a second application programming interface, wherein the response data carried in the data processing response includes: a target reply, the target reply is generated after natural language understanding of the query question using a target language processing model, and the target language processing model is trained using the model training method of any one of the embodiments of the present application.

[0186] Optionally, the computer-readable storage medium is also configured to store program code for executing the following steps: obtaining a currently input data processing dialogue request, wherein the request data carried in the data processing dialogue request includes: the question to be queried in the target application scenario; in response to the data processing dialogue request, returning a data processing dialogue reply, wherein the information carried in the data processing dialogue reply includes: a target reply, the target reply is generated after natural language understanding of the query question using a target language processing model, and the target language processing model is trained using the model training method of any one of the embodiments of the present application; and displaying the target reply in a graphical user interface.

[0187] According to the embodiment of the present application, the original language processing model is pre-trained by the original data set, so that it can learn the information and constraints in the original data pair, thereby establishing a basic language processing capability. Then, the pre-trained language processing model is fine-tuned by the extended data set, and the extended constraints are introduced so that the model can better adapt to the problems in the target application scenario. Finally, the fine-tuned language processing model is improved by the preference data set, and the model performance is further improved so that it can better understand and reply to the user's query questions. Thus, through the methods of pre-training, fine-tuning and preference improvement, a target language processing model that performs better in the target application scenario can be obtained, which can more accurately understand the user's query questions and generate more appropriate replies, thereby improving the practicality and reliability of the model. Through continuous optimization and iteration, the performance of the model can be further improved, so that it can better adapt to different application scenarios and needs, thereby achieving the technical effect of improving the model's compliance effect on diversity instructions and the model processing performance, thereby solving the technical problems of poor compliance effect of the model on diversity instructions and low model processing performance in the related technology.

[0188] The embodiment of the present application further provides a computer program product. Optionally, in this embodiment, the computer program product may include a computer program, and the computer program implements the method provided in the embodiment when executed by a processor.

[0189] The embodiments of the present application also provide a computer program product. Optionally, the computer program product may include a non-volatile computer-readable storage medium, which may be used to store a computer program, and when the computer program is executed by a processor, the method provided in the embodiments is implemented.

[0190] The embodiment of the present application further provides a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, the method provided in the above embodiment is implemented.

[0191] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0192] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0193] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0194] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0195] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc., which can store program code.

[0196] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A model training method, characterized in that: include: Pre-training an original language processing model using an original data set to obtain a first language processing model, wherein the original data set includes: a plurality of original data pairs, and original prompt texts in the plurality of original data pairs include original constraints; The first language processing model is fine-tuned using an extended data set to obtain a second language processing model, wherein the extended data set includes: a plurality of extended data pairs, wherein the extended prompt texts in the plurality of extended data pairs include extended constraints, and the extended constraints are obtained by expanding the original constraints; The preference data set is used to perform preference improvement on the second language processing model to obtain a target language processing model, wherein the preference data set includes: a plurality of preference data pairs, the plurality of preference data pairs are extended data pairs in which the degree of compliance of the extended reply text with the extended constraint meets a preset verification condition, and the target language processing model is used to perform natural language understanding on the query question in the target application scenario to generate a target reply; Among them, the model training method also includes: based on a preset constraint library, expanding the original constraints contained in the original prompt text to obtain the extended prompt text; using a third language processing model to obtain the extended reply text corresponding to the extended prompt text; and generating the extended data set based on the extended prompt text and the extended reply text.

2. The model training method according to claim 1, characterized in that: The model training method also includes: Clustering the multiple known constraints to obtain multiple constraint classes, wherein each constraint class in the multiple constraint classes respectively includes multiple constraint subclasses, and the multiple constraint classes include at least some of the following classes: content constraint class, scenario constraint class, style constraint class, format constraint class, and example constraint class; The preset constraint class library is constructed based on the multiple constraint classes.

3. The model training method according to claim 1 or 2, characterized in that: Based on the preset constraint library, the original constraint contained in the original prompt text is expanded to obtain the extended prompt text including: Performing a constraint query on the preset constraint library to obtain a query result, wherein the query result is used to determine a constraint category to be used; The original constraint contained in the original prompt text is expanded according to the query result to obtain the extended prompt text.

4. The model training method according to claim 1, characterized in that: Using the third language processing model to obtain the extended reply text corresponding to the extended prompt text includes: extracting the extension constraint from the extension prompt text using the third language processing model; The third language processing model is used to perform natural language understanding on the extended prompt text and at least one of the extended constraints to obtain the extended reply text.

5. The model training method according to claim 1, characterized in that: The model training method also includes: Perform constraint verification on at least one of the extended constraints based on the extended reply text to obtain a verification result, wherein the verification result is used to evaluate the degree of compliance of the extended reply text with the at least one constraint; The extended data set is screened according to the verification result to obtain the preference data set.

6. The model training method according to claim 5, characterized in that: The extended data set is screened according to the verification result to obtain the preference data set including: Sorting the multiple extended data pairs contained in the extended data set according to the verification result to obtain a sorting result, wherein the multiple extended data pairs are sorted from high to low according to the compliance degree in the sorting result; The preference data set is obtained by filtering the extended data set based on the sorting result.

7. The model training method according to claim 1, characterized in that: Using the preference data set to perform preference improvement on the second language processing model to obtain the target language processing model includes: A direct preference optimization method is adopted to control the second language processing model to perform preference learning on the preference data set to obtain the target language processing model.

8. A data processing method, characterized in that: include: Obtaining a query request, wherein the data carried in the query request includes: a question to be queried in a target application scenario; In response to the query request, a target language processing model is used to perform natural language understanding on the question to be queried to generate a target response, wherein the target language processing model is trained using the model training method described in any one of claims 1 to 7.

9. A data processing method, characterized in that: include: Obtaining a query request, wherein the data carried in the query request includes: intelligent customer service questions to be queried in an intelligent customer service scenario; In response to the query request, a target language processing model is used to perform natural language understanding on the intelligent customer service question to be queried, and a target intelligent customer service reply is generated, wherein the target language processing model is trained using the model training method described in any one of claims 1 to 7.

10. A data processing method, characterized in that: include: Obtaining a data processing request through a first application programming interface, wherein the request data carried in the data processing request includes: a question to be queried in a target application scenario; A data processing response is returned through a second application programming interface, wherein the response data carried in the data processing response includes: a target reply, wherein the target reply is generated by performing natural language understanding on the question to be queried using a target language processing model, and the target language processing model is trained using the model training method described in any one of claims 1 to 7.

11. A data processing method, characterized in that: include: Acquire a currently input data processing dialogue request, wherein the request data carried in the data processing dialogue request includes: a question to be queried in a target application scenario; In response to the data processing dialogue request, a data processing dialogue reply is returned, wherein the information carried in the data processing dialogue reply includes: a target reply, the target reply is generated by performing natural language understanding on the query question using a target language processing model, and the target language processing model is trained using the model training method according to any one of claims 1 to 7; The target response is displayed within a graphical user interface.

12. An electronic device, characterized in that: include: A memory storing an executable program; A processor, used to run the program, wherein when the program is run, the model training method described in any one of claims 1 to 7 or the data processing method described in any one of claims 8 to 11 is executed.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein, when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the model training method described in any one of claims 1 to 7 or the data processing method described in any one of claims 8 to 11.

14. A computer program product, characterized in that It includes a computer program which, when executed by a processor, implements the model training method described in any one of claims 1 to 7 or the data processing method described in any one of claims 8 to 11.

Images