Model training method and device, data processing method and device, equipment, medium and program product

Abstract

The embodiment of the invention provides a model training method and device, a data processing method and device, equipment, a medium and a program product, which are used for improving the accuracy of a model and realizing that the model does not collapse in a multi-round self-alignment process. The method can be applied to artificial intelligence. Comprising the steps of obtaining N training question and answer data sets, question example sets, verification sets and initial models; training the initial model by using a first training question and answer data set and the question example set to obtain a first model, wherein the first training question and answer data set is included in N training question and answer data sets; performing verification processing on the first model by using the verification set to obtain a collapse probability value; when the collapse probability value is smaller than a threshold value, training the first model by using a second training question and answer data set different from the first training question and answer data set and a question example set to obtain a second model; performing verification processing on the second model by using the verification set to update the collapse probability value; and repeating the operation until the collapse probability value is greater than or equal to a threshold value, and outputting the target model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a model training, data processing method, device, equipment, medium and program product. Background Art

[0002] In recent years, the continuous development and progress of artificial intelligence technology has brought great convenience to people's lives. Among them, the application of large language models based on artificial intelligence technology is particularly extensive. For example, they are used in various chat robots, smart assistants, and various content generation products. The wide application of large language models has improved people's experience in knowledge acquisition in multiple fields.

[0003] As users have increasing requirements for the accuracy of the answers output by large language models and the relevance of the answers to the corresponding questions, it is urgent to try to tap the model's own capabilities during the model training process to improve the accuracy of the large language model, thereby making the answers output by the large language model of higher quality. Summary of the invention

[0004] The embodiments of the present application provide a model training, data processing method, device, equipment, medium and program product for improving the accuracy of the model and ensuring that the model does not collapse during multiple rounds of self-alignment.

[0005] In view of this, on the one hand, the present application provides a model training method, including: obtaining N training question and answer data sets, a question example set, a verification set and an initial model, wherein each training question and answer data set has different training question and answer data, and the question example set includes M question examples, and N and M are positive integers; using the first training question and answer data set and the question example set to train the initial model to obtain a first model, wherein the first training question and answer data set is one of the N training question and answer data sets; using the verification set to verify the first model to obtain a collapse probability value; when the collapse probability value is less than a threshold, using the second training question and answer data set and the question example set to train the first model to obtain a second model, wherein the second training question and answer data set is a data set in the N training subsets that is different from the first training question and answer data set, and the perplexity of the second training question and answer data set is higher than the perplexity of the first training question data set; using the verification set to verify the second model to update the collapse probability value; repeating the above operations until the collapse probability value of the target model on the verification set is greater than or equal to the threshold, and outputting the target model.

[0006] On the other hand, the present application provides a model training device, including: an acquisition module, used to acquire N training question and answer data sets, a question example set, a verification set and an initial model, wherein each training question and answer data set has different training question and answer data, the question example set includes M question examples, and the N and the M are positive integers;

[0007] A processing module is used to train the initial model using a first training question and answer data set and the problem example set to obtain a first model, wherein the first training question and answer data set is one of the N training question and answer data sets; use the verification set to verify the first model to obtain a collapse probability value; when the collapse probability value is less than a threshold, use the second training question and answer data set and the problem example set to train the first model to obtain a second model, wherein the second training question and answer data set is a data set in the N training subsets that is different from the first training question and answer data set, and the perplexity of the second training question and answer data set is higher than the perplexity of the first training question data set; use the verification set to verify the second model to update the collapse probability value; repeat the above operations until the collapse probability value of the target model on the verification set is greater than or equal to the threshold, and output the target model.

[0008] In one possible design, in another implementation of another aspect of the embodiment of the present application, the acquisition module is used to acquire X question and answer data and the perplexity of the X question and answer data, where X is a positive integer greater than N;

[0009] According to the perplexity, the X question-answer data are sorted in order from easy to difficult to obtain a sorted data set;

[0010] The sorted data set is divided and processed in sequence according to the X / Y rule to obtain the N training question and answer data sets, where Y is used to indicate the number of training question and answer data in each of the N training question and answer data sets, and Y is a positive integer.

[0011] In a possible design, in another implementation of another aspect of the embodiment of the present application, the acquisition module is used to acquire an initial question example set, where the initial question example set includes M initial question examples;

[0012] The M initial question examples are modified according to preset rules to obtain the question example set, where the preset rules include at least one of increasing the richness of the question examples and reducing the rejected examples in the question example set.

[0013] In one possible design, in another implementation of another aspect of the embodiment of the present application, the acquisition module is used to modify the M initial question examples according to a rule of increasing information type, information completeness, information detail, and information timeliness of the questions to obtain the question example set;

[0014] or,

[0015] The number of rejection examples in the M initial question examples is reduced, and the target rejection examples are modified according to the rules of increasing the information type, information completeness, information detail and information timeliness of the questions to obtain the question example set;

[0016] or,

[0017] The M initial question examples are modified according to the rules of increasing the information type, information completeness, information details and information timeliness of the questions to obtain an intermediate question example set;

[0018] The number of rejected examples in the intermediate problem example set is reduced to obtain the M problem example set.

[0019] In one possible design, in another implementation of another aspect of the embodiment of the present application, the processing module is used to randomly sample S question examples from the M question example set to obtain a first question example subset, where S is a positive integer less than M;

[0020] Selecting the first training question-answering data set from the N training question-answering data sets in order from easy to difficult;

[0021] Selecting a first training problem data subset from the first training problem data set in order from easy to difficult;

[0022] Calling the first training problem data subset and the first problem example subset to train the initial model to obtain a first intermediate model;

[0023] Randomly sampling S problem examples from the M problem example set to obtain a second problem example subset, and selecting a second training problem data subset from the first training problem data set in order from easy to difficult, wherein the perplexity of each training problem data in the second training problem data subset is higher than the perplexity of each training problem data in the first training problem data subset;

[0024] Training the first intermediate model using the second training problem data subset and the second problem example subset to obtain a second intermediate model;

[0025] Repeat the above steps until the first training problem data set is processed, and then output the first model.

[0026] In one possible design, in another implementation of another aspect of the embodiment of the present application, the processing module is used to call the initial model to perform prediction processing on the first training question data subset based on the first question example subset to obtain a first prediction answer subset;

[0027] Modifying the first prediction answer subset according to the principles of usefulness, honesty and uselessness to obtain a second prediction answer subset;

[0028] The parameters of the initial model are adjusted according to the second predicted answer subset to obtain the first intermediate model.

[0029] In one possible design, in another implementation of another aspect of the embodiment of the present application, the processing module is used to calculate the loss value according to the second predicted answer subset, the first question example subset and the first training question data subset;

[0030] The parameters of the initial model are adjusted according to the loss value to obtain the first intermediate model.

[0031] In one possible design, in another implementation of another aspect of the embodiment of the present application, the processing module is used to call the first model to perform prediction processing on the verification set to obtain the output probability of the terminator of the response data corresponding to the verification set, and use the output probability as the collapse probability value.

[0032] In one possible design, in another implementation of another aspect of the embodiment of the present application, the initial model is a pre-trained large language model.

[0033] In one possible design, in another implementation of another aspect of the embodiment of the present application, the processing module is used to output the first model as the target model when the collapse probability value is greater than or equal to the threshold.

[0034] Another aspect of the present application provides a data processing method, comprising: obtaining data to be processed;

[0035] Calling the target model to perform prediction processing on the data to be processed to obtain the response data corresponding to the data to be processed, the target model is trained based on the above method;

[0036] The reply data is output.

[0037] On the other hand, the present application provides a data processing device, including: an acquisition module, used to acquire data to be processed;

[0038] A processing module is used to call a target model to perform prediction processing on the data to be processed to obtain response data corresponding to the data to be processed, and the target model is trained based on the above method;

[0039] The output module is used for outputting the reply data.

[0040] Another aspect of the present application provides a computer device, including: a memory, a processor, and a bus system;

[0041] Wherein, the memory is used to store programs;

[0042] The processor is used to execute the program in the memory, and the processor is used to execute the above-mentioned methods according to the instructions in the program code;

[0043] The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

[0044] Another aspect of the present application provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer is enabled to execute the above-mentioned methods.

[0045] Another aspect of the present application provides a computer program product or a computer program, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided by the above aspects.

[0046] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages: multiple rounds of self-alignment training are performed on the model using different training question and answer data sets, so that the generation ability and classification ability of the target model obtained by training are continuously improved with the rounds, and at the same time, each training question and answer data set is also ranked from easy to difficult as a whole, so that in the training process, the model does not face difficult problems in the early stage, thereby reducing the accumulation of errors in the model and improving the accuracy of the model. At the same time, in the self-alignment process, the validation set is used to analyze whether the model needs to stop training in advance, thereby preventing the model from collapsing in the later stage. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a schematic diagram of an exemplary architecture of a model-based question-answering system in an embodiment of the present application;

[0048] Figure 2 This is a schematic diagram of an embodiment of the model training method in the embodiment of the present application;

[0049] Figure 3A schematic diagram of the architecture of the model training method in the embodiment of the present application;

[0050] Figure 4 This is a schematic diagram of an embodiment of a data processing method in an embodiment of the present application;

[0051] Figure 5 This is a schematic diagram of an embodiment of a model training device in an embodiment of the present application;

[0052] Figure 6 This is a schematic diagram of an embodiment of a data processing device in an embodiment of the present application;

[0053] Figure 7 This is a schematic diagram of an embodiment of a server in an embodiment of the present application;

[0054] Figure 8 This is a schematic diagram of an embodiment of a terminal device in an embodiment of the present application. DETAILED DESCRIPTION

[0055] The embodiments of the present application provide a model training, data processing method, device, equipment, medium and program product for improving the accuracy of the model and ensuring that the model does not collapse during multiple rounds of self-alignment.

[0056] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "corresponding to" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0057] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0058] In recent years, the continuous development and progress of artificial intelligence technology has brought great convenience to people's lives. Among them, the application of large language models based on artificial intelligence technology is particularly extensive. For example, they are used in various chat robots, intelligent assistants, and various content generation products. The wide application of large language models has improved people's experience in knowledge acquisition in multiple fields. As users' requirements for the accuracy of the answers output by large language models and the relevance of answers to corresponding questions continue to increase, large language models usually need to align the requirements of large language models and model application scenarios during training. When aligning large language models with model requirements, a large amount of manually annotated data is required, and the acquisition of high-quality and large data sets is very expensive. Therefore, a self-alignment solution is usually used to train large language models so that they can be applied to specific application scenarios. Therefore, the goal of self-alignment is to fully explore the knowledge that the large language model itself has but is difficult to express, so as to reduce the cost of manual annotation as much as possible while ensuring the reliability of the model capability. Existing self-alignment methods mainly explore from three aspects: (1) try to make the model generate high-quality data by itself; (2) make full use of the existing high-quality corpus and let the model label these corpora with questions; (3) on the basis of the existing, fully explore the model's ability to output high-quality corpora. The above methods have a common feature: they aim to generate high-quality data through the pre-trained model and then directly perform supervised fine-tuning from the pre-trained model. In this way, the self-alignment process is usually only performed once based on the corpus information, so the model's capabilities are not fully explored.

[0059] In order to solve this technical problem, the present application provides the following technical problem: obtaining N training question and answer data sets, question example sets, verification sets and initial models, wherein each training question and answer data set has different training question and answer data, and the question example set includes M question examples, where N and M are positive integers; using the first training question and answer data set and the question example set to train the initial model to obtain a first model, wherein the first training question and answer data set is one of the N training question and answer data sets; using the verification set to verify the first model to obtain a collapse probability value; when the collapse probability value is less than a threshold, using the second training question and answer data set and the question example set to train the first model to obtain a second model, wherein the second training question and answer data set is a data set in the N training subsets that is different from the first training question and answer data set, and the perplexity of the second training question and answer data set is higher than the perplexity of the first training question data set; using the verification set to verify the second model to update the collapse probability value; repeating the above operations until the collapse probability value of the target model on the verification set is greater than or equal to the threshold, outputting the target model. In this way, the model is trained for multiple rounds of self-alignment using different training question-answering datasets, so that the generation and classification capabilities of the target model obtained through training are continuously improved with each round. At the same time, the training question-answering datasets are also ranked from easy to difficult overall, so that during the training process, the model will not face difficult problems in the early stages, thereby reducing the accumulation of errors in the model and improving the accuracy of the model. At the same time, during the self-alignment process, the validation set is used to analyze whether the model needs to stop training early, thereby preventing the model from collapsing in the later stages.

[0060] The large language model used in the embodiments of the present application relates to natural language processing technology in the field of artificial intelligence. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0061] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction systems, mechatronics, etc. Among them, the pre-trained model is also called a large language model or a basic model. After fine-tuning, it can be widely used in downstream tasks in various major directions of artificial intelligence. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0062] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0063] Optionally, the method provided in the embodiment of the present application also involves natural language processing, for example, a trained large language model can perform natural language processing tasks.

[0064] Natural language processing (NLP) is an important direction in the field of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing involves natural language, that is, the language people use in daily life, which is closely related to linguistics research; it also involves computer science and mathematics. The pre-training model, an important technology for model training in the field of artificial intelligence, is developed from the large language model (LLM) in the field of NLP. After fine-tuning, the large language model can be widely used in downstream tasks. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0065] Optionally, the data processing involved in the method provided in the embodiment of the present application can also be implemented based on cloud technology.

[0066] For example, various calculations designed in the training process of large language models can be implemented using cloud computing technology, and training question-answering data sets can be stored in the cloud. Cloud computing is a computing model that distributes computing tasks on a resource pool composed of a large number of computers, enabling various application systems to obtain computing power, storage space and information services as needed. The network that provides resources is called the "cloud". From the user's perspective, the resources in the "cloud" can be infinitely expanded, and can be obtained at any time, used on demand, expanded at any time, and paid for by use. Cloud storage is a new concept that extends and develops from the concept of cloud computing. Cloud storage can bring together a large number of storage devices of various types (storage devices are also called storage nodes) in the network through application software or application interfaces to work together to provide data storage and business access functions to the outside world.

[0067] The following describes several exemplary embodiments to illustrate the technical solutions of the embodiments of the present application and the technical effects produced by the technical solutions of the present application. It should be noted that the following embodiments can refer to, draw on or combine with each other, and the same terms, similar features and similar implementation steps in different embodiments will not be described repeatedly.

[0068] The model trained by the embodiment of the present application can be applied to a question-answering system based on the model. An exemplary structure of the system can be found in Figure 1 The model-based question-answering system includes a terminal 100, a server 200, a database 300 and a network 400. The terminal 100, the server 200 and the database 300 are connected via the network 400. The database 300 is used to provide the question-answering system with text (knowledge) related to the questions received by the question-answering system. Figure 1 The number of terminals, servers, and databases in the system shown is for example only. For example, the number of terminals, servers, and databases may be multiple, and the embodiments of the present application do not limit the number of terminals, servers, and databases.

[0069] The above-mentioned question-and-answer system is used to provide question-and-answer services for users, and the question-and-answer services can be intelligent customer service services, online education services, and intelligent medical services, etc., which are not limited in the embodiments of the present application. During the question-and-answer process, the user and the question-and-answer system can conduct continuous question-and-answer in the form of one question and one answer, and the multiple question-and-answer pairs generated by the continuous question-and-answer can be stored in the server 200. It should be noted that the pending questions and multiple question-and-answer pairs have all been free of personal information and authorized by the user or fully authorized by all parties.

[0070] Among them, the terminal 100 communicates with the server 200 through the network, and the database 300 can be integrated on the server 200, or placed on the cloud or other servers. In the above-mentioned question-and-answer process, the terminal 100 and the server 200 can interact with each other. For example, the user inputs the question to be processed through the terminal 100, and the terminal 100 sends the question to be processed to the server 200. In the above-mentioned question-and-answer process, different modules or applications in the server 200 can also interact with each other. For example, if the server 200 reports an error during the operation of a certain application, the server 200 collects the error information and generates the question to be processed based on the error information. According to the question to be processed, the server 200 obtains the reference text related to the question to be processed from the database 300, and the reference text can also be called supporting material. The server 200 calls the large language model to process the question to be processed and the reference text of the question to be processed, obtains the answer to the question to be processed, and returns the answer to the terminal 100.

[0071] Among them, the terminal 100 is a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, an intelligent voice interaction device, a smart home appliance, a car terminal, etc., but is not limited to this. The server 200 is an independent physical server, and can also be a server cluster or a distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), big data and artificial intelligence platforms. The large language model is a deep learning model with huge scale and many parameters. It can learn language structure, grammatical knowledge and semantics from a large-scale data set, and output answers based on the input questions.

[0072] Database 300, in short, can be regarded as an electronic filing cabinet - a place to store electronic files, where users can add, query, update, delete, etc. the data in the files. The so-called "database" is a collection of data that is stored together in a certain way, can be shared with multiple users, has as little redundancy as possible, and is independent of the application program. Database Management System (DBMS) is a computer software system designed for managing databases, generally with basic functions such as storage, interception, security, and backup. Database management systems can be classified according to the database model it supports, such as relational, Extensible Markup Language (XML); or according to the type of computer supported, such as server clusters, mobile phones; or according to the query language used, such as Structured Query Language (SQL), XQuery; or according to the performance impulse focus, such as maximum scale, maximum operating speed; or other classification methods. Regardless of the classification method used, some DBMS can cross categories, for example, supporting multiple query languages ​​at the same time.

[0073] It will be understood that in the specific implementation of the present application, relevant data such as training question and answer data sets, question example sets, and reference files of questions to be processed and answer data are required. When the above embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0074] Combined with the above introduction, the following is an introduction to the model training method in this application with the server as the execution subject. Please refer to Figure 2 , an embodiment of the model training method in the embodiment of the present application includes:

[0075] 201. Obtain N training question-answering data sets, a question example set, a validation set, and an initial model, wherein each training question-answering data set has different training question-answering data, the question example set includes M question examples, and N and M are positive integers.

[0076] In this embodiment, in order to implement multiple rounds of self-alignment training for the model, N training question-answering data sets need to be obtained, where N can be equal to the number of rounds of self-alignment or greater than the number of rounds of self-alignment, which is not specifically limited here. The training question-answering data set can be provided by a third-party data center or by a service application provider, which is not specifically limited here.

[0077] During the training process, in order to prevent the model from facing difficult problems in the early stage, thereby reducing the accumulation of model errors, the various question and answer data in the training question and answer data set can be sorted from easy to difficult. That is, the training question and answer data are sorted according to the perplexity, and then divided in sequence according to the predetermined number and perplexity from high to low or from low to high to obtain multiple training question and answer data sets. In an exemplary scheme, the process of obtaining the training question and answer data set can be as follows: obtain X question and answer data and the perplexity of the X question and answer data, where X is a positive integer greater than N; sort the X question and answer data from easy to difficult according to the perplexity to obtain a sorted data set; divide the sorted data set in sequence according to the X / Y rule to obtain the N training question and answer data sets, where Y is used to indicate the number of training question and answer data in each training question and answer data set in the N training question and answer data sets, and Y is a positive integer.

[0078] Among them, the perplexity is an indicator that measures the complexity or difficulty of understanding of text. It is often used to evaluate the performance of language models, especially in the fields of natural language processing and machine learning. In theory, the lower the perplexity, the stronger the predictive ability of the model. Perplexity is usually calculated by cross-entropy, which measures the difference between the probability distribution of each word in the model predicted text and the actual distribution. In the fields of machine translation, text generation, speech recognition, etc., perplexity is used to evaluate the performance of the model. Perplexity can be used to compare the difference between machine-generated text and human-written text. Generally, human writing has lower perplexity because they are more consistent with the statistical characteristics of natural language. In practical applications, perplexity can help developers understand and improve the performance of applications such as chatbots, automatic summary generators, and content recommendation systems. Perplexity is an important technical metric, but it is only one of the multiple dimensions for evaluating the performance of text or language models. In practical applications, other factors such as the readability, accuracy, and relevance of the text need to be considered.

[0079] At the same time, in order to enhance the model's ability to learn context, a question example set is used to support the model's predictive responses. In order to avoid overfitting the model to a single format during multiple rounds of self-alignment, the question example set can be enriched with information and change the rules for using the question example set during the self-alignment process. In an exemplary scheme, the following scheme can be adopted in the process of obtaining the question example set: obtaining an initial question example set, the initial question example set includes M initial question examples; modifying the M initial question examples according to preset rules to obtain the question example set, the preset rules including at least one of increasing the richness of the question examples and reducing the rejection examples in the question example set.

[0080] Based on this scheme, when modifying the initial question example according to preset rules, the specific scheme may be as follows: modify the M initial question examples according to the rules of increasing the information type, information completeness, information detail and information timeliness of the question to obtain the question example set; or, reduce the number of rejected examples in the M initial question examples, and modify the target rejected examples according to the rules of increasing the information type, information completeness, information detail and information timeliness of the question to obtain the question example set; or, modify the M initial question examples according to the rules of increasing the information type, information completeness, information detail and information timeliness of the question to obtain an intermediate question example set; reduce the number of rejected examples in the intermediate question example set to obtain the M question example set.

[0081] In an exemplary solution, the original initial question example set includes 48 context examples. For example, question A and answer a are used as a context example, and then answer a is enriched according to the rules of information type, information completeness, information detail, and information timeliness. For example, a question example can be as follows: How is the weather today? The original answer is "It will be sunny tomorrow", and the modified question example can be as follows: How is the weather today? The answer is "It will be sunny in your area tomorrow, the temperature will be 28 degrees, the ultraviolet intensity will be high, short sleeves and long pants are suitable for clothing, pay attention to heatstroke prevention when traveling, and bring sunshades."

[0082] If the 48 context examples originally include 24 rejection examples, 20 of them can be modified to become context examples of normal responses. At the same time, the responses of the remaining rejection examples can be enriched. For example, the rejection example is as follows: Question B, and its response is "I don't quite understand what you mean, I can't give an answer", while the modified rejection example is as follows: Question B, and its response is "I don't quite understand what you mean, but according to a certain word in the question, you can refer to a certain document or increase the focus of the description of the question".

[0083] On the other hand, in order to analyze and explain the later performance of the model in the self-alignment training process, a validation set can be provided to verify the performance of the model.

[0084] Optionally, the initial model can be a pre-trained large language model, so that when targeting different application scenarios, only the pre-trained large language model needs to be fine-tuned according to the training question and answer dataset, thereby improving the training efficiency of the model.

[0085] 202. Train the initial model using a first training question and answer data set and the question example set to obtain a first model, wherein the first training question and answer data set is one of the N training question and answer data sets.

[0086] In this embodiment, the server randomly samples S question examples from the M question example sets to obtain a first question example subset, where S is a positive integer less than M; selects the first training question and answer data set from the N training question and answer data sets in order from easy to difficult; selects a first training question data subset from the first training question data set in order from easy to difficult; calls the first training question data subset and the first question example subset to train the initial model to obtain a first intermediate model; randomly samples S question examples from the M question example sets to obtain a second question example subset, and selects a second training question data subset from the first training question data set in order from easy to difficult, the perplexity of each training question data in the second training question data subset is higher than the perplexity of each training question data in the first training question data subset; uses the second training question data subset and the second question example subset to train the first intermediate model to obtain a second intermediate model; repeats the above operations until the first training question data set is processed, and then outputs the first model.

[0087] In an exemplary solution, the N training question and answer data sets include 5 training question and answer data sets, namely, training question and answer data set 1, training question and answer data set 2, training question and answer data set 3, training question and answer data set 4 and training question and answer data set 5. Among them, the perplexity of the above 5 training question and answer data sets is ranked from low to high as follows: training question and answer data set 1, training question and answer data set 2, training question and answer data set 3, training question and answer data set 4, training question and answer data set 5. At the same time, the question example set includes 48 question examples. In the self-alignment process, in the first round, the server determines the training question and answer data set 1 as the first training question and answer data set, and obtains the first batch of model inputs in the order of the perplexity of the question and answer data set from low to high, and randomly samples 5 question examples from the 48 question example sets as the first batch of model inputs; then, the initial model is trained based on the 5 question examples and the first batch of training question and answer data to obtain the intermediate model after the first batch of training; repeat the above operation until the processing of each question and answer data in the training question and answer data set 1 is completed, and then the first model is output.

[0088] In this embodiment, in order to ensure that the model in the training process outputs higher quality response data, thereby improving the accuracy of the model, after the model predicts the training question and answer data to obtain predicted response data, the model can also modify the predicted response data according to the principles of usefulness, honesty and uselessness to obtain reflected and modified predicted response data; finally, the parameters of the model are adjusted according to the predicted response data.

[0089] In an exemplary solution, when the server adjusts the parameters of the model according to the predicted response data, the server calculates the loss value according to the predicted response data and the input data of the model, and then adjusts the parameters of the model according to the loss value. The calculation formula of the loss value can be as follows:

[0090]

[0091] Among them, the L t (θ) is used to indicate the loss value of self-alignment in the tth round; t It is used to indicate the training question-answering dataset of the tth round. t To indicate the t The training question answering dataset is sampled from r It is used to indicate a subset of problem examples randomly sampled from the problem example set. t ′ is used to indicate the predicted answer data output by the model. θ Used to indicate the model.

[0092] 203. Use the verification set to perform verification processing on the first model to obtain a collapse probability value.

[0093] In this embodiment, the collapse probability value can be understood as the risk probability value of a sharp drop in the performance of the model when facing test data; or, the collapse probability value can be understood as the risk probability value of a sharp drop in the adaptability of the model when facing test data (that is, the collapse probability value can be used to indicate the generalization ability of the model); or, the collapse probability value can be understood as the risk probability value of the model parameters being updated to cause unstable model performance (that is, the collapse probability value is used to indicate the risk probability value of the model losing stability); or, the collapse probability value can be understood as an evaluation indicator of the model (that is, when the collapse probability value indicates that the accuracy or loss value of the model has dropped significantly, it can indicate that the model has collapsed).

[0094] Based on the above description, in an exemplary scheme, when verifying the collapse probability value, the following scheme can be adopted: call the first model to perform verification processing on the verification set to obtain the output probability of the terminator of the response data corresponding to the verification set, and use the output probability as the collapse probability value.

[0095] 204 . Determine whether the collapse probability value is greater than a threshold value. If it is greater than the threshold value, execute step 208 ; if it is less than the threshold value, execute step 205 .

[0096] In this embodiment, in order to facilitate the description of the performance of the model, the collapse probability value can be directly limited by a threshold. If it exceeds the threshold, it means that the performance of the model is about to collapse; if it does not exceed the threshold, it means that the performance of the model meets the requirements. Therefore, after each round of self-alignment, the collapse probability value of the output model can be verified. If the collapse probability value is greater than or equal to the threshold, the model output by this round of self-alignment is output as the target model; if the collapse probability value is less than the threshold, the next round of self-alignment can be continued.

[0097] 205. Train the first model using a second training question and answer dataset and the question example set to obtain a second model, wherein the second training question and answer dataset is a dataset in the N training subsets that is different from the first training question and answer dataset, and the perplexity of the second training question and answer dataset is higher than the perplexity of the first training question dataset.

[0098] In this embodiment, the server randomly samples S question examples from the M question example sets to obtain a third question example subset, where S is a positive integer less than M; selects the second training question and answer data set from the N training question and answer data sets in order from easy to difficult; selects a third training question data subset from the second training question data set in order from easy to difficult; calls the third training question data subset and the second question example subset to train the first model to obtain a third intermediate model; randomly samples S question examples from the M question example sets to obtain a fourth question example subset, and selects a fourth training question data subset from the second training question data set in order from easy to difficult, the perplexity of each training question data in the fourth training question data subset is higher than the perplexity of each training question data in the third training question data subset; uses the fourth training question data subset and the fourth question example subset to train the third intermediate model to obtain a fourth intermediate model; repeats the above operations until the second training question data set is processed, and then outputs the second model.

[0099] In an exemplary solution, the N training question and answer data sets include 5 training question and answer data sets, namely, training question and answer data set 1, training question and answer data set 2, training question and answer data set 3, training question and answer data set 4 and training question and answer data set 5. Among them, the perplexity of the above 5 training question and answer data sets is ranked from low to high as follows: training question and answer data set 1, training question and answer data set 2, training question and answer data set 3, training question and answer data set 4, training question and answer data set 5. At the same time, the question example set includes 48 question examples. In the self-alignment process, in the second round, the server determines the training question and answer data set 2 as the second training question and answer data set, and at the same time obtains the first batch of model inputs in the order of the perplexity of the question and answer data set from low to high, and randomly samples 5 question examples from the 48 question example sets as the first batch of model inputs; then, the initial model is trained based on the 5 question examples and the first batch of training question and answer data to obtain the intermediate model after the first batch of training; repeat the above operation until the processing of each question and answer data in the training question and answer data set 2 is completed, and then the second model is output.

[0100] In this embodiment, in order to ensure that the model in the training process outputs higher quality response data, thereby improving the accuracy of the model, after the model predicts the training question and answer data to obtain predicted response data, the model can also modify the predicted response data according to the principles of usefulness, honesty and uselessness to obtain reflected and modified predicted response data; finally, the parameters of the model are adjusted according to the predicted response data.

[0101] In an exemplary solution, when the server adjusts the parameters of the model according to the predicted response data, the server calculates the loss value according to the predicted response data and the input data of the model, and then adjusts the parameters of the model according to the loss value. The calculation formula of the loss value can be as follows:

[0102]

[0103] Among them, the L t (θ) is used to indicate the loss value of self-alignment in the tth round; t It is used to indicate the training question-answering dataset of the tth round. t To indicate the t The training question answering dataset is sampled from r Used to indicate a subset of problem examples randomly sampled from the problem example set. t ′ is used to indicate the predicted answer data output by the model. θ Used to indicate the model.

[0104] 206. Perform verification processing on the second model using the verification set to update the collapse probability value.

[0105] In this embodiment, the second model is called to perform verification processing on the verification set to obtain the output probability of the terminator of the response data corresponding to the verification set, and the output probability is used as the second collapse probability value.

[0106] 207 , determining whether the second collapse probability value is greater than the threshold value, if it is greater than the threshold value, executing step 208 ; if it is less than the threshold value, performing iterative training according to the above steps 202 to 207 .

[0107] In this embodiment, in order to facilitate the description of the performance of the model, the collapse probability value can be directly limited by a threshold. If it exceeds the threshold, it means that the performance of the model is about to collapse; if it does not exceed the threshold, it means that the performance of the model meets the requirements. Therefore, after each round of self-alignment, the collapse probability value of the output model can be verified. If the collapse probability value is greater than or equal to the threshold, the model output by this round of self-alignment is output as the target model; if the collapse probability value is less than the threshold, the next round of self-alignment can be continued.

[0108] 208. Output the target model.

[0109] The server ends the self-alignment process and outputs the target model.

[0110] Below Figure 3 The system architecture shown here illustrates the model training process of this application:

[0111] Develop a context corpus (i.e., a set of question examples) that includes multiple context examples, such as 48 carefully curated and rich context examples.

[0112] A plurality of training question-answering data sets are obtained at the same time, wherein the plurality of training question-answering data sets are input into the model in order from easy to difficult.

[0113] During the training process, context examples are randomly extracted and input into the model together with the training question-answering dataset. The model outputs the response data through reasoning and reflection modification. The model parameters are then adjusted based on the response data. The training is then iterated according to the above process until the convergence condition is reached (for example, the collapse probability value of the model on the validation set reaches the threshold), and the target model is output.

[0114] The following is an explanation of the data processing method in this application using the terminal device as the execution subject. Figure 4 As shown, an embodiment of the data processing method in the present application includes:

[0115] 401. Get data to be processed.

[0116] The terminal device obtains the data to be processed through a third party or a program input interface running on the terminal device. In this embodiment, the data to be processed includes but is not limited to text questions, text data and text questions, text summaries, and the like.

[0117] 402. Calling a target model to perform prediction processing on the data to be processed to obtain response data corresponding to the data to be processed, wherein the target model is trained based on the above method.

[0118] The terminal device calls the target model to perform predictive processing on the data to be processed to obtain response data corresponding to the data to be processed.

[0119] 403. Output the response data.

[0120] The terminal device displays the response data through a display interface.

[0121] The model training device in this application is described in detail below. Figure 5 , Figure 5 This is a schematic diagram of an embodiment of a model training device in an embodiment of the present application. The model training device 20 includes:

[0122] An acquisition module 201 is used to acquire N training question-answering data sets, a question example set, a verification set, and an initial model, wherein each training question-answering data set has different training question-answering data, the question example set includes M question examples, and N and M are positive integers;

[0123] The processing module 202 is used to train the initial model using the first training question and answer data set and the problem example set to obtain a first model, wherein the first training question and answer data set is one of the N training question and answer data sets; verify the first model using the verification set to obtain a collapse probability value; when the collapse probability value is less than a threshold, train the first model using the second training question and answer data set and the problem example set to obtain a second model, wherein the second training question and answer data set is a data set in the N training subsets that is different from the first training question and answer data set, and the perplexity of the second training question and answer data set is higher than the perplexity of the first training question data set; verify the second model using the verification set to update the collapse probability value; repeat the above operations until the collapse probability value of the target model on the verification set is greater than or equal to the threshold, and output the target model.

[0124] In an embodiment of the present application, a model training device is provided. The above device is used to perform multiple rounds of self-alignment training on the model using different training question and answer data sets, so that the generation ability and classification ability of the target model obtained by training are continuously improved with the rounds. At the same time, each training question and answer data set is also ranked from easy to difficult as a whole, so that in the training process, the model does not face difficult problems in the early stage, thereby reducing the accumulation of errors in the model and improving the accuracy of the model. At the same time, in the self-alignment process, the validation set is used to analyze whether the model needs to stop training early, so as to prevent the model from collapsing in the later stage.

[0125] Optionally, in the above Figure 5 On the basis of the corresponding embodiment, in another embodiment of the model training device 20 provided in the embodiment of the present application,

[0126] The acquisition module 201 is used to acquire X question-answer data and the perplexity of the X question-answer data, where X is a positive integer greater than N;

[0127] According to the perplexity, the X question-answer data are sorted in order from easy to difficult to obtain a sorted data set;

[0128] The sorted data set is divided and processed in sequence according to the X / Y rule to obtain the N training question and answer data sets, where Y is used to indicate the number of training question and answer data in each of the N training question and answer data sets, and Y is a positive integer.

[0129] In an embodiment of the present application, a model training device is provided. Using the above device, the question and answer data are sorted from easy to difficult according to the perplexity of the question and answer data, so that each training question and answer data set is also sorted from easy to difficult as a whole, so that in the training process, the model will not face difficult problems in the early stage, thereby reducing the accumulation of model errors.

[0130] Optionally, in the above Figure 5 On the basis of the corresponding embodiment, in another embodiment of the model training device 20 provided in the embodiment of the present application, the acquisition module 201 is used to acquire an initial question example set, and the initial question example set includes M initial question examples;

[0131] The M initial question examples are modified according to preset rules to obtain the question example set, where the preset rules include at least one of increasing the richness of the question examples and reducing the rejected examples in the question example set.

[0132] In an embodiment of the present application, a model training device is provided. The device is used to increase the richness and quantity of problem examples, thereby improving the diversity of information in the model training process.

[0133] Optionally, in the above Figure 5 On the basis of the corresponding embodiment, in another embodiment of the model training device 20 provided in the embodiment of the present application,

[0134] The acquisition module 201 is used to modify the M initial question examples according to the rules of increasing the information type, information completeness, information detail and information timeliness of the questions to obtain the question example set;

[0135] or,

[0136] The number of rejection examples in the M initial question examples is reduced, and the target rejection examples are modified according to the rules of increasing the information type, information completeness, information detail and information timeliness of the questions to obtain the question example set;

[0137] or,

[0138] The M initial question examples are modified according to the rules of increasing the information type, information completeness, information details and information timeliness of the questions to obtain an intermediate question example set;

[0139] The number of rejected examples in the intermediate problem example set is reduced to obtain the M problem example set.

[0140] In an embodiment of the present application, a model training device is provided. The device is used to increase the richness and quantity of problem examples, thereby improving the diversity of information in the model training process.

[0141] Optionally, in the above Figure 5 On the basis of the corresponding embodiment, in another embodiment of the model training device 20 provided in the embodiment of the present application,

[0142] The processing module 202 is used to randomly sample S question examples from the M question example set to obtain a first question example subset, where S is a positive integer less than M;

[0143] Selecting the first training question-answering data set from the N training question-answering data sets in order from easy to difficult;

[0144] Selecting a first training problem data subset from the first training problem data set in order from easy to difficult;

[0145] Calling the first training problem data subset and the first problem example subset to train the initial model to obtain a first intermediate model;

[0146] Randomly sampling S problem examples from the M problem example set to obtain a second problem example subset, and selecting a second training problem data subset from the first training problem data set in order from easy to difficult, wherein the perplexity of each training problem data in the second training problem data subset is higher than the perplexity of each training problem data in the first training problem data subset;

[0147] Training the first intermediate model using the second training problem data subset and the second problem example subset to obtain a second intermediate model;

[0148] Repeat the above steps until the first training problem data set is processed, and then output the first model.

[0149] In an embodiment of the present application, a model training device is provided. By using the above device, a fixed set of problem examples is changed into a large set of problem examples that can be randomly sampled, so that the problem examples learned during the training process can be varied, thereby increasing the amount of information and diversity in the model training process. At the same time, during the training process, the input and self-alignment processes of the model are carried out in order from easy to difficult, so that the model does not face difficult problems in the early stage, thereby reducing the accumulation of errors in the model and improving the ability of the model.

[0150] Optionally, in the above Figure 5 On the basis of the corresponding embodiment, in another embodiment of the model training device 20 provided in the embodiment of the present application, the processing module 202 is used to call the initial model to perform prediction processing on the first training question data subset based on the first question example subset to obtain a first prediction answer subset;

[0151] Modifying the first prediction answer subset according to the principles of usefulness, honesty and uselessness to obtain a second prediction answer subset;

[0152] The parameters of the initial model are adjusted according to the second predicted answer subset to obtain the first intermediate model.

[0153] In the embodiment of the present application, a model training device is provided. By using the above device, the reflection ability of the model is introduced during the model training process, that is, the response data output by the model is reflected and modified, thereby improving the output quality and further improving the quality of model training.

[0154] Optionally, in the above Figure 5 On the basis of the corresponding embodiment, in another embodiment of the model training device 20 provided in the embodiment of the present application, the processing module 202 is used to calculate the loss value according to the second predicted answer subset, the first question example subset and the first training question data subset;

[0155] The parameters of the initial model are adjusted according to the loss value to obtain the first intermediate model.

[0156] In an embodiment of the present application, a model training device is provided. Using the above device, a loss value is determined based on the modified response data, a randomly sampled subset of question examples, and a currently input subset of training question data, and finally the loss value is used to adjust the parameters of the model, thereby improving the training results of the model.

[0157] Optionally, in the above Figure 5 On the basis of the corresponding embodiment, in another embodiment of the model training device 20 provided in the embodiment of the present application, the processing module 202 is used to call the first model to perform prediction processing on the verification set to obtain the output probability of the terminator of the response data corresponding to the verification set, and use the output probability as the collapse probability value.

[0158] In an embodiment of the present application, a model training device is provided. Using the above device, during multiple rounds of self-alignment, a validation set is used to analyze whether the model needs to stop training in advance, thereby preventing the model from collapsing in the later stage.

[0159] Optionally, in the above Figure 5 On the basis of the corresponding embodiment, in another embodiment of the model training device 20 provided in the embodiment of the present application, the initial model is a pre-trained large language model.

[0160] In an embodiment of the present application, a model training device is provided. Using the above device and a pre-trained large language model can effectively reduce the training process and improve the training efficiency.

[0161] Optionally, in the above Figure 5 On the basis of the corresponding embodiment, in another embodiment of the model training device 20 provided in the embodiment of the present application,

[0162] The processing module 202 is configured to output the first model as the target model when the collapse probability value is greater than or equal to the threshold.

[0163] In an embodiment of the present application, a model training device is provided. Using the above device, during multiple rounds of self-alignment, a validation set is used to analyze whether the model needs to stop training in advance, thereby preventing the model from collapsing in the later stage.

[0164] The model training device in this application is described in detail below. Figure 6 , Figure 6 This is a schematic diagram of an embodiment of a data processing device in an embodiment of the present application. The data processing device 30 includes:

[0165] An acquisition module 301 is used to acquire data to be processed;

[0166] The processing module 302 is used to call the target model to perform prediction processing on the data to be processed to obtain the response data corresponding to the data to be processed, and the target model is trained based on the above method;

[0167] The output module 303 is used to output the reply data.

[0168] In an embodiment of the present application, a model training device is provided. The above device is used to perform multiple rounds of self-alignment training on the model using different training question and answer data sets, so that the generation ability and classification ability of the target model obtained by training are continuously improved with the rounds. At the same time, each training question and answer data set is also ranked from easy to difficult as a whole, so that in the training process, the model does not face difficult problems in the early stage, thereby reducing the accumulation of errors in the model and improving the accuracy of the model. At the same time, in the self-alignment process, the validation set is used to analyze whether the model needs to stop training early, so as to prevent the model from collapsing in the later stage.

[0169] The model training device and data processing device provided in this application can be used in the server, please refer to Figure 7 , Figure 7 : is a schematic diagram of a server structure provided in an embodiment of the present application. The server 300 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 322 (for example, one or more processors) and memory 332, and one or more storage media 330 (for example, one or more mass storage devices) storing application programs 342 or data 344. Among them, the memory 332 and the storage medium 330 can be short-term storage or permanent storage. The program stored in the storage medium 330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 322 can be configured to communicate with the storage medium 330 to execute a series of instruction operations in the storage medium 330 on the server 300.

[0170] The server 300 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input and output interfaces 358, and / or one or more operating systems 341, such as Windows Server 2000. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM etc.

[0171] The steps performed by the model training device and the data processing device in the above embodiment can be based on the Figure 7 The server structure shown.

[0172] The model training device and data processing device provided in this application can be used in terminal devices, please refer to Figure 8 For the sake of convenience, only the parts related to the embodiments of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiments of the present application. In the embodiments of the present application, the terminal device is a smart phone as an example for explanation:

[0173] Figure 8 The block diagram shows a partial structure of a smart phone related to the terminal device provided in the embodiment of the present application. Figure 8 The smart phone includes: a radio frequency (RF) circuit 410, a memory 420, an input unit 430, a display unit 440, a sensor 450, an audio circuit 460, a wireless fidelity (WiFi) module 470, a processor 480, and a power supply 490. Those skilled in the art will appreciate that Figure 8 The structure of the smartphone shown in the figure does not constitute a limitation of the smartphone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0174] Combine the following Figure 8 A detailed introduction to the various components of a smartphone:

[0175] The RF circuit 410 can be used for receiving and sending signals during the process of sending and receiving information or making calls. In particular, after receiving the downlink information of the base station, it is sent to the processor 480 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 410 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit 410 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communication can use any communication standard or protocol, including but not limited to the global system of mobile communication (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), long term evolution (LTE), email, short messaging service (SMS), etc.

[0176] The memory 420 can be used to store software programs and modules. The processor 480 executes various functional applications and data processing of the smartphone by running the software programs and modules stored in the memory 420. The memory 420 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the smartphone (such as audio data, a phone book, etc.), etc. In addition, the memory 420 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0177] The input unit 430 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the smart phone. Specifically, the input unit 430 may include a touch panel 431 and other input devices 432. The touch panel 431, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 431 or near the touch panel 431 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 431 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 480, and can receive and execute commands sent by the processor 480. In addition, the touch panel 431 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 431, the input unit 430 may also include other input devices 432. Specifically, other input devices 432 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, etc.

[0178] The display unit 440 can be used to display information input by the user or information provided to the user and various menus of the smartphone. The display unit 440 may include a display panel 441. Optionally, the display panel 441 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like. Further, the touch panel 431 may cover the display panel 441. When the touch panel 431 detects a touch operation on or near it, it is transmitted to the processor 480 to determine the type of touch event. Subsequently, the processor 480 provides a corresponding visual output on the display panel 441 according to the type of touch event. Although in Figure 8 In the embodiment, the touch panel 431 and the display panel 441 are used as two independent components to implement the input and output functions of the smartphone, but in some embodiments, the touch panel 431 and the display panel 441 can be integrated to implement the input and output functions of the smartphone.

[0179] The smartphone may also include at least one sensor 450, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 441 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 441 and / or the backlight when the smartphone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the smartphone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in smartphones, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0180] The audio circuit 460, the speaker 461, and the microphone 462 can provide an audio interface between the user and the smartphone. The audio circuit 460 can transmit the received audio data to the speaker 461 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 462 converts the collected sound signal into an electrical signal, which is received by the audio circuit 460 and converted into audio data, and then the audio data is output to the processor 480 for processing, and then sent to another smartphone through the RF circuit 410, or the audio data is output to the memory 420 for further processing.

[0181] WiFi is a short-range wireless transmission technology. Smartphones can help users send and receive emails, browse web pages, and access streaming media through WiFi module 470, which provides users with wireless broadband Internet access. Figure 8 A WiFi module 470 is shown, but it is understandable that it is not an essential component of the smartphone and can be omitted as needed without changing the essence of the invention.

[0182] Processor 480 is the control center of the smartphone, which uses various interfaces and lines to connect various parts of the entire smartphone, and executes various functions of the smartphone and processes data by running or executing software programs and / or modules stored in memory 420, and calling data stored in memory 420, thereby monitoring the smartphone as a whole. Optionally, processor 480 may include one or more processing units; optionally, processor 480 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into processor 480.

[0183] The smart phone also includes a power source 490 (such as a battery) for supplying power to various components. Optionally, the power source can be logically connected to the processor 480 through a power management system, thereby managing functions such as charging, discharging, and power consumption management through the power management system.

[0184] Although not shown, the smartphone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0185] The steps performed by the terminal device in the above embodiment can be based on the Figure 8 The terminal device structure shown.

[0186] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the methods described in the aforementioned embodiments.

[0187] The present application also provides a computer program product including a program, which, when executed on a computer, enables the computer to execute the methods described in the aforementioned embodiments.

[0188] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0189] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0190] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0191] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0192] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0193] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A model training method, characterized in that: include: Obtain N training question-answering data sets, a question example set, a validation set, and an initial model, wherein each training question-answering data set has different training question-answering data, the question example set includes M question examples, and N and M are positive integers; Training the initial model using a first training question-and-answer data set and the question example set to obtain a first model, wherein the first training question-and-answer data set is one of the N training question-and-answer data sets; Using the verification set to verify the first model to obtain a collapse probability value; When the collapse probability value is less than a threshold, training the first model using a second training question and answer data set and the question example set to obtain a second model, wherein the second training question and answer data set is a data set in the N training subsets that is different from the first training question and answer data set, and the perplexity of the second training question and answer data set is higher than the perplexity of the first training question data set; Using the verification set to perform verification processing on the second model to update the collapse probability value; Repeat the above operation until the collapse probability value of the target model on the validation set is greater than or equal to the threshold, and output the target model.

2. The method according to claim 1, characterized in that The obtaining of N training question-answering data sets includes: Obtain X question-and-answer data and the perplexity of the X question-and-answer data, where X is a positive integer greater than N; Sorting the X question-answer data from easy to difficult according to the perplexity to obtain a sorted data set; The sorted data set is divided and processed in sequence according to the X / Y rule to obtain the N training question and answer data sets, where Y is used to indicate the number of training question and answer data in each of the N training question and answer data sets, and Y is a positive integer.

3. The method according to claim 1, characterized in that The acquisition question example set includes: Obtaining an initial question example set, wherein the initial question example set includes M initial question examples; The M initial question examples are modified according to preset rules to obtain the question example set, wherein the preset rules include at least one of increasing the richness of the question examples and reducing the rejected examples in the question example set.

4. The method according to claim 3, characterized in that The M initial question examples are modified according to a preset rule to obtain the question example set including: Modifying the M initial question examples according to the rules of increasing the information type, information completeness, information detail and information timeliness of the questions to obtain the question example set; or, Reducing the number of rejection examples in the M initial question examples, and modifying the target rejection examples according to the rules of increasing the information type, information completeness, information detail, and information timeliness of the questions, so as to obtain the question example set; or, Modify the M initial question examples according to the rules of increasing the information type, information completeness, information detail information and information timeliness of the questions to obtain an intermediate question example set; The number of rejected examples in the intermediate problem example set is reduced to obtain the M problem example set.

5. The method according to claim 1, characterized in that: The step of training the initial model using the first training question-answering data set and the question example set to obtain a first model includes: Randomly sampling S question examples from the M question example sets to obtain a first question example subset, where S is a positive integer less than M; Selecting the first training question-answering data set from the N training question-answering data sets in order from easy to difficult; Selecting a first training problem data subset from the first training problem data set in order from easy to difficult; Calling the first training question data subset and the first question example subset to train the initial model to obtain a first intermediate model; Randomly sampling S question examples from the M question example sets to obtain a second question example subset, and selecting a second training question data subset from the first training question data set in order from easy to difficult, wherein the perplexity of each training question data in the second training question data subset is higher than the perplexity of each training question data in the first training question data subset; Training the first intermediate model using the second training question data subset and the second question example subset to obtain a second intermediate model; Repeat the above steps until the first training problem data set is processed, and then output the first model.

6. The method according to claim 5, characterized in that The step of training the initial model using the first training problem data subset and the first problem example subset to obtain a first intermediate model comprises: Calling the initial model to perform prediction processing on the first training question data subset based on the first question example subset to obtain a first prediction answer subset; Modifying the first prediction answer subset according to the principles of usefulness, honesty and uselessness to obtain a second prediction answer subset; The parameters of the initial model are adjusted according to the second subset of predicted answers to obtain the first intermediate model.

7. The method according to claim 6, characterized in that The adjusting the parameters of the initial model according to the second prediction answer subset to obtain the first intermediate model comprises: Calculate a loss value based on the second predicted answer subset, the first question example subset, and the first training question data subset; The parameters of the initial model are adjusted according to the loss value to obtain the first intermediate model.

8. The method according to any one of claims 1 to 7, characterized in that Using the verification set to verify the first model to obtain a collapse probability value includes: The first model is called to perform prediction processing on the verification set to obtain the output probability of the terminator of the reply data corresponding to the verification set, and the output probability is used as the collapse probability value.

9. The method according to any one of claims 1 to 7, characterized in that The initial model is a pre-trained large language model.

10. The method according to any one of claims 1 to 7, characterized in that The method further comprises: When the collapse probability value is greater than or equal to the threshold, the first model is output as the target model.

11. A data processing method, characterized in that: include: Get the data to be processed; Calling a target model to perform prediction processing on the data to be processed to obtain response data corresponding to the data to be processed, wherein the target model is trained based on the methods described in claims 1 to 10 above; The reply data is output.

12. A model training device, characterized in that: include: An acquisition module is used to acquire N training question-answering data sets, a question example set, a verification set, and an initial model, wherein each training question-answering data set has different training question-answering data, the question example set includes M question examples, and N and M are positive integers; A processing module, used to train the initial model using a first training question and answer data set and the problem example set to obtain a first model, wherein the first training question and answer data set is one of the N training question and answer data sets; use the verification set to verify the first model to obtain a collapse probability value; when the collapse probability value is less than a threshold, use a second training question and answer data set and the problem example set to train the first model to obtain a second model, wherein the second training question and answer data set is a data set in the N training subsets that is different from the first training question and answer data set, and the perplexity of the second training question and answer data set is higher than the perplexity of the first training question data set; use the verification set to verify the second model to update the collapse probability value; repeat the above operations until the collapse probability value of the target model on the verification set is greater than or equal to the threshold, and output the target model.

13. A data processing device, characterized in that: include: An acquisition module is used to acquire data to be processed; A processing module, used for calling a target model to perform prediction processing on the data to be processed to obtain response data corresponding to the data to be processed, wherein the target model is trained based on the methods described in claims 1 to 10 above; An output module is used for outputting the reply data.

14. A computer device, characterized in that: include: Memory, processor, and bus system; Wherein, the memory is used to store programs; The processor is used to execute the program in the memory, and the processor is used to execute the method according to any one of claims 1 to 11 according to the instructions in the program code; The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

15. A computer-readable storage medium comprising instructions, which, when executed on a computer, causes the computer to perform the method according to any one of claims 1 to 11.

16. A computer program product comprising a computer program, characterized in that The computer program is used by a processor to execute the method according to any one of claims 1 to 11.

Images