Model training method and device, and data processing method and device

Abstract

The present disclosure provides a model training method and device, and a data processing method and device. The method of the present disclosure uses a reference model to screen an existing instruction dataset for an instruction whose model fitting difficulty meets a preset condition, and the method can screen the instruction dataset for a challenging instruction having high model fitting difficulty as a seed instruction. A plurality of similar instruction samples are generated by means of expansion on the basis of the seed instruction, thereby obtaining more challenging instruction samples, a training set comprising the instruction samples and reference responses for the instruction samples is constructed, and knowledge distillation can be achieved by using the reference model on the basis of the existing instruction dataset, thereby obtaining a training set containing higher-quality instruction data. Furthermore, using the training set to train a deep learning model enhances the capability of the deep learning model to handle more complex and challenging tasks, and improves the performance of a trained target model.

Description

[0001] The present disclosure claims the priority of Chinese patent application No. 202410732637.3, which was filed with the Chinese Patent Office on June 5, 2024, and entitled "Model training and data processing method and device", the entire content of which is incorporated herein by reference. TECHNICAL FIELD The present disclosure relates to the technical field of computer technology, and particularly relates to a model training and data processing method and device. BACKGROUND Large language models have the ability to generalize to tasks not in the training set after instruction fine-tuning. Instruction fine-tuning aligns a pre-trained language model with user preferences by fine-tuning the language model model under instruction data set supervision (SFT). Unlike traditional specific task fine-tuning, instruction fine-tuning utilizes the intrinsic ability of the model to train the model with much less data than pre-training and the extensive knowledge accumulated in the pre-training phase. With the emergence and use of powerful proprietary language models, various knowledge distillation (KD) methods have been proposed. With the knowledge distillation technology, a language model with larger parameters and higher capabilities (as a teacher model) can be used to fine-tune a language model with smaller parameters (as a student model), so as to compress the knowledge and capabilities of the teacher model into a smaller and more efficient student model, so that the model with smaller parameters can maintain relatively high performance while reducing the demand for computing resources. The deep learning model obtained by the existing instruction fine-tuning method will encounter difficulties in processing more complex and challenging tasks (such as logical reasoning, code generation, etc.), and the performance is poor. CONTENT The present disclosure provides a model training and data processing method and device to solve the problem that the deep learning model encounters difficulties in processing more complex and challenging tasks and has poor performance. In a first aspect, the present disclosure provides a model training method, comprising: filtering instructions with a model fitting difficulty meeting a preset condition from an existing instruction data set as seed instructions by means of a reference model; generating a plurality of instruction samples based on the seed instructions, and constructing a training set containing the instruction samples and reference responses of the instruction samples; training a deep learning model using the training set to obtain a target model trained.In a second aspect, the present disclosure provides a data processing method, comprising: receiving a calling request of an end-side device to the target model, the calling request containing an input instruction; inputting the input instruction into the target model, and generating a response result of the input instruction by the target model, the target model being trained by the following manner: screening seed instructions with a model fitting difficulty satisfying a preset condition from an existing instruction data set by a reference model, generating a plurality of instruction samples based on the seed instructions, constructing a training set containing the instruction samples and reference responses of the instruction samples, training a deep learning model using the training set, and obtaining a trained target model; and returning the response result of the input instruction to the end-side device. In a third aspect, the present disclosure provides a server, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the server to perform the method provided in any one of the preceding aspects. In a fourth aspect, the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer execution instructions, and when a processor executes the computer execution instructions, the method provided in any one of the preceding aspects is implemented. In a fifth aspect, the present disclosure provides a computer program product, comprising a computer program, and the computer program is executed by a processor to implement the method provided in any one of the preceding aspects. The model training and data processing method and device provided by the present disclosure can screen instructions with a higher model fitting difficulty and challenging from an instruction data set as seed instructions by screening instructions with a model fitting difficulty satisfying a preset condition from an existing instruction data set by a reference model; a plurality of similar instruction samples can be generated based on the seed instructions, and a training set containing the instruction samples and reference responses of the instruction samples can be constructed, so that knowledge distillation can be implemented based on the existing instruction data set by the reference model, and a training set with higher quality and challenging instruction data can be obtained; further, the ability of a deep learning model to process more complex and challenging tasks can be improved by training the deep learning model using the training set, and the performance of the target model after training can be improved. BRIEF DESCRIPTION OF DRAWINGS The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, further serve to explain the principles of the present disclosure.FIG. 1 is a schematic diagram of an example system architecture to which the present disclosure is applicable; FIG. 2 is a flowchart of a method for training a neural network model according to an example embodiment of the present disclosure; FIG. 3 is a flowchart of a process for screening seed instructions according to an example embodiment of the present disclosure; FIG. 4 is a flowchart of a process for constructing a training set based on seed instructions according to an example embodiment of the present disclosure; FIG. 5 is a schematic diagram of the proportion of each task type in an existing instruction data set according to an example embodiment of the present disclosure; FIG. 6 is a schematic diagram of a preset task proportion of each task type according to an example embodiment of the present disclosure; FIG. 7 is a flowchart of a method for training a model according to another example embodiment of the present disclosure; FIG. 8 is a framework diagram of a model training according to an example embodiment of the present disclosure; FIG. 9 is a flowchart of a data processing method according to an example embodiment of the present disclosure; and FIG. 10 is a schematic diagram of the structure of a server according to an example embodiment of the present disclosure. The above-described figures show the specific embodiments of the present disclosure, which will be described in more detail below. These figures and written descriptions are not intended to limit the scope of the concept of the present disclosure in any way, but to illustrate the concept of the present disclosure to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION The example embodiments will be described in detail below, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers on different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following example embodiments are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present disclosure as detailed in the appended claims. It should be noted that the user information (including but not limited to user device information, user attribute information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards, and provide corresponding operation portals for user selection of authorization or rejection. First, the terms involved in the present disclosure are explained: Instruction fine-tuning: also known as instruction tuning, using user instruction data and reference responses (responses) to fine-tune large language models to generate more user-preferred responses. Thought chain: to improve the ability of a language model to solve reasoning problems such as arithmetic or common sense reasoning problems by guiding the language model to simulate the reasoning steps in the human thinking process. Curriculum learning: a machine learning training strategy designed to improve the learning efficiency and final performance of a model by having the model first learn simple concepts and then gradually transition to complex concepts.

[0002] Alpaca dataset: An open-source instruction fine-tuning dataset based on artificial handwriting seed data and model expansion, covering multiple tasks and fields such as grammar, fashion, mathematics, and food. Visual question answering task: Determine the answer to a question from the visual information in an input image. Image description task: Generate a description of an input image. Visual entailment task: Predict the semantic relevance between an input image and text, i.e., entailment, neutral, or contradiction. Pointer representation and understanding task: Locate the image region corresponding to the input text in the input image. Image generation task: Generate an image based on the input description text. Text-based sentiment classification task: Predict the sentiment classification information of the input text. Text summarization task: Generate summary information of the input text. Multimodal task: A downstream task involving multiple modalities of data such as images and text, such as visual question answering, image description, visual entailment, pointer representation and understanding, image generation, etc. Multimodal pre-training model: A pre-training model that involves multiple modalities of data such as images and text, which can be applied to multimodal tasks after fine-tuning. Pre-training language model: A pre-training model obtained by pre-training a large language model (LLM). Large model refers to a deep learning model with a large number of model parameters, usually containing hundreds of millions, even tens of billions of model parameters. Large model can also be called foundation model (FM), which is pre-trained on a large amount of unlabeled corpus, resulting in a pre-training model with hundreds of millions of parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as large language model (LLM) and multimodal pre-training model.A large model can be widely applied in natural language processing (NLP) and computer vision, and can be applied to computer vision tasks such as visual question answering (VQA), image caption (IC), image generation, and natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc. Research shows that increasing the diversity and complexity of instruction data can improve model performance. However, for models with different capacities and parameters, the optimal complexity of instruction data is still an unsolved problem. Existing solutions attempt to maximize data diversity by using sentence embeddings. However, this method does not address the issue of model capacity imbalance. Deep learning models fine-tuned using these methods sometimes struggle with more complex and challenging tasks (such as logical reasoning, code generation, etc.), perform poorly, and generate low-quality response results. To address the aforementioned technical problems, the present disclosure provides a model training method that filters instructions with a model fitting difficulty that meets a predetermined condition from an existing instruction data set using a reference model. This allows the selection of instructions with a higher model fitting difficulty and challenges from the instruction data set as seed instructions. Based on the seed instructions, a plurality of similar instruction samples are generated, which can obtain more challenging instruction samples. A training set containing instruction samples and reference responses of instruction samples is constructed, and knowledge distillation can be achieved based on the existing instruction data set using the reference model to obtain a training set of challenging instruction data. Further, training a deep learning model using the training set can improve the ability of the deep learning model to handle more complex and challenging tasks and improve the performance of the target model after training. The model fitting difficulty of an instruction refers to the difficulty of learning (or fitting) the ability to process the instruction during model training. The higher the model fitting difficulty of an instruction, the more difficult it is for the model to learn (or fit) the ability to process the instruction, i.e., the more difficult it is for the model to fit the ability to generate a correct response to the instruction. FIG. 1 is a schematic diagram of an example system architecture applicable to the present disclosure. As shown in FIG. 1, the system architecture includes a server and an end-side device. The server and the end-side device have a communicable communication link between them, enabling communication between the server and the end-side device.The server is a computing device deployed in the cloud or locally, such as a cloud cluster. The server is responsible for obtaining and storing a deep learning model to be trained and an existing instruction data set. Based on the existing instruction data set, the server can filter out instructions with a model fitting difficulty satisfying a preset condition from the existing instruction data set as seed instructions with the aid of a reference model, generate a plurality of instruction samples based on the seed instructions, and construct a training set containing the instruction samples; train the deep learning model using the training set to obtain a target model trained. The deep learning model can be a deep learning model for natural language processing based on information of at least one modality of an image, a voice, or text, such as various human-computer interaction models, models for implementing a text-to-text generation task (such as text summarization, translation, and question answering), a voice-to-text model (such as a speech recognition model), an image-to-text model (such as an OCR model and an image abstract), a natural language understanding (NLU) model for emotion analysis, and the like. The deep learning model can be a lightweight model, a pre-trained model, or a large language model with a relatively small number of parameters, a multi-modal language model, and the like. The reference model is a pre-trained model with strong capabilities. Compared with the deep learning model, the reference model has a larger number of parameters and stronger inference capabilities. The reference model can be selected according to the specific application scenario of the target model. For example, a large-scale or super-large-scale language model. The reference model can run on another cloud server (not shown in FIG. 1). The server for training the deep learning model in FIG. 1 can call the reference model through an application program interface (API) of the reference model to use the inference capabilities of the reference model. In addition, the reference model can also run on the server for training the deep learning model in FIG. 1. The end-side device can be an electronic device running a downstream application. The end-side device can be a hardware device with network communication functions, computing functions, and information display functions, including but not limited to a smartphone, a tablet computer, a desktop computer, a local server, a cloud server, and the like. The end-side device needs to use the inference capabilities of the target model when running the downstream application. The downstream application running on the end-side device can implement a natural language processing task such as text summarization, translation, and question answering, or a voice-to-text task, or an image-to-text task, or a natural language understanding (NLU) task such as emotion analysis, and the like.In implementing the functions of downstream applications, the inference capability of the target model needs to be used to generate a response result of the input information by the target model based on the given input information. The end-side device implements the subsequent processing logic of the downstream application based on the response result of the input information. Based on the system architecture shown in FIG. 1, the end-side device sends the input information of a user to the server. The server receives the input information, inputs the input information into the target model, generates a response result of the input information through the target model, and returns the response result of the input information to the end-side device. The end-side device receives the response result of the input information returned by the server, and continues to execute the subsequent processing logic based on the response result of the input information. Illustratively, the server can provide the API of the trained target model to the end-side device. The end-side device can send a calling request for the target model to the server based on the API of the target model, and the calling request contains the input information of the user. The server receives the calling request for the target model sent by the end-side device, extracts the input information of the user from the calling request, generates a response result of the output information through the target model, and returns the response result to the end-side device. In addition, the server can also provide the trained target model (or the parameters of the trained target model) to the end-side device, and the end-side device can use the trained target model locally to implement specific tasks of downstream applications based on the inference capability of the target model, such as natural language processing tasks such as text summarization, translation, question answering, or speech-to-text tasks, or image-to-text tasks, or natural language understanding (NLU) tasks such as emotion analysis, etc. Of course, the trained target model can also be applied to other tasks in the field of artificial intelligence, which is not limited here. The technical solutions of the present disclosure and how the technical solutions of the present disclosure solve the above technical problems will be described in detail in the specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes mayAs shown in FIG. 2, the method specifically comprises the following steps: in step S201, a reference model is used to screen instructions with a model fitting difficulty satisfying a preset condition from an existing instruction data set as seed instructions. The existing instruction data set can be derived from a public instruction data set, such as an Alpaca data set, a Guanaco data set, or any other open source data set used for fine-tuning of a large language model instruction. The existing instruction data set can also be derived from a private data set provided by a user through an end-side device, or an instruction data automatically constructed by a large model, which is not specifically limited in this embodiment. The instruction data set usually includes instruction-response pairs, and each instruction-response pair includes an instruction and a response to the instruction. In this embodiment, the reference model for knowledge distillation can be configured and specified by a related technical personnel according to user requirements, or specified by the user through an end-side device. The reference model refers to a pre-trained model with strong capabilities. Compared with a deep learning model, the reference model has a larger parameter quantity and stronger inference capability. Specifically, a pre-trained model with strong capabilities applied in the same scenario can be selected as the reference model according to the specific application scenario of the target model, such as a large-scale or super-large-scale language model. The deep learning model to be trained can be specified by the user through an end-side device, or configured and specified by a related technical personnel according to user requirements. The deep learning model can be a deep learning model for natural language processing based on information of at least one modality, such as various human-computer interaction models, models for implementing text-to-text generation tasks (such as text summarization, translation, and question answering), voice-to-text models (such as speech recognition models), image-to-text models (such as OCR models and image summaries), natural language understanding (NLU) models for emotion analysis, and the like. The deep learning model can be a lightweight model, a pre-trained model, or a large model with a relatively small parameter quantity, a multi-modal language model, and the like, which is not specifically limited in this embodiment. In an example scenario, when a user needs to fine-tune an instruction of a deep learning model, a model training request is sent to a server through an end-side device, and the model training request contains information of the deep learning model to be trained and the reference model. In response to the model training request of the end-side device, the server can determine the reference model specified by the end-side device and the deep learning model to be trained based on the information of the deep learning model to be trained and the reference model. The parameter quantity of the deep learning model is less than that of the reference model. In another example scenario, in order to reduce the computing cost, a lightweight deep learning model is used to replace a large model to provide services to an end-side device on the server side.The server can use the large model as a reference model, distill knowledge from the reference model, and train a lightweight deep learning model. The target model obtained after training can maintain relatively high performance while reducing the demand for computing resources. The server uses the target model obtained after training to replace the large model to provide services to the terminal device. In this step, the model fitting difficulty of the instructions in the instruction data set is evaluated based on the reference model, and instructions with model fitting difficulty meeting a preset condition are selected from the existing instruction data set to obtain challenging instructions as seed instructions for constructing a challenging training set. The preset condition can be that the model fitting difficulty is greater than or equal to a difficulty threshold. Based on the preset condition, challenging instructions with relatively large model fitting difficulty can be selected. The difficulty threshold 6 can be configured and adjusted according to the value range of the instruction response quality evaluation value and combined with the actual application scenario. For example, the value range of the instruction response quality evaluation value can be [1, 10], and the difficulty threshold 6 can be 2 or 3, etc. Here, no specific limitation is made. In another optional embodiment, the preset condition can be that the task type of the instruction belongs to a predefined task type set. The task type set includes various task types with relatively large model fitting difficulty and specific challenges, including but not limited to: mathematical, logical reasoning, and programming related tasks. In this step, the server judges the task type of the instructions in the existing instruction data set based on the reference model, and selects instructions with task types belonging to the predefined task type set from the instruction data set as instructions with model fitting difficulty meeting the preset condition, to obtain challenging instructions as seed instructions for constructing a challenging training set. Step S202, based on the seed instructions, a plurality of instruction samples are generated, and a training set containing the instruction samples and the reference responses of the instruction samples is constructed. After selecting the seed instructions with relatively large model fitting difficulty, the server generates a plurality of similar instructions based on the seed instructions by using the reference model, as instruction samples for constructing the training set. Further, the server generates the reference responses of the instruction samples based on the reference model to obtain instruction sample-reference response pairs, and constructs a training set containing the instruction samples and the reference responses of the instruction samples (i.e., instruction sample-reference response pairs). The training set can be used for instruction fine-tuning and contains specific challenging instruction data. The reference response of the instruction sample is a high-quality response information to the instruction sample, which can be understood as the correct response or the labeled response of the instruction sample. Step S203, a deep learning model is trained using the training set to obtain a target model obtained after training. After obtaining the challenging training set, the deep learning model is fine-tuned using the training set to obtain a target model obtained after training.In an optional implementation of the step, the server inputs the instruction sample in the training set into the deep learning model to perform response prediction, and determines a probability that the deep learning model predicts the reference response of the instruction sample. According to the resampling weight of the instruction sample and the probability that the deep learning model predicts the reference response of the instruction sample, the first loss is calculated. The parameters of the deep learning model are adjusted according to the first loss, and a target model trained is obtained. The resampling weight of the instruction sample corresponds to the task type of the instruction sample, and instruction samples of the same task type have the same resampling weight. The resampling weights corresponding to different task types can be different. For a complex and challenging task type, such as a mathematical, logical reasoning, programming related task type, a larger resampling weight can be configured, and a smaller resampling weight is configured for other simpler task types. In this implementation, on the basis of the autoregressive causal language modeling loss used in the existing instruction fine-tuning, the resampling weight is added to the instruction sample of different task types, the loss of different instruction samples is weighted, the model has task perception ability in the instruction fine-tuning process, and the model can pay more attention to complex and challenging tasks, so as to improve the ability of the model to process more complex and challenging tasks. Exemplarily, the first loss can be calculated according to the resampling weight of the instruction sample and the probability that the deep learning model predicts the reference response of the instruction sample by using the following formula (1): L (0) = -∑i∈T∑j∈T (x i,x j ) e D s P r [T (x i,x j ) ] log P r [y i,y j ] (1) where L (0) represents the first loss, represents the parameters of the deep learning model, T represents the training set constructed based on the seed instruction, represents an arbitrary instruction sample-reference response pair in the training set T, represents the instruction sample, represents the reference response of the instruction sample x i, and represents T (x i,x j ). Pr[T(x £ ,y £) represents the resampling weight corresponding to the task type T(i, yi). Pr[y| i, 1] represents the probability that the reference response force of the instruction sample is predicted based on the deep learning model, that is, the probability that the corresponding correct prediction result is predicted by the deep learning model. In another optional embodiment of this step, the server inputs the instruction sample in the training set into the deep learning model to perform response prediction, determines the probability that the reference response of the instruction sample is predicted by the deep learning model, calculates a second loss according to the probability that the reference response of the instruction sample is predicted by the deep learning model, adjusts the parameters of the deep learning model according to the second loss, and obtains a trained target model. In this embodiment, the server directly uses the autoregressive causal language modeling loss as the second loss to train the deep learning model. Illustratively, the second loss can be calculated according to the probability that the reference response of the instruction sample is predicted by the deep learning model by using the following formula (2): log Pr[y £ |x £, <I)] (2) Wherein, L2(0) represents the second loss, the meanings of other characters in formula (2) are consistent with those in formula (1), and details are not described herein. In this embodiment, the training strategy used for training the deep learning model using the training set can be configured and adjusted according to the actual application scenario, including but not limited to the selected optimization algorithm, learning rate and the like, and details are not limited herein. After obtaining the trained target model, the server can provide the target model to the end-side device. Illustratively, after obtaining the trained target model, the server can provide a calling interface of the target model to the end-side device. The end-side device can send a calling request of the target model to the server based on the calling interface of the target model, and the calling request contains the input information of the user. The server receives the calling request of the target model sent by the end-side device, extracts the input information of the user from the calling request, generates a response result of the output information through the target model, and returns the response result to the end-side device. Illustratively, in the case that the end-side device stores the structure of the deep learning model, after obtaining the trained target model, the server can send the parameters of the target model to the end-side device. The end-side device initializes the parameters of the locally stored deep learning model according to the parameters of the target model, and obtains the trained target model. Illustratively, the server can also send the download information of the target model to the end-side device. The end-side device downloads the pre-trained target model to the local according to the received download information of the target model. The server provides the target model obtained through the knowledge distillation to the end-side device, so that the end-side device can store the target model locally, and the end-side device can use the inference ability of the target model locally. The scheme of this embodiment can filter out instructions with a model fitting difficulty satisfying a preset condition from the existing instruction data set by means of the reference model, can filter out instructions with a higher model fitting difficulty as seed instructions, can obtain more challenging instruction samples by expanding and generating a plurality of similar instruction samples based on the seed instructions, and can construct a training set containing the instruction samples and the reference responses of the instruction samples. The training set of challenging instruction data is obtained by means of the reference model based on the existing instruction data set through the knowledge distillation. Training the deep learning model using the training set can improve the ability of the deep learning model to process more complex and challenging tasks, and improve the performance of the target model after training.In an optional embodiment, in the step S201, the instructions in the existing instruction dataset that meet the preset condition in terms of fitting difficulty are screened out as seed instructions by means of the reference model, which can be implemented in the following manner: a deep learning model is fine-tuned using the instruction dataset to obtain a fine-tuned model; a predicted response of an instruction in the instruction dataset is generated by the fine-tuned model, and a reference response of the instruction is generated by the reference model; a model fitting difficulty (MFD for short) of the instruction is determined according to the quality difference between the predicted response and the reference response, and the instructions that meet the preset condition in terms of model fitting difficulty are screened out as seed instructions. The preset condition can be that the model fitting difficulty is greater than or equal to a difficulty threshold value. The difficulty threshold value 6 can be configured and adjusted according to the value range of the instruction response quality evaluation value and in combination with the actual application scenario, for example, the value range of the instruction response quality evaluation value can be [1, 10], and the difficulty threshold value 6 can be 2 or 3, etc., which is not limited here. FIG. 3 is a flowchart of screening seed instructions provided by an embodiment of the present disclosure. As shown in FIG. 3, the specific process of screening seed instructions that meet the preset condition in terms of fitting difficulty from the existing instruction dataset by means of the reference model is as follows: Step S300, an existing instruction dataset is obtained. In this embodiment, the existing instruction dataset can be derived from a public instruction dataset, such as an Alpaca dataset, a Guanaco dataset, or any other open-source dataset for fine-tuning of a large language model instruction. The existing instruction dataset can also be derived from a private dataset provided by a user through an end-side device, or an instruction data automatically constructed by a large model, which is not limited here. The instruction dataset usually includes instruction-response pairs, which include instructions and responses. Step S301, a deep learning model is fine-tuned using the instruction dataset to obtain a fine-tuned model. In this step, the existing instruction dataset is used to fine-tune a deep learning model (denoted as S) to obtain a fine-tuned model (denoted as So). The process of instruction fine-tuning can be implemented by any existing instruction fine-tuning method, which is not limited here. Step S302, a predicted response of an instruction in the instruction dataset is generated by the fine-tuned model. In this step, a response of an instruction in the instruction dataset is generated by the fine-tuned model So after instruction fine-tuning using the instruction dataset, which is referred to as a predicted response. This step aims to evaluate the basic ability of the deep learning model to learn from the instruction dataset without complex training. Step S303, a reference response of an instruction in the instruction dataset is generated by the reference model. In this step, a response of an instruction in the instruction dataset is generated by the reference model, which is referred to as a reference response.In step S304, the quality of the predicted response and the reference response of the instruction is evaluated by a judging model, to obtain a quality evaluation value of the predicted response and a quality evaluation value of the reference response. The judging model is used to evaluate the response quality of the predicted response generated by the fine-tuning model S0 and the reference response generated by the reference model, to obtain the quality evaluation value of the predicted response and the quality evaluation value of the reference response. The judging model can be implemented using any existing algorithm / model that can evaluate the response quality of an instruction, for example, the judging model can be implemented using a large language model LLM. Illustratively, preconfigured quality evaluation prompt information is obtained, the instruction, the predicted response and the reference response of the instruction are filled into the quality evaluation prompt information, and then the judging model is input. The judging model scores the predicted response and the reference response of the instruction based on the prompt of the quality evaluation prompt information, to obtain the quality evaluation value (i.e. the score) of the predicted response and the reference response of the instruction. In addition, the quality evaluation prompt information can also define other rules or requirements for quality evaluation, such as output format, evaluation basis / reason, etc., which are not limited here. For example, the following is an example of a quality evaluation prompt information:

[0003] You are an AI assistant that evaluates the quality of responses from other AI assistants based on the given user instruction: {Instruction}, the response from AI assistant 1: {answer}, and the response from AI assistant 2: {answer_2}. We want you to provide feedback on the performance of both AI assistants in responding to the given user instruction. Please rate their answers on usefulness, relevance, accuracy, and detail. Each AI assistant will receive a score between 1 and 10, where a higher score indicates better overall performance. Please first explain your evaluation, avoiding any potential biases, and ensure that the order of the responses does not affect your judgment. Then, output the scores for AI assistant 1 and AI assistant 2 in two separate lines. The output format is as follows: Evaluation based on: 〈your evaluation explanation here〉 Score for AI assistant 1: VScore1 > Score for AI assistant 2: VScore2 > Where {Instruction} represents the instruction to be filled, {answer} and {answer_2} represent the predicted and reference responses to be filled, 〈your evaluation explanation here〉 represents the evaluation basis (reason) output by the model, VScore1 > represents the quality evaluation value of the predicted response {answer} output by the model, and VScore2 > represents the quality evaluation value of the reference response {answer_2} output by the model. In this example, the quality evaluation value of each response is constrained to be within the interval [1, 10]. Based on the quality evaluation prompt information of this example, the judge model can obtain the quality evaluation values of the two responses (including the predicted response and the reference response) of the instruction at one time through one inference, and the quality evaluation values of the predicted response and the reference response are more comparable. In another optional embodiment, the judge model can evaluate the predicted response of the instruction based on the instruction and the predicted response of the instruction to obtain the quality evaluation value of the predicted response of the instruction; the judge model evaluates the reference response of the instruction based on the instruction and the reference response of the instruction to obtain the quality evaluation value of the reference response of the instruction. By independently evaluating the quality of the predicted response and the reference response of the instruction, the quality evaluation values of the two responses are obtained. Step S305, determining the model fitting difficulty of the instruction based on the difference between the quality evaluation value of the reference response of the instruction and the quality evaluation value of the predicted response. After obtaining the quality evaluation value of the reference response of the instruction and the quality evaluation value of the predicted response, the reference response of the instruction is taken as the reference to calculate the difference between the quality evaluation value of the reference response of the instruction and the quality evaluation value of the predicted response. The larger the difference, the worse the quality of the predicted response generated by the deep learning model compared to the reference response generated by the reference model.The smaller the difference (greater than 0) is, the closer the quality of the predicted response generated by the deep learning model to the quality of the reference response generated by the reference model is, but still worse than the quality of the reference response generated by the reference model. When the difference is less than 0, the quality of the predicted response generated by the deep learning model has surpassed the quality of the reference response generated by the reference model. Further, the difference between the quality evaluation value of the reference response and the quality evaluation value of the predicted response of the instruction is taken as the model fitting difficulty (MFD) of the instruction. Optionally, the difference between the quality evaluation value of the reference response and the quality evaluation value of the predicted response of the instruction can also be multiplied by a scaling coefficient to be taken as the model fitting difficulty (MFD) of the instruction. The scaling coefficient can be configured according to actual application requirements and experience values, which is not specifically limited here. In the embodiment, in order to construct a challenging seed instruction, the model fitting difficulty (MFD) evaluation index is introduced to measure the model fitting difficulty of the instruction, that is, the difficulty of the model to correctly respond to the instruction. Through the foregoing steps S304-S305, the model fitting difficulty of the instruction is determined according to the quality difference between the predicted response and the reference response of the instruction. Based on the model fitting difficulty (MFD) of the instruction, a challenging seed instruction can be selectively screened from the existing instruction data set. Step S306: Screen the instructions whose model fitting difficulty meets a preset condition from the instruction data set as seed instructions. In the embodiment, in order to screen a challenging instruction, a preset condition is configured as a screening condition. After obtaining the model fitting difficulty (MFD) of each instruction in the instruction data set, in this step, the instructions whose model fitting difficulty (MFD) meets the preset condition are screened from the instruction data set as seed instructions according to the preset condition. The preset condition can be that the model fitting difficulty is greater than or equal to a difficulty threshold, and based on the preset condition, a challenging instruction with a greater model fitting difficulty can be screened. The difficulty threshold 6 can be configured and adjusted according to the value range of the instruction response quality evaluation value and combined with the actual application scenario, for example, the value range of the instruction response quality evaluation value can be [1, 10], the difficulty threshold 6 can take a value of 2 or 3, etc., which is not specifically limited here. In another optional embodiment, the preset condition can be that the task type of the instruction belongs to a predefined task type set. The task type set contains various task types with greater model fitting difficulty and specific challenges, including but not limited to: mathematical, logical reasoning, programming related tasks.In this step, the server determines the task type of the instructions in the existing instruction dataset by referring to the reference model, and filters out the instructions whose task types belong to the predefined task type set from the instruction dataset as instructions whose model fitting difficulty satisfies the preset condition, obtains the challenging instructions as the seed instructions for constructing the challenging training set. The method of the embodiment generates the predicted response of the instructions in the instruction dataset by using the fine-tuned model of the deep learning model after fine-tuning on the existing instruction dataset, and generates the reference response of the instructions in the instruction dataset by referring to the reference model; obtains the instruction evaluation value of the reference response and the predicted response of the instructions by the evaluation of the adjudication model, determines the model fitting difficulty (MFD) of the instructions based on the difference between the reference response and the predicted response of the instructions as an index for measuring the model fitting difficulty and the challenge of the instructions, and uses the model fitting difficulty (MFD) to filter out the challenging seed instructions to construct the challenging training set, which can encourage the deep learning model to process more challenging tasks, thereby avoiding the possibility that the model is biased towards simple tasks, improving the ability of the deep learning model to process more complex and challenging tasks, and improving the performance of the target model after training. FIG. 4 is a flowchart of constructing a training set based on seed instructions provided by an example embodiment of the disclosure. In an optional embodiment, based on the filtered challenging seed instructions, strategic instruction distillation is performed, and a training set is constructed to ensure the balance of the distribution of instructions of different task types in the training set, so that the performance of the model on each task type is more balanced. In the embodiment, as shown in FIG. 4, in the foregoing step S202, a plurality of instruction samples are generated based on the seed instructions, and a training set containing the instruction samples is constructed, which can be realized in the following manner: In step S2021, the reference model is used to generate similar instructions and responses of the same task type based on the seed instructions and the task type of the seed instructions as instruction samples and reference responses of the instruction samples. In the embodiment, the reference model is used to generate similar instructions based on the seed instructions. The similar instructions have the same task type as the seed instructions, and are also comparable to the seed instructions in terms of instruction length and difficulty.For example, the following is an example of an instruction generation prompt:

[0004] "You play the role of an instruction creator. Your goal is to draw inspiration from the given seed instruction and create a brand new instruction. This new instruction should have the same task type as the given seed instruction: [{task_type}]. The length and difficulty of the new instruction should be similar to the given seed instruction. The content of the new instruction should be different from the given seed instruction. The new instruction must be reasonable and must be understood and responded to by a human. Given seed instruction: {Instruction} New instruction: 〈instruction〉". Where {Instruction} represents the seed instruction to be filled in. {task_type} represents the task type to be filled in, which is the task type of the seed instruction. 〈instruction> represents the new instruction generated by the reference model. Further, after obtaining the generated instruction sample, a reference response of the instruction sample is generated using the reference model. Illustratively, a pre-configured response generation prompt is obtained, the instruction sample is filled into the response generation prompt, and the reference model is input. The reference response of the instruction sample is generated by the reference model based on the prompt of the response generation prompt. The response generation prompt can use the prompt of any existing question and answer model, which is not specifically limited here. For example, an example of a response generation prompt is as follows:

[0005] You are an intelligent assistant capable of responding to given instructions. Given instructions: {Instruction} Your response: <answer>. Wherein {Instruction} represents an instruction sample to be filled in, and <answer> represents a reference response generated by the reference model. In an optional embodiment, when generating instruction samples, the reference model is required to generate instruction samples in a task proportion that matches a balanced quota, that is, the proportion of instruction samples of each task type in the generated instruction samples matches the preset task proportion. It is found through research that the performance of the language model on the three types of tasks of mathematics, logical reasoning and programming will continue to grow with the increase of the amount of instruction fine-tuning data, while the performance on other tasks will slow down after the instruction fine-tuning data reaches a certain amount. Therefore, in an optional embodiment, the task proportion of each task type is configured, wherein the proportion of key task types such as mathematics, logical reasoning and programming is increased. For example, FIG. 5 provides a schematic diagram of the proportion of instruction data of each task type in the existing instruction data set, and FIG. 6 provides a schematic diagram of the configured task proportion of each task type. In FIG. 5 and FIG. 6, the horizontal coordinate represents the amount of instruction data, and the vertical coordinate represents different task types of instruction data. As shown in FIG. 5, in the existing instruction data set, a large proportion of instruction data belongs to simple tasks such as common sense (Common-Sense), technology (Technology), grammar (Grammar), literature (Literature) and others (Others), while instruction data of challenging task types such as mathematics (Math), logical reasoning (Reasoning), code (including code generation (Code Generation) and code debugging (Code Debug) ) is relatively small. Compared with the proportion of each task type in the existing instruction data set shown in FIG. 5, the task proportion of each task type configured in the method of the present embodiment is shown in FIG. 6, that is, instruction data of challenging task types such as mathematics (Math), logical reasoning (Reasoning), code (including code generation (Code Generation) and code debugging (Code Debug) ) is dominant, the proportion of key task types such as mathematics, logical reasoning and programming is increased, and the proportion of instruction data of other simple tasks is small.Exemplarily, the task proportions of each task type can be configured as follows: the proportions of mathematics, logical reasoning, code generation and code debugging are 0.167:0.167:0.083:0.083, and the four task types account for 50%, and the remaining tasks are evenly distributed in the remaining 50% quota. Each task type and the task proportion of each task type can be configured according to the needs of the actual application scene, which is not limited here. It should be noted that in addition to the above-mentioned challenging task types such as mathematics (Math), logical reasoning (Reasoning), code (including code generation (Code Generation) and code debugging (Code Debug)), writing (Writing), computer science (Computer Science), law (Law), medicine (Medicine), art (Art), music (Music) and other task types in FIG. 5 and FIG. 6, other task types in FIG. 5 and FIG. 6 may not be completely consistent. As for the instruction data set in addition to the challenging task types such as mathematics (Math), logical reasoning (Reasoning), code (including code generation (Code Generation) and code debugging (Code Debug)), the more comprehensive the task types covered are, and as for which other task types the instruction data includes, it can be configured according to actual needs, which will not affect the technical effects of the present scheme. FIG. 5 and FIG. 6 emphasize the proportion of instruction data of challenging task types such as mathematics (Math), logical reasoning (Reasoning), code (including code generation (Code Generation) and code debugging (Code Debug)), and other task types not mentioned in FIG. 5 and FIG. 6 can refer to the instruction data of the corresponding task types in the existing public data set, which will not be mentioned here. Optionally, the server can configure the resampling weight of each task type according to the preset task proportion, sample seed instructions of each task type from the seed instructions according to the resampling weight of each task type, and generate similar instructions of the same task type as instruction samples according to the sampled seed instructions. The number of similar instructions generated based on each seed instruction is the same. For task types with greater resampling weights, the number of seed instructions of the corresponding task type is sampled more frequently, and the number of instruction samples of the corresponding task type is greater. In this way, the proportion of instruction data of each task type in the constructed training set can be controlled.Optionally, the server can randomly sample seed instructions of each task type to generate instruction samples of the corresponding task type, and construct a training set containing all generated instruction samples. Then, according to the preset task proportion, some instruction samples in the training set are deleted, so that the proportion of instruction data of each task type in the adjusted training set matches the preset task proportion. In an optional embodiment, when generating the reference response of the instruction sample by referring to the model, the server obtains the prompt information of the corresponding task type according to the task type of the instruction sample. The prompt information of different task types can be different. The prompt information of any task type contains the detailed requirements for responding to the instructions of the corresponding task type, which is used to prompt the reference model to generate better reference responses for the corresponding task type. For complex task types such as mathematics, logical reasoning, code generation or code debugging, etc. The prompt information of any complex task type can also contain a chain of thought (Chain of Thought, CoT), which splits the reasoning process of the instructions of the corresponding task type into multiple steps to prompt the reference model to generate the reference response of the instruction sample in steps, which can improve the quality of the generated reference response. Further, the reference model generates the reference response of the instruction sample according to the prompt information of the corresponding task type. For example, a simple example of the prompt information of the code generation type is as follows:

[0006] "You are an intelligent assistant who can respond to given task instructions. Please provide a code snippet with annotations, and then explain it. Task instruction: {Instruction} Your response: < answer > ". Where {Instruction} represents a to-be-filled instruction sample, and < answer > represents the reference response generated by the reference model. The prompt information "Please provide a code snippet with annotations, and then explain it" is a special requirement for the code generation task, which requires the generated code snippet to give annotations and give explanations of the code. For example, a simple example of the prompt information containing a chain of thought is as follows:

[0007] "You are an intelligent assistant who can respond to given task instructions. Please follow the steps below to think sequentially and answer in detail: Step 1..; Step 2.. Task instruction: {Instruction} Your response: < answer >". Where {Instruction} represents a sample instruction to be filled in, and < answer > represents the reference response generated by the reference model. The prompt information "Please follow the steps below to think sequentially and answer in detail: Step 1. ……; Step 2. ……" gives the reasoning steps of the thinking chain definition, and the reference model is required to think sequentially based on the reasoning steps of the thinking chain definition and answer in detail, that is, to give the reference response by step-by-step reasoning. Here is just an example, the specific content of each step is omitted, and the specific steps can be configured according to the actual task of the application. In this embodiment, if the instruction data set does not contain the task type of the instruction data, the reference model can be used to classify the task of the instruction. Illustratively, after filling in the instruction into the task classification prompt information, inputting the reference model, and determining the task type of the instruction according to the prompt of the task classification prompt information through the reference model. The task classification prompt information is used to prompt the reference model to classify the given instruction, and the specific content of the task classification prompt information can be configured according to the requirements of the actual application scene, which is not limited here. For example, based on the task types shown in FIG. 6, an example of the task classification prompt information is as follows:

[0008] "You are an intelligent assistant who can classify the task type or field of a given instruction and explain the basis for the classification: The task type or field is as follows: ['Math', 'Code Generation 5 , 'Writing', 'Computer

[0009] Science\ 'Reasoning', 'Complex Format', 5 Code Debug 5 ,

[0010] ' Common-Sense , , 5 Counterf actual 5 , 'Multilingual', 'Roleplay',

[0011] ' Biology 5 ,,Technology',,Ethics',, Sport',,Law',

[0012] ' Medicine 5 , 'Literature', 'Entertainment',,Art', 'Music',

[0013] ' Toxicity', 'Economy', 'Physics', 'History', 'Chemistry',

[0014] ' Philosophy', 'Health', 'Ecology', 'Grammar', 'Paraphrase', ’Others'] Instruction: {Instruction} Task Type: < classification>. Wherein, {Instruction} represents the instruction to be filled in, and < classification> represents the reference response generated by the reference model. The list of task types in the prompt information can be configured and challenged according to the actual application scenario, which is not specifically limited here. Step S2022, constructing a training set containing instruction samples and reference responses of instruction samples. After obtaining the instruction samples and reference responses of the instruction samples matched with the preset task proportion, a training set containing these instruction samples and reference responses of the instruction samples is constructed. The method of the embodiment generates similar instructions and responses of the same task type based on the seed instructions and the task types of the seed instructions by the reference model, as the instruction samples and the reference responses of the instruction samples, strategically instructs the instruction distillation based on the selected challenging seed instructions, and constructs a more challenging training set, which can improve the ability of the model to handle more complex and challenging tasks. In addition, the preset task proportion is configured in the embodiment, wherein the proportion of key task types such as mathematics, logical reasoning and programming is increased, and the proportion of the instruction samples of each task type in the generated instruction samples is matched with the preset task proportion, so as to ensure the balance of the distribution of instructions of different task types in the constructed training set, so that the performance of the model on each task type is more balanced, and the performance and instruction response quality of the model can be improved. Fig. 7 is a flow chart of the method for training the model provided by another exemplary embodiment of the present disclosure. Based on any of the foregoing embodiments, in the first round of iterative training, based on the fine-tuned model obtained by fine-tuning the deep learning model using the existing instruction data set, the reference model is used to select seed instructions from the instruction data set whose model fitting difficulty meets the preset condition as the seed instructions of the first round, the first round of training set is constructed based on the seed instructions of the first round, and the initial deep learning model is trained using the first round of training set. In the subsequent rounds of iterative training, based on the deep learning model obtained by the previous round of iterative training, the reference model is used to select instruction samples from the training set of the previous round whose model fitting difficulty meets the preset condition as the seed instructions of the current round, the training set of the current round is constructed based on the seed instructions of the current round, and the deep learning model obtained by the previous round of iterative training is further iteratively trained based on the training set of the current round. Until the iterative training is completed, the trained deep learning model is taken as the target model of the training.As shown in FIG. 7, the method specifically comprises the following steps: step S700, obtaining an existing instruction dataset. The existing instruction dataset can be derived from a public instruction dataset, such as an Alpaca dataset, a Guanaco dataset, or any other open-source dataset used for fine-tuning of instructions for a large language model. The existing instruction dataset can also be derived from a private dataset provided by a user through an end-side device, or an instruction data automatically constructed by a large model, which is not specifically limited in this embodiment. The instruction dataset generally includes instruction-response pairs, and each instruction-response pair includes an instruction and a response to the instruction. In this embodiment, in the first round of iterative training, a fine-tuned model is obtained by fine-tuning a deep learning model based on the existing instruction dataset, and a seed instruction with a model fitting difficulty meeting a preset condition is filtered from the instruction dataset by a reference model, as a seed instruction of the first round. A training set of the first round is constructed based on the seed instruction of the first round, and an initial deep learning model is trained using the training set of the first round. For details, see steps S701-S703. Step S701, filtering an instruction with a model fitting difficulty meeting a preset condition from the existing instruction dataset by a reference model, as a seed instruction of the first round. In this step, the implementation principle of filtering an instruction with a model fitting difficulty meeting a preset condition from the existing instruction dataset by a reference model is consistent with the implementation principle of step S201, and for details, see the related contents of the foregoing embodiments, which will not be repeated here. In this embodiment, the instruction with a model fitting difficulty meeting a preset condition filtered from the existing instruction dataset is used as a seed instruction in the first round of iterative training, which is referred to as a seed instruction of the first round. Step S702, generating a plurality of instruction samples based on the seed instruction of the first round, and constructing a training set of the first round. The implementation principle of this step is consistent with the implementation principle of step S202, and for details, see the related contents of the foregoing embodiments, which will not be repeated here. Step S703, training an initial deep learning model using the training set of the first round. In the process of the first round of iterative training, the deep learning model is fine-tuned using the training set of the first round, and a deep learning model obtained in the first round of iterative training is obtained. In an optional implementation of this step, on the basis of the autoregressive causal language modeling loss used in the existing instruction fine-tuning, a resampling weight is added to the instruction samples of different tasks, and the losses of different instruction samples are weighted, so that the model has task perception ability in the instruction fine-tuning process, and can pay more attention to complex and challenging tasks, thereby improving the ability of the model to process more complex and challenging tasks.Specifically, the server inputs the instruction samples in the first round of the training set into the deep learning model to predict responses, and determines probabilities of the deep learning model predicting reference responses of the instruction samples. According to the resampling weights of the instruction samples and the probabilities of the deep learning model predicting reference responses of the instruction samples, a third loss is calculated. According to the third loss, the parameters of the deep learning model are adjusted to obtain the deep learning model after the first round of iterative training. The resampling weight of an instruction sample corresponds to the task type of the instruction sample, and instruction samples of the same task type have the same resampling weight. The resampling weights corresponding to different task types can be different. For a more complex and challenging task type, such as a mathematical, logical reasoning, programming related task type, a larger resampling weight can be configured, and a smaller resampling weight is configured for other simpler task types. The principle of calculating the third loss is consistent with the principle of calculating the first loss in the foregoing step S203, and details are referred to the related content of the foregoing embodiments, which will not be described here. In another optional implementation of the step, the server uses an autoregressive causal language modeling loss to train the deep learning model. Specifically, the server inputs the instruction samples in the first round of the training set into the deep learning model to predict responses, and determines probabilities of the deep learning model predicting reference responses of the instruction samples. According to the probabilities of the deep learning model predicting reference responses of the instruction samples, a fourth loss is calculated. According to the fourth loss, the parameters of the deep learning model are adjusted to obtain the deep learning model after the first round of iterative training. The principle of calculating the fourth loss is consistent with the principle of calculating the second loss in the foregoing step S203, and details are referred to the related content of the foregoing embodiments, which will not be described here. In the second and subsequent rounds of iterative training, based on the deep learning model obtained in the previous round of iterative training, the server screens instruction samples with a model fitting difficulty meeting a preset condition from the training set of the previous round by means of the reference model, as seed instructions of the current round, constructs a training set of the current round based on the seed instructions of the current round, and continues to perform iterative training on the deep learning model obtained in the previous round of iterative training based on the training set of the current round. Details are referred to steps S704-S707o until the iterative training ends, and the trained deep learning model is taken as a target model after training. In step S704, based on the deep learning model obtained in the previous round of iterative training, the server screens instruction samples with a model fitting difficulty meeting a preset condition from the training set of the previous round by means of the reference model, as seed instructions of the current round.In this step, the deep learning model obtained through the previous round of iterative training generates the predicted response of the instruction sample in the previous round of training set; the reference model generates the reference response of the instruction sample; the model fitting difficulty of the instruction sample is determined according to the quality difference between the predicted response and the reference response of the instruction sample; the instruction sample whose model fitting difficulty meets the preset condition is selected from the training set of the previous round as the seed instruction of this round. Wherein, the model fitting difficulty of the instruction sample is determined according to the quality difference between the predicted response and the reference response of the instruction sample, which is similar to the implementation principle of steps S304-S305 described above, and specific reference is made to the related contents of the previous embodiments, which will not be repeated here. Wherein, the implementation principle of selecting the instruction sample whose model fitting difficulty meets the preset condition from the training set of the previous round is similar to the implementation principle of step S306 described above, and specific reference is made to the related contents of the previous embodiments, which will not be repeated here. Step S705, constructing the training set of this round based on the seed instruction of this round. In this step, based on the seed instruction of this round and the task type of the seed instruction, similar instructions of the same task type are generated as instruction samples of this round, and the proportion of instruction samples of each task type in the instruction samples of this round matches the preset task proportion; the reference model generates the reference response of the instruction sample of this round; the training set containing the instruction sample of this round and the reference response of the instruction sample of this round is constructed as the training set of this round. The specific implementation principle of this step is similar to the implementation principle type of the previous step S202 of generating a plurality of instruction samples based on the seed instruction and constructing a training set containing the instruction sample, and the difference is that the seed instruction based on which the training set is constructed in this step is the seed instruction of this round. Specific reference is made to the related contents of the previous embodiments, which will not be repeated here. In this embodiment, the server can sample a plurality of instruction samples and reference responses of the instruction samples from the training set of the previous round and add them to the training set of this round, so that the training set not only contains the challenging instructions generated in this round, but also contains the relatively simple (relatively simple for the deep learning model after the first round of training) instructions in the training set of the previous round, so as to avoid the catastrophic forgetting of the model to the basic task (simple task in addition to the complex challenging task) by contacting the relatively simple instructions. Specifically, according to the preconfigured configuration proportion of the newly generated instructions corresponding to each round and the number of the newly generated instruction samples of this round currently contained in the training set of this round, a plurality of instruction samples and reference responses of the instruction samples are sampled from the training set of the previous round and added to the training set of this round, so that the proportion of the newly generated instruction samples in the training set of this round matches the configuration proportion of the newly generated instructions corresponding to this round. Wherein, the configuration proportion of the newly generated instructions corresponding to each round increases round by round.In this way, starting from the second iteration, in each subsequent training iteration round, the proportion of challenging instructions generated in the current round in the constructed training set is continuously increased, and the proportion of instructions in the training set of the previous round is continuously reduced, gradually transitioning from a training set with a low proportion of challenging instructions to a training set with a higher proportion of challenging instructions, and improving the ability of the model to handle complex challenging tasks through continuous iteration training. For example, the server can set a proportion factor a representing the proportion of newly generated instructions (challenging instructions generated in the current round) in each round of training set. The initial value of the proportion factor a is the proportion of newly generated instructions in the second round. In subsequent iterations, the proportion factor a is increased by a fixed proportion increment, that is, a(r + 1) = a(r)(l + Aa), where a(r) represents the proportion factor of the rth round, a(r + 1) represents the proportion factor of the (r + 1)th round, and Aa represents the fixed proportion increment. For example, the proportion factor a is 30% in the second round, and in subsequent iterations, the proportion factor a is increased by a fixed proportion increment of 10%. Therefore, the proportion factor a becomes 30% X (1+10%) in the third round, and the proportion factor a becomes 30% X (1+10%) X (1+10%) in the fourth round. For example, the server can configure the configuration proportion of newly generated instructions corresponding to each iteration. For example, the proportion of newly generated instructions in the training set of the second round is 30%, and the proportion of newly generated instructions in the training set is increased by a preset increment (such as 10%, or 20%, etc.) in subsequent iterations. For example, if the preset increment is 10%, the proportion of newly generated instructions in the training set of the third round is 40%, and the proportion of newly generated instructions in the training set of the fourth round is 50%, and so on, until the proportion of newly generated instructions in the training set reaches 100% and no longer increases. Optionally, the server can also configure the proportion of newly generated instructions in the training set not to exceed a preset maximum proportion (such as 100%, or 90%, etc.), wherein the preset maximum proportion can be configured according to the needs and experience values of the actual application scenario, which is not limited here. In an optional embodiment of step S706, the server inputs the instruction samples in the training set of the current round into the deep learning model to perform response prediction, determines the probability of the deep learning model predicting the reference response of the instruction sample, calculates a fifth loss according to the resampling weight of the instruction sample and the probability of the deep learning model predicting the reference response of the instruction sample, adjusts the parameters of the deep learning model according to the fifth loss, and obtains a trained target model.The resampling weight of the instruction sample corresponds to the task type of the instruction sample, and instruction samples of the same task type have the same resampling weight. The resampling weights corresponding to different task types can be different. For more complex and challenging task types such as mathematical, logical reasoning, programming related tasks and the like, a larger resampling weight can be configured, and a smaller resampling weight is configured for other simpler task types. In the implementation, on the basis of the autoregressive causal language modeling loss used in the existing instruction fine-tuning, the resampling weight is added to the instruction samples of different task types, the losses of different instruction samples are weighted, the model has task perception ability in the instruction fine-tuning process, and more attention can be paid to more complex and challenging tasks, so as to improve the ability of the model to process more complex and challenging tasks. Illustratively, the probability of the deep learning model predicting the reference response of the instruction sample can be calculated according to the resampling weight of the instruction sample and the following formula (3) to calculate the fifth loss: logPr [y £ |x £ , <t>] (3) Wherein, L5(0) represents the fifth loss, ① represents the parameters of the deep learning model. D, represents the training set of the rth round, r represents the round of this round. (x, y) represents an arbitrary instruction sample-reference response pair in the training set, x represents the instruction sample, y £ represents the reference response of the instruction sample x. T(x £ , y £ ) represents the task type of the instruction sample-reference response pair (x £ , y £ ). Pr[T(x £ , y £ , ①] represents the probability that the deep learning model predicts the reference response of the instruction sample based on the instruction sample. In other words, the probability that the deep learning model predicts the correct prediction result corresponding to the instruction sample. In another optional implementation of this step, the server can directly use the autoregressive causal language modeling loss as the sixth loss to train the deep learning model. Specifically, the server inputs the instruction sample in the training set of the current round into the deep learning model to perform response prediction, determines the probability that the deep learning model predicts the reference response of the instruction sample, calculates the sixth loss according to the probability that the deep learning model predicts the reference response of the instruction sample, adjusts the parameters of the deep learning model according to the sixth loss, and obtains the target model trained. Step S707, determine whether the iteration training end condition is met. The iteration training end condition can be model performance convergence, reaching the maximum number of iterations, etc. The specific configuration can be based on the actual application scenario and experience, which is not limited here. If the determination result is that the iteration training end condition is not met, the iteration training is not completed, and steps S704-S707 are executed in a loop to perform the next round of iteration training. If the determination result is that the iteration training end condition is met, the iteration training is completed, and step S708 is executed to take the deep learning model trained as the target model trained. Step S708, take the deep learning model trained as the target model trained. In the first round of iteration training, the method of this embodiment is based on the fine-tuned model obtained by fine-tuning the deep learning model using the existing instruction data set, screens the seed instruction with a model fitting difficulty meeting the preset condition from the instruction data set with the aid of the reference model, takes the seed instruction of the first round as the seed instruction of the first round, constructs the training set of the first round based on the seed instruction of the first round, and trains the initial deep learning model using the training set of the first round. In each subsequent round of iteration training, the deep learning model obtained in the previous round of iteration training is used to screen the instruction sample with a model fitting difficulty meeting the preset condition from the training set of the previous round with the aid of the reference model, take the seed instruction of the current round as the seed instruction of the current round, construct the training set of the current round based on the seed instruction of the current round, and continue to perform iteration training on the deep learning model obtained in the previous round of iteration training based on the training set of the current round. Until the iteration training is completed, the deep learning model trained is taken as the target model trained. By introducing the multi-round course planning idea, the model fitting difficulty of the training set is dynamically adjusted in the continuous multiple rounds of iteration training, and the ability of the model to handle complex challenging tasks is improved through continuous rounds of iteration training. FIG. 8 is a framework diagram of model training provided by this embodiment. As shown in FIG. 8, based on the instruction in the existing instruction data set, the predicted response of the instruction is generated by the deep learning model (fine-tuned by the instruction data set), and the reference response of the instruction is generated by the reference model.In addition, when generating the reference response of the instruction by referring to the model, prompt information containing a thinking chain or task type specific prompt information containing a detailed description of problem solving can be used to improve the quality of the generated reference response. Further, based on the predicted response of the instruction and the reference response, the screening and distillation of the instruction is performed by means of the judge model to obtain the challenging instruction and construct the training set. Specifically, the predicted response of the instruction and the reference response are evaluated by the judge model, the model fitting difficulty of the instruction is determined based on the difference between the quality evaluation values of the predicted response of the instruction and the reference response, and the seed instruction with higher model fitting difficulty is screened out. Further, the challenging instruction data is constructed based on the seed instruction by the reference model, and the training set containing the challenging instruction data is constructed. The instruction data of each task type in the constructed training set matches the preset task proportion. Further, in the subsequent rounds of iterative training, the seed instruction with higher model fitting difficulty is screened out from the training set of the previous round (such as the r-1 round shown in the figure) by means of the reference model, the training set of the current round (such as the r round shown in the figure) is constructed based on the screened seed instruction, and the task resampling is performed to match the resampled training set of each round with the preset task proportion. The deep learning model is trained using the training set of the current round. In addition, when constructing the training set of the current round, part of the instruction data is extracted from the training set of the previous round and added to the training set of the current round, and the proportion of the newly generated instruction in each round of the training set is increased round by round, so that the proportion of the challenging instruction generated in the current round in the training set used in the subsequent rounds will increase, and the proportion of the instruction retained in the training set of the previous round will decrease, gradually transitioning from the training set with a low proportion of challenging instructions to the training set with a higher proportion of challenging instructions, increasing the difficulty of the training set, and improving the ability of the model to handle complex challenging tasks through continuous rounds of iterative training. FIG. 9 is a flowchart of a data processing method provided by an example embodiment of the present disclosure. The execution subject of the embodiment is the server in the system architecture described above. As shown in FIG. 9, the specific steps of the method are as follows: Step S901, receiving a calling request of a target model by an end-side device, the calling request containing an input instruction. In the embodiment, the server provides the API of the trained target model to the end-side device. The end-side device can send a calling request of the target model to the server based on the API of the target model, and the calling request contains the input information of the user. The server receives the calling request of the target model sent by the end-side device, and extracts the input instruction of the user from the calling request.The input instruction can be a natural language processing task such as text summarization, translation, question and answer, or a natural language understanding (NLU) task such as speech-to-text, image-to-text, or emotion analysis, or an instruction of other types, which is not limited here. In step S902, the input instruction is input into the target model, and a response result of the input instruction is generated by the target model. The target model is trained in the following manner: seed instructions with a model fitting difficulty satisfying a preset condition are selected from an existing instruction data set by means of a reference model, a plurality of instruction samples are generated based on the seed instructions, a training set containing the instruction samples is constructed, a deep learning model is trained using the training set, and a trained target model is obtained. The training process of the target model is described in the foregoing embodiments, which will not be repeated here. In step S903, the response result of the input instruction is returned to the terminal device. After the response result of the input instruction is generated, the server returns the response result of the input instruction to the terminal device. The terminal device displays the response result of the input instruction to the user. In the method of this embodiment, the target model is trained in the following manner: instructions with a model fitting difficulty satisfying a preset condition are selected from an existing instruction data set by means of a reference model, instructions with a high model fitting difficulty and challenges can be selected from the instruction data set as seed instructions; a plurality of similar instruction samples are generated based on the seed instructions, i.e., more instruction samples with challenges are obtained, a training set containing the instruction samples and reference responses of the instruction samples is constructed, knowledge distillation can be implemented based on the existing instruction data set by means of the reference model, and a training set of challenging instructions is obtained; further, a deep learning model is trained using the training set, the ability of the deep learning model to process more complex and challenging tasks can be improved, the performance of the target model after training can be improved, and the quality of the generated response result can be improved. FIG. 10 is a structural schematic diagram of a server provided in an embodiment of the present disclosure. As shown in FIG. 10, the server includes a memory 1001 and a processor 1002. The memory 1001 is configured to store computer execution instructions and can be configured to store other various data to support operations on the server. The processor 1002 is in communication connection with the memory 1001 and is configured to execute the computer execution instructions stored in the memory 1001 to implement the technical solutions provided in any of the method embodiments, which have similar specific functions and technical effects that can be achieved, and will not be repeated here. Optionally, as shown in FIG. 10, the server further includes a firewall 1003, a load balancer 1004, a communication component 1005, a power supply component 1006, and other components.The server is not limited to the components shown in FIG. 10. The server can be deployed locally or in the cloud. The computer readable storage medium stores computer executable instructions. When the processor executes the computer executable instructions, the method of any of the preceding embodiments is implemented. The specific functions and technical effects are not repeated here. The computer program product includes a computer program. When the processor executes the computer program, the method of any of the preceding embodiments is implemented. The computer program is stored in a readable storage medium. The at least one processor of the server can read the computer program from the readable storage medium. The at least one processor executes the computer program to enable the server to implement the technical solutions of any of the preceding methods. The specific functions and technical effects are not repeated here. The chip includes a processing module and a communication interface. The processing module can execute the technical solutions of the server in the preceding method embodiments. Optionally, the chip further includes a storage module (e.g., a memory). The storage module is configured to store instructions. The processing module is configured to execute the instructions stored in the storage module. The execution of the instructions stored in the storage module enables the processing module to execute the technical solutions of any of the preceding method embodiments. The integrated modules in the form of software function modules can be stored in a computer readable storage medium. The software function modules are stored in a storage medium. The software function modules include a plurality of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute part of the steps of the methods of the embodiments of the disclosure. It should be understood that the processor can be a central processing unit (CPU), a graphics processing unit (GPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the disclosed methods can be directly embodied in the hardware processor for execution, or be executed by a combination of hardware and software modules in at least one processor.The storage can include a high-speed Random Access Memory (RAM), and can also include a non-volatile storage, such as at least one disk memory, and can also be a U disk, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc. The storage can be an Object Storage Service (OSS). The storage can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a Static Random Access Memory (SRAM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), an Erasable Programmable Read-Only Memory (EPROM), a Programmable Read-Only Memory (PROM), a Read-Only Memory (ROM), a magnetic storage, a flash memory, a magnetic disk or an optical disk. The communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as a mobile hotspot (WiFi), a second generation mobile communication system (2G), a third generation mobile communication system (3G), a fourth generation mobile communication system (4G) / Long Term Evolution (LTE), a fifth generation mobile communication system (5G), etc. mobile communication network, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, infrared technology, Ultra Wide Band (UWB) technology, Bluetooth technology and other technologies. The power supply component provides power for various components of the device where the power supply component is located.The power component can include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device in which the power component is located. The storage medium can be implemented in any type of volatile or non-volatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM), magnetic storage, flash memory, magnetic or optical disks. The storage medium can be any available media that can be accessed by a general purpose or special purpose computer. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. Of course, the storage medium can be a part of the processor. The processor and the storage medium can be located in a single location, such as a special purpose integrated circuit, or in more than one location, such as a distributed computing environment. It should be understood that the terms "comprise", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without limitation, an element preceded by "comprises... a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the element. The order of the above-described implementations of the present disclosure is merely an example and does not represent the order of the implementations. In some of the processes described in the above-described implementations and accompanying drawings, multiple operations can be performed in the order described in this specification or in a different order, or in parallel. The order of the operations should not be interpreted as a limitation unless specifically stated. The processes can include more or fewer operations than those described in this specification. The operations can be performed in serial, in parallel, or in some other order. It should be understood that the terms "first", "second", and the like used in this specification are used to distinguish different messages, devices, modules, and the like, and do not represent a specific order. The terms "first" and "second" are not limited to different types. The term "multiple" means two or more, unless otherwise specifically limited. Through the above description of the implementations, those skilled in the art can clearly understand that the above-described implementation methods can be implemented by software and a necessary general hardware platform, and of course, can also be implemented by hardware. However, the former is a better implementation in many cases.Based on such understanding, the technical solutions of the present disclosure can be embodied in the form of a software product in essence or in the part of contribution to the prior art, the computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a plurality of instructions to make a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) execute the method of various embodiments of the present disclosure. Other implementation solutions of the present disclosure will be easily conceived by those skilled in the art after considering the specification and practicing the disclosed invention. The present disclosure is intended to cover any variations, uses, or adaptive changes of the present disclosure, which follow the general principles of the present disclosure and include the common knowledge or conventional technical means in the technical field not disclosed by the present disclosure. The above is only the preferred embodiment of the present disclosure, and does not limit the patent scope of the present disclosure, and any equivalent structure or equivalent process transformation using the content of the specification and drawings of the present disclosure, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present disclosure.< / t>

Claims

CLAIM 1. A method of model training, wherein, The method comprises the steps of: screening, by means of a reference model, instructions with a preset model fitting difficulty from an existing instruction data set as seed instructions; generating a plurality of instruction samples based on the seed instructions, and constructing a training set containing the instruction samples and reference responses of the instruction samples; training a deep learning model using the training set to obtain a trained target model.

2. The method of claim 1, wherein, The method of screening, by means of a reference model, instructions with a preset model fitting difficulty from an existing instruction data set as seed instructions comprises the steps of: fine-tuning the deep learning model using the instruction data set to obtain a fine-tuned model; generating predicted responses of instructions in the instruction data set by means of the fine-tuned model; generating reference responses of the instructions by means of a reference model; determining the model fitting difficulty of the instructions according to the quality difference between the predicted responses and the reference responses of the instructions, and screening instructions with a preset model fitting difficulty as seed instructions.

3. The method according to claim 2, wherein, The method of determining the model fitting difficulty of the instructions according to the quality difference between the predicted responses and the reference responses of the instructions comprises the steps of: performing quality evaluation on the predicted responses and the reference responses of the instructions by means of a referee model to obtain a quality evaluation value of the predicted responses and a quality evaluation value of the reference responses; and determining the model fitting difficulty of the instructions according to the difference between the quality evaluation value of the reference responses and the quality evaluation value of the predicted responses.

4. The method according to any one of claims 1 to 3, wherein, The method of generating a plurality of instruction samples based on the seed instructions, and constructing a training set containing the instruction samples and reference responses of the instruction samples comprises the steps of: generating similar instructions and responses of the same task type as instruction samples and reference responses of the instruction samples by means of a reference model based on the seed instructions and the task type of the seed instructions, and the proportion of instruction samples of each task type in the generated instruction samples matches a preset task proportion; and constructing a training set containing the instruction samples and reference responses of the instruction samples.

5. The method according to claim 4, wherein, The method of generating similar instructions of the same task type as instruction samples based on the seed instructions and the task type of the seed instructions comprises the steps of: configuring resampling weights of each task type according to a preset task proportion; sampling seed instructions of each task type from the seed instructions according to the resampling weights of each task type, and generating similar instructions of the same task type as instruction samples according to the sampled seed instructions.

6. The method according to claim 4 or 5, wherein, The method of generating reference responses of the instruction samples by means of a reference model comprises the steps of: obtaining prompt information of a corresponding task type according to the task type of the instruction samples, wherein the prompt information of the corresponding task type contains a thinking chain, and the thinking chain is used to prompt the reference model to generate reference responses of the instruction samples step by step; and generating reference responses of the instruction samples step by step by means of a reference model according to the prompt information of the corresponding task type.

7. The method according to any one of claims 1 to 6, wherein, The training of the deep learning model using the training set obtains a trained target model, and the method comprises the following steps: According to the resampling weight of the instruction sample and the probability of the deep learning model predicting the reference response of the instruction sample, a first loss is calculated; and the parameters of the deep learning model are adjusted according to the first loss to obtain the trained target model.

8. The method according to any one of claims 1-7, wherein, Further comprising: In the first round of iterative training, a fine-tuned model is obtained by fine-tuning the deep learning model using the existing instruction data set, and a seed instruction with a model fitting difficulty meeting a preset condition is screened from the instruction data set by the reference model as a seed instruction of the first round; a training set of the first round is constructed based on the seed instruction of the first round, and an initial deep learning model is trained using the training set of the first round; in each subsequent round of iterative training, a deep learning model is obtained based on the deep learning model of the previous round of iterative training, and an instruction sample with a model fitting difficulty meeting a preset condition is screened from the training set of the previous round by the reference model as a seed instruction of the current round; a training set of the current round is constructed based on the seed instruction of the current round, and the deep learning model of the previous round of iterative training is further iteratively trained based on the training set of the current round; until the iterative training is completed, the trained deep learning model is taken as the trained target model.

9. The method of claim 8, wherein, The deep learning model obtained based on the deep learning model of the previous round of iterative training screens an instruction with a model fitting difficulty meeting a preset condition from the training set of the previous round by the reference model as a seed instruction of the current round, and comprises the following steps: the deep learning model of the previous round of iterative training generates a predicted response of an instruction sample in the training set of the previous round; the reference model generates a reference response of the instruction sample; the model fitting difficulty of the instruction sample is determined according to the quality difference between the predicted response and the reference response of the instruction sample; and the instruction sample with the model fitting difficulty meeting the preset condition is screened from the training set of the previous round as the seed instruction of the current round.

10. The method according to claim 8 or 9, wherein, The training set of the current round is constructed based on the seed instruction of the current round, and comprises the following steps: similar instructions of the same task type are generated as instruction samples of the current round based on the seed instruction of the current round and the task type of the seed instruction, and the proportion of the instruction samples of each task type in the instruction samples of the current round matches a preset task proportion; the reference model generates reference responses of the instruction samples of the current round; a training set containing the instruction samples of the current round and the reference responses of the instruction samples of the current round is constructed as the training set of the current round; and a plurality of instruction samples and reference responses of the instruction samples are sampled from the training set of the previous round and added to the training set of the current round.

11. The method according to claim 10, wherein, The sampling of the plurality of instruction samples and the reference responses of the instruction samples from the training set of the previous round and adding to the training set of the current round comprises: according to the configuration proportion of the newly generated instructions corresponding to each round and the number of the newly generated instruction samples currently contained in the training set of the current round, sampling a plurality of instruction samples and reference responses of the instruction samples from the training set of the previous round and adding to the training set of the current round, so that the proportion of the newly generated instruction samples in the training set of the current round matches the configuration proportion of the newly generated instructions corresponding to the current round; wherein the configuration proportion of the newly generated instructions corresponding to each round is increased round by round.

12. The method according to any one of claims 1-11, wherein, Further comprising: in response to a model training request of the edge-side device, determining a reference model specified by the edge-side device and a deep learning model to be trained, the parameter quantity of the deep learning model being less than the parameter quantity of the reference model.

13. The method according to any one of claims 1 to 12, wherein, After obtaining the target model trained, Further comprising: providing a calling interface of the target model to the edge-side device; or, sending parameters of the target model to the edge-side device; or, sending download information of the target model to the edge-side device.

14. A data processing method, wherein, Comprise: Receiving a calling request of a target model by an edge-side device, the calling request containing an input instruction; inputting the input instruction into the target model, generating a response result of the input instruction by the target model, the target model being trained by the following way: filtering out seed instructions with a model fitting difficulty meeting a preset condition from an existing instruction data set by a reference model, generating a plurality of instruction samples based on the seed instructions, and constructing a training set containing the instruction samples and reference responses of the instruction samples, training a deep learning model using the training set, obtaining a target model trained, and returning the response result of the input instruction to the edge-side device.

15. A server, wherein, Comprise: At least one processor; And a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions are executed by the at least one processor, so that the server executes the method of any one of claims 1-14.

16. A computer readable storage medium, wherein, The computer-readable storage medium stores computer-executable instructions, when the processor executes the computer-executable instructions, the method of any one of claims 1-14 is realized.

17. A computer program product comprising a computer program, wherein, The computer program is executed by the processor to realize the method of any one of claims 1-14.

Images