Deep learning model generation method, electronic equipment and storage medium

Abstract

The invention discloses a deep learning model generation method, electronic equipment and a storage medium. The method comprises the steps that a prediction result is obtained, the prediction result is obtained after input data are predicted through a pre-training language model, and the input data are real request data input in advance in a preset application scene; obtaining user feedback data according to the prediction result, the user feedback data being used for recording acceptance feedback and rejection feedback of the prediction result; performing model alignment on the pre-trained language model and the to-be-optimized language model based on the user feedback data to obtain a supplemented language model; and determining the supplemented language model as a target language model. The technical problems that a deep learning model generation method provided by related technologies is low in training efficiency, and a model obtained through training is poor in performance in a specific scene are solved.

Description

Technical Field

[0001] The present application relates to the fields of computer technology and artificial intelligence technology, and specifically, to a deep learning model generation method, electronic device, and storage medium. Background Art

[0002] In recent years, the rapid development of artificial intelligence and deep learning technologies has promoted major breakthroughs in large-scale pre-trained language models (LLM) in the field of natural language processing (NLP). However, according to relevant technologies, in the process of supervised fine-tuning (SFT) of LLM, reinforcement learning is usually based on additional reward models or complex proximal policy optimization (PPO) methods, which leads to low model training efficiency; in addition, relevant technologies rarely or do not mention how to optimize the specific task alignment in the LLM training process, which makes it difficult for the trained LLM to achieve the expected performance when performing specific tasks in application scenarios.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a deep learning model generation method, an electronic device, and a storage medium to at least solve the technical problems that the deep learning model generation method provided by the related technology has low training efficiency and the trained model has poor performance in specific scenarios.

[0005] According to one aspect of an embodiment of the present application, a deep learning model generation method is provided, including: obtaining a prediction result, wherein the prediction result is obtained by predicting input data using a pre-trained language model, and the input data is real request data pre-input in a preset application scenario; obtaining user feedback data based on the prediction result, wherein the user feedback data is used to record acceptance feedback and rejection feedback of the prediction result, the acceptance feedback is used to represent a first part of the results in the prediction result, the first part of the results is the accepted result, and the rejection feedback is used to represent a second part of the results in the prediction result, the second part of the results is the rejected result; based on the user feedback data, the pre-trained language model and the language model to be optimized are model aligned to obtain a completed language model; and the completed language model is determined as the target language model.

[0006] According to another aspect of an embodiment of the present application, a deep learning model generation method is also provided, including: obtaining a text prediction result, wherein the text prediction result is obtained by predicting pre-input text data using a pre-trained text prediction model, and the text data is a real request text input in a text question and answer scenario; obtaining user feedback data based on the text prediction result, wherein the user feedback data is used to record acceptance text feedback and rejection text feedback of the text prediction result, the acceptance text feedback is used to represent the first part of the results in the text prediction result, and the first part of the results are the accepted results, and the rejection text feedback is used to represent the second part of the results in the text prediction result, and the second part of the results are the rejected results; based on the user feedback data, the pre-trained text prediction model and the text prediction model to be optimized are model aligned to obtain a completed text prediction model; and the completed text prediction model is determined as the target text prediction model.

[0007] According to another aspect of an embodiment of the present application, a model prediction method is also provided, including: obtaining current input data; using a target language model to predict the current input data to obtain a target prediction result; wherein the target language model is generated after aligning a pre-trained language model with a language model to be optimized based on user feedback data, and the user feedback data is obtained based on the prediction result, and the prediction result is obtained after predicting the input data using the pre-trained language model, the input data is real request data pre-input in a preset application scenario, and the user feedback data is used to record acceptance feedback and rejection feedback of the prediction result, the acceptance feedback is used to represent the first part of the results in the prediction result, and the first part of the results is the accepted result, and the rejection feedback is used to represent the second part of the results in the prediction result, and the second part of the results is the rejected result.

[0008] According to another aspect of an embodiment of the present application, an electronic device is also provided, including: a memory storing an executable program; and a processor for running the program, wherein when the program is running, any one of the above-mentioned deep learning model generation methods or model prediction methods is executed.

[0009] According to another aspect of an embodiment of the present application, a computer-readable storage medium is also provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned deep learning model generation methods or model prediction methods.

[0010] According to the technical solution provided by the present application, by obtaining a prediction result, wherein the prediction result is obtained by predicting input data using a pre-trained language model, and the input data is real request data pre-input in a preset application scenario; further obtaining user feedback data based on the prediction result, wherein the user feedback data is used to record acceptance feedback and rejection feedback of the prediction result, the acceptance feedback is used to represent the first part of the results in the prediction result, the first part of the results are the accepted results, and the rejection feedback is used to represent the second part of the results in the prediction result, the second part of the results are the rejected results; a method of aligning the pre-trained language model with the language model to be optimized based on the user feedback data is used to obtain a completed language model, and then the aligned language model is determined as the target language model.

[0011] It is easy to understand that in the embodiments of the present application, the prediction results and their corresponding user feedback data are obtained for the real request data input by the input end (such as the user) in the preset application scenario. Since the user feedback data can represent the optimization requirements of the language model according to the accepted part and the rejected part in the prediction result, the model generation through the user feedback data can achieve the purpose of generating the target language model based on the input feedback of the preset application scenario, thereby achieving the technical effect of improving the model generation efficiency, improving the alignment accuracy of the model for specific tasks, and enhancing the model performance in specific scenarios, thereby solving the technical problems that the deep learning model generation method provided by the related technology has low training efficiency and the trained model has poor performance in specific scenarios.

[0012] It is easy to notice that the above general description and the following detailed description are only for the purpose of exemplifying and explaining the present application, and do not constitute a limitation of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0014] Figure 1 It is a schematic diagram of the application scenario of the deep learning model generation method in this application;

[0015] Figure 2 is a flowchart of a deep learning model generation method according to Example 1 of the present application;

[0016] Figure 3 is a schematic diagram of an optional deep learning model generation process according to Example 1 of the present application;

[0017] Figure 4 is a flowchart of another deep learning model generation method according to Example 2 of the present application;

[0018] Figure 5 is a flow chart of a model prediction method according to Example 3 of the present application;

[0019] Figure 6 is a structural schematic diagram of a deep learning model generating device according to Example 4 of the present application;

[0020] Figure 7 is a structural schematic diagram of an optional deep learning model generating device according to Example 4 of the present application;

[0021] Figure 8 is a structural schematic diagram of another deep learning model generating device according to Example 4 of the present application;

[0022] Fig. 9 is a schematic diagram of the structure of a model prediction device according to Example 4 of the present application;

[0023] Fig.10 is a schematic structural diagram of an optional model prediction device according to Example 4 of the present application;

[0024] Fig.11 It is a structural block diagram of a computer terminal according to Example 5 of the present application. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] The technical solution provided in this application is mainly implemented by large-scale model technology. The large model here refers to a deep learning model with large-scale model parameters, which can usually contain hundreds of millions, tens of billions, hundreds of billions, trillions or even more than 10 trillion model parameters. The large model can also be called a cornerstone model / foundation model (Foundation Model). The large model is pre-trained through large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as large-scale language model (Large Language Model, LLM), multi-modal pre-training model (multi-modal pre-training model), etc.

[0028] It should be noted that when the large model is actually applied, the pre-trained model can be fine-tuned through a small number of samples, so that the large model can be applied to different tasks. For example, the large model can be widely used in natural language processing (NLP), computer vision and other fields, specifically in computer vision tasks such as visual question answering (VQA), image caption (IC), image generation, etc., and can also be widely used in natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. Therefore, the main application scenarios of the large model include but are not limited to digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0029] First, some nouns or terms that appear in the process of describing the embodiments of the present application are subject to the following explanations.

[0030] Large Language Model (LLM): refers to an artificial intelligence model with a certain degree of general capabilities.

[0031] Pretrained Language Model (PLM): refers to a model that learns the statistical characteristics of a language by pre-training on a large amount of text data.

[0032] Training process: The process by which the model generates a new model through the input of the data set and back-propagation to update the parameters.

[0033] Model alignment: refers to the process of adjusting the output of the model to be consistent with the given information.

[0034] In recent years, the rapid development of artificial intelligence and deep learning technology has promoted LLM to make major breakthroughs in the field of NLP. In the related technologies, the commonly used deep learning model generation methods are: the first is to generate models based on multiple different outputs predicted for each sample; the second is to generate models through reinforcement learning based on reward models or complex PPO methods. However, although the first method does not require additional training costs, its reasoning time is very high; the training resources required for the second method are more than three times that of conventional SFT training and the training difficulty is high. In addition, both of the above two deep learning model generation methods have a large room for optimization in terms of training efficiency, and the trained LLM is difficult to achieve the expected performance when performing specific tasks in application scenarios.

[0035] With respect to the problems caused by the defects of the above-mentioned related technologies, no effective solutions have been proposed in the related technical field before the present application.

[0036] Example 1

[0037] According to an embodiment of the present application, a deep learning model generation method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0038] Considering the huge number of model parameters of large models and the limited computing resources of mobile terminals, the above-mentioned deep learning model generation method provided in the embodiment of the present application can be applied to Figure 1 The application scenarios shown are not limited to these. Figure 1 In the application scenario shown, the large model is deployed in the server 10, and the server 10 can be connected to one or more client devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client device 20 here may include but is not limited to: a smart phone, a tablet computer, a laptop computer, a PDA, a personal computer, a smart home device, a vehicle-mounted device, etc. The client device 20 can interact with the user through a graphical user interface to implement the call of the large model, thereby implementing the method provided in the embodiment of the present application.

[0039] Under the above operating environment, this application provides Figure 2 The deep learning model generation method shown. Figure 2 is a flowchart of a deep learning model generation method according to Example 1 of the present application, such as Figure 2 As shown, the deep learning model generation method includes:

[0040] Step S21, obtaining a prediction result, wherein the prediction result is obtained by predicting input data using a pre-trained language model, and the input data is real request data pre-input in a preset application scenario;

[0041] Step S22, obtaining user feedback data according to the prediction results, wherein the user feedback data is used to record acceptance feedback and rejection feedback of the prediction results, the acceptance feedback is used to indicate the first part of the prediction results, the first part of the results are the accepted results, and the rejection feedback is used to indicate the second part of the prediction results, the second part of the results are the rejected results;

[0042] Step S23, aligning the pre-trained language model with the language model to be optimized based on the user feedback data to obtain a completed language model;

[0043] Step S24: determine the completed language model as the target language model.

[0044] The above-mentioned preset application scenarios may be, but are not limited to, scenarios involving the use of LLM in the fields of e-commerce, education, medical treatment, conferences, social networks, financial products, logistics, and navigation. Correspondingly, the above-mentioned target language model is a language model trained for the above-mentioned preset application scenarios, and the target language model can show higher performance on specific tasks in the corresponding preset application scenarios.

[0045] The deep learning model generation method of the embodiment of the present application can be run on the client, and the above input data is the real request data input by the user through the client in the above preset application scenario. The above deep learning model generation method can also be run on the server, and the above input data is the real request data obtained by the server from a database or data stream, and the database or data stream is input by the user through the client associated with the server. The above real request data is used to characterize the user's request information for a specific task in the preset application scenario.

[0046] The pre-trained language model is a language model pre-trained by SFT processing. The pre-trained language model can be a large-scale language model trained for a preset application scenario, or a large-scale language model for a general field. The prediction result includes one or more prediction results obtained after the input data can be predicted.

[0047] Since the above-mentioned pre-trained language model may not be trained for the preset application scenario, the prediction results obtained by the above-mentioned pre-trained language model may not fully meet the expected prediction results corresponding to the real request data in the preset application scenario. Therefore, in order to clarify the optimization requirements of the preset application scenario for the language model, user feedback data is obtained for model alignment. Furthermore, the target language model obtained by the model alignment process can meet the optimization requirements of the preset application scenario for the language model as much as possible, that is, the target language model can have a higher performance in processing specific tasks in the preset application scenario.

[0048] In an embodiment of the present application, a prediction result is obtained, wherein the prediction result is obtained by predicting input data using a pre-trained language model, and the input data is real request data pre-input in a preset application scenario; user feedback data is further obtained based on the prediction result, wherein the user feedback data is used to record acceptance feedback and rejection feedback of the prediction result, the acceptance feedback is used to represent the first part of the results in the prediction result, the first part of the results is the accepted result, and the rejection feedback is used to represent the second part of the results in the prediction result, the second part of the results is the rejected result; a pre-trained language model and a language model to be optimized are aligned based on the user feedback data to obtain a completed language model, and then the aligned language model is determined as the target language model.

[0049] It is easy to understand that in the embodiments of the present application, the prediction results and their corresponding user feedback data are obtained for the real request data input by the input end (such as the user) in the preset application scenario. Since the user feedback data can represent the optimization requirements of the language model according to the accepted part and the rejected part in the prediction result, the model generation through the user feedback data can achieve the purpose of generating the target language model based on the input feedback of the preset application scenario, thereby achieving the technical effect of improving the model generation efficiency, improving the alignment accuracy of the model for specific tasks, and enhancing the model performance in specific scenarios, thereby solving the technical problems that the deep learning model generation method provided by the related technology has low training efficiency and the trained model has poor performance in specific scenarios.

[0050] In an embodiment of the present application, a system consisting of a client device and a server can execute the following steps: the client device executes sending input data to the server, wherein the input data is real request data pre-input in a preset application scenario; the server executes the steps corresponding to the deep learning model generation method, uses a pre-trained language model to predict the input data to obtain a prediction result, obtains user feedback data based on the prediction result, and performs model alignment on the pre-trained language model and the language model to be optimized based on the user feedback data to obtain a completed language model, determines the completed language model as the target language model, and returns the target language model to the client (or provides the client with an interface for using the target language model), wherein the user feedback data is used to record acceptance feedback and rejection feedback of the prediction result, the acceptance feedback is used to represent the first part of the results in the prediction result, and the first part of the results is the accepted result, and the rejection feedback is used to represent the second part of the results in the prediction result, and the second part of the results is the rejected result.

[0051] It should be noted that, when the operating resources of the client device can meet the deployment and operating conditions of the large model, the embodiments of the present application can be carried out in the client device.

[0052] In a specific implementation of the embodiment of the present application, preference feedback modeling is performed on the SFT model to achieve feedback alignment of the model. The following takes the implementation scenario of preference feedback modeling on the SFT model as an example to further illustrate the above-mentioned deep learning model generation method of the embodiment of the present application.

[0053] In the above implementation scenario, the generation process of the target language model is as follows: Figure 3 As shown, the pre-trained language model is a large-scale language model processed by SFT, denoted by π SFT The language model to be optimized corresponding to the preset application scenario is denoted as π θ . Using the pre-trained language model π SFT The real request data is predicted to obtain the prediction results, and then the prediction results are labeled to determine the acceptance feedback and rejection feedback. The target loss is further calculated based on the acceptance feedback and rejection feedback, and the target loss is used to pre-train the language model π SFT And the language model to be optimized π θ The model is aligned to obtain the target language model.

[0054] In an optional embodiment, the deep learning model generation method further includes the following method steps:

[0055] Step S25: Initialize the model parameters of the language model to be optimized using the pre-trained language model.

[0056] In the above exemplary implementation scenario, the pre-trained language model π is used.SFT For the predetermined language model to be optimized π θ Initialize, that is, before optimizing the language model to be optimized, the language model to be optimized π θ The model parameters are the same as the above pre-trained language model π SFT Be consistent.

[0057] After initialization is completed, obtain the real request data x pre-entered by the user in the preset application scenario (can be obtained in real time), and use the pre-trained language model π SFT Predict the actual request data x (can be a real-time prediction) to obtain the prediction results, and select N results from the prediction results and record them as y = {y1,…,y N}.

[0058] Furthermore, after obtaining the prediction results, the process of generating a target language model for a preset application scenario can be divided into: the first stage, the preference feedback collection stage; the second stage, the model alignment stage.

[0059] In an optional embodiment, in step S22, obtaining user feedback data according to the prediction result includes the following method steps:

[0060] Step S221, grouping and sampling the prediction results to obtain multiple groups of preference candidate data to be labeled;

[0061] Step S222: label the preference candidate data to obtain user feedback data.

[0062] In the above exemplary implementation scenario, the prediction result y = {y1, ..., y N} are sorted to obtain a sorting result, wherein the sorting result is used to sort y1,…,y N The data are sorted from high to low to see if they meet the user's expectations. Then, based on the sorting results, group sampling is performed to obtain multiple groups of candidate data (x, y) to be labeled. a ,y b ), where y a It means from y1,…,y N One of the prediction results sampled from multiple prediction results, y b It means from y1,…,y N Another prediction result sampled from multiple prediction results, and y a The predicted results are relative to y b For example, the above group sampling can be two-by-two sampling. Correspondingly, according to the permutation and combination rules, the number of groups of the above preference candidate data to be marked is

[0063] In the above exemplary implementation scenario, the labeling of the preference candidate data can be completed by a technician or by a pre-developed labeling tool, which can label the prediction results with acceptance / rejection labels according to user preferences. After the preference candidate data is labeled, the accepted part of the preference candidate data indicates that the corresponding prediction result is fed back as acceptance, and the rejected part of the preference candidate data indicates that the corresponding prediction result is fed back as rejection, thereby obtaining user feedback data. For each set of preference candidate data (x, y a ,y b ), the corresponding user feedback data can be expressed as (x,y win ,y lose ), where the subscript win of the prediction result y indicates that the prediction result predicted by the pre-trained language model is accepted by the user, whereas the subscript lose indicates that the prediction result is rejected by the user.

[0064] It is easy to notice that in the above exemplary implementation scenario, in the preference feedback collection stage, user feedback data is determined by collecting the user's acceptance / rejection of the prediction results generated by the pre-trained language model. The user feedback data serves as valuable data for aligning the pre-trained language model with the language model to be optimized.

[0065] In an optional embodiment, in step S23, the pre-trained language model and the language model to be optimized are aligned based on the user feedback data to obtain a completed language model, including the following method steps:

[0066] Step S231, determining the target loss between the pre-trained language model and the language model to be optimized through user feedback data;

[0067] Step S232: align the pre-trained language model with the language model to be optimized based on the target loss to obtain a completed language model.

[0068] In the above exemplary implementation scenario, in the model alignment phase, based on a specific loss function, the target loss (denoted as L(π θ ; π SFT )), and then use the target loss to make the language model to be optimized and the pre-trained language model as aligned as possible to obtain the completed language model, and then determine the completed language model as the target language model.

[0069] The above two stages complete the model generation for the preset application scenario, avoiding the complex reinforcement learning process, simplifying the training process and improving the training efficiency.

[0070] Experiments were conducted in specific application scenarios, and it was found that the performance of the target language model generated by the deep learning model generation method provided in the embodiment of the present application was significantly better than the language model generated by related technologies (deep learning model generation methods based on reward models or PPO reinforcement learning).

[0071] In an optional embodiment, in step S231, determining the target loss between the pre-trained language model and the language model to be optimized through user feedback data includes the following method steps:

[0072] Step S311, obtaining the probability distribution difference between the pre-trained language model and the language model to be optimized in predicting the user feedback data;

[0073] Step S312: determining and acquiring a target loss based on the probability distribution difference.

[0074] The above target loss is used to characterize the difference between the pre-trained language model and the language model to be optimized. Therefore, in the above exemplary implementation scenario, the pre-trained language model and the language model to be optimized are used to respectively predict the probability distribution of the predicted user feedback data corresponding to the prediction result. The difference between the probability distributions of the pre-trained language model and the language model to be optimized can characterize the difference in the prediction performance of the pre-trained language model and the language model to be optimized for the preset application scenario, thereby determining the target loss L(π θ ; π SFT ).

[0075] In an optional embodiment, the above-mentioned deep learning model generation method further includes:

[0076] Step S3111, obtaining the probability distribution difference between the language model to be optimized and the pre-trained language model for predicting acceptance of feedback and rejection of feedback respectively;

[0077] Step S3112, normalize the probability distribution difference to obtain the target loss.

[0078] In the above exemplary implementation scenario, the pre-trained language model π is used. SFT And the language model to be optimized π θ Accepting feedback (y win |x) and rejection feedback (y lose |x) respectively perform probability distribution prediction, and after normalizing the probability distribution difference, the normalized result is used as the target loss to characterize the difference in prediction performance between the pre-trained language model and the language model to be optimized for the preset application scenario.

[0079] In an optional embodiment, in step S3111, obtaining the probability distribution difference between the language model to be optimized and the pre-trained language model for predicting acceptance of feedback and rejection of feedback respectively includes the following method steps:

[0080] Step S31111, under given input data conditions, determining a first probability that the pre-trained language model predicts receiving feedback, and determining a second probability that the language model to be optimized predicts receiving feedback;

[0081] Step S31112, under given input data conditions, determining a third probability that the pre-trained language model predicts a rejection of feedback, and determining a fourth probability that the language model to be optimized predicts a rejection of feedback;

[0082] Step S31113, obtaining a first probability ratio by calculating the first probability and the second probability, and obtaining a second probability ratio by calculating the third probability and the fourth probability;

[0083] Step S31114, calculating the probability distribution difference based on the first probability ratio and the second probability ratio.

[0084] In the above exemplary implementation scenario, for a given input data (ie, x), a pre-trained language model π is used. SFT Predict the first probability π SFT (y win |x); using the language model to be optimized π θ Predict the second probability π θ (y win |x); using the pre-trained language model π SFT Predict the third probability π SFT (y lose |x); using the language model to be optimized π θ The fourth probability π is predicted θ (y lose |x).

[0085] Furthermore, for the first probability π SFT (y win |x) and the second probability π θ (y win |x) is divided and then the logarithm is taken to obtain the first probability ratio, which is expressed as For the third probability π SFT (y lose |x) and the fourth probability π θ (y lose |x) is divided and then the logarithm is taken to obtain the second probability ratio, which is expressed as

[0086] In an optional embodiment, in step S3112, the probability distribution difference is normalized to obtain the target loss, including the following method steps:

[0087] Step S31121, constructing a loss function based on the first probability ratio, the second probability ratio, the first adjustment parameter and the second adjustment parameter, wherein the first adjustment parameter and the second adjustment parameter are used to adjust the weight of the probability distribution difference calculated by the first probability ratio and the second probability ratio;

[0088] Step S31122, normalize the probability distribution difference through the loss function to obtain the target loss.

[0089] In the above exemplary implementation scenario, the first adjustment parameter and the second adjustment parameter are adjustable hyperparameters of the neural network model, the first adjustment parameter is denoted as α, and the second adjustment parameter is denoted as β. The loss function constructed above can be an activation function, and in this example, the activation function is a sigmoid function (denoted as σ).

[0090] Based on this, a loss function is constructed as shown in the following formula (1):

[0091]

[0092] In the above loss function, π SFT (y win |x) and π SFT (y loss |x) can be used to provide auxiliary constraints to prevent the language model to be optimized from being inconsistent with the pre-trained language model π during the optimization process. SFT Too much deviation.

[0093] In an optional embodiment, in step S23, the pre-trained language model and the language model to be optimized are aligned based on the target loss to obtain a completed language model, including the following method steps:

[0094] Step S231, adjusting the first adjustment parameter and the second adjustment parameter based on the target loss until the probability distribution difference is minimized to obtain a completed language model.

[0095] In the above exemplary implementation scenario, based on the target loss L(π θ ; π SFT ) is used to dynamically adjust the first adjustment parameter α and the second adjustment parameter β, so that L(π θ ; π SFT ) is taken to the minimum value to obtain the completed language model, and the completed language model at this time is used as the target language model.

[0096] It is easy to notice that the deep learning model generation method provided in the embodiment of the present application can achieve model alignment without using an additional reward model, and also avoids using complex approximate strategy optimization for reinforcement learning. The training efficiency is high and the language model obtained by training has good performance for specific scenarios.

[0097] In summary, the deep learning model generation method provided in the embodiment of the present application can optimize the pre-trained language model, achieve more accurate task alignment in the preset application scenario, and enhance model performance. Specifically, based on the user feedback data corresponding to the prediction result, the pre-trained language model and the language model to be optimized are subjected to model alignment processing to obtain a target language model that can perform well on the specific tasks of the preset application scenario. It is easy to find that the deep learning model generation method provided in the embodiment of the present application does not need to rely on an additional reward model or PPO reinforcement learning method, but uses preference feedback for input data to accurately align the language model to specific tasks in the preset application scenario in a concise and efficient manner.

[0098] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0099] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0100] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0101] Example 2

[0102] In the operating environment as in Example 1, the present application provides Figure 4 Another deep learning model generation method shown in . Figure 4 is a flowchart of another deep learning model generation method according to Example 2 of the present application, such as Figure 4 As shown, the deep learning model generation method includes:

[0103] Step S41, obtaining a text prediction result, wherein the text prediction result is obtained by predicting pre-input text data using a pre-trained text prediction model, and the text data is a real request text input in a text question and answer scenario;

[0104] Step S42, obtaining user feedback data according to the text prediction result, wherein the user feedback data is used to record the acceptance text feedback and rejection text feedback of the text prediction result, the acceptance text feedback is used to represent the first part of the text prediction result, and the first part of the result is the accepted result, and the rejection text feedback is used to represent the second part of the text prediction result, and the second part of the result is the rejected result;

[0105] Step S43, aligning the pre-trained text prediction model with the text prediction model to be optimized based on the user feedback data to obtain a completed text prediction model;

[0106] Step S44: determine the completed text prediction model as the target text prediction model.

[0107] The above-mentioned text question answering scenario can be, but is not limited to, scenarios involving the use of LLM for text question answering in the fields of e-commerce, education, medical care, conferences, social networks, financial products, logistics, and navigation. Correspondingly, the above-mentioned target text prediction model is a text prediction model trained for the above-mentioned text question answering scenario, and the target text prediction model can show higher performance on specific tasks in the corresponding text question answering scenario.

[0108] The deep learning model generation method of the embodiment of the present application can be run on the client, and the pre-input text data is the real request text input by the user through the client in the text question and answer scenario. The deep learning model generation method can also be run on the server, and the pre-input text data is the real request text obtained by the server from a database or data stream, and the database or data stream is input by the user through the client associated with the server. The real request text is used to represent the user's request information for a specific task in the text question and answer scenario.

[0109] The pre-trained text prediction model is a text prediction model pre-trained by SFT processing. The pre-trained text prediction model can be a large-scale text prediction model trained for text question-answering scenarios, or a large-scale text prediction model for general fields. The text prediction results include one or more text prediction results obtained after predicting the pre-input text data.

[0110] Since the above-mentioned pre-trained text prediction model may not be trained for the text question-answering scenario, the text prediction results predicted by the above-mentioned pre-trained text prediction model may not fully meet the expected text prediction results corresponding to the real request text in the text question-answering scenario. Therefore, in order to clarify the optimization requirements of the text question-answering scenario for the text prediction model, user feedback data is obtained for model alignment. Furthermore, the completed text prediction model obtained by the model alignment process is used as the target text prediction model. The target text prediction model can meet the optimization requirements of the text question-answering scenario for the text prediction model as much as possible, that is, the target text prediction model can have a higher performance in processing specific tasks in the text question-answering scenario.

[0111] It should be noted that the above-mentioned text question and answer scenario can also be expanded to an audio question and answer scenario, a video question and answer scenario, a virtual reality / augmented reality question and answer scenario, etc.

[0112] In an embodiment of the present application, a text prediction result is first obtained, wherein the text prediction result is obtained by predicting pre-input text data using a pre-trained text prediction model, and the text data is a real request text input in a text question and answer scenario; user feedback data is further obtained based on the text prediction result, wherein the user feedback data is used to record acceptance text feedback and rejection text feedback of the text prediction result, the acceptance text feedback is used to represent the first part of the results in the text prediction result, and the first part of the results are the accepted results, and the rejection text feedback is used to represent the second part of the results in the text prediction result, and the second part of the results are the rejected results, and a pre-trained text prediction model and a text prediction model to be optimized are aligned based on the user feedback data to obtain a completed text prediction model, and then the completed text prediction model is determined as the target text prediction model.

[0113] It is easy to understand that in the embodiments of the present application, for the actual request text input by the input end (such as the user) in the text question and answer scenario, the text prediction result and its corresponding preference feedback result are obtained. Since the preference feedback result can characterize the optimization requirements of the text prediction model according to the accepted part and the rejected part of the text prediction result, model generation through the preference feedback result can achieve the purpose of generating a target text prediction model based on the input feedback of a specific text question and answer scenario, thereby achieving the technical effect of improving the model generation efficiency, improving the model's alignment accuracy for specific tasks, and enhancing the model performance in specific scenarios, thereby solving the technical problems that the deep learning model generation method provided by the related technology has low training efficiency and the trained model has poor performance in specific scenarios.

[0114] In an embodiment of the present application, a system consisting of a client device and a server can perform the following steps: the client device executes sending pre-input text data to the server, wherein the pre-input text data is a real request text pre-input in a text question and answer scenario; the server executes the steps corresponding to the deep learning model generation method, uses a pre-trained text prediction model to predict the pre-input text data to obtain a text prediction result, obtains user feedback data based on the text prediction result, and aligns the pre-trained text prediction model with the text prediction model to be optimized based on the user feedback data to obtain a completed text prediction model, determines the completed text prediction model as the target text prediction model, and returns the target text prediction model to the client (or provides the client with a usage interface of the target text prediction model), wherein the user feedback data is used to record the acceptance text feedback and rejection text feedback of the text prediction result, the acceptance text feedback is used to represent the first part of the results in the text prediction result, and the first part of the results is the accepted result, and the rejection text feedback is used to represent the second part of the results in the text prediction result, and the second part of the results is the rejected result.

[0115] It should be noted that, when the operating resources of the client device can meet the deployment and operating conditions of the large model, the embodiments of the present application can be carried out in the client device.

[0116] It should be noted that the preferred implementation of this embodiment can refer to the relevant description in Example 1, which will not be repeated here.

[0117] Example 3

[0118] In the operating environment as in Example 1, the present application provides Figure 5 A model prediction method is shown. Figure 5 is a flow chart of a model prediction method according to Example 3 of the present application, such as Figure 5 As shown, the model prediction method includes:

[0119] Step S51, obtaining current input data;

[0120] Step S52, using the target language model to predict the current input data to obtain a target prediction result;

[0121] Among them, the target language model is generated after aligning the pre-trained language model with the language model to be optimized based on the user feedback data. The user feedback data is obtained according to the prediction results. The prediction results are obtained by predicting the input data using the pre-trained language model. The input data is the real request data pre-input in the preset application scenario. The user feedback data is used to record the acceptance feedback and rejection feedback of the prediction results. The acceptance feedback is used to represent the first part of the prediction results, and the first part of the results is the accepted results. The rejection feedback is used to represent the second part of the prediction results, and the second part of the results is the rejected results.

[0122] According to an embodiment of the present application, a model prediction service is provided, which can be run on a client or a server, and the server can be an independent or distributed server, or a server set up in the cloud. After the above-mentioned model prediction service obtains the current input data, it uses the target language model corresponding to the preset application scenario generated according to the above-mentioned deep learning model generation method to predict the current input data, obtains the target prediction result, and outputs the target prediction result to the service caller.

[0123] In the embodiment of the present application, the deep learning model generation method based on the aforementioned embodiment can not only achieve the purpose of generating a target language model based on input feedback of a preset application scenario, but also realize the technical effects of improving model generation efficiency, improving the alignment accuracy of the model for specific tasks, and enhancing the model performance in specific scenarios, thereby solving the technical problems that the deep learning model generation method provided by the related technology has low training efficiency and the trained model has poor performance in specific scenarios, but also can conveniently provide model prediction services for specific tasks in the preset application scenarios, which is user-friendly and conducive to application in specific scenarios.

[0124] In an optional embodiment, a graphical user interface is provided by an electronic device, and the content displayed by the graphical user interface at least partially includes a model prediction scenario. The model prediction method further includes the following method steps:

[0125] Step S531, displaying a request input box and a request category and a request example corresponding to a preset application scenario in a graphical user interface, wherein the request category and the request example are used to prompt a method of initiating a request in the request input box;

[0126] Step S532, in response to the first touch operation on the graphical user interface, obtaining current input data input in the request input box based on the request category and the request example;

[0127] Step S533 , in response to the second touch operation on the graphical user interface, a target prediction result corresponding to the current input data is fed back in the graphical user interface.

[0128] In the above optional embodiments, the electronic device may be a terminal device or a cloud device. The graphical user interface provided by the electronic device can realize interaction with the user. Through the model prediction scenario displayed by the graphical user interface, the request input box and the request category and request example corresponding to the preset application scenario are displayed to the user, wherein the request category includes multiple candidate categories preset in the model prediction scenario, and the request example includes at least one question example preset in the model prediction scenario.

[0129] When the above-mentioned first touch operation is detected, the current input data entered in the request input box is obtained based on the request category and the request example, wherein the first touch operation is an editing request operation (such as a selection operation performed on multiple candidate categories and at least one question example, an input operation performed on the request input box, etc.), and the above-mentioned request category, request example and current input data are used as input to the target language model, so as to perform model prediction and obtain the target prediction result.

[0130] Furthermore, when the second touch operation is detected, a target prediction result corresponding to the current input data is fed back in the graphical user interface, wherein the second touch operation is an operation of submitting a request, and the second touch operation is used to trigger an event in which the target language model starts prediction.

[0131] Through the above optional embodiments, the present application provides a visual implementation method of a model prediction method, which can be implemented on the client to support users in specifying request categories, request examples and current input data, and controlling the start of model prediction. This method is user-friendly and convenient, and has a good user experience.

[0132] It should be noted that the preferred implementation of this embodiment can refer to the relevant description in Example 1, which will not be repeated here.

[0133] Example 4

[0134] According to an embodiment of the present application, a device embodiment for implementing the above-mentioned deep learning model generation method is also provided. Figure 6 is a structural diagram of a deep learning model generation device according to Example 4 of the present application, such as Figure 6 As shown, the device comprises:

[0135] A first acquisition module 601 is used to acquire a prediction result, wherein the prediction result is obtained by predicting input data using a pre-trained language model, and the input data is real request data pre-input in a preset application scenario;

[0136] The second acquisition module 602 is used to acquire user feedback data according to the prediction result, wherein the user feedback data is used to record the acceptance feedback and rejection feedback of the prediction result, the acceptance feedback is used to indicate the first part of the prediction result, and the first part of the prediction result is the accepted result, and the rejection feedback is used to indicate the second part of the prediction result, and the second part of the prediction result is the rejected result;

[0137] A generation module 603 is used to align the pre-trained language model with the language model to be optimized based on the user feedback data to obtain a completed language model;

[0138] The determination module 604 determines the completed language model as the target language model.

[0139] Optionally, the second acquisition module 602 is further used to: perform group sampling on the prediction results to obtain multiple groups of preference candidate data to be labeled; and label the preference candidate data to obtain user feedback data.

[0140] Optionally, the generation module 603 is further used to: determine a target loss between the pre-trained language model and the language model to be optimized through user feedback data; and perform model alignment on the pre-trained language model and the language model to be optimized based on the target loss to obtain a completed language model.

[0141] Optionally, the generation module 603 is further used to: obtain the probability distribution difference between the pre-trained language model and the language model to be optimized in respectively predicting the user feedback data; and determine the target loss based on the probability distribution difference.

[0142] Optionally, the generation module 603 is further used to: obtain the probability distribution difference between the language model to be optimized and the pre-trained language model predicting acceptance of feedback and rejection of feedback respectively; and normalize the probability distribution difference to obtain a target loss.

[0143] Optionally, the above-mentioned generation module 603 is also used to: determine the first probability that the pre-trained language model predicts acceptance of feedback under given input data conditions, and determine the second probability that the language model to be optimized predicts acceptance of feedback; determine the third probability that the pre-trained language model predicts rejection of feedback under given input data conditions, and determine the fourth probability that the language model to be optimized predicts rejection of feedback; calculate a first probability ratio by calculating the first probability and the second probability, and calculate a second probability ratio by calculating the third probability and the fourth probability; calculate the probability distribution difference based on the first probability ratio and the second probability ratio.

[0144] Optionally, the above-mentioned generation module 603 is also used to: construct a loss function based on the first probability ratio, the second probability ratio, the first adjustment parameter and the second adjustment parameter, wherein the first adjustment parameter and the second adjustment parameter are used to adjust the weight of the probability distribution difference calculated by the first probability ratio and the second probability ratio; the probability distribution difference is normalized by the loss function to obtain the target loss.

[0145] Optionally, the generation module 603 is further used to: adjust the first adjustment parameter and the second adjustment parameter based on the target loss until the probability distribution difference is minimized to obtain the completed language model.

[0146] Optionally, Figure 7 is a schematic diagram of the structure of an optional deep learning model generation device according to Example 4 of the present application, such as Figure 7 As shown, the device includes Figure 6 In addition to all the modules shown, the present invention also includes: an initialization module 605, which is used to initialize the model parameters of the language model to be optimized using the pre-trained language model.

[0147] It should be noted that the first acquisition module 601, the second acquisition module 602, the generation module 603 and the determination module 604 correspond to steps S21 to S24 in Example 1, and the four modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n), and the above-mentioned modules may also be part of the device and may be run in the computer terminal 10 provided in Example 1.

[0148] In an embodiment of the present application, a prediction result is obtained, wherein the prediction result is obtained by predicting input data using a pre-trained language model, and the input data is real request data pre-input in a preset application scenario; user feedback data is further obtained based on the prediction result, wherein the user feedback data is used to record acceptance feedback and rejection feedback of the prediction result, the acceptance feedback is used to represent the first part of the results in the prediction result, and the first part of the results is the accepted result, and the rejection feedback is used to represent the second part of the results in the prediction result, and the second part of the results is the rejected result; a pre-trained language model and a language model to be optimized are aligned based on the user feedback data to obtain a completed language model, and then the completed language model is determined as the target language model.

[0149] It is easy to understand that in the embodiments of the present application, the prediction results and their corresponding user feedback data are obtained for the real request data input by the input end (such as the user) in the preset application scenario. Since the user feedback data can represent the optimization requirements of the language model according to the accepted part and the rejected part in the prediction result, the model generation through the user feedback data can achieve the purpose of generating the target language model based on the input feedback of the preset application scenario, thereby achieving the technical effect of improving the model generation efficiency, improving the alignment accuracy of the model for specific tasks, and enhancing the model performance in specific scenarios, thereby solving the technical problems that the deep learning model generation method provided by the related technology has low training efficiency and the trained model has poor performance in specific scenarios.

[0150] According to an embodiment of the present application, another device embodiment for implementing the deep learning model generation method in the above-mentioned embodiment 2 is also provided. Figure 8 is a schematic diagram of the structure of another deep learning model generation device according to Example 4 of the present application, such as Figure 8 As shown, the device comprises:

[0151] A first acquisition module 801 is used to acquire a text prediction result, wherein the text prediction result is obtained by predicting pre-input text data using a pre-trained text prediction model, and the text data is a real request text input in a text question and answer scenario;

[0152] The second acquisition module 802 is used to acquire user feedback data according to the text prediction result, wherein the user feedback data is used to record the acceptance text feedback and rejection text feedback of the text prediction result, the acceptance text feedback is used to represent the first part of the text prediction result, and the first part of the result is the accepted result, and the rejection text feedback is used to represent the second part of the text prediction result, and the second part of the result is the rejected result;

[0153] A generation module 803 is used to align the pre-trained text prediction model with the text prediction model to be optimized based on the user feedback data to obtain a completed text prediction model;

[0154] The determination module 804 determines the completed text prediction model as the target text prediction model.

[0155] It should be noted that the first acquisition module 801, the second acquisition module 802, the generation module 803 and the determination module 804 correspond to steps S41 to S44 in Example 2, and the four modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 2. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n), and the above-mentioned modules may also be part of the device and may be run in the computer terminal 10 provided in Example 1.

[0156] According to an embodiment of the present application, another device embodiment for implementing the model prediction method in the above-mentioned embodiment 3 is also provided. Fig. 9 is a schematic diagram of the structure of a model prediction device according to Example 4 of the present application, such as Fig. 9 As shown, the device comprises:

[0157] The acquisition module 901 is used to acquire the current input data;

[0158] Prediction module 902, used to predict current input data using a target language model to obtain a target prediction result;

[0159] Among them, the target language model is generated after aligning the pre-trained language model with the language model to be optimized based on the user feedback data. The user feedback data is obtained according to the prediction results. The prediction results are obtained by predicting the input data using the pre-trained language model. The input data is the real request data pre-input in the preset application scenario. The user feedback data is used to record the acceptance feedback and rejection feedback of the prediction results. The acceptance feedback is used to represent the first part of the prediction results, and the first part of the results is the accepted results. The rejection feedback is used to represent the second part of the prediction results, and the second part of the results is the rejected results.

[0160] Optionally, a graphical user interface is provided by the electronic device, and the content displayed by the graphical user interface at least partially includes a model prediction scenario. Fig.10 is a schematic diagram of the structure of an optional model prediction device according to Example 4 of the present application, such as Fig.10 As shown, the device includes Fig. 9In addition to all the modules shown, it also includes: a visualization module 903, which is used to display the request input box and the request category and request example corresponding to the preset application scenario in the graphical user interface, wherein the request category and request example are used to prompt the method of initiating a request in the request input box; in response to a first touch operation acting on the graphical user interface, obtain the current input data entered in the request input box based on the request category and the request example; in response to a second touch operation acting on the graphical user interface, feedback the target prediction result corresponding to the current input data in the graphical user interface.

[0161] It should be noted that the acquisition module 901 and the prediction module 902 correspond to steps S51 to S52 in Example 3, and the examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Example 3. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n), and the above-mentioned modules may also be part of the device and may be run in the computer terminal 10 provided in Example 1.

[0162] It should be noted that the preferred implementation of this embodiment can refer to the relevant description in Example 1 or Example 2, which will not be repeated here.

[0163] Example 5

[0164] According to an embodiment of the present application, a computer terminal is also provided, which can be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal can also be replaced by a terminal device such as a mobile terminal.

[0165] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of the computer network.

[0166] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the deep learning model generation method: obtaining a prediction result, wherein the prediction result is obtained by predicting the input data using a pre-trained language model, and the input data is real request data pre-input in a preset application scenario; obtaining user feedback data based on the prediction result, wherein the user feedback data is used to record acceptance feedback and rejection feedback of the prediction result, the acceptance feedback is used to represent the first part of the results in the prediction result, and the first part of the results is the accepted result, and the rejection feedback is used to represent the second part of the results in the prediction result, and the second part of the results is the rejected result; based on the user feedback data, the pre-trained language model and the language model to be optimized are model aligned to obtain a completed language model; and the completed language model is determined as the target language model.

[0167] Optionally, Fig.11 is a structural block diagram of a computer terminal according to Embodiment 5 of the present application, such as Fig.11 As shown, the computer terminal 110 may include: one or more (only one is shown in the figure) processors 1102, a memory 1104, a storage controller 1106, and a peripheral interface 1108, wherein the peripheral interface 1108 is connected to a radio frequency module, an audio module, and a display.

[0168] Among them, the memory 1104 can be used to store software programs and modules, such as the program instructions / modules corresponding to the deep learning model generation method and device in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned deep learning model generation method. The memory 1104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1104 may further include a memory remotely arranged relative to the processor, and these remote memories can be connected to the computer terminal 110 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0169] The processor 1102 can call the information and application stored in the memory through the transmission device to perform the following steps: obtain a prediction result, wherein the prediction result is obtained by predicting the input data using a pre-trained language model, and the input data is real request data pre-input in a preset application scenario; obtain user feedback data based on the prediction result, wherein the user feedback data is used to record acceptance feedback and rejection feedback of the prediction result, the acceptance feedback is used to represent the first part of the results in the prediction result, and the first part of the results are the accepted results, and the rejection feedback is used to represent the second part of the results in the prediction result, and the second part of the results are the rejected results; based on the user feedback data, the pre-trained language model and the language model to be optimized are model aligned to obtain a completed language model; and the completed language model is determined as the target language model.

[0170] Optionally, the processor 1102 may also execute program codes of the following steps: grouping and sampling the prediction results to obtain multiple groups of preference candidate data to be labeled; labeling the preference candidate data to obtain user feedback data.

[0171] Optionally, the processor 1102 may also execute the following program code: determining the target loss between the pre-trained language model and the language model to be optimized through user feedback data; aligning the pre-trained language model and the language model to be optimized based on the target loss to obtain a completed language model.

[0172] Optionally, the processor 1102 may further execute program codes of the following steps: obtaining a probability distribution difference between the pre-trained language model and the language model to be optimized that predict the user feedback data respectively; and determining a target loss based on the probability distribution difference.

[0173] Optionally, the processor 1102 may also execute program code of the following steps: obtaining the probability distribution difference between the language model to be optimized and the pre-trained language model predicting acceptance of feedback and rejection of feedback respectively; and normalizing the probability distribution difference to obtain a target loss.

[0174] Optionally, the processor 1102 may also execute program code for the following steps: under given input data conditions, determining a first probability that the pre-trained language model predicts acceptance of feedback, and determining a second probability that the language model to be optimized predicts acceptance of feedback; under given input data conditions, determining a third probability that the pre-trained language model predicts rejection of feedback, and determining a fourth probability that the language model to be optimized predicts rejection of feedback; calculating a first probability ratio by the first probability and the second probability, and calculating a second probability ratio by the third probability and the fourth probability; and calculating a probability distribution difference based on the first probability ratio and the second probability ratio.

[0175] Optionally, the processor 1102 may also execute the program code of the following steps: constructing a loss function based on the first probability ratio, the second probability ratio, the first adjustment parameter and the second adjustment parameter, wherein the first adjustment parameter and the second adjustment parameter are used to adjust the weight of the probability distribution difference calculated by the first probability ratio and the second probability ratio; normalizing the probability distribution difference through the loss function to obtain the target loss.

[0176] Optionally, the processor 1102 may further execute program code of the following steps: adjusting the first adjustment parameter and the second adjustment parameter based on the target loss until the probability distribution difference is minimized to obtain a target language model.

[0177] Optionally, the processor 1102 may also execute program code of the following steps: initializing model parameters of the language model to be optimized using a pre-trained language model.

[0178] The processor 1102 can call the information and application stored in the memory through the transmission device to perform the following steps: obtaining a text prediction result, wherein the text prediction result is obtained by predicting pre-input text data using a pre-trained text prediction model, and the text data is a real request text input in a text question and answer scenario; obtaining user feedback data based on the text prediction result, wherein the user feedback data is used to record acceptance text feedback and rejection text feedback of the text prediction result, the acceptance text feedback is used to represent the first part of the results in the text prediction result, and the first part of the results are accepted results, and the rejection text feedback is used to represent the second part of the results in the text prediction result, and the second part of the results are rejected results; based on the user feedback data, the pre-trained text prediction model and the text prediction model to be optimized are model aligned to generate a target text prediction model.

[0179] The processor 1102 can call the information and application stored in the memory through the transmission device to perform the following steps: obtain current input data; use the target language model to predict the current input data to obtain a target prediction result; wherein the target language model is generated after aligning the pre-trained language model and the language model to be optimized based on the user feedback data, the user feedback data is obtained according to the prediction result, and the prediction result is obtained after predicting the input data using the pre-trained language model, the input data is the real request data pre-input in the preset application scenario, and the user feedback data is used to record the acceptance feedback and rejection feedback of the prediction result, the acceptance feedback is used to represent the first part of the results in the prediction result, and the first part of the results is the accepted result, and the rejection feedback is used to represent the second part of the results in the prediction result, and the second part of the results is the rejected result.

[0180] Optionally, the processor 1102 may also execute program code for the following steps: displaying a request input box and request categories and request examples corresponding to preset application scenarios in a graphical user interface, wherein the request categories and request examples are used to prompt a method for initiating a request in the request input box; in response to a first touch operation on the graphical user interface, obtaining current input data input in the request input box based on the request category and the request example; and in response to a second touch operation on the graphical user interface, feeding back a target prediction result corresponding to the current input data in the graphical user interface.

[0181] Using the embodiment of the present application, a computer terminal solution for implementing the above-mentioned deep learning model generation method or model prediction method is provided. By obtaining the prediction result, wherein the prediction result is obtained by predicting the input data using a pre-trained language model, and the input data is the real request data pre-input in the preset application scenario; further obtaining user feedback data based on the prediction result, wherein the user feedback data is used to record the acceptance feedback and rejection feedback of the prediction result, the acceptance feedback is used to represent the first part of the results in the prediction result, the first part of the results is the accepted result, and the rejection feedback is used to represent the second part of the results in the prediction result, and the second part of the results is the rejected result; using the method of aligning the pre-trained language model with the language model to be optimized based on the user feedback data, the completed language model is obtained, and then the aligned language model is determined as the target language model.

[0182] It is easy to understand that in the embodiments of the present application, the prediction results and their corresponding user feedback data are obtained for the real request data input by the input end (such as the user) in the preset application scenario. Since the user feedback data can represent the optimization requirements of the language model according to the accepted part and the rejected part in the prediction result, the model generation through the user feedback data can achieve the purpose of generating the target language model based on the input feedback of the preset application scenario, thereby achieving the technical effect of improving the model generation efficiency, improving the alignment accuracy of the model for specific tasks, and enhancing the model performance in specific scenarios, thereby solving the technical problems that the deep learning model generation method provided by the related technology has low training efficiency and the trained model has poor performance in specific scenarios.

[0183] It can be understood by those skilled in the art that Fig.11 The structure shown is for illustration only, and the computer terminal may also be a terminal device such as a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, and a mobile Internet device (Mobile Internet Devices, MID). Fig.11 The structure of the computer terminal is not limited. For example, the computer terminal 110 may also include Fig.11 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Fig.11 Different configurations shown.

[0184] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which can include: a flash drive, ROM, RAM, a magnetic disk or an optical disk, etc.

[0185] Example 6

[0186] According to an embodiment of the present application, a computer-readable storage medium is also provided. Optionally, in this embodiment, the above storage medium can be used to store the program code executed by the deep learning model generation method provided in the above embodiment 1 or embodiment 2.

[0187] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0188] Optionally, in this embodiment, the computer-readable storage medium is configured to store a program code for executing the following steps: obtaining a prediction result, wherein the prediction result is obtained by predicting input data using a pre-trained language model, and the input data is real request data pre-input in a preset application scenario; obtaining user feedback data based on the prediction result, wherein the user feedback data is used to record acceptance feedback and rejection feedback of the prediction result, the acceptance feedback is used to represent the first part of the results in the prediction result, and the first part of the results are the accepted results, and the rejection feedback is used to represent the second part of the results in the prediction result, and the second part of the results are the rejected results; based on the user feedback data, the pre-trained language model and the language model to be optimized are model aligned to obtain a completed language model; and the completed language model is determined as the target language model.

[0189] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: grouping and sampling the prediction results to obtain multiple groups of preference candidate data to be labeled; labeling the preference candidate data to obtain user feedback data.

[0190] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: determining a target loss between a pre-trained language model and a language model to be optimized through user feedback data; and performing model alignment on the pre-trained language model and the language model to be optimized based on the target loss to obtain a completed language model.

[0191] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: obtaining the probability distribution difference between the pre-trained language model and the language model to be optimized for predicting user feedback data respectively; and determining the target loss based on the probability distribution difference.

[0192] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: obtaining the probability distribution difference between the language model to be optimized and the pre-trained language model predicting acceptance feedback and rejection feedback respectively; normalizing the probability distribution difference to obtain the target loss.

[0193] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: under given input data conditions, determining a first probability that the pre-trained language model predicts acceptance of feedback, and determining a second probability that the language model to be optimized predicts acceptance of feedback; under given input data conditions, determining a third probability that the pre-trained language model predicts rejection of feedback, and determining a fourth probability that the language model to be optimized predicts rejection of feedback; calculating a first probability ratio by the first probability and the second probability, and calculating a second probability ratio by the third probability and the fourth probability; and calculating a probability distribution difference based on the first probability ratio and the second probability ratio.

[0194] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: constructing a loss function based on a first probability ratio, a second probability ratio, a first adjustment parameter and a second adjustment parameter, wherein the first adjustment parameter and the second adjustment parameter are used to adjust the weight of the probability distribution difference calculated by the first probability ratio and the second probability ratio; normalizing the probability distribution difference through the loss function to obtain a target loss.

[0195] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: adjusting the first adjustment parameter and the second adjustment parameter based on the target loss until the probability distribution difference is minimized to obtain the target language model.

[0196] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: initializing model parameters of the language model to be optimized using a pre-trained language model.

[0197] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: obtaining text prediction results, wherein the text prediction results are obtained by predicting pre-input text data using a pre-trained text prediction model, and the text data is a real request text input in a text question and answer scenario; obtaining user feedback data based on the text prediction results, wherein the user feedback data is used to record acceptance text feedback and rejection text feedback of the text prediction results, the acceptance text feedback is used to represent the first part of the results in the text prediction results, and the first part of the results are accepted results, and the rejection text feedback is used to represent the second part of the results in the text prediction results, and the second part of the results are rejected results; based on the user feedback data, the pre-trained text prediction model and the text prediction model to be optimized are model aligned to generate a target text prediction model.

[0198] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: obtaining current input data; using a target language model to predict the current input data to obtain a target prediction result; wherein the target language model is generated after aligning a pre-trained language model with a language model to be optimized based on user feedback data, the user feedback data is obtained based on the prediction result, the prediction result is obtained after predicting the input data using the pre-trained language model, the input data is real request data pre-input in a preset application scenario, the user feedback data is used to record acceptance feedback and rejection feedback of the prediction result, the acceptance feedback is used to represent the first part of the results in the prediction result, the first part of the results is the accepted result, and the rejection feedback is used to represent the second part of the results in the prediction result, the second part of the results is the rejected result.

[0199] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: displaying a request input box and request categories and request examples corresponding to preset application scenarios in a graphical user interface, wherein the request categories and request examples are used to prompt a method for initiating a request in the request input box; in response to a first touch operation applied to the graphical user interface, obtaining current input data input into the request input box based on the request category and the request example; and in response to a second touch operation applied to the graphical user interface, feeding back a target prediction result corresponding to the current input data in the graphical user interface.

[0200] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0201] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0202] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0203] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0204] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0205] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, or all or part of the technical solution. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, ROM, RAM, mobile hard disks, magnetic disks or optical disks.

[0206] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A deep learning model generation method, characterized in that: include: Obtaining a prediction result, wherein the prediction result is obtained by predicting input data using a pre-trained language model, and the input data is real request data pre-input in a preset application scenario; Acquire user feedback data according to the prediction result, wherein the user feedback data is used to record acceptance feedback and rejection feedback of the prediction result, the acceptance feedback is used to represent a first part of the prediction result, the first part of the result is an accepted result, and the rejection feedback is used to represent a second part of the prediction result, the second part of the result is a rejected result; Based on the user feedback data, the pre-trained language model and the language model to be optimized are aligned to obtain a completed language model; The completed language model is determined as the target language model.

2. The deep learning model generation method according to claim 1, characterized in that: Acquiring the user feedback data according to the prediction result includes: The prediction results are grouped and sampled to obtain multiple groups of preference candidate data to be labeled; The preference candidate data is labeled to obtain the user feedback data.

3. The deep learning model generation method according to claim 1, characterized in that: Performing model alignment on the pre-trained language model and the language model to be optimized based on the user feedback data to obtain the completed language model includes: Determining a target loss between the pre-trained language model and the language model to be optimized through the user feedback data; The pre-trained language model and the language model to be optimized are aligned based on the target loss to obtain the completed language model.

4. The deep learning model generation method according to claim 3, characterized in that: Determining the target loss between the pre-trained language model and the language model to be optimized by using the user feedback data includes: Obtaining a difference in probability distribution of the user feedback data predicted by the pre-trained language model and the language model to be optimized; The target loss is determined based on the probability distribution difference.

5. The deep learning model generation method according to claim 4, characterized in that: The deep learning model generation method also includes: Obtaining the probability distribution difference between the acceptance feedback and the rejection feedback predicted by the language model to be optimized and the pre-trained language model respectively; The probability distribution difference is normalized to obtain the target loss.

6. The deep learning model generation method according to claim 5, characterized in that: Obtaining the probability distribution difference between the language model to be optimized and the pre-trained language model respectively predicting the acceptance feedback and the rejection feedback includes: Under the given input data condition, determining a first probability that the pre-trained language model predicts the acceptance of feedback, and determining a second probability that the language model to be optimized predicts the acceptance of feedback; Given the input data, determining a third probability that the pre-trained language model predicts the rejection feedback, and determining a fourth probability that the language model to be optimized predicts the rejection feedback; A first probability ratio is calculated by the first probability and the second probability, and a second probability ratio is calculated by the third probability and the fourth probability; The probability distribution difference is calculated based on the first probability ratio and the second probability ratio.

7. The deep learning model generation method according to claim 6, characterized in that: The probability distribution difference is normalized to obtain the target loss, which includes: Constructing a loss function based on the first probability ratio, the second probability ratio, a first adjustment parameter, and a second adjustment parameter, wherein the first adjustment parameter and the second adjustment parameter are used to adjust the weight of the probability distribution difference calculated by the first probability ratio and the second probability ratio; The probability distribution difference is normalized by the loss function to obtain the target loss.

8. The deep learning model generation method according to claim 7, characterized in that: Performing model alignment on the pre-trained language model and the language model to be optimized based on the target loss to obtain the completed language model includes: The first adjustment parameter and the second adjustment parameter are adjusted based on the target loss until the probability distribution difference is minimized to obtain the padded language model.

9. The deep learning model generation method according to claim 1, characterized in that: The deep learning model generation method also includes: The pre-trained language model is used to initialize model parameters of the language model to be optimized.

10. A deep learning model generation method, characterized in that: include: Obtaining a text prediction result, wherein the text prediction result is obtained by predicting pre-input text data using a pre-trained text prediction model, and the text data is a real request text input in a text question and answer scenario; Acquire user feedback data according to the text prediction result, wherein the user feedback data is used to record acceptance text feedback and rejection text feedback of the text prediction result, the acceptance text feedback is used to represent a first part of the results in the text prediction result, the first part of the results is the accepted result, and the rejection text feedback is used to represent a second part of the results in the text prediction result, the second part of the results is the rejected result; Based on the user feedback data, the pre-trained text prediction model and the text prediction model to be optimized are aligned to obtain a completed text prediction model; The completed text prediction model is determined as the target text prediction model.

11. A model prediction method, characterized in that: include: Get the current input data; Using a target language model to predict the current input data to obtain a target prediction result; Wherein, the target language model is generated using the deep learning model generation method described in any one of claims 1 to 9.

12. The model prediction method according to claim 11, characterized in that: A graphical user interface is provided by an electronic device, wherein the content displayed by the graphical user interface at least partially includes a model prediction scenario, and the model prediction method further includes: Displaying a request input box and a request category and a request example corresponding to the preset application scenario in the graphical user interface, wherein the request category and the request example are used to prompt a method of initiating a request in the request input box; In response to a first touch operation on the graphical user interface, obtaining the current input data input in the request input box based on the request category and the request example; In response to a second touch operation applied to the graphical user interface, the target prediction result corresponding to the current input data is fed back in the graphical user interface.

13. An electronic device, characterized in that: include: A memory storing an executable program; A processor for running the program, wherein the program, when running, executes the deep learning model generation method described in any one of claims 1 to 10 or the model prediction method described in any one of claims 11 to 12.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein, when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the deep learning model generation method described in any one of claims 1 to 10 or the model prediction method described in any one of claims 11 to 12.

Images