Deep learning model generation method, electronic device and storage medium

Abstract

The present application discloses a deep learning model generation method, an electronic device and a storage medium. The method comprises: acquiring a prediction result, wherein the prediction result is obtained by using a pre-trained language model to predict input data, and the input data is real request data pre-input in a preset application scenario; acquiring user feedback data on the basis of the prediction result, wherein the user feedback data is used for recording acceptance feedback and rejection feedback of the prediction result; on the basis of the user feedback data, performing model alignment on the pre-trained language model and a language model to be optimized, to obtain a supplemented language model; and determining the supplemented language model as a target language model. The present application solves the technical problems in the related art of low training efficiency of a deep learning model generation method provided and poor performance of a trained model in a specific scenario.

Description

Deep learning model generation method, electronic device and storage medium Technical Field

[0001] The present application relates to the fields of computer technology and artificial intelligence technology, and specifically to a deep learning model generation method, electronic device, and storage medium. Background Art

[0002] In recent years, the rapid development of artificial intelligence and deep learning technologies has driven significant breakthroughs in the field of natural language processing (NLP) using large-scale pre-trained language models (LLMs). However, according to related technologies, the supervised fine-tuning (SFT) process of LLMs typically relies on additional reward models or complex proximal policy optimization (PPO) methods for reinforcement learning, resulting in low model training efficiency. Furthermore, related technologies rarely or entirely address how to optimize task-specific alignment during LLM training, resulting in the trained LLMs struggling to achieve the expected performance when performing specific tasks in application scenarios.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far.

[0004] Summary of the Invention

[0005] The embodiments of the present application provide a deep learning model generation method, an electronic device, and a storage medium to at least solve the technical problems that the deep learning model generation method provided by the related art has low training efficiency and poor performance of the trained model in specific scenarios.

[0006] According to one aspect of an embodiment of the present application, a deep learning model generation method is provided, including: obtaining a prediction result, wherein the prediction result is obtained by predicting input data using a pre-trained language model, and the input data is real request data pre-input in a preset application scenario; obtaining user feedback data based on the prediction result, wherein the user feedback data is used to record acceptance feedback and rejection feedback of the prediction result, the acceptance feedback is used to represent a first part of the results in the prediction result, the first part of the results being the accepted results, and the rejection feedback is used to represent a second part of the results in the prediction result, the second part of the results being the rejected results; performing model alignment on the pre-trained language model and the language model to be optimized based on the user feedback data to obtain a completed language model; and determining the completed language model as the target language model.

[0007] According to another aspect of an embodiment of the present application, a deep learning model generation method is also provided, including: obtaining a text prediction result, wherein the text prediction result is obtained by predicting pre-input text data using a pre-trained text prediction model, and the text data is a real request text input in a text question-and-answer scenario; obtaining user feedback data based on the text prediction result, wherein the user feedback data is used to record acceptance text feedback and rejection text feedback of the text prediction result, the acceptance text feedback is used to represent the first part of the results in the text prediction result, and the first part of the results is the accepted result, and the rejection text feedback is used to represent the second part of the results in the text prediction result, and the second part of the results is the rejected result; based on the user feedback data, the pre-trained text prediction model and the text prediction model to be optimized are aligned to obtain a completed text prediction model; and the completed text prediction model is determined as the target text prediction model.

[0008] According to another aspect of an embodiment of the present application, a model prediction method is also provided, including: obtaining current input data; using a target language model to predict the current input data to obtain a target prediction result; wherein the target language model is generated after aligning a pre-trained language model with a language model to be optimized based on user feedback data, and the user feedback data is obtained based on the prediction result, and the prediction result is obtained after predicting the input data using the pre-trained language model, the input data is real request data pre-input in a preset application scenario, and the user feedback data is used to record acceptance feedback and rejection feedback of the prediction result, the acceptance feedback is used to represent the first part of the results in the prediction result, and the first part of the results is the accepted result, and the rejection feedback is used to represent the second part of the results in the prediction result, and the second part of the results is the rejected result.

[0009] According to another aspect of an embodiment of the present application, an electronic device is also provided, including: a memory storing an executable program; and a processor for running the program, wherein when the program is running, any one of the above-mentioned deep learning model generation methods or model prediction methods is executed.

[0010] According to another aspect of an embodiment of the present application, a computer-readable storage medium is also provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned deep learning model generation methods or model prediction methods.

[0011] According to the technical solution provided in the present application, by obtaining a prediction result, wherein the prediction result is obtained by predicting the input data using a pre-trained language model, and the input data is real request data pre-input in a preset application scenario; further obtaining user feedback data based on the prediction result, wherein the user feedback data is used to record acceptance feedback and rejection feedback of the prediction result, the acceptance feedback is used to represent the first part of the results in the prediction result, the first part of the results is the accepted result, and the rejection feedback is used to represent the second part of the results in the prediction result, the second part of the results is the rejected result; adopting a method of model alignment between the pre-trained language model and the language model to be optimized based on the user feedback data to obtain a completed language model, and then determining the aligned language model as the target language model.

[0012] It is easy to understand that in the embodiments of the present application, the prediction results and their corresponding user feedback data are obtained for the real request data input by the input end (such as the user) in the preset application scenario. Since the user feedback data can represent the optimization requirements of the language model according to the accepted part and the rejected part of the prediction result, model generation through the user feedback data can achieve the purpose of generating the target language model based on the input feedback of the preset application scenario, thereby achieving the technical effects of improving the model generation efficiency, improving the alignment accuracy of the model for specific tasks, and enhancing the model performance in specific scenarios, thereby solving the technical problems of the deep learning model generation method provided by the related technology, which has low training efficiency and poor performance of the trained model in specific scenarios.

[0013] It is easy to notice that the above general description and the following detailed description are merely for the purpose of exemplifying and explaining the present application, and do not constitute a limitation of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0015] FIG1 is a schematic diagram of an application scenario of the deep learning model generation method in this application;

[0016] FIG2 is a flow chart of a method for generating a deep learning model according to Example 1 of the present application;

[0017] FIG3 is a schematic diagram of an optional deep learning model generation process according to Example 1 of the present application;

[0018] FIG4 is a flowchart of another deep learning model generation method according to Example 2 of the present application;

[0019] FIG5 is a flow chart of a model prediction method according to Example 3 of the present application;

[0020] FIG6 is a schematic structural diagram of a deep learning model generation device according to Example 4 of the present application;

[0021] FIG7 is a schematic structural diagram of an optional deep learning model generation device according to Example 4 of the present application;

[0022] FIG8 is a schematic structural diagram of another deep learning model generation device according to Example 4 of the present application;

[0023] FIG9 is a schematic structural diagram of a model prediction device according to Example 4 of the present application;

[0024] FIG10 is a schematic structural diagram of an optional model prediction device according to Example 4 of the present application;

[0025] FIG11 is a structural block diagram of a computer terminal according to Example 5 of the present application. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] The technical solution provided in this application is mainly implemented using large-scale model technology. The large model here refers to a deep learning model with large-scale model parameters, which can usually contain hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than ten trillion model parameters. The large model can also be called a cornerstone model / foundation model. It is pre-trained by using large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks and has good generalization capabilities, such as large-scale language models (LLMs) and multi-modal pre-training models.

[0029] It should be noted that when large models are actually applied, the pre-trained model can be fine-tuned with a small number of samples, so that the large model can be applied to different tasks. For example, large models can be widely used in fields such as natural language processing (NLP) and computer vision. Specifically, they can be applied to computer vision tasks such as visual question answering (VQA), image captioning (IC), and image generation. They can also be widely used in natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. Therefore, the main application scenarios of large models include but are not limited to digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0030] First, some nouns or terms that appear in the process of describing the embodiments of the present application are subject to the following explanations.

[0031] Large Language Model (LLM): refers to an artificial intelligence model with a certain degree of general capabilities.

[0032] Pretrained Language Model (PLM): refers to a model that learns the statistical characteristics of a language by pretraining on a large amount of text data.

[0033] Training process: The process in which the model generates a new model through the input of the data set and backpropagation to update the parameters.

[0034] Model alignment: refers to the process of adjusting the model output to be consistent with the given information.

[0035] In recent years, the rapid development of artificial intelligence and deep learning technologies has driven LLM to achieve significant breakthroughs in the field of NLP. Among the related technologies, the commonly used deep learning model generation methods are: the first is to generate models based on multiple different outputs predicted for each sample; the second is to generate models through reinforcement learning based on reward models or complex PPO methods. However, although the first method does not require additional training costs, its inference time is very high; the training resources required for the second method are more than three times that of conventional SFT training and the training is difficult. In addition, both of the above deep learning model generation methods have a large room for optimization in terms of training efficiency, and the trained LLMs are difficult to achieve the expected performance when performing specific tasks in application scenarios.

[0036] With respect to the problems caused by the defects of the above-mentioned related technologies, no effective solutions have been proposed in the related technical field before the present application.

[0037] Example 1

[0038] According to an embodiment of the present application, a deep learning model generation method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0039] Taking into account the huge amount of model parameters of the large model and the limited computing resources of the mobile terminal, the above-mentioned deep learning model generation method provided in the embodiment of the present application can be applied to the application scenario shown in Figure 1, but is not limited to this. In the application scenario shown in Figure 1, the large model is deployed in the server 10, and the server 10 can be connected to one or more client devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client device 20 here can include but is not limited to: smart phones, tablet computers, laptops, PDAs, personal computers, smart home devices, car-mounted devices, etc. The client device 20 can interact with the user through a graphical user interface to implement the call to the large model, thereby implementing the method provided in the embodiment of the present application.

[0040] In the above operating environment, the present application provides a deep learning model generation method as shown in Figure 2. Figure 2 is a flow chart of a deep learning model generation method according to Example 1 of the present application. As shown in Figure 2, the deep learning model generation method includes:

[0041] Step S21, obtaining a prediction result, wherein the prediction result is obtained by using a pre-trained language model to predict input data, where the input data is real request data pre-input in a preset application scenario;

[0042] Step S22: obtaining user feedback data based on the prediction results, wherein the user feedback data is used to record acceptance feedback and rejection feedback of the prediction results, where the acceptance feedback is used to indicate the first part of the prediction results, which is the accepted result, and the rejection feedback is used to indicate the second part of the prediction results, which is the rejected result;

[0043] Step S23: aligning the pre-trained language model with the language model to be optimized based on user feedback data to obtain a completed language model;

[0044] Step S24: determining the completed language model as the target language model.

[0045] The aforementioned predefined application scenarios may include, but are not limited to, scenarios involving the use of LLMs in areas such as e-commerce, education, healthcare, conferencing, social networking, financial products, logistics, and navigation. Correspondingly, the aforementioned target language model is a language model trained for the aforementioned predefined application scenarios, and the target language model is capable of performing high performance on specific tasks within the corresponding predefined application scenarios.

[0046] The deep learning model generation method of the embodiment of the present application can be run on the client, and the above-mentioned input data is the real request data input by the user through the client in the above-mentioned preset application scenario. The above-mentioned deep learning model generation method can also be run on the server, and the above-mentioned input data is the real request data obtained by the server from a database or data stream, and the database or data stream is input by the user through the client associated with the server. The above-mentioned real request data is used to represent the user's request information for a specific task in the preset application scenario.

[0047] The pre-trained language model is a language model pre-trained through SFT processing. The pre-trained language model can be a large-scale language model trained for a preset application scenario or a large-scale language model for a general domain. The prediction results include one or more prediction results obtained after predicting the input data.

[0048] Because the pre-trained language model may not be trained specifically for the pre-defined application scenario, the prediction results generated by the pre-trained language model may not fully meet the expected prediction results corresponding to the actual request data in the pre-defined application scenario. Therefore, in order to clarify the optimization requirements of the pre-defined application scenario for the language model, user feedback data is obtained for model alignment. Furthermore, the target language model obtained through model alignment can meet the optimization requirements of the pre-defined application scenario as much as possible. In other words, the target language model can achieve high performance for processing specific tasks in the pre-defined application scenario.

[0049] In an embodiment of the present application, a prediction result is obtained, wherein the prediction result is obtained by predicting input data using a pre-trained language model, and the input data is real request data pre-input in a preset application scenario; user feedback data is further obtained based on the prediction result, wherein the user feedback data is used to record acceptance feedback and rejection feedback of the prediction result, the acceptance feedback is used to represent the first part of the results in the prediction result, the first part of the results is the accepted result, and the rejection feedback is used to represent the second part of the results in the prediction result, the second part of the results is the rejected result; the pre-trained language model and the language model to be optimized are aligned based on the user feedback data to obtain a completed language model, and then the aligned language model is determined as the target language model.

[0050] It is easy to understand that in the embodiments of the present application, the prediction results and their corresponding user feedback data are obtained for the real request data input by the input end (such as the user) in the preset application scenario. Since the user feedback data can represent the optimization requirements of the language model according to the accepted part and the rejected part of the prediction result, model generation through the user feedback data can achieve the purpose of generating the target language model based on the input feedback of the preset application scenario, thereby achieving the technical effects of improving the model generation efficiency, improving the alignment accuracy of the model for specific tasks, and enhancing the model performance in specific scenarios, thereby solving the technical problems of the deep learning model generation method provided by the related technology, which has low training efficiency and poor performance of the trained model in specific scenarios.

[0051] In an embodiment of the present application, a system consisting of a client device and a server can perform the following steps: the client device sends input data to the server, wherein the input data is real request data pre-input in a preset application scenario; the server executes the steps corresponding to the deep learning model generation method, uses a pre-trained language model to predict the input data to obtain a prediction result, obtains user feedback data based on the prediction result, aligns the pre-trained language model with the language model to be optimized based on the user feedback data to obtain a completed language model, determines the completed language model as the target language model, and returns the target language model to the client (or provides the client with an interface for using the target language model), wherein the user feedback data is used to record acceptance feedback and rejection feedback of the prediction result, the acceptance feedback is used to represent the first part of the result in the prediction result, and the first part of the result is the accepted result, and the rejection feedback is used to represent the second part of the result in the prediction result, and the second part of the result is the rejected result.

[0052] It should be noted that, when the operating resources of the client device can meet the deployment and operating conditions of the large model, the embodiments of the present application can be carried out in the client device.

[0053] In a specific implementation of an embodiment of the present application, preference feedback modeling is performed on the SFT model to achieve feedback alignment of the model. The following uses the implementation scenario of preference feedback modeling on the SFT model as an example to further illustrate the above-mentioned deep learning model generation method of the embodiment of the present application.

[0054] In the above implementation scenario, the generation process of the target language model is shown in Figure 3. The pre-trained language model is a large-scale language model processed by SFT, denoted as π SFT The language model to be optimized corresponding to the preset application scenario is denoted as π θ . Using pre-trained language model π SFT Predict the real request data to get the prediction results, then mark the prediction results, determine the acceptance feedback and rejection feedback, and further calculate the target loss based on the acceptance feedback and rejection feedback, and use the target loss to pre-train the language model π SFT And the language model to be optimized π θ The model is aligned to obtain the target language model.

[0055] In an optional embodiment, the deep learning model generation method further includes the following method steps:

[0056] Step S25: Initialize the model parameters of the language model to be optimized using the pre-trained language model.

[0057] In the above exemplary implementation scenario, the pre-trained language model π is used. SFT For the predetermined language model to be optimized π θ Initialize, that is, before optimizing the language model to be optimized, the language model to be optimized π θ The model parameters are the same as the above pre-trained language model π SFT Stay consistent.

[0058] After initialization is completed, obtain the real request data x pre-entered by the user in the preset application scenario (can be obtained in real time), and use the pre-trained language model π SFT Predict the actual request data x (can be real-time prediction), get the prediction results, select N results from the prediction results and record them as y = {y1,…,y N}.

[0059] Furthermore, after obtaining the prediction results, the process of generating a target language model for a preset application scenario can be divided into: the first stage, the preference feedback collection stage; the second stage, the model alignment stage.

[0060] In an optional embodiment, in step S22, obtaining user feedback data based on the prediction result includes the following method steps:

[0061] Step S221, grouping and sampling the prediction results to obtain multiple groups of preference candidate data to be labeled;

[0062] Step S222: Label the preference candidate data to obtain user feedback data.

[0063] In the above exemplary implementation scenario, the prediction result y={y1,…,y N} are sorted to obtain the sorting result, wherein the sorting result is used to sort y1,…,y N The data are sorted from high to low to see if they meet the user's expectations. Then, based on the sorting results, group sampling is performed to obtain multiple groups of preference candidate data (x, y a ,y b ), where y a Indicates that from y1,…,y N One of the prediction results sampled from multiple prediction results, y b Indicates that from y1,…,y N Another prediction result sampled from multiple prediction results, and y a The predicted results relative to y b For example, the above group sampling can be two-by-two sampling. Correspondingly, according to the permutation and combination rules, the number of groups of the above preference candidate data to be marked is

[0064] In the above exemplary implementation scenario, the labeling of the preference candidate data can be completed by a technician or by a pre-developed labeling tool, which can label the prediction results with acceptance / rejection labels according to the user's preferences. After the preference candidate data are labeled, the accepted part of the preference candidate data indicates that the corresponding prediction result is fed back as acceptance, and the rejected part of the preference candidate data indicates that the corresponding prediction result is fed back as rejection, thereby obtaining user feedback data. For each set of preference candidate data (x, y a ,y b ), the corresponding user feedback data can be expressed as (x,y win ,y lose ), where the subscript win of the prediction result y indicates that the prediction result predicted by the pre-trained language model is accepted by the user, whereas the subscript lose indicates that the prediction result is rejected by the user.

[0065] It is easy to notice that in the above exemplary implementation scenario, in the preference feedback collection stage, user feedback data is determined by collecting the user's acceptance / rejection of the prediction results generated by the pre-trained language model. The user feedback data serves as valuable data for aligning the pre-trained language model with the language model to be optimized.

[0066] In an optional embodiment, in step S23, the pre-trained language model and the language model to be optimized are aligned based on the user feedback data to obtain a completed language model, including the following method steps:

[0067] Step S231, determining the target loss between the pre-trained language model and the language model to be optimized through user feedback data;

[0068] In step S232 , the pre-trained language model and the language model to be optimized are aligned based on the target loss to obtain a completed language model.

[0069] In the above exemplary implementation scenario, in the model alignment phase, based on a specific loss function, the target loss (denoted as L(π θ ; π SFT )), and then use the target loss to make the language model to be optimized and the pre-trained language model as aligned as possible to obtain the padded language model, and then determine the padded language model as the target language model.

[0070] The above two stages complete the model generation for the preset application scenario, avoiding the complex reinforcement learning process, simplifying the training process and improving training efficiency.

[0071] Experiments were conducted in specific application scenarios, and it was found that the performance of the target language model generated by the deep learning model generation method provided in the embodiments of the present application was significantly better than the language model generated by related technologies (deep learning model generation methods based on reward models or PPO reinforcement learning).

[0072] In an optional embodiment, in step S231, determining the target loss between the pre-trained language model and the language model to be optimized through user feedback data includes the following method steps:

[0073] Step S311, obtaining the difference in probability distribution of user feedback data predicted by the pre-trained language model and the language model to be optimized;

[0074] Step S312: determining and acquiring a target loss based on the probability distribution difference.

[0075] The above target loss is used to characterize the difference between the pre-trained language model and the language model to be optimized. Therefore, in the above exemplary implementation scenario, the pre-trained language model and the language model to be optimized are used to respectively predict the probability distribution of the predicted user feedback data corresponding to the prediction results. The difference between the probability distributions of the pre-trained language model and the language model to be optimized can characterize the difference in prediction performance of the pre-trained language model and the language model to be optimized for the preset application scenario, thereby determining the target loss L(π θ ; π SFT ).

[0076] In an optional embodiment, the above-mentioned deep learning model generation method further includes:

[0077] Step S3111, obtaining the difference in probability distribution of accepting feedback and rejecting feedback predicted by the language model to be optimized and the pre-trained language model respectively;

[0078] Step S3112: normalize the probability distribution difference to obtain the target loss.

[0079] In the above exemplary implementation scenario, the pre-trained language model π is used. SFT And the language model to be optimized π θ Accepting feedback (y win |x) and rejection feedback (y lose |x) respectively perform probability distribution prediction, and after normalizing the probability distribution difference, the normalized result is used as the target loss to characterize the difference in prediction performance between the pre-trained language model and the language model to be optimized for the preset application scenario.

[0080] In an optional embodiment, in step S3111, obtaining the difference in probability distribution of the language model to be optimized and the pre-trained language model respectively predicting acceptance of feedback and rejection of feedback includes the following method steps:

[0081] Step S31111: Under given input data conditions, determining a first probability that the pre-trained language model predicts that feedback will be accepted, and determining a second probability that the language model to be optimized predicts that feedback will be accepted;

[0082] Step S31112: Under given input data conditions, determining a third probability that the pre-trained language model predicts a rejection of the feedback, and determining a fourth probability that the language model to be optimized predicts a rejection of the feedback;

[0083] Step S31113, calculating a first probability ratio by the first probability and the second probability, and calculating a second probability ratio by the third probability and the fourth probability;

[0084] Step S31114: Calculate the probability distribution difference based on the first probability ratio and the second probability ratio.

[0085] In the above exemplary implementation scenario, for a given input data (ie, x), a pre-trained language model π is used. SFT Predict the first probability π SFT (y win |x); using the language model to be optimized π θ Predict the second probability π θ (y win |x); using pre-trained language model π SFT Predict the third probability π SFT (y lose |x); using the language model to be optimized π θ The fourth probability π is predicted θ (y lose |x).

[0086] Furthermore, for the first probability π SFT (y win |x) and the second probability π θ (y win |x) is divided and then the logarithm is taken to obtain the first probability ratio, which is expressed as For the third probability π SFT (y lose |x) and the fourth probability π θ (y lose |x) is divided and then the logarithm is taken to obtain the second probability ratio, which is expressed as

[0087] In an optional embodiment, in step S3112, normalizing the probability distribution difference to obtain the target loss includes the following method steps:

[0088] Step S31121: constructing a loss function based on the first probability ratio, the second probability ratio, the first adjustment parameter, and the second adjustment parameter, wherein the first adjustment parameter and the second adjustment parameter are used to adjust the weight of the probability distribution difference calculated by the first probability ratio and the second probability ratio;

[0089] Step S31122: Normalize the probability distribution difference using the loss function to obtain the target loss.

[0090] In the exemplary implementation scenario described above, the first and second adjustment parameters are adjustable neural network model hyperparameters. The first adjustment parameter is denoted as α, and the second adjustment parameter is denoted as β. The loss function constructed above can be an activation function. In this example, the activation function is a sigmoid function (denoted as σ).

[0091] Based on this, the loss function shown in the following formula (1) is constructed:

[0092] In the above loss function, π SFT (y win |x) and π SFT (y loss |x) can be used to provide auxiliary constraints to prevent the language model to be optimized from being inconsistent with the pre-trained language model π during the optimization process. SFT Too much deviation.

[0093] In an optional embodiment, in step S23, the pre-trained language model and the language model to be optimized are aligned based on the target loss to obtain a completed language model, including the following method steps:

[0094] In step S231 , the first adjustment parameter and the second adjustment parameter are adjusted based on the target loss until the probability distribution difference is minimized to obtain a completed language model.

[0095] In the above exemplary implementation scenario, based on the target loss L(π θ ; π SFT ) is used to dynamically adjust the first adjustment parameter α and the second adjustment parameter β so that L(π θ ; π SFT ) is taken to the minimum value to obtain the padded language model, and the padded language model at this time is used as the target language model.

[0096] It is easy to notice that the deep learning model generation method provided in the embodiment of the present application can achieve model alignment without using an additional reward model, and also avoids the use of complex approximate strategy optimization for reinforcement learning. The training efficiency is high, and the language model obtained by training has good performance for specific scenarios.

[0097] In summary, the deep learning model generation method provided in the embodiment of the present application can optimize the pre-trained language model, achieve more accurate task alignment in the preset application scenario, and enhance model performance. Specifically, based on the user feedback data corresponding to the prediction results, the pre-trained language model and the language model to be optimized are subjected to model alignment processing to obtain a target language model that can perform well on the specific tasks of the preset application scenario. It is easy to find that the deep learning model generation method provided in the embodiment of the present application does not need to rely on an additional reward model or PPO reinforcement learning method, but uses preference feedback for input data to accurately align the language model to the specific tasks in the preset application scenario in a concise and efficient manner.

[0098] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0099] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0100] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0101] Example 2

[0102] In the operating environment as in Example 1, the present application provides another deep learning model generation method as shown in Figure 4. Figure 4 is a flow chart of another deep learning model generation method according to Example 2 of the present application. As shown in Figure 4, the deep learning model generation method includes:

[0103] Step S41: obtaining a text prediction result, wherein the text prediction result is obtained by using a pre-trained text prediction model to predict pre-input text data, where the text data is the actual request text input in the text question-answering scenario;

[0104] Step S42: obtaining user feedback data based on the text prediction result, wherein the user feedback data is used to record acceptance text feedback and rejection text feedback of the text prediction result, wherein the acceptance text feedback is used to represent the first part of the text prediction result, which is the accepted result, and the rejection text feedback is used to represent the second part of the text prediction result, which is the rejected result;

[0105] Step S43: aligning the pre-trained text prediction model with the text prediction model to be optimized based on the user feedback data to obtain a completed text prediction model;

[0106] Step S44: determining the completed text prediction model as the target text prediction model.

[0107] The aforementioned text question-answering scenarios include, but are not limited to, scenarios involving the use of LLM for text question-answering in e-commerce, education, healthcare, conferences, social networks, financial products, logistics, and navigation. Accordingly, the aforementioned target text prediction model is a text prediction model trained for the aforementioned text question-answering scenarios, and this target text prediction model demonstrates high performance on specific tasks within the corresponding text question-answering scenarios.

[0108] The deep learning model generation method of the embodiment of the present application can be run on a client, and the pre-input text data is the actual request text entered by the user through the client in the text question and answer scenario. The deep learning model generation method can also be run on a server, and the pre-input text data is the actual request text obtained by the server from a database or data stream, and the database or data stream is input by the user through a client associated with the server. The actual request text is used to represent the user's request information for a specific task in the text question and answer scenario.

[0109] The pre-trained text prediction model is a text prediction model pre-trained through SFT processing. The pre-trained text prediction model can be a large-scale text prediction model trained for text question-answering scenarios, or a large-scale text prediction model for general domains. The text prediction results include one or more text prediction results obtained by predicting pre-input text data.

[0110] Because the pre-trained text prediction model may not have been trained specifically for text question-answering scenarios, the text prediction results generated by the pre-trained text prediction model may not fully meet the expected text prediction results for the actual request text in text question-answering scenarios. Therefore, to clarify the optimization requirements of the text prediction model for text question-answering scenarios, user feedback data is obtained for model alignment. Furthermore, the padded text prediction model obtained through model alignment is used as the target text prediction model. The target text prediction model can best meet the optimization requirements of the text question-answering scenario for the text prediction model, that is, the target text prediction model can achieve high performance in handling specific tasks in text question-answering scenarios.

[0111] It should be noted that the above-mentioned text question-and-answer scenario can also be expanded to audio question-and-answer scenarios, video question-and-answer scenarios, virtual reality / augmented reality question-and-answer scenarios, etc.

[0112] In an embodiment of the present application, a text prediction result is first obtained, wherein the text prediction result is obtained by predicting pre-input text data using a pre-trained text prediction model, and the text data is a real request text input in a text question-and-answer scenario; user feedback data is further obtained based on the text prediction result, wherein the user feedback data is used to record acceptance text feedback and rejection text feedback of the text prediction result, the acceptance text feedback is used to represent the first part of the results in the text prediction result, and the first part of the results is the accepted result; the rejection text feedback is used to represent the second part of the results in the text prediction result, and the second part of the results is the rejected result. The pre-trained text prediction model and the text prediction model to be optimized are aligned based on the user feedback data to obtain a completed text prediction model, and then the completed text prediction model is determined as the target text prediction model.

[0113] It is easy to understand that in the embodiments of the present application, for the actual request text input by the input end (such as the user) in the text question and answer scenario, the text prediction result and its corresponding preference feedback result are obtained. Since the preference feedback result can represent the optimization requirements of the text prediction model according to the accepted part and the rejected part of the text prediction result, model generation through the preference feedback result can achieve the purpose of generating a target text prediction model based on the input feedback of a specific text question and answer scenario, thereby achieving the technical effect of improving the model generation efficiency, improving the model's alignment accuracy for specific tasks, and enhancing the model performance in specific scenarios, thereby solving the technical problems of the deep learning model generation method provided by the related technology, which has low training efficiency and poor performance of the trained model in specific scenarios.

[0114] In an embodiment of the present application, a system consisting of a client device and a server can perform the following steps: the client device executes sending pre-input text data to the server, wherein the pre-input text data is a real request text pre-input in a text question and answer scenario; the server executes the steps corresponding to the deep learning model generation method, uses a pre-trained text prediction model to predict the pre-input text data to obtain a text prediction result, obtains user feedback data based on the text prediction result, aligns the pre-trained text prediction model with the text prediction model to be optimized based on the user feedback data to obtain a completed text prediction model, determines the completed text prediction model as the target text prediction model, and returns the target text prediction model to the client (or provides the client with an interface for using the target text prediction model), wherein the user feedback data is used to record acceptance text feedback and rejection text feedback of the text prediction result, the acceptance text feedback is used to represent the first part of the results in the text prediction result, and the first part of the results is the accepted result, and the rejection text feedback is used to represent the second part of the results in the text prediction result, and the second part of the results is the rejected result.

[0115] It should be noted that, when the operating resources of the client device can meet the deployment and operating conditions of the large model, the embodiments of the present application can be carried out in the client device.

[0116] It should be noted that the preferred implementation of this embodiment can be found in the relevant description in Example 1 and will not be repeated here.

[0117] Example 3

[0118] In the operating environment as in Example 1, the present application provides a model prediction method as shown in Figure 5. Figure 5 is a flow chart of a model prediction method according to Example 3 of the present application. As shown in Figure 5, the model prediction method includes:

[0119] Step S51, obtaining current input data;

[0120] Step S52, using the target language model to predict the current input data to obtain a target prediction result;

[0121] Among them, the target language model is generated after aligning the pre-trained language model and the language model to be optimized based on user feedback data. The user feedback data is obtained according to the prediction results. The prediction results are obtained by predicting the input data using the pre-trained language model. The input data is the real request data pre-input in the preset application scenario. The user feedback data is used to record the acceptance feedback and rejection feedback of the prediction results. The acceptance feedback is used to represent the first part of the prediction results, and the first part of the results is the accepted result. The rejection feedback is used to represent the second part of the prediction results, and the second part of the results is the rejected result.

[0122] According to an embodiment of the present application, a model prediction service is provided. The model prediction service can be run on a client or a server. The server can be a standalone or distributed server, or a server located in the cloud. After obtaining the current input data, the model prediction service uses a target language model corresponding to a preset application scenario generated according to the deep learning model generation method to predict the current input data, obtain a target prediction result, and output the target prediction result to the service caller.

[0123] In the embodiment of the present application, the deep learning model generation method based on the aforementioned embodiment can not only achieve the purpose of generating a target language model based on input feedback of a preset application scenario, but also realize the technical effects of improving model generation efficiency, improving the alignment accuracy of the model for specific tasks, and enhancing the model performance in specific scenarios, thereby solving the technical problems of low training efficiency and poor performance of the trained model in specific scenarios of the deep learning model generation method provided by the related technology, but also can conveniently provide model prediction services for specific tasks in the preset application scenario, which is user-friendly and conducive to application in specific scenarios.

[0124] In an optional embodiment, a graphical user interface is provided by an electronic device, and content displayed by the graphical user interface at least partially includes a model prediction scenario. The model prediction method further includes the following method steps:

[0125] Step S531: Displaying a request input box and request categories and request examples corresponding to preset application scenarios in a graphical user interface, wherein the request categories and request examples are used to prompt a method for initiating a request in the request input box;

[0126] Step S532 , in response to the first touch operation on the graphical user interface, obtaining current input data entered in the request input box based on the request type and the request example;

[0127] Step S533 , in response to the second touch operation on the graphical user interface, the target prediction result corresponding to the current input data is fed back in the graphical user interface.

[0128] In the above optional embodiment, the electronic device may be a terminal device or a cloud device. The graphical user interface provided by the electronic device enables interaction with the user. Through the model prediction scenario displayed by the graphical user interface, the request input box and the request category and request example corresponding to the preset application scenario are presented to the user, wherein the request category includes multiple candidate categories preset in the model prediction scenario, and the request example includes at least one question example preset in the model prediction scenario.

[0129] When the above-mentioned first touch operation is detected, the current input data entered in the request input box is obtained based on the request category and the request example, wherein the first touch operation is an editing request operation (such as a selection operation performed on multiple candidate categories and at least one question example, an input operation performed on the request input box, etc.), and the above-mentioned request category, request example and current input data are used as input to the target language model, so as to perform model prediction and obtain the target prediction result.

[0130] Furthermore, when the second touch operation is detected, the target prediction result corresponding to the current input data is fed back in the graphical user interface, wherein the second touch operation is an operation of submitting a request, and the second touch operation is used to trigger an event in which the target language model starts prediction.

[0131] Through the above optional embodiments, the present application provides a visual implementation method of a model prediction method, which can be implemented on the client to support users in specifying request categories, request examples, and current input data, as well as controlling the start of model prediction. This method is user-friendly, convenient, and has a good user experience.

[0132] It should be noted that the preferred implementation of this embodiment can be found in the relevant description in Example 1 and will not be repeated here.

[0133] Example 4

[0134] According to an embodiment of the present application, an embodiment of a device for implementing the above-mentioned deep learning model generation method is also provided. Figure 6 is a structural schematic diagram of a deep learning model generation device according to embodiment 4 of the present application. As shown in Figure 6, the device includes:

[0135] A first acquisition module 601 is configured to acquire a prediction result, wherein the prediction result is obtained by using a pre-trained language model to predict input data, where the input data is real request data pre-input in a preset application scenario;

[0136] A second acquisition module 602 is configured to acquire user feedback data based on the prediction results, wherein the user feedback data is used to record acceptance feedback and rejection feedback of the prediction results, where acceptance feedback is used to indicate the first portion of the prediction results, which is the accepted result, and rejection feedback is used to indicate the second portion of the prediction results, which is the rejected result;

[0137] A generation module 603 is configured to align the pre-trained language model with the language model to be optimized based on user feedback data to obtain a completed language model;

[0138] The determination module 604 determines the completed language model as the target language model.

[0139] Optionally, the second acquisition module 602 is further configured to: perform group sampling on the prediction results to obtain multiple groups of preference candidate data to be labeled; and label the preference candidate data to obtain user feedback data.

[0140] Optionally, the generation module 603 is further configured to: determine a target loss between the pre-trained language model and the language model to be optimized through user feedback data; and perform model alignment on the pre-trained language model and the language model to be optimized based on the target loss to obtain a completed language model.

[0141] Optionally, the generation module 603 is further configured to: obtain a difference in probability distribution of user feedback data predicted by the pre-trained language model and the language model to be optimized; and determine a target loss based on the difference in probability distribution.

[0142] Optionally, the generation module 603 is further configured to: obtain a probability distribution difference between the language model to be optimized and the pre-trained language model predicting acceptance of feedback and rejection of feedback; and normalize the probability distribution difference to obtain a target loss.

[0143] Optionally, the above-mentioned generation module 603 is also used to: determine, under given input data conditions, a first probability that the pre-trained language model predicts acceptance of feedback, and determine a second probability that the language model to be optimized predicts acceptance of feedback; under given input data conditions, determine a third probability that the pre-trained language model predicts rejection of feedback, and determine a fourth probability that the language model to be optimized predicts rejection of feedback; calculate a first probability ratio by dividing the first probability by the second probability, and calculate a second probability ratio by dividing the third probability by the fourth probability; and calculate a probability distribution difference based on the first probability ratio and the second probability ratio.

[0144] Optionally, the above-mentioned generation module 603 is also used to: construct a loss function based on the first probability ratio, the second probability ratio, the first adjustment parameter and the second adjustment parameter, wherein the first adjustment parameter and the second adjustment parameter are used to adjust the weight of the probability distribution difference calculated by the first probability ratio and the second probability ratio; and normalize the probability distribution difference through the loss function to obtain the target loss.

[0145] Optionally, the generating module 603 is further configured to adjust the first adjustment parameter and the second adjustment parameter based on the target loss until the probability distribution difference is minimized to obtain a completed language model.

[0146] Optionally, Figure 7 is a structural diagram of an optional deep learning model generation device according to Example 4 of the present application. As shown in Figure 7, in addition to all the modules shown in Figure 6, the device also includes: an initialization module 605, which is used to initialize the model parameters of the language model to be optimized using a pre-trained language model.

[0147] It should be noted that the first acquisition module 601, the second acquisition module 602, the generation module 603, and the determination module 604 correspond to steps S21 to S24 in Example 1. The four modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in Example 1. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors. The above modules can also be part of the device and can run in the operating environment provided in Example 1.

[0148] In an embodiment of the present application, a prediction result is obtained, wherein the prediction result is obtained by predicting input data using a pre-trained language model, and the input data is real request data pre-input in a preset application scenario; user feedback data is further obtained based on the prediction result, wherein the user feedback data is used to record acceptance feedback and rejection feedback of the prediction result, the acceptance feedback is used to represent the first part of the results in the prediction result, and the first part of the results is the accepted result, and the rejection feedback is used to represent the second part of the results in the prediction result, and the second part of the results is the rejected result; the pre-trained language model and the language model to be optimized are aligned based on the user feedback data to obtain a completed language model, and then the completed language model is determined as the target language model.

[0149] It is easy to understand that in the embodiments of the present application, the prediction results and their corresponding user feedback data are obtained for the real request data input by the input end (such as the user) in the preset application scenario. Since the user feedback data can represent the optimization requirements of the language model according to the accepted part and the rejected part of the prediction result, model generation through the user feedback data can achieve the purpose of generating the target language model based on the input feedback of the preset application scenario, thereby achieving the technical effects of improving the model generation efficiency, improving the alignment accuracy of the model for specific tasks, and enhancing the model performance in specific scenarios, thereby solving the technical problems of the deep learning model generation method provided by the related technology, which has low training efficiency and poor performance of the trained model in specific scenarios.

[0150] According to an embodiment of the present application, another device embodiment for implementing the deep learning model generation method in the above-mentioned embodiment 2 is also provided. Figure 8 is a structural schematic diagram of another deep learning model generation device according to embodiment 4 of the present application. As shown in Figure 8, the device includes:

[0151] A first acquisition module 801 is configured to acquire a text prediction result, wherein the text prediction result is obtained by predicting pre-input text data using a pre-trained text prediction model, where the text data is a real request text input in a text question-answering scenario;

[0152] A second acquisition module 802 is configured to acquire user feedback data based on the text prediction results, wherein the user feedback data is used to record acceptance text feedback and rejection text feedback of the text prediction results, wherein the acceptance text feedback is used to represent the first part of the text prediction results, which is the accepted result, and the rejection text feedback is used to represent the second part of the text prediction results, which is the rejected result;

[0153] A generation module 803 is configured to perform model alignment on the pre-trained text prediction model and the text prediction model to be optimized based on user feedback data to obtain a completed text prediction model;

[0154] The determination module 804 determines the completed text prediction model as the target text prediction model.

[0155] It should be noted that the first acquisition module 801, the second acquisition module 802, the generation module 803, and the determination module 804 described above correspond to steps S41 to S44 in Example 2. The examples and application scenarios implemented by the four modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 2. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors. The above modules can also be part of the device and can run in the operating environment provided in Example 1.

[0156] According to an embodiment of the present application, another embodiment of a device for implementing the model prediction method in the above embodiment 3 is also provided. FIG9 is a schematic structural diagram of a model prediction device according to embodiment 4 of the present application. As shown in FIG9 , the device includes:

[0157] Acquisition module 901, used to obtain current input data;

[0158] Prediction module 902, used to predict the current input data using the target language model to obtain a target prediction result;

[0159] Among them, the target language model is generated after aligning the pre-trained language model and the language model to be optimized based on user feedback data. The user feedback data is obtained according to the prediction results. The prediction results are obtained by predicting the input data using the pre-trained language model. The input data is the real request data pre-input in the preset application scenario. The user feedback data is used to record the acceptance feedback and rejection feedback of the prediction results. The acceptance feedback is used to represent the first part of the prediction results, and the first part of the results is the accepted result. The rejection feedback is used to represent the second part of the prediction results, and the second part of the results is the rejected result.

[0160] Optionally, a graphical user interface is provided by an electronic device, and the content displayed by the graphical user interface at least partially includes a model prediction scenario. Figure 10 is a structural diagram of an optional model prediction device according to Example 4 of the present application. As shown in Figure 10, in addition to all the modules shown in Figure 9, the device also includes: a visualization module 903, which is used to display the request input box and the request category and request example corresponding to the preset application scenario in the graphical user interface, wherein the request category and request example are used to prompt the method of initiating a request in the request input box; in response to a first touch operation acting on the graphical user interface, obtain the current input data entered in the request input box based on the request category and request example; in response to a second touch operation acting on the graphical user interface, feedback the target prediction result corresponding to the current input data in the graphical user interface.

[0161] It should be noted that the acquisition module 901 and the prediction module 902 correspond to steps S51 to S52 in Example 3. The examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 3. It should be noted that the modules or units can be hardware components or software components stored in a memory and processed by one or more processors. The modules can also be part of the device and can run in the operating environment provided in Example 1.

[0162] It should be noted that the preferred implementation of this embodiment can refer to the relevant description in Example 1 or Example 2, and will not be repeated here.

[0163] Example 5

[0164] According to an embodiment of the present application, an electronic device is also provided. The electronic device may be a computer terminal. The computer terminal may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal may be replaced by a terminal device such as a mobile terminal.

[0165] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of a computer network.

[0166] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the deep learning model generation method: obtaining a prediction result, wherein the prediction result is obtained by predicting the input data using a pre-trained language model, and the input data is real request data pre-input in a preset application scenario; obtaining user feedback data based on the prediction result, wherein the user feedback data is used to record acceptance feedback and rejection feedback of the prediction result, the acceptance feedback is used to represent the first part of the results in the prediction result, and the first part of the results is the accepted result, and the rejection feedback is used to represent the second part of the results in the prediction result, and the second part of the results is the rejected result; based on the user feedback data, the pre-trained language model and the language model to be optimized are aligned to obtain a completed language model; and the completed language model is determined as the target language model.

[0167] Optionally, Figure 11 is a structural block diagram of a computer terminal according to Example 5 of the present application. As shown in Figure 11, the computer terminal 110 may include: one or more (only one is shown in the figure) processors 1102, a memory 1104, a storage controller 1106, and a peripheral interface 1108, wherein the peripheral interface 1108 is connected to the radio frequency module, the audio module and the display.

[0168] Among them, the memory 1104 can be used to store software programs and modules, such as the program instructions / modules corresponding to the deep learning model generation method and device in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned deep learning model generation method. The memory 1104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1104 may further include a memory remotely located relative to the processor, and these remote memories can be connected to the computer terminal 110 via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0169] The processor 1102 can call the information and application stored in the memory through the transmission device to perform the following steps: obtain a prediction result, wherein the prediction result is obtained by predicting the input data using a pre-trained language model, and the input data is the real request data pre-input in a preset application scenario; obtain user feedback data based on the prediction result, wherein the user feedback data is used to record the acceptance feedback and rejection feedback of the prediction result, the acceptance feedback is used to represent the first part of the results in the prediction result, the first part of the results is the accepted result, and the rejection feedback is used to represent the second part of the results in the prediction result, the second part of the results is the rejected result; based on the user feedback data, the pre-trained language model and the language model to be optimized are aligned to obtain a completed language model; and the completed language model is determined as the target language model.

[0170] Optionally, the processor 1102 may further execute program codes of the following steps: performing group sampling on the prediction results to obtain multiple groups of preference candidate data to be labeled; and labeling the preference candidate data to obtain user feedback data.

[0171] Optionally, the processor 1102 may also execute the program code of the following steps: determining the target loss between the pre-trained language model and the language model to be optimized through user feedback data; and performing model alignment on the pre-trained language model and the language model to be optimized based on the target loss to obtain a completed language model.

[0172] Optionally, the processor 1102 may further execute program code of the following steps: obtaining a probability distribution difference between the pre-trained language model and the language model to be optimized for respectively predicting user feedback data; and determining a target loss based on the probability distribution difference.

[0173] Optionally, the processor 1102 may further execute program code of the following steps: obtaining the probability distribution difference between the language model to be optimized and the pre-trained language model for predicting acceptance of feedback and rejection of feedback respectively; and normalizing the probability distribution difference to obtain a target loss.

[0174] Optionally, the processor 1102 may also execute program code for the following steps: determining, under given input data conditions, a first probability that the pre-trained language model predicts acceptance of feedback, and determining a second probability that the language model to be optimized predicts acceptance of feedback; determining, under given input data conditions, a third probability that the pre-trained language model predicts rejection of feedback, and determining a fourth probability that the language model to be optimized predicts rejection of feedback; calculating a first probability ratio by the first probability and the second probability, and calculating a second probability ratio by the third probability and the fourth probability; and calculating a probability distribution difference based on the first probability ratio and the second probability ratio.

[0175] Optionally, the processor 1102 may also execute the program code of the following steps: constructing a loss function based on the first probability ratio, the second probability ratio, the first adjustment parameter, and the second adjustment parameter, wherein the first adjustment parameter and the second adjustment parameter are used to adjust the weight of the probability distribution difference calculated by the first probability ratio and the second probability ratio; normalizing the probability distribution difference through the loss function to obtain the target loss.

[0176] Optionally, the processor 1102 may further execute program code of the following steps: adjusting the first adjustment parameter and the second adjustment parameter based on the target loss until the probability distribution difference is minimized to obtain a target language model.

[0177] Optionally, the processor 1102 may further execute program code of the following steps: initializing model parameters of the language model to be optimized using a pre-trained language model.

[0178] The processor 1102 can call the information and application stored in the memory through the transmission device to perform the following steps: obtaining a text prediction result, wherein the text prediction result is obtained by predicting pre-input text data using a pre-trained text prediction model, and the text data is the actual request text input in a text question and answer scenario; obtaining user feedback data based on the text prediction result, wherein the user feedback data is used to record the acceptance text feedback and rejection text feedback of the text prediction result, the acceptance text feedback is used to represent the first part of the results in the text prediction result, and the first part of the results is the accepted result, and the rejection text feedback is used to represent the second part of the results in the text prediction result, and the second part of the results is the rejected result; based on the user feedback data, the pre-trained text prediction model and the text prediction model to be optimized are aligned to generate a target text prediction model.

[0179] The processor 1102 can call the information and application stored in the memory through the transmission device to perform the following steps: obtain current input data; use the target language model to predict the current input data to obtain a target prediction result; wherein, the target language model is generated after aligning the pre-trained language model and the language model to be optimized based on the user feedback data, and the user feedback data is obtained based on the prediction result. The prediction result is obtained by predicting the input data using the pre-trained language model. The input data is the real request data pre-input in the preset application scenario. The user feedback data is used to record the acceptance feedback and rejection feedback of the prediction result. The acceptance feedback is used to represent the first part of the result in the prediction result, and the first part of the result is the accepted result. The rejection feedback is used to represent the second part of the result in the prediction result, and the second part of the result is the rejected result.

[0180] Optionally, the processor 1102 may also execute the program code for the following steps: displaying a request input box and a request category and a request example corresponding to a preset application scenario in a graphical user interface, wherein the request category and the request example are used to prompt a method for initiating a request in the request input box; responding to a first touch operation on the graphical user interface, obtaining current input data input in the request input box based on the request category and the request example; responding to a second touch operation on the graphical user interface, feeding back a target prediction result corresponding to the current input data in the graphical user interface.

[0181] According to an embodiment of the present application, a computer terminal solution for implementing the above-mentioned deep learning model generation method or model prediction method is provided. By obtaining a prediction result, wherein the prediction result is obtained by predicting input data using a pre-trained language model, and the input data is real request data pre-input in a preset application scenario; further obtaining user feedback data based on the prediction result, wherein the user feedback data is used to record acceptance feedback and rejection feedback of the prediction result, the acceptance feedback is used to represent the first part of the results in the prediction result, the first part of the results is the accepted result, and the rejection feedback is used to represent the second part of the results in the prediction result, the second part of the results is the rejected result; using the user feedback data to align the pre-trained language model with the language model to be optimized, a completed language model is obtained, and then the aligned language model is determined as the target language model.

[0182] It is easy to understand that in the embodiments of the present application, the prediction results and their corresponding user feedback data are obtained for the real request data input by the input end (such as the user) in the preset application scenario. Since the user feedback data can represent the optimization requirements of the language model according to the accepted part and the rejected part of the prediction result, model generation through the user feedback data can achieve the purpose of generating the target language model based on the input feedback of the preset application scenario, thereby achieving the technical effects of improving the model generation efficiency, improving the alignment accuracy of the model for specific tasks, and enhancing the model performance in specific scenarios, thereby solving the technical problems of the deep learning model generation method provided by the related technology, which has low training efficiency and poor performance of the trained model in specific scenarios.

[0183] Those skilled in the art will appreciate that the structure shown in FIG11 is merely illustrative, and the computer terminal may also be a terminal device such as a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, or a mobile internet device (MID). FIG11 does not limit the structure of the computer terminal. For example, the computer terminal 110 may include more or fewer components (such as a network interface, a display device, etc.) than those shown in FIG11, or may have a configuration different from that shown in FIG11.

[0184] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which can include: a flash drive, ROM, RAM, a magnetic disk or an optical disk, etc.

[0185] Example 6

[0186] According to an embodiment of the present application, a computer-readable storage medium is further provided. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the deep learning model generation method provided in the above embodiment 1 or embodiment 2.

[0187] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0188] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: obtaining a prediction result, wherein the prediction result is obtained by predicting input data using a pre-trained language model, and the input data is real request data pre-input in a preset application scenario; obtaining user feedback data based on the prediction result, wherein the user feedback data is used to record acceptance feedback and rejection feedback of the prediction result, the acceptance feedback is used to represent the first part of the results in the prediction result, and the first part of the results is the accepted result, and the rejection feedback is used to represent the second part of the results in the prediction result, and the second part of the results is the rejected result; based on the user feedback data, the pre-trained language model and the language model to be optimized are aligned to obtain a completed language model; and the completed language model is determined as the target language model.

[0189] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: grouping and sampling the prediction results to obtain multiple groups of preference candidate data to be labeled; labeling the preference candidate data to obtain user feedback data.

[0190] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: determining the target loss between the pre-trained language model and the language model to be optimized through user feedback data; and performing model alignment on the pre-trained language model and the language model to be optimized based on the target loss to obtain a completed language model.

[0191] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: obtaining the probability distribution difference between the pre-trained language model and the language model to be optimized for respectively predicting user feedback data; and determining the target loss based on the probability distribution difference.

[0192] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: obtaining the probability distribution difference between the language model to be optimized and the pre-trained language model for predicting acceptance of feedback and rejection of feedback respectively; normalizing the probability distribution difference to obtain the target loss.

[0193] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: under given input data conditions, determining a first probability that the pre-trained language model predicts acceptance of feedback, and determining a second probability that the language model to be optimized predicts acceptance of feedback; under given input data conditions, determining a third probability that the pre-trained language model predicts rejection of feedback, and determining a fourth probability that the language model to be optimized predicts rejection of feedback; calculating a first probability ratio by dividing the first probability by the second probability, and calculating a second probability ratio by dividing the third probability by the fourth probability; and calculating a probability distribution difference based on the first probability ratio and the second probability ratio.

[0194] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: constructing a loss function based on the first probability ratio, the second probability ratio, the first adjustment parameter and the second adjustment parameter, wherein the first adjustment parameter and the second adjustment parameter are used to adjust the weight of the probability distribution difference calculated by the first probability ratio and the second probability ratio; normalizing the probability distribution difference through the loss function to obtain the target loss.

[0195] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: adjusting the first adjustment parameter and the second adjustment parameter based on the target loss until the probability distribution difference is minimized to obtain the target language model.

[0196] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: initializing model parameters of the language model to be optimized using a pre-trained language model.

[0197] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: obtaining text prediction results, wherein the text prediction results are obtained by predicting pre-input text data using a pre-trained text prediction model, and the text data is a real request text input in a text question and answer scenario; obtaining user feedback data based on the text prediction results, wherein the user feedback data is used to record acceptance text feedback and rejection text feedback of the text prediction results, the acceptance text feedback is used to represent the first part of the results in the text prediction results, and the first part of the results is the accepted result, and the rejection text feedback is used to represent the second part of the results in the text prediction results, and the second part of the results is the rejected result; based on the user feedback data, the pre-trained text prediction model and the text prediction model to be optimized are model aligned to generate a target text prediction model.

[0198] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: obtaining current input data; using a target language model to predict the current input data to obtain a target prediction result; wherein the target language model is generated after aligning a pre-trained language model with a language model to be optimized based on user feedback data, the user feedback data is obtained based on the prediction result, and the prediction result is obtained after predicting the input data using the pre-trained language model, the input data is real request data pre-input in a preset application scenario, and the user feedback data is used to record acceptance feedback and rejection feedback of the prediction result, the acceptance feedback is used to represent the first part of the results in the prediction result, and the first part of the results is the accepted result, and the rejection feedback is used to represent the second part of the results in the prediction result, and the second part of the results is the rejected result.

[0199] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: displaying a request input box and a request category and a request example corresponding to a preset application scenario in a graphical user interface, wherein the request category and the request example are used to prompt a method for initiating a request in the request input box; in response to a first touch operation applied to the graphical user interface, obtaining current input data input in the request input box based on the request category and the request example; in response to a second touch operation applied to the graphical user interface, feeding back a target prediction result corresponding to the current input data in the graphical user interface.

[0200] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0201] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0202] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0203] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0204] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0205] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, ROM, RAM, mobile hard drives, magnetic disks or optical disks.

[0206] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A deep learning model generation method, comprising: Obtaining a prediction result, wherein the prediction result is obtained by predicting input data using a pre-trained language model, and the input data is real request data pre-input in a preset application scenario; Acquire user feedback data according to the prediction result, wherein the user feedback data is used to record acceptance feedback and rejection feedback of the prediction result, the acceptance feedback is used to represent a first part of the prediction result, the first part of the result is an accepted result, and the rejection feedback is used to represent a second part of the prediction result, the second part of the result is a rejected result; Based on the user feedback data, the pre-trained language model and the language model to be optimized are aligned to obtain a completed language model; The completed language model is determined as the target language model.

2. The deep learning model generation method according to claim 1, wherein: Acquiring the user feedback data according to the prediction result includes: The prediction results are grouped and sampled to obtain multiple groups of preference candidate data to be labeled; The preference candidate data is labeled to obtain the user feedback data.

3. The deep learning model generation method according to claim 1, wherein: Performing model alignment on the pre-trained language model and the language model to be optimized based on the user feedback data to obtain the completed language model includes: Determining a target loss between the pre-trained language model and the language model to be optimized through the user feedback data; The pre-trained language model and the language model to be optimized are aligned based on the target loss to obtain the completed language model.

4. The deep learning model generation method according to claim 3, wherein: Determining the target loss between the pre-trained language model and the language model to be optimized by using the user feedback data includes: Obtaining a difference in probability distribution of the user feedback data predicted by the pre-trained language model and the language model to be optimized; The target loss is determined based on the probability distribution difference.

5. The deep learning model generation method according to claim 4, wherein: The deep learning model generation method also includes: Obtaining the probability distribution difference between the acceptance feedback and the rejection feedback predicted by the language model to be optimized and the pre-trained language model respectively; The probability distribution difference is normalized to obtain the target loss.

6. The deep learning model generation method according to claim 5, wherein: Obtaining the probability distribution difference between the language model to be optimized and the pre-trained language model respectively predicting the acceptance feedback and the rejection feedback includes: Under the given input data condition, determining a first probability that the pre-trained language model predicts the acceptance of feedback, and determining a second probability that the language model to be optimized predicts the acceptance of feedback; Given the input data, determining a third probability that the pre-trained language model predicts the rejection feedback, and determining a fourth probability that the language model to be optimized predicts the rejection feedback; A first probability ratio is calculated by the first probability and the second probability, and a second probability ratio is calculated by the third probability and the fourth probability; The probability distribution difference is calculated based on the first probability ratio and the second probability ratio.

7. The deep learning model generation method according to claim 6, wherein: The probability distribution difference is normalized to obtain the target loss, which includes: Constructing a loss function based on the first probability ratio, the second probability ratio, a first adjustment parameter, and a second adjustment parameter, wherein the first adjustment parameter and the second adjustment parameter are used to adjust the weight of the probability distribution difference calculated by the first probability ratio and the second probability ratio; The probability distribution difference is normalized by the loss function to obtain the target loss.

8. The deep learning model generation method according to claim 7, wherein: Performing model alignment on the pre-trained language model and the language model to be optimized based on the target loss to obtain the completed language model includes: The first adjustment parameter and the second adjustment parameter are adjusted based on the target loss until the probability distribution difference is minimized to obtain the padded language model.

9. The deep learning model generation method according to claim 1, wherein: The deep learning model generation method also includes: The pre-trained language model is used to initialize model parameters of the language model to be optimized.

10. A deep learning model generation method, comprising: Obtaining a text prediction result, wherein the text prediction result is obtained by predicting pre-input text data using a pre-trained text prediction model, and the text data is a real request text input in a text question and answer scenario; Acquire user feedback data according to the text prediction result, wherein the user feedback data is used to record acceptance text feedback and rejection text feedback of the text prediction result, the acceptance text feedback is used to represent a first part of the results in the text prediction result, the first part of the results is the accepted result, and the rejection text feedback is used to represent a second part of the results in the text prediction result, the second part of the results is the rejected result; Based on the user feedback data, the pre-trained text prediction model and the text prediction model to be optimized are aligned to obtain a completed text prediction model; The completed text prediction model is determined as the target text prediction model.

11. A model prediction method, comprising: Get the current input data; Using a target language model to predict the current input data to obtain a target prediction result; Wherein, the target language model is generated using the deep learning model generation method described in any one of claims 1 to 9.

12. The model prediction method according to claim 11, wherein: A graphical user interface is provided by an electronic device, wherein the content displayed by the graphical user interface at least partially includes a model prediction scenario, and the model prediction method further includes: Displaying a request input box and a request category and a request example corresponding to the preset application scenario in the graphical user interface, wherein the request category and the request example are used to prompt a method of initiating a request in the request input box; In response to a first touch operation on the graphical user interface, obtaining the current input data input in the request input box based on the request category and the request example; In response to a second touch operation applied to the graphical user interface, the target prediction result corresponding to the current input data is fed back in the graphical user interface.

13. An electronic device, comprising: A memory storing an executable program; A processor, configured to run the program, wherein when the program is run, the deep learning model generation method described in any one of claims 1 to 10 or the model prediction method described in any one of claims 11 to 12 is executed.

14. A computer-readable storage medium comprising a stored executable program, wherein: When the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the deep learning model generation method described in any one of claims 1 to 10 or the model prediction method described in any one of claims 11 to 12.

Images