Large Language Model Determination Methods, Devices, and Program Products

Abstract

This disclosure provides a method, apparatus, and program product for determining a large language model, relating to the field of computer technology. The method includes: in the data collection stage, constructing a preference dataset for multiple dimensions; constructing a loss function corresponding to the large language model based on the probability of generating data for each dimension at the large language model training node; performing adaptive multi-objective training on the preference dataset until the loss function converges to obtain a target large language model; the obtained target large language model can improve the balance of responses across multiple dimensions.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to an apparatus and program product for determining a large language model. Background Technology

[0002] Large language models refer to models built using artificial intelligence techniques that can automatically learn, adapt, and optimize to achieve more accurate and efficient predictions and decisions. They are typically built on machine learning, deep learning, and other technologies, enabling them to learn patterns and rules from large amounts of data and make predictions and decisions. With the development of technology, the application of large language models has become increasingly widespread; however, current responses from these models do not consistently emphasize different dimensions, resulting in an unbalanced dimensionality in the responses.

[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] This disclosure provides a method, apparatus, and program product for determining large language models, which at least improves the balance of responses from large language models across multiple dimensions.

[0005] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0006] According to one aspect of this disclosure, a method for determining a large language model is provided, comprising:

[0007] During the data collection phase, preference datasets are constructed across multiple dimensions;

[0008] During the training phase of the large language model, a loss function corresponding to the large language model is constructed based on the probability of generating data in each dimension. Adaptive multi-objective training is then performed on the preference dataset until the loss function converges, resulting in the target large language model.

[0009] In one embodiment of this disclosure, the method further includes:

[0010] If the loss function value corresponding to the loss function does not converge, the weight of each dimension is determined based on the probability of generating data for each dimension and the Gaussian sampling model.

[0011] Adjust the weights corresponding to each dimension to reduce the differences between the multiple dimensions of the response output by the large language model;

[0012] The loss function is updated based on the adjusted weights for each dimension.

[0013] In one embodiment of this disclosure, the weights corresponding to each dimension are determined based on the probability of generating data for each dimension and a Gaussian sampling model, including:

[0014] The mean and variance of the probability of generating data for each dimension are determined based on the probability of generating data for each dimension.

[0015] A Gaussian distribution model is constructed based on the probability mean and probability variance of each dimension of data.

[0016] Gaussian sampling is performed on the Gaussian distribution model to obtain the weights corresponding to each dimension.

[0017] In one embodiment of this disclosure, a loss function corresponding to the large language model is constructed based on the probability of generating data in each dimension, including:

[0018] A loss function is constructed based on the weights corresponding to each dimension and the mean probability of generating data for each dimension.

[0019] In one embodiment of this disclosure, the method further includes:

[0020] If the loss function value does not converge, adjust the model parameters to reduce the differences between the multiple dimensions of the response output by the large language model.

[0021] In one embodiment of this disclosure,

[0022] The preference dataset includes prompt words, multiple responses to the prompt words, and multiple dimensions of data for each response. The preference dataset is constructed based on these multiple dimensions, including:

[0023] Based on the multi-dimensional data corresponding to each reply, multiple replies are divided into winning replies and losing replies;

[0024] By concatenating the multi-dimensional prompts with the victory and failure replies respectively, we obtain the reconstructed multiple failure and victory prompts.

[0025] In one embodiment of this disclosure, the method further includes:

[0026] The reconstructed failure and victory prompts are input into the large language model. The probability of generating data for each dimension is determined based on the probability of the victory prompt in the response of each dimension output by the large language model.

[0027] In one embodiment of this disclosure, the method further includes:

[0028] In response to the user entering a question at the large language model input terminal, a multi-dimensional and balanced response corresponding to the question is displayed.

[0029] According to another aspect of this disclosure, a large language model determination apparatus is provided, comprising:

[0030] Build modules are used to construct preference datasets across multiple dimensions during the data collection phase;

[0031] The training module is used in the training phase of the large language model to construct the loss function corresponding to the large language model based on the probability of generating data in each dimension, and to perform adaptive multi-objective training on the preference dataset until the loss function converges to obtain the target large language model.

[0032] In one embodiment of this disclosure, the apparatus further includes:

[0033] The first determining module is used to determine the weight of each dimension based on the probability of generating data for each dimension and the Gaussian sampling model when the loss function value corresponding to the loss function has not converged.

[0034] The adjustment module is used to adjust the weights corresponding to each dimension in order to reduce the differences between the multiple dimensions of data in the response output by the large language model.

[0035] The update module is used to update the loss function based on the adjusted weights corresponding to each dimension.

[0036] In one embodiment of this disclosure, the first determining module includes:

[0037] The determination unit is used to determine the mean probability and variance of generating data for each dimension based on the probability of generating data for each dimension.

[0038] The first building unit is used to construct the Gaussian distribution model generation space based on the probability mean and probability variance of generating data in each dimension.

[0039] The sampling unit is used to perform Gaussian weight sampling on the Gaussian distribution model generation space to obtain the weight corresponding to each dimension.

[0040] In one embodiment of this disclosure, the training module includes:

[0041] The second building unit is used to construct a loss function based on the weights corresponding to each dimension and the mean probability of generating data for each dimension, so that the differences between multiple dimensions of data are reduced as the loss function value gradually converges.

[0042] In one embodiment of this disclosure, the apparatus further includes:

[0043] The adjustment module is used to adjust the model parameters when the loss function value corresponding to the loss function has not converged, so as to reduce the differences between the multiple dimensions of data corresponding to the output of the large language model.

[0044] In one embodiment of this disclosure, the building module further includes:

[0045] The partitioning unit is used to divide multiple replies into winning replies and losing replies based on multiple dimensions of data corresponding to each reply;

[0046] The splicing single eye is used to splice multi-dimensional prompts with victory and failure replies respectively, resulting in multiple reconstructed failure and victory prompts.

[0047] In one embodiment of this disclosure, the apparatus further includes:

[0048] The second determining module is used to input the reconstructed multiple failure prompt words and victory prompt words into the large language model, and determine the probability of generating data for each dimension based on the probability of the victory prompt word in the response of each dimension output by the large language model.

[0049] In one embodiment of this disclosure, the apparatus further includes:

[0050] The display module is used to respond to the user's input of a question at the large language model input terminal and display a multi-dimensional and balanced response corresponding to the question.

[0051] According to another aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform any of the above-described large language model determination methods by executing the executable instructions.

[0052] According to another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements any of the model determination methods described above.

[0053] According to another aspect of this disclosure, a computer program product is provided, comprising a computer program or computer instructions, which are loaded and executed by a processor to enable a computer to implement any of the above-described large language model determination methods.

[0054] In this embodiment of the disclosure, during the data collection phase, a preference dataset is constructed for multiple dimensions. At the large language model training node, a loss function corresponding to the large language model is constructed based on the probability of generating data for each dimension. Adaptive multi-objective training is performed on the preference dataset until the loss function converges to obtain the target large language model. The obtained target large language model can improve the balance of responses across multiple dimensions.

[0055] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0056] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0057] Figure 1 The diagram shows a schematic of a large language model determination system structure according to an embodiment of this disclosure.

[0058] Figure 2 A flowchart of a method for determining a large language model is shown in an embodiment of this disclosure.

[0059] Figure 3 A flowchart of another large language model determination method in an embodiment of this disclosure is shown.

[0060] Figure 4 A flowchart of another method for determining a large language model is shown in an embodiment of this disclosure.

[0061] Figure 5 A flowchart of another method for determining a large language model is shown in this embodiment of the present disclosure.

[0062] Figure 6 A flowchart of another method for determining a large language model is shown in this embodiment of the present disclosure.

[0063] Figure 7 A schematic diagram of a large language model determination device is shown in an embodiment of this disclosure.

[0064] Figure 8 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0065] Figure 9 A schematic diagram of a computer-readable storage medium according to an embodiment of the present disclosure is shown. Detailed Implementation

[0066] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0067] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0068] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0069] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0070] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0071] To improve the balance of responses from large language models across multiple dimensions, several approaches have been proposed. One approach is the instruction control method, which controls each preference by inserting control markers into the prompts; however, this method has poor adaptability. Another approach is the multi-model ensemble method, which optimizes each preference dimension by training multiple reward models; however, this method is highly complex.

[0072] To address the aforementioned issues, this disclosure provides a method, apparatus, and computer program product for determining large language models.

[0073] It should be noted that, where there is no conflict, the embodiments of this disclosure and the technical features in the embodiments can be combined with each other.

[0074] The specific implementation methods of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.

[0075] Figure 1 A schematic diagram of a model determination system for large languages ​​is shown in an embodiment of this disclosure. This system can be applied to the model determination methods and apparatus in various embodiments of this disclosure.

[0076] like Figure 1As shown, the model determination system 10 may include a front-end display module 101 and a model processing module 102. In some embodiments, the front-end display module 101 and the model processing module 102 may be located on different devices. The front-end display module 101 may be a module on an electronic device with data acquisition capabilities. The model processing module 102 may be a module on an electronic device with data processing capabilities, such as a computer or server. The front-end display module 101 and the model processing module 102 may also be located on the same device, such as an input module and a processing module on a computer or mobile phone.

[0077] The front-end display module 101 and the model processing module 102 communicate via a network, which can be a wired network or a wireless network.

[0078] Optionally, the aforementioned wireless or wired networks use standard communication technologies and / or protocols. The network is typically the Internet, but can also be any network, including but not limited to Local Area Networks (LANs), Metropolitan Area Networks (MANs), Wide Area Networks (WANs), mobile, wired or wireless networks, private networks, or any combination of virtual private networks. In some embodiments, technologies and / or formats including Hyper Text Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network. Furthermore, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Networks (VPNs), and Internet Protocol Security (IPsec) can be used to encrypt all or some links. In other embodiments, custom and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0079] The following describes the case where the front-end display module 101 and the model processing module 102 are located on two different devices.

[0080] The front-end display module 101 can be located on a terminal device, which can be various electronic devices, including but not limited to smartphones, tablets, laptops, desktop computers, wearable devices, augmented reality devices, virtual reality devices, etc.

[0081] Optionally, the client for the application installed on different terminal devices can be the same, or the client for the same type of application based on different operating systems. Depending on the terminal platform, the specific form of the application client can also differ; for example, the application client can be a mobile client, a PC client, etc.

[0082] The model processing module 102 can be located on a server. The server can be a server that provides various services, such as a backend management server that supports the operation of the device by the user using the terminal device. The backend management server can analyze and process the received request data and feed the processing results back to the terminal device.

[0083] Optionally, the server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0084] Figure 2 A flowchart illustrating a method for determining a large language model according to an embodiment of this disclosure is shown. Figure 2 As shown, the method for determining the large language model in this embodiment of the disclosure may include:

[0085] S210, during the data collection phase, construct a preference dataset for multiple dimensions.

[0086] In some embodiments, the preference dataset may include prompts, replies, and multidimensional data for each reply across multiple dimensions.

[0087] Cue words can include information that the user inputs into the large language model. Cues can be keywords, phrases, or sentences used to guide the model to generate specific types or styles of text, perform specific tasks, or provide specific information.

[0088] In some embodiments, the response may include the large language model's reply to the input prompts. The response may be text of a specific type or style generated by the model, keywords, phrases, or sentences that perform a specific task or provide specific information.

[0089] In some embodiments, multi-dimensional data may include dimensions and dimension scores. Dimensions may include perspectives on evaluating responses; for example, multiple dimensions may be helpfulness, correctness, and instruction compliance. Dimension scores may be helpfulness scores, correctness scores, and instruction compliance scores.

[0090] In some embodiments, prompt words can be input into a large language model to obtain multiple responses, and then instructions can be sent to the large language model to enable it to evaluate the responses from multiple dimensions and obtain multi-dimensional data.

[0091] For example, a prompt could be: How many r in strawberry.

[0092] Inputting the above prompts into the large language model, the first response is: Counting: 1.s 2.t 3.r 4.a 5.w 6.b 7.e 8.r 9.r 10.y There are 3 “r” in the word “strawberry”. The second response is: there are 2 “r” in the word “strawberry”.

[0093] Then, the above prompts and their corresponding two responses are input into the large language model, with the following inputs: "Please score the helpfulness according to the scoring criteria," "Please score the correctness according to the scoring criteria," and "Please score the instruction compliance according to the scoring criteria." This guides the large language model to generate multi-dimensional data.

[0094] S220: During the training phase of the large language model, a loss function corresponding to the large language model is constructed based on the probability of generating data in each dimension. Adaptive multi-objective training is performed on the preference dataset until the loss function converges, thus obtaining the target large language model.

[0095] In some embodiments, the probability of generating data for each dimension can be determined based on a large language model.

[0096] In some embodiments, the loss function for constructing a large language model based on the probability of generating data in each dimension may include determining the loss function based on the mean and variance of the probability of generating data in each dimension.

[0097] In some embodiments, the loss function for constructing a large language model based on the probability of each dimension of data can use the generation metric as an implicit reward, enabling the large language model constructed based on the above loss function to inherently understand and align the meanings of different dimensions.

[0098] In some embodiments, the loss function is a function used to measure the difference between the prediction results of a large language model and the true values, and is used to optimize the large language model.

[0099] In some embodiments, constructing a loss function corresponding to the large language model based on the probability of generating data for each dimension enables the target large language model optimized based on the above loss function to align multiple dimensions.

[0100] In some embodiments, adaptive multi-objective training may include inputting prompts into a model during training to obtain the model's output responses and multiple dimensions of data corresponding to each response. Then, based on a loss function, the model's output is compared with the multiple responses and multiple dimensions of data corresponding to each response in the training samples. The model's output includes the model's output responses and the multiple dimensions of data corresponding to each response.

[0101] In this embodiment of the disclosure, during the data collection phase, a preference dataset is constructed for multiple dimensions. At the large language model training node, a loss function corresponding to the large language model is constructed based on the probability of generating data for each dimension. Adaptive multi-objective training is performed on the preference dataset until the loss function converges to obtain the target large language model. The obtained target large language model can improve the balance of responses across multiple dimensions.

[0102] Figure 3 A flowchart of another large language model determination method according to an embodiment of this disclosure is shown. Figure 3 As shown, the method for determining the large language model in this embodiment of the disclosure may include:

[0103] S310, if the loss function value corresponding to the loss function has not converged, determine the weight corresponding to each dimension based on the probability of generating data for each dimension and the Gaussian sampling model.

[0104] In some embodiments, loss function value convergence may include the difference between multiple consecutively acquired loss function values ​​being less than a preset threshold.

[0105] In some embodiments, S310 may include:

[0106] The mean and variance of the probability of generating data for each dimension are determined based on the probability of generating data for each dimension.

[0107] A Gaussian distribution model is constructed to generate a space based on the probability mean and probability variance of each dimension of data.

[0108] Gaussian weight sampling is performed on the Gaussian distribution model generation space to obtain the weights corresponding to each dimension.

[0109] In some embodiments, the Gaussian distribution of the probability can be determined after the probability mean and probability variance are determined.

[0110] In some embodiments, Gaussian sampling refers to the process of randomly drawing samples from a Gaussian distribution.

[0111] For example, Gaussian sampling can be performed on the Gaussian distribution model based on the Box-Muller transform, the Ziggurat algorithm, and the central limit theorem.

[0112] In some embodiments, after Gaussian sampling, the sampled data can be used as the weights for each dimension.

[0113] In some embodiments, after Gaussian sampling, the sampled data can be mathematically transformed and used as the weights for each dimension.

[0114] S320, adjust the weights corresponding to each dimension to reduce the differences between the multiple dimensions of the response output by the large language model.

[0115] In some embodiments, the weights of each dimension can be adjusted according to a preset period, or the weights of each dimension can be adjusted in real time based on user definitions.

[0116] In some embodiments, the dimensional data includes dimensional scores, and reducing the differences between each dimensional data may include reducing the differences between the dimensional scores of each dimensional data.

[0117] In some embodiments, the weights of each dimension can be adjusted based on the dimension data corresponding to each dimension. For example, the weights of each dimension can be adjusted based on the inverse proportional relationship and the dimension data corresponding to each dimension.

[0118] S330 updates the loss function based on the adjusted weights corresponding to each dimension.

[0119] In some embodiments, the loss function consists of the weights corresponding to each dimension and the mean probability of generating data for each dimension.

[0120] For example, the loss function can be:

[0121]

[0122] Where x is the prompt word, y a As the first reply, y b For the second response, m, h, c, and if represent different dimensions, πθ represents the large language model, and β is a temperature parameter used to scale the difference in implicit rewards between preferred responses. These are the prompts for different dimensions after the reconstruction of the first reply. Corresponding to the different dimensions of the prompts after the reconstruction of the second reply, α {i} These are weighting factors for different dimensions.

[0123] In some embodiments, in order to ensure that the constructed target large language model can obtain corresponding rewards while producing multi-dimensional balanced responses, this disclosure constructs an objective function to obtain the optimal solution. This disclosure sets the weight vector α = [α1, ..., α...]. k The weight vector is obtained through policy α ~ ρ, where, And α k >0. Then, the Pareto optimal solution is determined by solving a scalar optimization problem:

[0124]

[0125] in, Let y be the preference consistency function for the k-th dimension. w In response to the victory, y l This is a failed response. 'x' is a prompt word. k ρ represents the modified prompt word that focuses on the k-th target, ρ represents sampling, i.e., sampling a value from the distribution, and π represents the output of the target large language model.

[0126] To effectively optimize and balance multiple objectives, this disclosure employs dynamic weighted sampling. Unlike traditional methods that fuse scores from multiple dimensions into a single score, this disclosure uses a dynamic weighted summation of scores across different dimensions in its target large language model, achieving score balance across multiple dimensions and enabling adaptive objective optimization.

[0127]

[0128] In some embodiments, α is derived from the weighting strategy ρ. Output y for the target large language model w The probability is greater than y l r k For a scoring model, f represents the input x and dimension d, and the output is a rewrite of x with respect to dimension d.

[0129] In this embodiment of the disclosure, the mean and variance of each dimension k can be calculated based on the word generation probability in the target large language model. These data are used to parameterize a Gaussian distribution ρ. k Sampling weight α from it k , average value μ k and variance It can be:

[0130]

[0131] in, It is the probability of the t-th label in dimension k.

[0132] The formula for the log-likelihood of generating an output sequence y from a given output sequence x can be:

[0133]

[0134] Combining formulas (3), (4), and (5), we obtain the loss function of this disclosure, namely formula (1).

[0135] In some embodiments, since the loss function is constructed from the weights corresponding to each dimension and the probability mean of the data in each dimension, the loss function can be updated according to the adjusted weights corresponding to each dimension.

[0136] In this embodiment of the disclosure, the weight corresponding to each dimension is determined by the probability of generating data for each dimension, and the weight corresponding to each dimension is adjusted to reduce the difference between the multiple dimensions of data corresponding to the response output by the model. The loss function is updated based on the adjusted weights corresponding to each dimension, which enables the response output by the large language model trained based on the loss function to be multi-dimensionally balanced.

[0137] Figure 4 A flowchart illustrating another method for determining a large language model according to an embodiment of this disclosure is shown. Figure 4 As shown, the model determination method in this embodiment may include:

[0138] S410, During the data collection phase, a preference dataset is constructed for multiple dimensions;

[0139] S420 constructs a loss function based on the weights corresponding to each dimension and the mean probability of generating data for each dimension, so that the difference between data from multiple dimensions is reduced as the loss function value gradually converges.

[0140] S430 performs adaptive multi-objective training on the preference dataset until the loss function converges, resulting in a target large language model.

[0141] In some embodiments, the adjusted model parameters may include: model weights and biases, learning rate, regularization parameters, batch size, optimizer parameters, network structure parameters, and initialization parameters.

[0142] In this embodiment of the disclosure, when the loss function corresponding to the model does not reach the preset threshold, adjusting the model parameters can reduce the differences between the multiple dimensions of the model output response, making the model output response more balanced in multiple dimensions.

[0143] Figure 5A flowchart illustrating yet another method for determining a large language model according to an embodiment of this disclosure is shown. Figure 5 As shown, the method for determining the large language model in this embodiment of the disclosure may include:

[0144] S510 categorizes multiple replies into winning replies and losing replies based on multiple dimensions of data corresponding to each reply.

[0145] In some embodiments, responses can be determined as either successful or unsuccessful based on the magnitude of the dimensional score.

[0146] In some embodiments, since each response can correspond to multiple dimensions of scores, the scores of multiple dimensions can be weighted and averaged, and then sorted according to the size of the weighted average. The response corresponding to the weighted average at the top of the sorted sequence is determined as the winning response.

[0147] In some embodiments, the prompt words may include prompt words corresponding to multiple dimensions.

[0148] S520 concatenates the multi-dimensional prompts with the victory and failure replies respectively to obtain multiple reconstructed failure and victory prompts.

[0149] In some embodiments, the method of using target prompt words, target responses corresponding to target prompt words, and target dimension data corresponding to each target response as training samples can be the same as the method of generating training samples based on prompt words, multiple responses corresponding to prompt words, and multiple dimension data corresponding to each response.

[0150] In some embodiments, the constructed victory prompts can be: Please focus on helpfulness, score according to the scoring criteria, Counting: 1.s 2.t 3.r 4.a 5.w 6.b 7.e 8.r 9.r 10.y There are 3 "r" in this word "strawberry". How many points can this reply get?; Please focus on correctness, score according to the scoring criteria, Counting: 1.s 2.t 3.r 4.a 5.w 6.b 7.e 8.r 9.r 10.y There are 3 "r" in this word "strawberry". How many points can this reply get?; and Please focus on instruction compliance, score according to the scoring criteria, Counting: 1.s 2.t 3.r 4.a 5.w 6.b 7.e 8.r 9.r 10.y There are 3 "r" in this word. How many points can the word "strawberry" get? The constructed failure prompts can be: Please focus on helpfulness, score according to the scoring criteria, there are 2 "r" in this word "strawberry" how many points can this reply get?; Please focus on correctness, score according to the scoring criteria, there are 2 "r" in this word "strawberry" how many points can this reply get?; and Please focus on instruction compliance, score according to the scoring criteria, there are 2 "r" in this word "strawberry" how many points can this reply get?. The reconstructed prompts are input into the target large language model according to their dimensions, so that the target large language model outputs a reply with a higher score. For example, in the correctness dimension, you can use "Please focus on correctness, score according to the scoring criteria," Counting: 1.s 2.t 3.r 4.a 5.w 6.b 7.e 8.r 9.r 10.y There are 3 "r" in this word "strawberry" how many points can this reply get? and "Please focus on correctness, score according to the scoring criteria, there are 2 "r" in this word "strawberry" how many points can this reply get? The algorithm calculates the score for the response "are2“r” in the word "strawberry" and inputs the responses with higher scores into a large language model. It then determines the probability of the winning response appearing in the model's output. This probability is the probability of the correctness dimension.

[0151] In this embodiment of the disclosure, multiple replies are divided into victory replies and failure replies based on multiple dimension data corresponding to each reply. Then, the multi-dimensional prompt words are concatenated with the victory replies and failure replies respectively to obtain multiple reconstructed failure prompt words and victory prompt words. The above prompt words are input into the large language model to obtain the reply output by the large language model, thereby determining the probability of each dimension data, making the determined probability of the dimension data more accurate.

[0152] Figure 6 A flowchart illustrating yet another method for determining a large language model according to an embodiment of this disclosure is shown. Figure 6 As shown, the model determination method in this embodiment may include:

[0153] S610, during the data collection phase, construct a preference dataset for multiple dimensions;

[0154] During the training phase of the large language model, a loss function corresponding to the large language model is constructed based on the probability of generating data in each dimension. Adaptive multi-objective training is then performed on the preference dataset until the loss function converges, thus obtaining the target large language model.

[0155] S620: During the training phase of the large language model, a loss function corresponding to the large language model is constructed based on the probability of generating data in each dimension. Adaptive multi-objective training is performed on the preference dataset until the loss function converges, thus obtaining the target large language model.

[0156] The S630 responds to the user's input of a question at the large language model input terminal and displays a multi-dimensional and balanced response corresponding to the question.

[0157] In some embodiments, after the large language model is trained, a preference-based large language model can be installed in an electronic device. After the user inputs a question, the electronic device sends the input question to the input end of the preference-based large language model, which then processes the input question to obtain a multi-dimensional, balanced response. The electronic device then displays the response.

[0158] Based on the same inventive concept, this disclosure also provides a large language model determination device, as shown in the following embodiments. Since the principle by which this device solves the problem is similar to that of the above-described method embodiments, the implementation of this device embodiment can refer to the implementation of the above-described method embodiments, and repeated details will not be elaborated further.

[0159] Figure 7 This diagram illustrates a large language model determination device according to an embodiment of the present disclosure, such as... Figure 7 As shown, the large language model determination device 700 includes:

[0160] Module 710 is used to build preference datasets across multiple dimensions during the data collection phase;

[0161] Training module 720 is used in the training phase of the large language model to construct the loss function corresponding to the large language model based on the probability of generating data in each dimension, and to perform adaptive multi-objective training on the preference dataset until the loss function converges to obtain the target large language model.

[0162] In one embodiment of this disclosure, the apparatus further includes:

[0163] The first determining module is used to determine the weight of each dimension based on the probability of generating data for each dimension and the Gaussian sampling model when the loss function value corresponding to the loss function has not converged.

[0164] The adjustment module is used to adjust the weights corresponding to each dimension in order to reduce the differences between the multiple dimensions of data in the response output by the large language model.

[0165] The update module is used to update the loss function based on the adjusted weights corresponding to each dimension.

[0166] In one embodiment of this disclosure, the first determining module includes:

[0167] The determination unit is used to determine the mean probability and variance of generating data for each dimension based on the probability of generating data for each dimension.

[0168] The first building unit is used to construct the Gaussian distribution model generation space based on the probability mean and probability variance of generating data in each dimension.

[0169] The sampling unit is used to perform Gaussian weight sampling on the Gaussian distribution model generation space to obtain the weight corresponding to each dimension.

[0170] In one embodiment of this disclosure, the training module includes:

[0171] The second building unit is used to construct a loss function based on the weights corresponding to each dimension and the mean probability of generating data for each dimension, so that the differences between multiple dimensions of data are reduced as the loss function value gradually converges.

[0172] In one embodiment of this disclosure, the apparatus further includes:

[0173] The adjustment module is used to adjust the model parameters when the loss function value corresponding to the loss function has not converged, so as to reduce the differences between the multiple dimensions of data corresponding to the output of the large language model.

[0174] In one embodiment of this disclosure, the building module further includes:

[0175] The partitioning unit is used to divide multiple replies into winning replies and losing replies based on multiple dimensions of data corresponding to each reply;

[0176] The splicing single eye is used to splice multi-dimensional prompts with victory and failure replies respectively, resulting in multiple reconstructed failure and victory prompts.

[0177] In one embodiment of this disclosure, the apparatus further includes:

[0178] The second determining module is used to input the reconstructed multiple failure prompt words and victory prompt words into the large language model, and determine the probability of generating data for each dimension based on the probability of the victory prompt word in the response of each dimension output by the large language model.

[0179] In one embodiment of this disclosure, the apparatus further includes:

[0180] The display module is used to respond to the user's input of a question at the large language model input terminal and display a multi-dimensional and balanced response corresponding to the question.

[0181] In this embodiment of the disclosure, during the data collection phase, a preference dataset is constructed for multiple dimensions. At the large language model training node, a loss function corresponding to the large language model is constructed based on the probability of generating data for each dimension. Adaptive multi-objective training is performed on the preference dataset until the loss function converges to obtain the target large language model. The obtained target large language model can improve the balance of responses across multiple dimensions.

[0182] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0183] This disclosure provides an electronic device comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform any of the above-described display methods by executing the executable instructions.

[0184] For example, see below. Figure 8 To describe an electronic device 800 according to such an embodiment of the present disclosure. Figure 8 The electronic device 800 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0185] like Figure 8As shown, the electronic device 800 is manifested in the form of a general-purpose computing device. The components of the electronic device 800 may include, but are not limited to: at least one processing unit 810, at least one storage unit 820, and a bus 830 connecting different system components (including storage unit 820 and processing unit 810).

[0186] The storage unit stores program code, which can be executed by the processing unit 810, causing the processing unit 810 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 810 can perform the following steps of the above method embodiments:

[0187] During the data collection phase, preference datasets are constructed across multiple dimensions;

[0188] During the training phase of the large language model, a loss function corresponding to the large language model is constructed based on the probability of generating data in each dimension. Adaptive multi-objective training is then performed on the preference dataset until the loss function converges, thus obtaining the target large language model.

[0189] Storage unit 820 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 8201 and / or cache memory 8202, and may further include a read-only memory (ROM) 8203.

[0190] The storage unit 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0191] Bus 830 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0192] Electronic device 800 can also communicate with one or more external devices 840 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 800, and / or with any device that enables electronic device 800 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 850. Furthermore, electronic device 800 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 860. As shown, network adapter 860 communicates with other modules of electronic device 800 via bus 830. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0193] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0194] In the disclosed exemplary embodiments, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. Figure 9 This illustration shows a schematic diagram of a computer-readable storage medium according to an embodiment of the present disclosure, such as... Figure 9 As shown, the computer-readable storage medium 900 stores a program product capable of implementing the methods described above.

[0195] In some possible implementations, various aspects of this disclosure may also be implemented as a program product comprising program code that, when run on a terminal device, causes the terminal device to perform the steps described in the foregoing “Detailed Description” section of this specification according to various exemplary embodiments of this disclosure.

[0196] More specific examples of computer-readable storage media in this disclosure may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0197] In this disclosure, a computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device.

[0198] Optionally, the program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0199] In practical implementation, program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0200] This disclosure provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the display methods provided in various alternative embodiments of this disclosure.

[0201] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0202] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0203] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope of this disclosure is indicated by the appended claims.

Claims

1. A method for determining a large language model, characterized in that, include: During the data collection phase, preference datasets are constructed across multiple dimensions; During the training phase of the large language model, a loss function corresponding to the large language model is constructed based on the probability of generating data in each dimension. Adaptive multi-objective training is performed on the preference dataset until the loss function converges to obtain the target large language model. The method further includes: If the loss function value corresponding to the loss function does not converge, the weight of each dimension is determined based on the probability of generating data for each dimension and the Gaussian sampling model. Adjust the weights corresponding to each dimension to reduce the differences between the multiple dimensions of data corresponding to the output response of the large language model; The loss function is updated based on the adjusted weights corresponding to each dimension. The determination of the weights for each dimension based on the probability of generating data for each dimension and the Gaussian sampling model includes: Based on the probability of generating data for each dimension, determine the mean and variance of the probability of generating data for each dimension. A Gaussian distribution model generation space is constructed based on the probability mean and probability variance of each dimension of data. Gaussian weight sampling is performed on the Gaussian distribution model generation space to obtain the weights corresponding to each dimension.

2. The method according to claim 1, characterized in that, The loss function for constructing the large language model based on the probability of generating data in each dimension includes: The loss function is constructed based on the weights corresponding to each dimension and the mean probability of generating data for each dimension, so that the differences between multiple dimensions of data decrease as the loss function value gradually converges.

3. The method according to claim 1, characterized in that, The method further includes: If the loss function value corresponding to the loss function does not converge, the model parameters are adjusted to reduce the differences between the multiple dimensions of data corresponding to the response output by the large language model.

4. The method according to claim 1, characterized in that, The preference dataset includes prompt words, multiple responses corresponding to the prompt words, and multiple dimensions of data corresponding to each response. The construction of the preference dataset for these multiple dimensions includes: Based on the multi-dimensional data corresponding to each reply, multiple replies are divided into winning replies and losing replies; By concatenating the multi-dimensional prompt words with the victory and failure replies respectively, we obtain the reconstructed multiple failure and victory prompt words.

5. The method according to claim 4, characterized in that, The method further includes: The reconstructed multiple failure prompts and victory prompts are input into the large language model, and the probability of generating data for each dimension is determined based on the probability of the victory prompt in the response of each dimension output by the large language model.

6. The method according to claim 4, characterized in that, The method further includes: In response to a user entering a question at the input terminal of the large language model, a multi-dimensional and balanced response corresponding to the question is displayed.

7. A large language model determination device, characterized in that, include: Build modules are used to construct preference datasets across multiple dimensions during the data collection phase; The training module is used in the training phase of the large language model to construct the loss function corresponding to the large language model based on the probability of generating data in each dimension, and to perform adaptive multi-objective training on the preference dataset until the loss function converges to obtain the target large language model. Also includes: If the loss function value corresponding to the loss function does not converge, the weight of each dimension is determined based on the probability of generating data for each dimension and the Gaussian sampling model. Adjust the weights corresponding to each dimension to reduce the differences between the multiple dimensions of data corresponding to the output response of the large language model; The loss function is updated based on the adjusted weights corresponding to each dimension. The determination of the weights for each dimension based on the probability of generating data for each dimension and the Gaussian sampling model includes: Based on the probability of generating data for each dimension, determine the mean and variance of the probability of generating data for each dimension. A Gaussian distribution model generation space is constructed based on the probability mean and probability variance of each dimension of data. Gaussian weight sampling is performed on the Gaussian distribution model generation space to obtain the weights corresponding to each dimension.

8. A computer program product, said computer program product comprising a computer program or computer instructions, characterized in that, The computer program or the computer instructions are loaded and executed by the processor to enable the computer to implement the large language model determination method as described in any one of claims 1-6.

Images