Methods, apparatuses, devices, media, and computer programs for updating large-scale language models

Abstract

This invention provides update methods, apparatus, devices, media, and products that improve the inference capabilities of large-scale language models. [Solution] The method for updating a large-scale language model (LLM) includes: autonomously generating data using an initial LLM to obtain synthetic data corresponding to multiple capabilities of the initial LLM; constructing preference samples based on the synthetic data corresponding to the multiple capabilities; and training the initial LLM using the preference samples to obtain a target LLM.

Description

Technical Field

[0001] The present disclosure relates to the technical field of artificial intelligence, particularly to the technical fields such as large-scale models and deep learning, and specifically to a method, apparatus, device, medium and computer program for updating a large language model.

Background Art

[0002] With the development of technology, inference tasks can be performed using large language models (Large Language Model, LLM). In order to obtain better inference results, it is necessary to improve the inference capability of LLM.

Summary of the Invention

Means for Solving the Problem

[0003] The present disclosure provides a method, apparatus, device, medium and computer program for updating a large language model.

[0004] According to one aspect of the present disclosure, there is provided a method for updating a large language model, comprising: performing autonomous data generation using an initial LLM to obtain synthetic data corresponding to a plurality of capabilities of the initial LLM; constructing preference samples based on the synthetic data corresponding to the plurality of capabilities; and training the initial LLM using the preference samples to obtain a target LLM.

[0005] According to another aspect of the present disclosure, there is provided an apparatus for updating a large language model, comprising: a generation module configured to perform autonomous data generation using an initial LLM to obtain synthetic data corresponding to a plurality of capabilities of the initial LLM; a construction module configured to construct preference samples based on the synthetic data corresponding to the plurality of capabilities; and a training module configured to train the initial LLM using the preference samples to obtain a target LLM.

[0006] Another aspect of the present disclosure provides an electronic device comprising at least one processor and a memory communicated with the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in any one of the above aspects.

[0007] Another aspect of the present disclosure provides a non-temporary computer-readable storage medium in which computer instructions are stored, the computer instructions being used to cause the computer to perform the method described in any one of the above aspects.

[0008] According to another aspect of the present disclosure, a computer program product is provided which includes a computer program, which, when executed by a processor, implements the method described in any one of the above aspects.

[0009] According to embodiments of this disclosure, the inference capabilities of large-scale language models can be improved.

[0010] It should be understood that the contents described in this section are not intended to identify any important features of the embodiments of the Disclosure or to limit the scope of the Disclosure. Other features of the Disclosure are more readily apparent in the following specification. [Brief explanation of the drawing]

[0011] The drawings are provided to better understand this technical proposal and do not constitute a limitation of this disclosure. [Figure 1] This is a schematic diagram of the first embodiment of the present disclosure. [Figure 2] This is a schematic diagram according to a second embodiment of the present disclosure. [Figure 3]This is a schematic diagram of composite data corresponding to multiple capabilities according to the embodiments of this disclosure. [Figure 4] This is a schematic diagram of a third embodiment of the present disclosure. [Figure 5] This is a schematic diagram of a fourth embodiment of the present disclosure. [Figure 6] This is a schematic diagram of an electronic device for realizing a method for updating a large-scale language model according to the embodiments of this disclosure. [Modes for carrying out the invention]

[0012] Hereinafter, exemplary embodiments of the present disclosure will be described with reference to the drawings. These will include various details of the embodiments of the present disclosure, which are intended to aid understanding, but should be considered merely exemplary. Accordingly, those skilled in the art should be aware that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and brevity, descriptions of known functions and structures will be omitted in the following description.

[0013] In related technologies, the inference capabilities of LLMs can be improved through methods such as reinforcement learning, but there is room for improvement in their effectiveness.

[0014] Figure 1 is a schematic diagram of a first embodiment of the present disclosure. This embodiment provides a method for updating a large-scale language model. As shown in Figure 1, the method includes the following: In 101, autonomous data generation is performed using an initial LLM to obtain synthetic data corresponding to multiple capabilities of the initial LLM. In 102, preference samples are constructed based on the synthetic data corresponding to the multiple capabilities. In 103, the initial LLM is trained using the preference samples to obtain a target LLM.

[0015] Here, "initial LLM" refers to an LLM that requires improvement of reasoning abilities. Specifically, the reasoning domain can include mathematical reasoning, physical reasoning, or general question-answering reasoning, among others.

[0016] Autonomous data generation refers to generating data using the initial LLM itself, without requiring manual annotation or generation by additionally introduced generation devices.

[0017] These multiple abilities can specifically include self-questioning ability, self-answering ability, and self-correction ability. The composite data corresponding to each of the above abilities can be called problem data, reasoning trajectory data, and error correction data, respectively.

[0018] After obtaining the aforementioned synthetic data, preference samples can be constructed based on the synthetic data.

[0019] Preference samples are training data used to train an initial LLM and obtain a target LLM.

[0020] In related technologies, evolutionary methods based on reinforcement learning require manual acquisition of training data, such as experts annotating high-quality problem data. This is inefficient and increases data acquisition costs. Furthermore, it requires the introduction of an additional reward model, which is used to obtain reward signals and train the model based on those signals, thus increasing training costs.

[0021] In this embodiment, autonomous data generation is performed using an initial LLM, synthetic data is obtained, preference samples are constructed based on the synthetic data, the initial LLM is trained using the preference samples, and a target LLM is obtained. By using the initial LLM itself for data synthesis and training, there is no need to manually annotate the training data, and there is no need to introduce additional models such as reward models. This enables highly efficient and high-quality self-evolution of the LLM and improves the inference ability of the LLM. Furthermore, by obtaining synthetic data corresponding to multiple abilities and constructing preference samples based on the synthetic data of multiple abilities, the richness of the samples can be increased, further improving the inference ability of the LLM.

[0022] Figure 2 is a schematic diagram of a second embodiment of this disclosure. This embodiment provides a method for updating a large-scale language model.

[0023] In this embodiment, the multiple capabilities include self-questioning ability, self-answering ability, and self-correction ability. Based on the above capabilities, the corresponding synthetic data are problem data, reasoning trajectory data, and error correction data, respectively.

[0024] The whole process can be divided into an inner loop and an outer loop. In the inner loop, preference samples are obtained based on the autonomous data generation of the initial LLM, and in the outer loop, the initial LLM is trained using the preference samples to obtain the target LLM. The inner and outer loops can be executed once or multiple times.

[0025] As shown in Figure 2, the method includes the following: In step 201, an initial LLM is used to ask self-questions and obtain problem data. In step 202, the initial LLM is used to provide self-answers to the problem data and obtain inference trajectory data. In step 203, a target problem is determined from the problem data, and a target answer is determined from the answer data corresponding to the target problem. In step 204, the initial LLM is used to perform self-correction on the target answer and obtain error correction data. In step 205, preference samples are constructed based on the problem data, the inference trajectory data, and the error correction data. In step 206, the initial LLM is trained using the preference samples to obtain a target LLM.

[0026] Processes 201-205 mentioned above belong to the inner loop, while process 206 belongs to the outer loop.

[0027] The inner and outer loops can be executed multiple times, and in each iteration t, the inner loop is initialized with LLM (model parameter π). θ t (represented by) through preference sample D t The external loop obtains preference sample D t The initial LLM model parameter π θ tUpdated the target LLM (model parameter is π θ t+1 We obtain (represented by ).

[0028] Regarding the internal loop, prompt information allows the initial LLM to play different roles, and based on these different roles, it is possible to generate composite data corresponding to multiple capabilities.

[0029] Figure 3 is a schematic diagram of composite data corresponding to multiple capabilities according to the embodiments of this disclosure.

[0030] As shown in Figure 3, prompt allows the initial LLM to perform the roles of generator, solver, and refiner, respectively. When the initial LLM functions as a generator, it obtains m (positive integer) problem data through the generator. When the initial LLM functions as a solver, it obtains n (positive integer) inference trajectory data for each problem data, which include the steps to solve the problem and the final answer data through the solver. It also determines the target problem from the m problem data based on the answer data contained in the n inference trajectory data. When the initial LLM functions as a refiner, it corrects the target answer corresponding to the target problem through the refiner and obtains error correction data.

[0031] In this embodiment, by acquiring problem data, inference trajectory data, and error correction data as synthetic data, the self-questioning ability, self-answering ability, and self-correction ability of a large-scale language model can be comprehensively considered, and after training based on this synthetic data, the aforementioned abilities of the large-scale language model can be synchronously improved.

[0032] Specifically, the initial LLM can be input with first prompt information. This first prompt information includes role function information corresponding to the generator, such as "generate a challenging reasoning task." Based on this first prompt information, the initial LLM can generate m problem data.

[0033] After obtaining m pieces of problem data, second prompt information can be input into the initial LLM. The second prompt information includes role function information corresponding to the solver, for example, information such as "generate inference trajectory data for the following problem data". The initial LLM generates n pieces of inference trajectory data for each piece of problem data based on the second prompt information.

[0034] Thereafter, one or more target problems can be acquired from the m pieces of problem data based on the answer data included in the inference trajectory data.

[0035] In some embodiments, the inference trajectory data includes answer data corresponding to the problem data. Determining a target problem from the problem data includes: clustering the answer data to obtain answer clusters; determining a target indicator of the problem data based on probabilities of the answer clusters, wherein the target indicator is used for measuring the execution difficulty of the problem data; and determining a target problem from the problem data based on the target indicator.

[0036] Here, for each piece of problem data, after obtaining n pieces of inference trajectory data, a total of n pieces of answer data included in each inference trajectory data can be obtained. Thereafter, n pieces of answer data can be clustered into k (a positive integer) answer clusters based on equivalence of the answer data (for example, similarity between pairs of answer data). Each answer cluster is a set of answer data and includes one or more pieces of answer data.

[0037] For each piece of problem data, after obtaining k answer clusters, a target indicator can be calculated based on the probability of each answer cluster. The target indicator is used for measuring the execution difficulty of the problem data.

[0038] Expressed the above process by a formula, the generator π g t is m pieces of problem data Q t = {q1 t ,...,q mt The goal is to generate}. Among them, q i t (1≦i≦m) represents the i-th problem data in the t-th iteration.

[0039] Each problem data q i t Regarding the solver π, s t This consists of n independent inference trajectory data Y. i t ={y i,1 t ,...,y i,n t Generates}. Among them, y i,j t (1≦j≦n) is the problem data q i t This is the j-th inference trajectory data corresponding to [the specified value].

[0040] Each problem data q i t For this, clustering is performed on the response data contained in n inference trajectory data, and response cluster C is formed. i t ={c i,1 1 ,...,c i,n t} is obtained. Among them c i,j t (1≦j≦k) is the problem data q i t This is the j-th response cluster corresponding to [the previous state].

[0041] The target metric calculated based on the response clusters may specifically be normalized Shannon entropy. The calculation formula is as follows:

number

[0042] Here,

number

[0043] After obtaining the target indicators described above, problem data in which the target indicators fall within the range of a predetermined first threshold and second threshold can be designated as target problems.

[0044] In other words, it can be divided into three areas based on the target indicators mentioned above. The areas of study are:

number

number

number

[0045] Based on the aforementioned domains, the target problems will be problem data located in the adjacent developmental domain.

[0046] In this embodiment, based on target indicators, the difficulty level of problem data can be accurately identified, and the target problem can be accurately determined, thereby increasing feasibility.

[0047] In some embodiments, determining a target answer from the answer data corresponding to the target problem includes determining target trajectory data from the inference trajectory data corresponding to the target problem based on the probability of the answer cluster corresponding to the target problem.

[0048] Here, a predetermined number of response clusters can be selected in order of their probability, and the response data within the selected response clusters can be used as the target response.

[0049] Assuming the predetermined number is 2, the response cluster c has a high probability. i,j t The response data within (j=1,2) will be used as the target response.

[0050] In this embodiment, by determining the target answer based on the probability of the answer cluster, the target answer can be obtained with high efficiency, and processing efficiency can be improved.

[0051] After determining the target problem and target answer, refiner π r t Error correction data is obtained using this method.

[0052] In some embodiments, there are multiple target answers. Performing self-correction on the target answers and obtaining the error correction data includes correcting the multiple target answers based on a counterfactual correction algorithm to obtain multiple sets of corrected trajectory data, and using the multiple sets of corrected trajectory data as error correction data when the answer data corresponding to the multiple sets of corrected trajectory data matches.

[0053] Here, the refiner verifies the accuracy of the inference by logical convergence rather than statistical agreement, based on an independent counterfactual correction mechanism. Specifically, this may include the following:

[0054] Each target problem (i.e., problem data located in the proximal developmental zone) q i t Regarding this, the two response clusters c with the highest probability i,1 t,c i,2 t Obtain the refiner π. r t This re-solves the task under counterfactual assumptions. It assumes that the previously submitted response data is erroneous and forces the model to rethink and construct independent inference trajectory data, thereby obtaining corrected trajectory data corresponding to the two response clusters mentioned above.

number

number

[0055] In this embodiment, when the response data corresponding to the multiple sets of correction trajectory data matches, the multiple sets of correction trajectory data are used as error correction data. This allows for the acquisition of error correction data based on internal logical consistency rather than statistical consistency, and further, by acquiring preference samples, the reliability of the training data can be improved, thereby enhancing the performance of large-scale models.

[0056] After obtaining the problem data, inference trajectory data, and error correction data mentioned above, preference samples can be constructed based on these three types of data.

[0057] In other words, based on the problem data, the first sample D g t We constructed the second sample D based on the inference trajectory data. s t We construct the third sample D based on the error correction data. r t It is possible to construct the first sample D. g t , Sample 2 D s t and 3 Sample Dr t Combining these to create preference sample D t Therefore, the preference sample is the set of the first sample, the second sample, and the third sample.

[0058] In this embodiment, by obtaining preference samples based on the first, second, and third samples described above, the richness of the preference samples can be improved, and the inference ability of the LLM can be further enhanced.

[0059] Since the composite data includes the three types mentioned above, the following methods can be used to determine the corresponding positive samples for different types: For problem data, target problems (problem data within the adjacent developmental domain) are treated as positive samples, and all others as negative samples. For inference trajectory data, inference trajectory data where the answer data is the true answer mentioned above are treated as positive samples, and all others as negative samples. For error correction data, only positive samples are retained, and the positive samples are corrected trajectory data where the incorrect answer has been corrected to the true answer.

[0060] For example, the target answer includes a first answer and a second answer. The refiner obtains the first corrected trajectory data based on the first answer (e.g., 0) and the second corrected trajectory data based on the second answer (e.g., 1 / 2). Assuming that the answer data corresponding to the first and second corrected trajectory data are identical (e.g., both are 0), the true answer is 0, the positive sample in the error correction data is the second corrected trajectory data, and the positive sample in the inference trajectory data is the inference trajectory data corresponding to answer 0.

[0061] A preference sample includes synthetic data and a preference label, which is used to represent the degree of preference for the synthetic data. For example, a preference label is represented by 1 or 0, where 1 indicates that the corresponding synthetic data is a positive sample and 0 indicates that the corresponding synthetic data is a negative sample.

[0062] Taking the problem data as an example, if the first problem data is a positive sample, the preference label corresponding to the first problem data is 1, and the first sample D is determined based on the first problem data and its corresponding preference label. g t The second and third samples can be obtained in the same manner.

[0063] In this way, preference labels are generated based on predetermined rules, and preference labels corresponding to synthetic data can be automatically determined, eliminating the need for manual annotation or determination based on additional models such as reward models, thereby reducing computational costs and improving processing efficiency.

[0064] Furthermore, when combined with the explanation described later, the KTO (Kahneman-Tversky optimization) training method can be adopted for the training process. In this training method, samples are triplets, meaning each triplet contains (prompt sample, response sample, and preference label). The prompt sample is the input to the LLM, the response sample is the output of the LLM, and the preference label corresponds to the response sample.

[0065] In other words, a preference sample can include a prompt sample, a response sample, and a preference label. The preference label is generated based on predetermined rules and is used to indicate whether the response sample is a positive or negative sample, for example, 1 represents a positive sample and 0 represents a negative sample.

[0066] The prompt sample is input to the initial LLM to obtain the output probability of the response sample. A loss function is constructed based on this output probability and the preference label corresponding to the response sample. The target LLM is obtained by adjusting the model parameters of the initial LLM based on the loss function.

[0067] Specifically, Sample 1 D g tIn this context, the prompt sample is prompt information corresponding to causing the LLM to act as a generator, the response sample is problem data generated by the LLM, and the preference label is a label of 1 or 0 determined based on the positive and negative samples in the problem data mentioned above.

[0068] Sample 2 D s t In this context, the prompt sample is prompt information corresponding to instructing the LLM to act as a solver, the response sample is inference trajectory data for the problem data generated by the LLM, and the preference label is a label of 1 or 0 determined based on the positive and negative samples in the above inference trajectory data.

[0069] Sample 3D r t In this case, the prompt sample is prompt information corresponding to having the LLM act as a refiner, the response sample is error correction data for the target answer to the target problem generated by the LLM, and includes only the correct samples in the error correction data, and the preference label is represented by 1.

[0070] After obtaining the preference samples mentioned above, the initial LLM is trained to obtain the target LLM.

[0071] In other words, the above prompt samples are input into the initial LLM to obtain the output probability of the response samples, a loss function is constructed based on the output probability and preference labels, and the model parameters of the initial LLM are adjusted based on the loss function to obtain the target LLM.

[0072] Referring to Figure 4 in combination with the above, the evolution (update) process of a large-scale language model can mainly consist of one or more iterative rounds. Each iterative round includes an inner loop and an outer loop. The inner loop mainly includes self-questioning, identification of nearest neighbor development areas, self-answering, and self-criticism (self-correction) through counterfactual logical verification, acquiring three types of synthetic data through the above process, and further constructing a preference dataset based on the synthetic data. The outer loop employs the KTO algorithm to perform optimization and enable the updating of model parameters. By repeating the above process, continuous evolution of the large-scale language model can be achieved.

[0073] Figure 5 is a schematic diagram of a third embodiment of the present disclosure. This embodiment provides a large-scale language model update device. As shown in Figure 5, the device 500 includes a generation module 501, a construction module 502, and a training module 503.

[0074] The generation module 501 is used to autonomously generate data using the initial LLM and to obtain synthetic data corresponding to multiple capabilities of the initial LLM. The construction module 502 is used to construct preference samples based on the synthetic data corresponding to the multiple capabilities. The training module 503 is used to train the initial LLM using the preference samples and to obtain a target LLM.

[0075] In this embodiment, data synthesis and training can be performed using the initial LLM itself. This eliminates the need for manual annotation of training data and the need to introduce additional models such as reward models. This enables efficient and high-quality self-evolution of the LLM and improves its inference capabilities. Furthermore, by acquiring synthetic data corresponding to multiple capabilities and constructing preference samples based on the synthetic data of multiple capabilities, the richness of the samples can be improved, further enhancing the inference capabilities of the LLM.

[0076] In some embodiments, the composite data corresponding to the multiple capabilities includes problem data, inference trajectory data, and error correction data.

[0077] The generation module 501 further performs self-questioning using the initial LLM, acquires the problem data, performs self-answering to the problem data using the initial LLM to acquire the inference trajectory data, determines the target problem from the problem data, determines the target answer from the answer data corresponding to the target problem, performs self-correction on the target answer using the initial LLM, and acquires the error correction data.

[0078] In this embodiment, by acquiring problem data, inference trajectory data, and error correction data as synthetic data, the self-questioning ability, self-answering ability, and self-correction ability of a large-scale language model can be comprehensively considered. Furthermore, after training based on this synthetic data, the above abilities of the large-scale language model can be improved synchronously.

[0079] In some embodiments, the inference trajectory data includes answer data corresponding to the problem data.

[0080] The generation module 501 further performs clustering on the response data to obtain response clusters, determines a target index for the problem data which is used to measure the difficulty of executing the problem data based on the probability of the response clusters, and determines a target problem from the problem data based on the target index.

[0081] In this embodiment, feasibility can be improved by accurately identifying the difficulty level of problem data based on target indicators and further accurately determining the target problem.

[0082] In some embodiments, the generation module 501 further clusters the response data based on the similarity between the response data to obtain response clusters containing at least one response data.

[0083] In this way, by clustering highly similar response data into the same response cluster, response clusters can be obtained efficiently and simply.

[0084] In some embodiments, the target index is a normalized Shannon entropy determined based on the probability, the probability being determined based on the quantity of response data included in the response cluster.

[0085] The generation module 501 further determines problem data in which the target indicators are located within predetermined first and second thresholds as target problems.

[0086] Thus, by using problem data from the optimal learning domain as the target problem, it is advantageous to train the initial LLM using the target problem, improve the learning ability of the initial LLM, and obtain a target LLM with better performance.

[0087] In some embodiments, the generation module 501 further determines a target answer from the answer data corresponding to the target problem based on the probability of the answer cluster corresponding to the target problem.

[0088] In this embodiment, by determining the target answer based on the probability of the answer cluster, it is possible to efficiently obtain the target answer and improve processing efficiency.

[0089] In some embodiments, the generation module 501 further selects a predetermined number of response clusters in order of decreasing probability, and uses the response data within the selected response clusters as target responses.

[0090] In this way, by using high-probability response data as the target response, the effectiveness of the target response can be improved. Furthermore, after making corrections to the target response, the correction effect can be improved, thereby enhancing the inference accuracy of the updated LLM.

[0091] In some embodiments, there are multiple target responses.

[0092] The generation module 501 further corrects the multiple target answers based on a counterfactual correction algorithm and obtains multiple sets of corrected trajectory data. When the response data corresponding to the multiple sets of corrected trajectory data matches, the multiple sets of corrected trajectory data are converted into error correction data.

[0093] In this embodiment, when the response data corresponding to the multiple sets of correction trajectory data matches, the multiple sets of correction trajectory data are used as error correction data. This allows for the acquisition of error correction data based on internal logical consistency rather than statistical consistency, and further, by acquiring preference samples, the reliability of the training data can be improved, thereby enhancing the performance of large-scale models.

[0094] In some embodiments, the composite data corresponding to the multiple capabilities includes problem data, inference trajectory data, and error correction data.

[0095] The construction module 502 further constructs a first sample based on the problem data, a second sample based on the inference trajectory data, a third sample based on the error correction data, and combines the first sample, the second sample, and the third sample to form the preference sample.

[0096] In this embodiment, by obtaining preference samples based on the first, second, and third samples described above, the richness of the preference samples can be improved, and the inference ability of the LLM can be further enhanced.

[0097] In some embodiments, the construction module 502 further holds the target problem in the problem data as the positive sample in the first sample, the other problem data as the negative sample in the first sample, and / or the inference trajectory data corresponding to the true answer in the answer data as the positive sample in the second sample, the other answer data as the negative sample in the second sample, and / or the positive sample in the error correction data as the third sample, wherein the positive sample is corrected trajectory data corrected from an incorrect answer to the true answer.

[0098] In this way, positive and negative samples can be determined through the above rules, eliminating the need for manual annotation of positive and negative samples, reducing costs, and enabling accurate and efficient acquisition of positive and negative samples.

[0099] In some embodiments, the preference sample includes a prompt sample, a response sample, and a preference label. The preference label is generated based on predetermined rules and is used to indicate whether the response sample is a positive or negative sample.

[0100] The training module 503 further inputs the prompt samples into an initial large-scale language model, obtains the output probabilities of the response samples, constructs a loss function based on the output probabilities and the preference labels, and adjusts the model parameters of the initial large-scale language model based on the loss function to obtain the target large-scale language model.

[0101] In this way, model training is performed based on the above prompt samples, response samples, and preference labels. Since the preference labels are determined based on predetermined rules, there is no need to introduce additional reward models or the like to determine reference signals, thus saving resources and improving training efficiency.

[0102] To ensure understanding, identical or similar content in different embodiments of this disclosure may be referenced to one another.

[0103] To ensure clarity, the terms "First," "Second," etc., used in the embodiments of this disclosure are used solely for distinction and do not indicate relative importance or chronological order.

[0104] To make it easier to understand, if the order of steps related to the process is not particularly limited, then the chronological relationships between these steps are not limited.

[0105] In the proposed technology disclosed herein, the collection, storage, use, processing, transmission, provision, and disclosure of each type of information, including relevant user personal information, all comply with the provisions of relevant laws and regulations and do not violate public order and morals.

[0106] Based on embodiments of this disclosure, the disclosure further provides electronic devices, readable storage media, and computer program products.

[0107] Figure 6 is a schematic block diagram of an electronic device 600 that may be used to carry out embodiments of the present disclosure. The electronic device 600 represents various forms of digital computers, such as laptops, desktop computers, workbenches, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may further represent various forms of mobile devices, such as PDAs, mobile phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure as described and / or requested herein.

[0108] As shown in Figure 6, the device 600 includes an arithmetic means 601 that can perform various appropriate operations and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage means 606 into a random access memory (RAM) 603. The RAM 603 may store various programs and data necessary for the operation of the electronic device 600. The arithmetic means 601, ROM 602, and RAM 603 are connected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0109] Multiple components of the electronic device 600, including, for example, input means 606 such as a keyboard and mouse, output means 607 such as various types of displays and speakers, storage means 608 such as magnetic disks and optical disks, and communication means 609 such as a network card, modem, and wireless communication transceiver, are connected to the I / O interface 605. The communication means 609 enables the electronic device 600 to exchange information / data with other devices, for example, via computer networks such as the Internet and / or various telecommunication networks.

[0110] The arithmetic means 601 may be a variety of general-purpose and / or dedicated processing components having processing and computational capabilities. Some examples of the arithmetic means 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), a variety of dedicated artificial intelligence (AI) computing chips, a variety of computing units for executing machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The arithmetic means 601 performs the various methods and processes described above, for example, a method for updating a large language model. For example, in some embodiments, a method for updating a large language model may be implemented as a computer software program physically embedded in a machine-readable medium, such as a storage means 608. In some embodiments, part or all of the computer program can be loaded and / or installed into the electronic device 600 via a ROM 602 and / or a communication means 609. Once the computer program is loaded into the RAM 603 and executed by the arithmetic means 601, one or more steps of the method for updating a large language model described in this disclosure can be performed. Alternatively, in other embodiments, the computing means 601 may be configured in any other suitable way (e.g., via firmware) to perform a method for updating a large language model.

[0111] Various embodiments of the systems and technologies described herein may be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), dedicated integrated circuits (ASICs), dedicated standard products (ASSPs), system-on-chip systems (SOCs), load-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include being implemented in one or more computer programs. The one or more computer programs can be executed and / or interpreted on a programmable system including at least one programmable processor. The programmable processor is a dedicated or general-purpose programmable processor capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, at least one input device, and at least one output device.

[0112] Program code for carrying out the methods of this disclosure can be written using any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a dedicated computer, or another programmable data processing device so that when the program code is executed by the processor or controller, it performs the functions / operations specified in the flowchart and / or block diagrams. The program code may run entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a standalone package, or entirely on a remote machine or server.

[0113] In the context of this disclosure, a machine-readable medium is a tangible medium that contains or stores programs used by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. More specific examples of machine-readable storage media include one or more line-based electrical connections, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0114] To provide user interaction, the systems and technologies described herein may be implemented on a computer equipped with a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) and a keyboard and pointing device (e.g., a mouse or trackball) for providing input from the user to the computer. Other types of devices may also be used to provide user interaction. For example, the feedback provided to the user may be any form of sensing feedback (e.g., visual feedback, auditory feedback, or haptic feedback), and input from the user may be received in any form (including sound input, voice input, or haptic input).

[0115] The systems and technologies described herein can be implemented in computing systems including backend components (e.g., data servers), computing systems including middleware components (e.g., application servers), or computing systems including frontend components (e.g., client computers having a graphical user interface or a web browser, through which users can interact with embodiments of the systems and technologies described herein), or in computing systems including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by digital data communication (e.g., communication networks) in any form or medium. Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), and internetworks.

[0116] A computer system can include clients and servers. Clients and servers are generally geographically distant from each other and typically interact through a communication network. The client-server relationship arises from computer programs running on corresponding computers that have a client-server relationship with each other. A server may be a cloud server, also called a cloud computing server or cloud host, and is a host product within a cloud computing service system, designed to address the drawbacks of traditional physical hosts and VPS services ("Virtual Private Server," abbreviated as "VPS"), such as high management difficulty and limited scalability. Furthermore, a server may be a server in a distributed system or a server integrated with blockchain technology.

[0117] It should be understood that steps can be rearranged, added, or deleted using the various forms of flows described above. For example, each step described in this application may be performed in a parallel or sequential order, or in a different order, as long as the desired results of the proposed technology disclosed herein are achieved.

[0118] The specific embodiments described above do not constitute limitations on the scope of protection of this application. Those skilled in the art will understand that various modifications, combinations, partial combinations, and substitutions are possible in accordance with design requirements and other factors. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for updating large-scale language models, Autonomous data generation is performed using an initial large-scale language model, and synthetic data corresponding to multiple capabilities of the initial large-scale language model is obtained. Based on the composite data corresponding to the aforementioned multiple capabilities, a preference sample is constructed, The initial large-scale language model is trained using the aforementioned preference samples to obtain the target large-scale language model. Methods that include...

2. The method according to claim 1, The composite data corresponding to the aforementioned multiple capabilities includes problem data, inference trajectory data, and error correction data. Using the aforementioned initial large-scale language model to perform autonomous data generation and obtain synthetic data corresponding to multiple capabilities of the aforementioned initial large-scale language model is, Using the aforementioned initial large-scale language model, self-questions are asked to obtain the aforementioned problem data, Using the aforementioned initial large-scale language model, the model performs self-responses to the problem data and obtains the aforementioned inference trajectory data. The process involves determining a target problem from the aforementioned problem data, and determining a target answer from the corresponding answer data for the aforementioned target problem. Using the aforementioned initial large-scale language model, self-correction is performed on the target response to obtain the error correction data. Methods that include...

3. The method according to claim 2, The aforementioned inference trajectory data includes answer data corresponding to the aforementioned problem data, Determining the target problem from the aforementioned problem data is The aforementioned response data is subjected to clustering to obtain response clusters, Based on the probability of the aforementioned response clusters, a target indicator for the problem data, which is used to measure the difficulty of executing the problem data, is determined. Based on the aforementioned target indicators, a target problem is determined from the aforementioned problem data. Methods that include...

4. The method according to claim 3, Performing clustering on the aforementioned response data to obtain response clusters is, A method comprising clustering the response data based on the similarity between the response data to obtain a response cluster containing at least one response data.

5. The method according to claim 4, The aforementioned target index is a normalized Shannon entropy determined based on the aforementioned probability, and the aforementioned probability is determined based on the number of response data included in the response cluster. Determining a target problem from the problem data based on the aforementioned target indicators is: A method comprising determining problem data in which the target indicator falls within a predetermined range of a first threshold and a second threshold as a target problem.

6. The method according to claim 3, Determining the target answer from the response data corresponding to the aforementioned target problem is A method comprising determining a target answer from among the answer data corresponding to the target problem, based on the probability of the answer cluster corresponding to the target problem.

7. The method according to claim 6, Determining a target answer from the response data corresponding to the target problem based on the probability of the response cluster corresponding to the target problem is: A method comprising selecting a predetermined number of response clusters in order of their probability of being response clusters, and using the response data within the selected response clusters as the target response.

8. The method according to claim 2, There are multiple target responses, Performing self-correction on the aforementioned target answer and obtaining the aforementioned error correction data is, Based on a counterfactual correction algorithm, corrections are made to the aforementioned multiple target responses, and multiple sets of corrected trajectory data are obtained. When the response data corresponding to the multiple sets of correction trajectory data matches, the multiple sets of correction trajectory data are used as error correction data. Methods that include...

9. A method according to any one of claims 3 to 7, The composite data corresponding to the aforementioned multiple capabilities includes problem data, inference trajectory data, and error correction data. Constructing preference samples based on the composite data corresponding to the aforementioned multiple capabilities is, To construct a first sample based on the aforementioned problem data, To construct a second sample based on the aforementioned inference trajectory data, To construct a third sample based on the aforementioned error correction data, The first sample, the second sample, and the third sample are combined to form the preference sample. Methods that include...

10. The method according to claim 9, Constructing a first sample based on the aforementioned problem data includes making the target problem in the problem data a positive sample in the first sample and the other problem data a negative sample in the first sample. and / or, Constructing a second sample based on the aforementioned inference trajectory data includes making the inference trajectory data corresponding to the true answer in the response data a positive sample in the second sample, and the other response data a negative sample in the second sample. and / or, A method for constructing a third sample based on the error correction data, comprising holding a correct sample in the error correction data as the third sample, wherein the correct sample is corrected trajectory data corresponding to being corrected from an incorrect answer to a true answer.

11. A method according to any one of claims 1 to 8, The preference sample includes a prompt sample, a response sample, and a preference label, the preference label being generated based on a predetermined rule and used to indicate whether the response sample is a positive or negative sample. Training the initial large-scale language model using the aforementioned preference samples to obtain the target large-scale language model is: The prompt sample is input into an initial large-scale language model, and the output probability of the response sample is obtained. Constructing a loss function based on the output probability and the preference label, The model parameters of the initial large-scale language model are adjusted based on the loss function to obtain the target large-scale language model. Methods that include...

12. A device for updating large-scale language models, A generation module for autonomously generating data using an initial large-scale language model and obtaining synthetic data corresponding to multiple capabilities of the initial large-scale language model, A construction module for constructing preference samples based on synthetic data corresponding to the aforementioned multiple capabilities, A training module for training the initial large-scale language model using the aforementioned preference samples to obtain a target large-scale language model, and A device equipped with the following features.

13. It is an electronic device, At least one processor, The system comprises at least one processor and a memory that is communicated with by it, The memory stores instructions that can be executed by the at least one processor, and when the instructions are executed by the at least one processor, the electronic device causes the at least one processor to perform the method according to any one of claims 1 to 8.

14. A non-temporary computer-readable storage medium storing computer instructions, wherein the computer instructions cause a computer to perform the method described in any one of claims 1 to 8.

15. It is a computer program, The computer program, when executed by the processor, implements the method described in any one of claims 1 to 8. Computer program.

Images