Method, apparatus, device, medium and program product for processing data

Abstract

The embodiment of the invention relates to a data processing method and device, equipment, a medium and a program product. The method includes obtaining a raw dataset for training a machine learning model, the raw dataset including a first plurality of cues usable as inputs for the machine learning model. The method further includes dividing the first plurality of cues into a plurality of cue groups, the similarity between two cues from the same cue group being greater than the similarity between two cues from different cue groups. The method also includes determining a plurality of distances from a second plurality of cues in a cue group of the plurality of cue groups to a center point of the cue group. The method further includes selecting, based on the plurality of distances, a partial cue from the cue group for adjusting the machine learning model. Through the method, the representative prompt is selected from the plurality of prompts of the original data set to adjust the model, so that the knowledge range is effectively reduced, the use of computing resources is reduced, and the data processing efficiency and the user experience are improved.

Description

Technical Field

[0001] Embodiments of the present disclosure generally relate to the field of machine learning models, and more particularly to methods, devices, equipment, media, and program products for processing data. Background Art

[0002] In recent years, reinforcement learning from human feedback (RLHF) has played an important role in the success of large language models (LLMs). RLHF aims to align the model with human values ​​and preferences in the post-training stage through a large amount of human annotated feedback. To enhance this alignment, multiple advanced training techniques have been introduced, including direct preference optimization (DPO), sequence likelihood calibration (SLiC), and Kahneman-Tversky optimization (KTO).

[0003] The large language model after reinforcement learning has many significant advantages. It improves the accuracy and quality of answers. This is because the large language model at this time enhances the understanding of human intentions and meets user needs. In addition, it can also improve the diversity of generated texts to adapt to various types of users. Therefore, the large language model after reinforcement learning has shown high application value and potential in many fields. Summary of the invention

[0004] Embodiments of the present disclosure provide a method, apparatus, device, medium, and program product for processing data.

[0005] According to a first aspect of the present disclosure, a method for processing data is provided. The method includes obtaining an original data set for training a machine learning model, the original data set including a first plurality of prompts that can be used as inputs to the machine learning model. The method also includes dividing the first plurality of prompts into a plurality of prompt groups, wherein the similarity between two prompts from the same prompt group is greater than the similarity between two prompts from different prompt groups. The method also includes determining a plurality of distances from a second plurality of prompts in a prompt group of the plurality of prompt groups to a center point of the prompt group. The method also includes selecting, based on the plurality of distances, a portion of the prompts from the prompt group for use in adjusting the machine learning model.

[0006] In a second aspect of the present disclosure, a device for processing data is provided. The device includes a data set acquisition module, configured to acquire an original data set for training a machine learning model, the original data set including a first plurality of prompts that can be used as inputs of the machine learning model; a prompt group division module, configured to divide the first plurality of prompts into a plurality of prompt groups, wherein the similarity between two prompts from the same prompt group is greater than the similarity between two prompts from different prompt groups; a distance determination module, configured to determine a plurality of distances from a second plurality of prompts in a prompt group of the plurality of prompt groups to a center point of the prompt group; and a prompt selection module, configured to select a portion of prompts from the prompt group based on the plurality of distances for adjusting the machine learning model.

[0007] In a third aspect of the present disclosure, an electronic device is provided, comprising at least one processor; and a storage device for storing at least one program, wherein when the at least one program is executed by the at least one processor, the at least one processor implements the method according to the first aspect of the present disclosure.

[0008] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method according to the first aspect of the present disclosure is implemented.

[0009] In a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the method according to the first aspect of the present disclosure when executed by a processor.

[0010] It should be understood that the contents described in this content section are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present disclosure.

[0012] Figure 1 A schematic diagram illustrating an example environment in which the apparatus and / or methods of some embodiments of the present disclosure may be implemented;

[0013] Figure 2 A schematic diagram illustrating an example method for processing data according to some embodiments of the present disclosure;

[0014] Figure 3 A schematic diagram illustrating an example process for processing data according to some embodiments of the present disclosure;

[0015] Figure 4 A schematic diagram illustrating a method for processing a preference data item according to some embodiments of the present disclosure is illustrated;

[0016] Figure 5 A schematic diagram illustrating an example process for processing preference data items according to some embodiments of the present disclosure;

[0017] Figure 6 A schematic block diagram of an apparatus for processing data according to some embodiments of the present disclosure is illustrated;

[0018] Figure 7 A schematic block diagram of an example device suitable for implementing various embodiments of the present disclosure is illustrated.

[0019] In the various drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0020] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0021] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0022] In the process of reinforcement learning of large language models, in addition to referring to the previous training techniques, the alignment process can also be improved from the data perspective. This involves using techniques such as data selection, sampling, and reorganization to enhance various aspects of the original preference learning dataset, such as quality, diversity, and difficulty. However, current data optimization methods mainly focus on a single component of the preference learning data (instructions or responses), and lack a unified method for systematically optimizing the overall preference data.

[0023] A traditional approach to improving alignment from a data perspective is reinforcement signal-direct preference optimization (RS-DPO). This approach first generates diverse responses from a supervised fine-tuned model. Then, it guides a reward model to identify contrast pairs based on the reward distribution and applies DPO to optimize the model's alignment. Another approach is Curry-DPO, which utilizes multiple preference pairs for each prompt and processes these preference pairs through a process from simple to complex tasks. It systematically ranks these pairs based on the difference in response quality and iteratively trains the model to achieve better performance than standard DPO in various benchmarks. Although these two methods have improved model performance to a certain extent, they also have some shortcomings in optimizing preference data. For example, RS-DPO may ignore the comprehensiveness of the data in the process of contrast pair identification, and although Curry-DPO enhances the model's capabilities through curriculum learning, its ranking and training process may require additional computing resources and does not fully consider the overall optimization of preference data. Importantly, these methods themselves are not specific to their respective models and are not dynamically selected by the model. Moreover, these methods do not mention the relationship between prompts and responses.

[0024] At least to address the above and other potential problems, an embodiment of the present disclosure proposes a method for processing data. In this method, a computing device first obtains an original data set for training a machine learning model. The original data set includes a first plurality of prompts that can be used as input to the machine learning model. Next, the computing device may divide the first plurality of prompts into a plurality of prompt groups. In these prompt groups, the similarity between two prompts from the same prompt group is greater than the similarity between two prompts from different prompt groups. The computing device may further determine a plurality of distances from a second plurality of prompts in a prompt group in the plurality of prompt groups to a center point of the prompt group, and select a portion of the prompts from the prompt group based on the plurality of distances for adjusting the machine learning model.

[0025] Through this method, by selecting representative prompts from multiple prompts in the original data set to adjust the model, the breadth of knowledge is effectively reduced, the use of computing resources is reduced, and the data processing efficiency and user experience are improved. In addition, the depth of knowledge can be dynamically enhanced to increase the depth of knowledge and provide users with more useful information.

[0026] The embodiments of the present disclosure will be described in detail below in conjunction with the accompanying drawings. Figure 1An example environment in which the devices and / or methods of embodiments of the present disclosure may be implemented is shown. In environment 100, computing device 104 may train or adjust a machine learning model using raw data set 102, such as aligning a large language model with human preferences.

[0027] The computing device 104 includes, but is not limited to, a personal computer, a server computer, a handheld or laptop device, a mobile device (such as a mobile phone, a personal digital assistant (PDA), a media player, etc.), a multiprocessor system, consumer electronics, a minicomputer, a mainframe computer, a distributed computing environment including any of the above systems or devices, etc.

[0028] like Figure 1 As shown, the original data set 102 is a data set that can be used to train or adjust the preferences of the machine learning model. The original data set 102 may include multiple prompts as inputs to the machine learning model. Each of the multiple prompts may be input data of the machine learning model to be adjusted, such as a user's question. Additionally, in addition to the prompts as inputs, the original data set 102 may also include corresponding responses. Each of the multiple prompts may have multiple responses. For example, for the same query to be input into the machine learning model, multiple different answers may be stored.

[0029] The computing device 104 can obtain the original data set 102. In one example, the computing device 104 obtains the original data set 102 from other computing devices. For example, the original data set is obtained from a server storing training data through a network. In another example, the computing device 104 can store the original data set 102 in a local storage device in advance and obtain it from the local storage device when it is used. The above examples are only used to describe the present disclosure, but are not specific limitations of the present disclosure.

[0030] After acquiring the original data set 102, the computing device 104 performs breadth compression on the original data set 102. Breadth compression refers to selecting representative prompts from the original data set, rather than using all the prompts in the original data set to adjust the machine learning model.

[0031] In this process, the plurality of prompts included in the original data set 102 can be divided into a plurality of prompt groups 106. For example, a pre-trained machine learning model is used to obtain a quantitative representation of each of the plurality of prompts, such as an embedding vector. Then, a clustering method based on the embedding vector can be used to divide the plurality of prompts into the plurality of prompt groups 106. For example, a K-means clustering algorithm is used to divide the vectors into the plurality of prompt groups. For each prompt group, a plurality of distances 110 between the plurality of prompts in the prompt group and a center point of the prompt group can be calculated, such as calculating the Euclidean distance between each prompt and the center point.

[0032] Then, the computing device 104 can select some prompts 112 that are closer to the center point of the prompt group. These prompts that are closer to the center point can be considered as representative prompts in the prompt group. This effectively reduces the breadth of knowledge and focuses on representative prompts. Then, the part of the prompt 112 is used for training or adjusting the machine learning model 114. In addition, for other prompt groups in the multiple prompt groups, the above method can also be used to select representative part of the prompts corresponding to the prompt group.

[0033] Additionally, in addition to the breadth compression of the original data set, the selected part of the prompts can be further processed, for example, to dynamically enhance the knowledge depth. In order to enhance the knowledge depth of the preference learning data set, the low-rank adaptation of large language models (LoRA) can be used to fine-tune the machine learning model to be adjusted (e.g., the base model). Then the fine-tuned machine learning model is used to generate multiple responses to the prompt. For example, a jailbreak prompt can be used to generate more diverse responses. The jailbreak prompt can also be called a limit-breaking prompt, which refers to some instructions or words entered by the user, the purpose of which is to bypass the security policy, moral code or usage rules pre-set by the language model. Next, a high-quality response pair can be constructed by a predetermined model (such as a large language model as a judge (LLM-as-a-judge)) and a rejection sampling method. Then, based on the previous unified depth enhancement, the gradient features of each preference data item (prompt-response pair) are used to dynamically allocate depth. Additionally, random projection can be used to reduce the dimension of the LoRA gradient to accelerate the subsequent clustering process. This gradient-based data representation enables measuring the contribution of each response pair to model optimization, thereby achieving more efficient learning by dynamically adjusting the allocated knowledge depth resources.

[0034] In the above process, K-means clustering can be used for gradient features to evaluate the knowledge depth required for each prompt. For each cluster, the part (prompt-response pair) closest to the center point or centroid can be selected to allocate more deep learning resources. This method dynamically adjusts the number of response pairs for different samples based on gradient clustering and selection, thereby optimizing the allocation of knowledge depth.

[0035] Finally, the DPO operation can be performed on the dynamically allocated data after the above adjustment to adjust the parameters of the machine learning model. This process has significant advantages in the preference alignment and performance improvement of the model. The above process can also be called balanced preference optimization (BPO), which dynamically enhances the knowledge depth of each sample through hierarchical sampling.

[0036] Through this method, by selecting representative prompts from multiple prompts in the original data set to adjust the model, the breadth of knowledge is effectively reduced, the use of computing resources is reduced, and the data processing efficiency and user experience are improved. After further dynamic knowledge depth enhancement, the depth of knowledge can be increased to provide users with more useful information.

[0037] Combined with the above Figure 1 A schematic diagram of an example environment in which the devices and / or methods of some embodiments of the present disclosure may be implemented is described below. Figure 2 A schematic diagram depicting an example method 200 for processing data according to some embodiments of the present disclosure. Figure 2 The method shown can be Figure 1 Executed by computing device 104 or any suitable computing device.

[0038] like Figure 2 As shown, in the example method 200, at box 202, the computing device 104 obtains an original data set for training a machine learning model, and the original data set includes a first plurality of prompts that can be used as inputs to the machine learning model. In order to adjust the preferences of the machine learning model, a pre-generated original data set can be obtained, which includes various prompts for the machine learning model, and these prompts can be used to adjust the preferences of the machine learning model.

[0039] At block 204, computing device 104 divides the first plurality of prompts into a plurality of prompt groups, wherein the similarity between two prompts from the same prompt group is greater than the similarity between two prompts from different prompt groups. Since there are a large number of similar prompts in the first plurality of prompts, the first plurality of prompts may be divided first according to the similarity of the prompts.

[0040] In some embodiments, the computing device 104 determines multiple quantized representations for the first plurality of prompts. For example, a corresponding vector is generated for each prompt, so multiple corresponding vectors can be generated for the first plurality of prompts. The computing device 104 then divides the first plurality of prompts into multiple prompt groups by performing a clustering operation on the multiple quantized representations. In one example, the computing device performs a K-means clustering operation on the multiple quantized representations (multiple vectors) to divide the first plurality of prompts into multiple prompt groups. In another example, the computing device 104 performs a K-center clustering operation on the multiple quantized representations (multiple vectors) to divide the first plurality of prompts into multiple prompt groups. The above examples are only used to describe the present disclosure, and are not specific limitations of the present disclosure. Those skilled in the art may use any suitable method to divide the multiple prompts into multiple prompt groups.

[0041] At block 206, a plurality of distances from a second plurality of prompts in a prompt group in the plurality of prompt groups to a center point of the prompt group are determined. For ease of description, the plurality of distances may also be referred to as a first plurality of distances.

[0042] In some embodiments, when determining multiple distances from the second plurality of prompts to the center point of the prompt group, the computing device 104 needs to first determine a quantitative representation of the center point. For example, the computing device 104 can determine a group of quantitative representations corresponding to the prompt group from multiple quantitative representations. Then, the computing device determines the center point of the prompt group based on the group of quantitative representations. For example, the mean of the group of quantitative representations is determined as the center point. Alternatively, other suitable methods can be used to determine the center point, such as selecting a quantitative representation in the group of quantitative representations as the center point. Then, the computing device 104 can use each quantitative representation and the obtained quantitative representation of the center point to determine multiple distances from the second plurality of prompts to the center point of the prompt group. For example, the Euclidean distance between vectors is calculated.

[0043] At block 208, computing device 104 selects a portion of prompts from the prompt group based on the plurality of distances for use in adjusting the machine learning model. In order to reduce the breadth of knowledge, computing device 104 may utilize the above distances to select representative prompts.

[0044] In some embodiments, when selecting partial prompts from a prompt group using multiple distances, the computing device 104 may first sort the prompts in the prompt group according to the sizes of the multiple distances. For example, the prompt group is sorted in descending order of the distances from the prompt to the center point or in descending order, and then a predetermined number of smaller prompts are selected from the sorted prompt group as partial prompts. In one example, the predetermined number is determined based on the number of prompt groups and a predetermined selection ratio. In another example, the predetermined number is directly predetermined.

[0045] In some embodiments, when selecting a partial prompt from a prompt group based on multiple distances, it can be determined for each of the multiple distances whether it is less than a threshold distance. If the distance is less than the threshold distance, the prompt corresponding to the distance is added to the partial prompt. If the distance is greater than or equal to the threshold distance, the prompt corresponding to the distance is not added to the partial prompt.

[0046] Additionally, for each of the plurality of prompt groups, the above method can be used to determine a portion of prompts for training the machine learning model. In addition, after the original data set is compressed in width, in order to provide more content information for each prompt, dynamic depth enhancement can be further performed. This process will be described in conjunction with Figure 4 and Figure 5 Give a description.

[0047] Through this method, representative prompts are selected from multiple prompts in the original data set to adjust the model, which effectively reduces the breadth of knowledge and the use of computing resources, thereby improving data processing efficiency and user experience.

[0048] Combined with the above Figure 2 Describes some exemplary methods for processing data according to some embodiments of the present disclosure. Figure 3 An example process for processing data according to some embodiments of the present disclosure is described. Figure 3 The method shown can be Figure 1 Executed by computing device 104 or any suitable computing device.

[0049] In example 300, knowledge breadth compression 302 may be performed on an original data set 304 used to train or adjust the preferences of a machine learning model. The original data set 304 includes a plurality of prompts, such as X1, X2, . . . , X n , where n is an integer. In addition, each of the plurality of prompts in the original data set 304 may have a set of corresponding responses. For example, for prompt X1, there is a corresponding response and Each response is an answer corresponding to a prompt. Next, the original data set 304 can be input into the vector converter 306 for processing. In the vector converter 306, each prompt in the original data set 304 can be converted into a quantitative representation, such as an embedded vector. Then, the vector converter 306 performs a clustering operation on these prompts, such as a K-means clustering operation. Then, the multiple prompts in the original data set 304 are divided into multiple prompt groups. Next, the distance from the prompt in each prompt group to the center point of the corresponding prompt group can be further calculated. For each prompt group, some prompts can be selected based on the distance, for example, the first 10% of the prompts with the smallest distance in each prompt group are selected as representatives of the prompt group. Then, the selected prompts form a representative prompt set 308. For example, the representative prompt set 308 includes X1, X2, ..., Xm, where m is an integer less than n.

[0050] Next, the computing device 104 may obtain responses corresponding to each prompt in the representative data set 308 from the original data set, thereby forming an adjusted data set 310. The number of prompts in the adjusted data set 310 is reduced relative to the original data set, thereby compressing the original data set in breadth. Then, the adjusted data set 310 may be applied to the DPO 310 to optimize the machine learning model.

[0051] Through this method, representative prompts are selected from multiple prompts in the original data set to adjust the model, which effectively reduces the breadth of knowledge and the use of computing resources, thereby improving data processing efficiency and user experience.

[0052] Combined with the above Figure 3 A schematic diagram of an example process for processing data according to some embodiments of the present disclosure is described below. Figure 4 A schematic diagram describing a method for processing a preference data item according to some embodiments of the present disclosure; Figure 4 The method shown can be Figure 1 The computing device 104 or any suitable computing device in the embodiment is executed. For the convenience of description, the original data set may be included in the above Figure 2 The multiple responses corresponding to the prompts in the partial prompts described in are called the first multiple responses.

[0053] In the example method 400, at block 402, the computing device 104 may also obtain a second plurality of responses by providing the prompt to the machine learning model. To increase the depth of knowledge, multiple responses may be further generated by the machine learning model to be adjusted. For ease of description, the multiple responses may also be referred to as the second plurality of responses. Figure 2The hints in the hint group described in the following are each hint in the hint group or Figure 3 Each prompt in the representative prompt set 308 described in is processed using a machine learning model to generate a corresponding plurality of responses.

[0054] In some embodiments, the machine learning model to be trained or adjusted can be fine-tuned using a pre-selected data set. The fine-tuned machine learning model is then used to process each prompt to generate a corresponding second plurality of responses.

[0055] At box 404, computing device 104 determines multiple preference data items corresponding to the prompt based on the prompt, the first plurality of responses, and the second plurality of responses, wherein the preference data items include the prompt, the first response to the prompt, and the second response to the prompt, the first response being preferred over the second response.

[0056] In some embodiments, when determining a plurality of preference data items corresponding to a prompt, the computing device may input the prompt, the first plurality of responses, and the second plurality of responses into a preference determination model to generate a plurality of preference data items. For example, the prompt, the first plurality of responses, and the second plurality of responses are input into a predetermined large language model to determine a plurality of response pairs corresponding to each prompt, each response pair including a relatively good response and a relatively poor response for the prompt. The prompt and the response pair may form a preference data item.

[0057] At block 406, the computing device 104 adjusts the machine learning model based on the plurality of preference data items. After obtaining the plurality of preference data items, the computing device 104 may also adjust the number of preference data items corresponding to each prompt, thereby dynamically adjusting the content depth.

[0058] In some embodiments, the computing device 104 uses multiple preference data items to determine multiple gradient features of the loss function for the machine learning model corresponding to the multiple preference data items. For ease of description, the multiple gradient features may also be referred to as the first multiple gradient features. For example, the computing device 104 may first determine the loss function of the DPO corresponding to the model adjustment, and then substitute the prompts, good responses, and poor responses in each of the multiple preference data items into the loss function to further determine the gradient features of the loss function corresponding to the preference data item. Then, the computing device 104 may use the multiple gradient features to select a group of preference data items from the multiple preference data items. Then, the machine learning model is adjusted using the selected group of preference data items.

[0059] When selecting a set of preference data items, the computing device may divide the plurality of gradient features into at least one gradient feature group. For example, the computing device 104 may divide the first plurality of gradient features into at least one gradient feature group by performing a clustering operation on the first plurality of gradient features, such as by K-means clustering or K-center point clustering. When clustering a plurality of gradient features of a prompt, the size of K may be dynamically set, so that the acquired preference data items may be dynamically adjusted. In the gradient feature groups obtained after clustering, the similarity between two gradient features from the same gradient feature group is greater than the similarity between two gradient feature groups from different gradient feature groups.

[0060] The computing device 102 may further determine a second plurality of distances from a second plurality of gradient features in a gradient feature group to a center point of the gradient feature group. The center point may be calculated using the mean of the gradient features in the gradient feature group, or one of the gradient features may be selected as the center point. The computing device 104 then calculates the vector distances from each gradient feature to the center point. The computing device 104 may determine a portion of the gradient features from the gradient feature group using the calculated second plurality of distances corresponding to the gradient feature group.

[0061] In some embodiments, the computing device 104 uses the calculated second multiple distances to sort the gradient features in the gradient feature group. Then, the computing device 104 selects a predetermined number of gradient features from the sorted gradient feature group as partial gradient features. For example, 10% of the gradient features are selected from each gradient feature group, or a predetermined number of gradient features are selected from each gradient feature group. In some embodiments, the computing device 104 may compare the distance in the second multiple distances with a threshold distance. If the distance is less than the threshold distance, the gradient feature may be added to the partial gradient feature. If the distance is greater than or equal to the threshold distance, the gradient feature is not added to the partial gradient feature. The above examples are only used to describe the present disclosure and are not specific limitations of the present disclosure.

[0062] Next, the computing device 104 may determine the preference data item corresponding to the partial gradient feature as a data item in a set of preference data items. Because after determining the partial gradient feature, the preference data item of the gradient feature can be found. Then, the computing device 104 determines the obtained preference data item as a data item in a set of preference data items for training the machine learning model. For each prompt, the above-mentioned method can be used to process to obtain a set of preference data items corresponding to each prompt for training the machine learning model.

[0063] Then, the computing device 104 uses the multiple sets of preference data items corresponding to these prompts to train the machine learning model, for example, using the DPO method to train the machine learning model. Through this method, the depth of knowledge can be further dynamically enhanced, thereby increasing the depth of knowledge and providing more useful information to users.

[0064] Combined with the above Figure 4 A schematic diagram of a method for processing a preference data item according to some embodiments of the present disclosure is described below; Figure 5 Describe an example process for processing preference data items according to some embodiments of the present disclosure; Figure 5 The method shown can be Figure 1 Executed by computing device 104 or any suitable computing device.

[0065] Example 500 may be Figure 3 A further improvement of the example 300 in FIG. 5, reference numerals 502, 504, 506 and 508 correspond to the contents and Figure 3 The contents of the figure marks 302, 304, 306 and 308 correspond to the same contents and will not be described in detail here.

[0066] After obtaining the representative data set 508, the computing device 104 may also use the pre-acquired seed data 516 to fine-tune the machine learning model to be subjected to preference adjustment to generate a supervised fine-tuning (SFT) strategy 514 or a supervised fine-tuning model. The seed data 516 may be data corresponding to a certain field or aspect, and the machine learning model may be a general basic model, which may be applicable to a predetermined field after fine-tuning.

[0067] In order to increase the depth of knowledge, the prompts in the representative data set 508 can be input into the SFT strategy 514 to further generate multiple responses, and the responses corresponding to these prompts in the original data set 504 are combined to generate an enhanced response set 512. For example, the response corresponding to X1 in the original data set 504 is and The response generated by SFT policy 514 is arrive Thus, the response corresponding to X1 in the enhanced response set is arrive Then, the enhanced response set 512 can be processed by a model such as a large language model to generate multiple response pairs for each prompt, and each prompt and a response pair can constitute a preference data item. These preference data items constitute a preference data item set 518. Each response pair includes a good prompt and a bad prompt for the corresponding prompt, for example, for prompt X1, It's a good response. It is a poor response.

[0068] For each preference data item, a gradient feature of the corresponding loss function may be calculated. Then, the gradient features corresponding to the preference data items in the preference data item set 518 may be clustered. In this process, multiple gradient features of multiple preference data items corresponding to each prompt may be clustered to be divided into different gradient feature groups.

[0069] In this process, a gradient clustering operation can be performed for each prompt, so that a gradient-based cluster 520 can be obtained. Then, a gradient feature is selected based on the distance from the gradient feature in each gradient feature group to the center point of the group. For example, the top 10% of gradient features with the smallest distance are selected. Then, the preference data items corresponding to the selected gradient features are used to form a target data set 522. For example, the response pair for X1 in the target data set 522 is and Indicates that the response is adjusted to represent the data. m Its response pair is and Its response is enhanced.The target dataset 522 is then used to perform DPO 524 to further train the fine-tuned machine learning model to adjust its preferences.

[0070] Combined with the above Figure 4 and Figure 5 A scheme for performing dynamic knowledge depth enhancement on a partial prompt or representative data set after breadth compression of the original data is described. Alternatively, the original data set may be directly enhanced with dynamic knowledge depth in a manner similar to the above manner.

[0071] For example, the computing device 104 may obtain multiple responses corresponding to each prompt in the original data set, and for the convenience of description, the multiple responses may also be referred to as the third multiple responses. In addition, the computing device 104 may also obtain multiple responses by providing the prompt to the machine learning model, and for the convenience of description, the multiple responses may be referred to as the fourth multiple responses. In addition, the computing device 104 may also use the third multiple responses and the fourth multiple responses to the prompt to determine multiple preference data items corresponding to the prompt, wherein the preference data items include the prompt, the third response to the prompt, and the fourth response to the prompt, and the third response is preferred over the fourth response. In some embodiments, when determining multiple preference data items corresponding to the prompt, the computing device may input the prompt, the third multiple responses, and the fourth multiple responses into the preference determination model to generate the multiple preference data items.

[0072] The computing device 104 may then adjust the machine learning model based on the multiple preference data items determined from the original data set. In some embodiments, when adjusting the machine learning model based on the multiple preference data items, the computing device may use the multiple preference data items to determine multiple gradient features of the loss function for the machine learning model corresponding to the multiple preference data items. For ease of description, the multiple gradient features may also be referred to as the third multiple gradient features. The computing device 104 then selects a group of preference data items from the multiple preference data items based on the multiple gradient features. Thus, the computing device 104 may use the group of preference data items to adjust the machine learning model.

[0073] In some embodiments, when selecting a group of preference data items from the plurality of preference data items, the computing device divides the third plurality of gradient features into at least one gradient feature group, and the similarity between two gradient features from the same gradient feature group is greater than the similarity between two gradient feature groups from different gradient feature groups. The computing device 104 may further determine a plurality of distances from a fourth plurality of gradient features in a gradient feature group in at least one gradient feature group to a center point of the gradient feature group, and for ease of description, the plurality of distances may also be referred to as a third plurality of distances. Then, the computing device determines a portion of the gradient features from the gradient feature group based on the third plurality of distances corresponding to the gradient feature group. Then, the computing device 104 determines the preference data item corresponding to the portion of the gradient features as a data item in the above-mentioned group of preference data items.

[0074] As described above, when the plurality of gradient features are divided into at least one gradient feature group, the third plurality of gradient features may be divided into at least one gradient feature group by performing a clustering operation on the third plurality of gradient features.

[0075] In some embodiments, when determining partial gradient features from the gradient feature group according to a third plurality of distances corresponding to the gradient feature group, the computing device 104 may sort the gradient features in the gradient feature group using the third plurality of distances. Then, the computing device 104 selects a predetermined number of gradient features from the sorted gradient feature group as partial gradient features. Figure 4 and Figure 5 The rest of the description also applies to the scenario of performing dynamic depth augmentation directly on the original dataset.

[0076] Through this method, the computing device can directly and dynamically enhance the knowledge depth of the original data set, thereby increasing the depth of knowledge and providing users with more useful information.

[0077] like Figure 6As shown, the device 600 includes a data set acquisition module 602, which is configured to acquire an original data set for training a machine learning model, wherein the original data set includes a first plurality of prompts that can be used as inputs to the machine learning model; a prompt group division module 604, which is configured to divide the first plurality of prompts into a plurality of prompt groups, wherein the similarity between two prompts from the same prompt group is greater than the similarity between two prompts from different prompt groups; a distance determination module 606, which is configured to determine a plurality of distances from a second plurality of prompts in a prompt group among the plurality of prompt groups to a center point of the prompt group; and a prompt selection module 608, which is configured to select a portion of prompts from the prompt group based on the plurality of distances for adjusting the machine learning model.

[0078] In some embodiments, the prompt group division module 604 includes: a quantitative representation determination module configured to determine multiple quantitative representations for the first multiple prompts; and a first clustering module configured to divide the first multiple prompts into multiple prompt groups by clustering the multiple quantitative representations.

[0079] In some embodiments, the clustering operation is a K-means clustering operation.

[0080] In some embodiments, the distance determination module 606 includes: a group of quantitative representation determination modules, configured to determine a group of quantitative representations corresponding to the prompt group from multiple quantitative representations; a center point determination module, configured to determine the center point of the prompt group based on the group of quantitative representations; and a first plurality of distance determination modules, configured to determine multiple distances from the second plurality of prompts to the center point of the prompt group based on the group of quantitative representations and the center point.

[0081] In some embodiments, the prompt selection module 608 includes: a sorting module configured to sort the prompts in the prompt group based on a plurality of distances; and a first selection module configured to select a predetermined number of prompts from the sorted prompt group as partial prompts.

[0082] In some embodiments, the prompt selection module 608 includes a distance determination module configured to determine whether a distance among multiple distances is less than a threshold distance; and an adding module configured to add a prompt corresponding to the distance to the partial prompt in response to the distance being less than the threshold distance.

[0083] In some embodiments, the original data set also includes a first plurality of responses corresponding to the prompts in the partial prompts, and the device 600 also includes: a second plurality of responses acquisition module, configured to obtain a second plurality of responses by providing the prompts to the machine learning model; and a plurality of preference data item determination module, configured to determine a plurality of preference data items corresponding to the prompts based on the prompts, the first plurality of responses, and the second plurality of responses, wherein the preference data items include the prompts, a first response to the prompts, and a second response to the prompts, the first response being preferred to the second response; and a model adjustment module, configured to adjust the machine learning model based on the plurality of preference data items.

[0084] In some embodiments, the plurality of preference data items determination module includes: a model generation module configured to input the prompt, the first plurality of responses, and the second plurality of responses into a preference determination model to generate the plurality of preference data items.

[0085] In some embodiments, the model adjustment module includes: a gradient feature determination module, configured to determine multiple gradient features of a loss function for a machine learning model corresponding to multiple preference data items based on multiple preference data items; a preference data item selection module, configured to select a group of preference data items from multiple preference data items based on multiple gradient features; and a first adjustment module, configured to adjust the machine learning model based on a group of preference data items.

[0086] In some embodiments, the multiple gradient features are a first multiple gradient features, the multiple distances are a first multiple distances, and the preference data item selection module includes: a feature gradient division module configured to divide the multiple gradient features into at least one gradient feature group, and the similarity between two gradient features from the same gradient feature group is greater than the similarity between two gradient feature groups from different gradient feature groups; a second multiple distance determination module configured to determine a second multiple distances from a second multiple gradient feature in a gradient feature group in at least one gradient feature group to a center point of the gradient feature group; and a partial gradient feature determination module configured to determine a partial gradient feature from the gradient feature group based on the second multiple distances corresponding to the gradient feature group; and a data item determination module configured to determine the preference data item corresponding to the partial gradient feature as a data item in a group of preference data items.

[0087] In some embodiments, the feature gradient division module includes: a second clustering module configured to divide the first plurality of gradient features into at least one gradient feature group by performing a clustering operation on the first plurality of gradient features.

[0088] In some embodiments, the partial gradient feature determination module includes: a second sorting module configured to sort the gradient features in the gradient feature group based on a second plurality of distances; and a gradient feature selection module configured to select a predetermined number of gradient features from the sorted gradient feature group as partial gradient features.

[0089] Figure 7 A schematic block diagram of an example device 700 that may be used to implement embodiments of the present disclosure is shown. Figure 1 The computing device 104 in the embodiment of the present invention can be implemented by using the device 700. As shown in the figure, the device 700 includes a processing unit (CPU) 701, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 702 or computer program instructions loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0090] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage page 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0091] The various processes and processing described above, such as methods 200 and 400, may be performed by the processing unit 701. For example, in some embodiments, the methods 200 and 400 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the CPU 701, one or more actions of the example methods 200 and 400 described above may be performed.

[0092] The present disclosure may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.

[0093] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (non-exhaustive list) of computer readable storage medium include: portable computer disk, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding device, such as a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination of the above. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (e.g., a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0094] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0095] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0096] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0097] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0098] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0099] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0100] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for processing data, comprising: Obtaining a raw data set for training a machine learning model, the raw data set comprising a first plurality of prompts that can be used as input to the machine learning model; dividing the first plurality of prompts into a plurality of prompt groups, a similarity between two prompts from the same prompt group being greater than a similarity between two prompts from different prompt groups; determining a plurality of distances from a second plurality of prompts in a prompt group in the plurality of prompt groups to a center point of the prompt group; as well as Based on the multiple distances, a portion of the prompts are selected from the prompt group for adjusting the machine learning model.

2. The method of claim 1, wherein dividing the first plurality of prompts into a plurality of prompt groups comprises: determining a plurality of quantized representations for the first plurality of cues; as well as The first plurality of prompts are divided into a plurality of prompt groups by performing a clustering operation on the plurality of quantized representations. The method according to claim 2 , wherein the clustering operation is a K-means clustering operation.

4. The method of claim 2, wherein determining a plurality of distances from a second plurality of cues in a cue group in the plurality of cue groups to a center point of the cue group comprises: determining a set of quantized representations from among the plurality of quantized representations that corresponds to the set of prompts; determining a center point of the prompt group based on the set of quantitative representations; as well as Based on the set of quantitative representations and the center point, a plurality of distances of the second plurality of cues to the center point of the set of cues are determined.

5. The method of claim 1 , wherein selecting a portion of prompts from the prompt group for use in adjusting the machine learning model based on the plurality of distances comprises: sorting the prompts in the prompt group based on the plurality of distances; as well as A predetermined number of prompts are selected from the sorted group of prompts as the partial prompts.

6. The method of claim 1, wherein selecting a portion of prompts from the prompt group for use in adjusting the machine learning model based on the plurality of distances comprises: determining whether a distance among the plurality of distances is less than a threshold distance; as well as In response to the distance being less than the threshold distance, a hint corresponding to the distance is added to the partial hint.

7. The method of claim 1, wherein the original data set further includes a first plurality of responses corresponding to prompts in the portion of prompts, and the method further comprises: obtaining a second plurality of responses by providing the prompt to the machine learning model; as well as determining, based on the prompt, the first plurality of responses, and the second plurality of responses, a plurality of preference data items corresponding to the prompt, wherein the preference data items include the prompt, a first response to the prompt, and a second response to the prompt, the first response being preferred to the second response; as well as Based on the plurality of preference data items, the machine learning model is adjusted.

8. The method of claim 7, wherein determining a plurality of preference data items corresponding to the prompt comprises: The prompt, the first plurality of responses, and the second plurality of responses are input into a preference determination model to generate the plurality of preference data items.

9. The method of claim 7, wherein based on the plurality of preference data items, adjusting the machine learning model comprises: Based on the plurality of preference data items, determining a plurality of gradient features of a loss function for the machine learning model corresponding to the plurality of preference data items; selecting a set of preference data items from the plurality of preference data items based on the plurality of gradient features; as well as Based on the set of preference data items, the machine learning model is adjusted.

10. The method of claim 9, wherein the plurality of gradient features is a first plurality of gradient features, the plurality of distances is a first plurality of distances, and selecting a set of preference data items from the plurality of preference data items comprises: Dividing the plurality of gradient features into at least one gradient feature group, wherein a similarity between two gradient features from the same gradient feature group is greater than a similarity between two gradient feature groups from different gradient feature groups; determining a second plurality of distances from a second plurality of gradient features in the at least one gradient feature group to a center point of the gradient feature group; and determining a partial gradient feature from the gradient feature group based on a second plurality of distances corresponding to the gradient feature group; and The preference data item corresponding to the partial gradient feature is determined as a data item in the set of preference data items.

11. The method according to claim 10, wherein dividing the plurality of gradient features into at least one gradient feature group comprises: The first plurality of gradient features are divided into at least one gradient feature group by performing a clustering operation on the first plurality of gradient features.

12. The method of claim 10, wherein determining a portion of the gradient features from the gradient feature group based on a second plurality of distances corresponding to the gradient feature group comprises: sorting the gradient features in the gradient feature group based on the second plurality of distances; as well as A predetermined number of gradient features are selected from the sorted group of gradient features as the partial gradient features.

13. An apparatus for processing data, comprising: a data set acquisition module configured to acquire an original data set for training a machine learning model, the original data set comprising a first plurality of prompts that can be used as inputs to the machine learning model; a prompt group division module configured to divide the first plurality of prompts into a plurality of prompt groups, wherein a similarity between two prompts from the same prompt group is greater than a similarity between two prompts from different prompt groups; A distance determination module configured to determine a plurality of distances from a second plurality of prompts in a prompt group of the plurality of prompt groups to a center point of the prompt group; as well as A prompt selection module is configured to select some prompts from the prompt group based on the multiple distances for adjusting the machine learning model.

14. An electronic device comprising: at least one processor; as well as A storage device, used for storing at least one program, when the at least one program is executed by the at least one processor, so that the at least one processor implements the method according to any one of claims 1-12.

15. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 12 when executed by a processor.

16. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 12.

Images