Data screening method, device and equipment and computer storage medium

Abstract

The embodiment of the invention provides a data screening method and device, equipment and a computer storage medium. The data screening method comprises the steps that a plurality of preference data samples are obtained, and the preference data samples comprise input information and positive output samples and negative output samples under at least one preference dimension parameter; consensus index values of the multiple preference data samples under different preference dimension parameters are determined, the preference data samples correspond to original preference dimension parameters, and the consensus index values are used for identifying the recognition degree of the preference data samples under non-original preference dimension parameters; same-strategy index values corresponding to the multiple preference data samples are determined respectively, wherein the same-strategy index values are used for identifying the adaptation degree between the preference data samples and the to-be-trained language model; and based on the consensus index values and the same strategy index values corresponding to the plurality of preference data samples, screening out a target training sample corresponding to the model fine tuning task from the plurality of preference data samples.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a data screening method and device, equipment and a computer storage medium. BACKGROUND

[0002] In the application scenarios of artificial intelligence, natural language processing (NLP) and deep learning, a language model usually refers to a neural network model used for processing, understanding or generating text. The language model often takes text as input, learns the statistical rules, semantic representation and context relationship of language through network structure, and generates text information meeting specific needs.

[0003] However, the inference effect of the language model depends largely on the quality of the training data used for model fine-tuning operation. At present, the training data often comes from real data generated in text processing operation, and the number of real data is relatively large and the noise is relatively large. Therefore, how to efficiently and cost-effectively screen out training data valuable for model fine-tuning operation from a large amount of noisy data source is a technical problem to be solved in the application scenario of the language model. SUMMARY

[0004] The embodiments of the present application provide a data screening method, device, equipment and computer storage medium, which can efficiently and cost-effectively screen out training data valuable for model fine-tuning operation, thereby facilitating to improve the efficiency and effect of preference optimization of the language model.

[0005] In a first aspect, the embodiments of the present application provide a data screening method, comprising: obtaining a plurality of preference data samples, wherein the preference data samples comprise input information, positive output samples and negative output samples under at least one preference dimension parameter; determining consensus index values of the plurality of preference data samples under different preference dimension parameters respectively, wherein the preference data samples correspond to original preference dimension parameters, and the consensus index values are used to identify the recognition degree of the preference data samples under non-original preference dimension parameters; determining same-strategy index values corresponding to the plurality of preference data samples respectively, wherein the same-strategy index values are used to identify the adaptation degree between the preference data samples and a language model to be trained; screening target training samples corresponding to the model fine-tuning task from the plurality of preference data samples based on the consensus index values and the same-strategy index values corresponding to the plurality of preference data samples respectively.

[0006] In a second aspect, the embodiments of the present application provide a data screening device, comprising: a first obtaining module, configured to obtain a plurality of preference data samples, wherein the preference data samples comprise input information, positive output samples and negative output samples under at least one preference dimension parameter; a first determining module, configured to respectively determine a consensus index value of the plurality of preference data samples under different preference dimension parameters, wherein the preference data samples correspond to original preference dimension parameters, and the consensus index value is used to identify an approval degree of the preference data samples under non-original preference dimension parameters; The first determining module is further configured to respectively determine a same-strategy index value corresponding to the plurality of preference data samples, and the same-strategy index value is used to identify an adaptation degree between the preference data samples and the language model to be trained. a first processing module, configured to filter out target training samples corresponding to the model fine-tuning task from the plurality of preference data samples based on the consensus index value and the same-strategy index value corresponding to the plurality of preference data samples respectively.

[0007] In a third aspect, an electronic device is provided, and the electronic device comprises a memory and a processor. The memory is configured to store one or more computer instructions. When the one or more computer instructions are executed by the processor, the method in the first aspect is implemented.

[0008] In a fourth aspect, a computer storage medium is provided, and the computer storage medium is configured to store a computer program. When the computer program is executed by a computer, the method in the first aspect is implemented.

[0009] In a fifth aspect, a computer program product is provided, and the computer program product comprises a computer readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors perform the steps in the method in the first aspect.

[0010] The data screening method, device and equipment and computer storage medium provided by the embodiment can obtain multiple preference data samples, determine consensus index values of the multiple preference data samples under different preference dimension parameters and same strategy index values corresponding to the multiple preference data samples respectively, and then screen out target training samples corresponding to the model fine-tuning task from the multiple preference data samples based on the consensus index values and the same strategy index values corresponding to the multiple preference data samples respectively. The target training samples are high-quality data with high consensus degrees screened out based on the consensus index values and the same strategy index values. Therefore, the training data valuable for model training operation can be efficiently and low-cost screened out, which can effectively improve the model training quality and effect of the to-be-trained language model when the target training samples are used to perform the training operation on the to-be-trained language model, and further ensures the practicability of the method. BRIEF DESCRIPTION OF DRAWINGS

[0011] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and serve to explain the principles of the application, and do not limit the application in any way. In the drawings: Figure 1 A scene schematic diagram of a data screening method provided by an exemplary embodiment of the application; Figure 2 A flowchart of a data screening method provided by an exemplary embodiment of the application; Figure 3 A flowchart of determining consensus index values of multiple preference data samples under different preference dimension parameters provided by an exemplary embodiment of the application; Figure 4 A flowchart of determining same strategy index values corresponding to multiple preference data samples provided by an exemplary embodiment of the application; Figure 5 A flowchart of screening out target training samples corresponding to the model fine-tuning task from the multiple preference data samples provided by an exemplary embodiment of the application; Figure 6 A principle schematic diagram of a preference data selection method based on consensus index values and same strategy index values provided by an exemplary application embodiment of the application; Figure 7 A structure schematic diagram of a data screening device provided by an exemplary embodiment of the application; Figure 8 A structure schematic diagram of an electronic device provided by an exemplary embodiment of the application. DETAILED DESCRIPTION

[0012] In order to make the purposes, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described clearly and completely below in combination with specific embodiments of the present application and corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0013] It should be noted that, in the case where the embodiments of the present application involve user information, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0014] The various models (including but not limited to language models or large models) involved in the present application comply with relevant laws and standards. Moreover, the technical solutions provided by the embodiments of the present application can adopt a deep learning model with relatively large model parameter scale, wherein the large model is only one model example, and the embodiments of the present application do not limit the number of model parameters supported by the deep learning model adopted to meet the actual demand. The deep learning model involved in the embodiments of the present application can be a language model (Language Model, LM for short) or a multimodal model (Multimodal Model, MM for short) based on artificial intelligence.

[0015] In addition, it should be noted that, in the case where the embodiments of the present application involve user interaction operations or trigger operations, the user interaction operations or trigger operations involved in the embodiments of the present application include but are not limited to: touch operations, gesture operations, voice operations, head movement operations, eye movement operations and various modes of interactive operations; wherein the touch operation includes but is not limited to: click operation, double-click operation, long-press operation, sliding operation, pinch operation or mouse hovering operation, etc. The sliding operation includes but is not limited to: straight line sliding, curve sliding, etc.

[0016] Term definition: Hand card: is the commodity explanation script used in the process of anchor live broadcast.

[0017] Large language model, Large Language Model, LLM for short, refers to a deep learning model trained based on massive text data and capable of performing multiple natural language processing tasks.

[0018] Intelligent live hand card: a professional model obtained through fine-tuning training based on LLM and live application data, used to generate high-quality hand cards (explanation scripts) for anchors that match product information.

[0019] Direct Preference Optimization (DPO) is a simpler and more efficient LLM alignment method that replaces Reinforcement Learning from Human Feedback (RLHF) by directly fine-tuning preference data on hand card models.

[0020] Consensus index value: an index value used to quantify the degree of evaluation consistency of a data sample in different preference dimensions.

[0021] Same strategy index value: an index value used to measure whether the generation strategy of a hand card sample matches the hand card model strategy, reflecting the degree of adaptation of hand card data to hand card model fine-tuning.

[0022] To facilitate understanding of the data screening method, device, equipment and computer storage medium provided by the embodiments of the present application, the related art will be briefly described first: In the application scenario of live streaming, small and medium-sized anchors often have limited experience and are difficult to prepare high-quality explanation scripts (i.e., "hand cards") for a large number of products in a short period of time. Therefore, a large language model is introduced to develop "intelligent live hand cards" to assist anchors in improving their explanation capabilities. However, the reasoning effect of the large language model is largely dependent on the quality of the training data used for fine-tuning tasks.

[0023] In related technology 1, the training data used for fine-tuning tasks of the large language model can be "full-fine-grained preference training data", i.e., all collected real preference data and synthetic preference data are mixed and directly used for direct preference optimization operation.

[0024] However, the data set contains a large amount of noise, conflicts and irrelevant information, and the large language model will be severely disturbed by low-quality data during the learning process, resulting in poor performance of the large language model, and even learning incorrect behavior patterns; at the same time, processing massive data also causes a huge waste of computing resources.

[0025] In related technology 2, the training data used for fine-tuning tasks of the large language model can be "based on a single overall preference granularity", i.e., the above training data relies on the annotation personnel to label which is better in the "single overall preference granularity", i.e., the overall degree.

[0026] However, a single overall preference labeling standard is too general and difficult to operate. For a large language model, the reply information output by the large language model may perform well in one aspect (such as comprehensive information) but perform poorly in another aspect (such as oralization). This makes it difficult for labelers to give consistent and reliable judgments, so that the preference data itself is full of "implicit" noise and conflicts, thereby limiting the final effect that the large language model can achieve when performing alignment operations (or model fine-tuning operations). If high-quality overall preference labeling is to be obtained, personnel need to perform one-by-one labeling operations, which makes the labeling cost of training data extremely high.

[0027] In summary, the noise of real data is relatively large, and the high-value hand card sample is sparse. The quality of synthetic data is relatively low, and a large amount of manpower is needed for data cleaning and filtering, which is costly and inefficient. Therefore, if real data or synthetic data is used as training data for fine-tuning of the hand card model, the relevance of the training data to the "intelligent hand card" task may not be strong, and the adaptability may be insufficient. Therefore, how to efficiently and cost-effectively filter out valuable training data for model fine-tuning from a large amount of noisy data sources is a technical problem that needs to be solved in the application scenario of a large language model.

[0028] To solve the above technical problems, an embodiment of the present application provides a data screening method, device, equipment and computer storage medium. Referring to FIG. 1, the data screening method comprises the following steps. Figure 1 As shown in the figure, the execution subject of the data screening method can be a data screening device 200, which can be implemented as a local server, a cloud server or an edge server. When the data screening device 200 is implemented as a cloud server, the data screening method can be executed in the cloud, and a plurality of computing nodes (cloud servers) can be deployed in the cloud, each computing node having computing, storage and other processing resources. In the cloud, a plurality of computing nodes can be organized to provide a certain service. Of course, one computing node can also provide one or more services. The cloud can provide the service in the form of a service interface, which can be called by a user to use the corresponding service. The service interface includes a model development kit (Software Development Kit, SDK), an application programming interface (Application Programming interface, API) and the like.

[0029] The data screening device 200 is in communication connection with the client 100, wherein the client 100 is used for the user to apply to be able to trigger the data screening operation. The client 100 can be any computing device with certain information interaction capability. In specific implementation, the client 100 can be a mobile phone, a personal computer PC, a tablet computer, a set application program, etc. In addition, the basic structure of the client 100 can include at least one processor. The number of processors depends on the configuration and type of the client. The client 100 can also include a memory, which can be volatile, such as a random access memory (RAM), or non-volatile, such as a read-only memory (ROM), a flash memory, etc., or can include both types. The memory usually stores an operating system (OS), one or more application programs, and program data, etc. In addition to the processing unit and the memory, the client 100 also includes some basic configurations, such as a network card chip, an IO bus, a display component, and some peripheral devices, etc. Optionally, some peripheral devices can include, for example, a keyboard, a mouse, a stylus, a printer, etc. Other peripheral devices are well known in the art and will not be described here.

[0030] The data screening device 200 refers to a device that can implement data screening operations in a network virtual environment, and usually refers to a device that uses a network to plan information and perform data screening operations. The data screening device 200 can be a data screening model for implementing data screening operations. In physical implementation, the data screening device 200 can be any device that can provide computing services and perform corresponding data screening operations, such as a processor, a server, etc. The composition of the data screening device 200 mainly includes a processor, a hard disk, a memory, a system bus, etc., which is similar to the general computer architecture.

[0031] In the above embodiment, the data screening device 200 and the client 100 are in network connection, which can be wireless or wired network connection. If the data screening device 200 and the client 100 are in communication connection, the network standard of the mobile network can be any one of 2G (Global System for Mobile Communications GSM), 2.5G (General Packet Radio Service GPRS), 3G (Wideband Code Division Multiple Access (WCDMA), Time Division Synchronous Code Division Multiple Access (TD-SCDMA), 4G (Long Term Evolution LTE), 4G+ (Enhanced Long Term Evolution LTE+), Worldwide Interoperability for Microwave Access (WiMax), 5G, 6G, etc.

[0032] In the embodiments of the present application, the client 100 is used for an application by a user to generate a data screening request or a data screening task for triggering a data screening operation. The data screening request or the data screening task can be generated based on a model training requirement of a preset language model. In order to implement the data screening operation, the data screening request or the data screening task can be sent to the data screening device 200, so that the data screening device 200 can perform a corresponding data screening operation based on the data screening request or the data screening task.

[0033] The data screening device 200 is used for obtaining the data screening request or the data screening task sent by the client 100, and then obtaining a plurality of preference data samples based on the obtained data screening request or data screening task. The plurality of preference data samples can be annotated sample data corresponding to a plurality of preference dimension parameters. The preference dimension parameters corresponding to different application scenarios can be different. For example, in the application scenario of hand card, the plurality of preference dimension parameters can include fine-grained dimension parameters such as "explanation skill", "information authenticity", "oralization" and the like. In the application scenario of chat, the plurality of preference dimension parameters can include fine-grained dimension parameters such as "logical continuity", "understanding accuracy" and the like.

[0034] Specifically, the preference data sample includes input information, positive output sample and / or negative output sample under at least one preference dimension parameter. For example, the preference data sample can be The above-mentioned is used to represent the preference data sample, is input information. In the application scenario of hand card, the input information can include commodity information and script generation instruction. may be a positive sample reply information that is determined to be more preferred under the preset preference dimension parameter, may be a negative sample reply information that is determined to be not preferred under the preset preference dimension parameter.

[0035] After obtaining the plurality of preference data samples, the plurality of preference data samples can be analyzed and processed to determine the consensus index values of the plurality of preference data samples under different preference dimension parameters respectively. The preference data sample can include the labeled data under the preset preference dimension parameter, i.e., the positive sample reply information and the negative sample reply information under the preset preference dimension parameter. The consensus index value is used to identify the recognition degree of the preference data sample under different preference dimension parameters, and is used to measure the consensus degree of the preference data sample under different preference dimension parameters. In addition to determining the consensus index values of the plurality of preference data samples under different preference dimension parameters, the plurality of preference data samples can also be analyzed and processed to determine the same strategy index values corresponding to the plurality of preference data samples respectively. The same strategy index value is used to identify the adaptation degree between the preference data sample and the language model to be trained.

[0036] After determining the consensus index values and the same strategy index values corresponding to the plurality of preference data samples respectively, data screening operations can be performed based on the consensus index values and the same strategy index values corresponding to the plurality of preference data samples, to screen out target training samples corresponding to the model fine-tuning task from the plurality of preference data samples, thereby effectively realizing efficient and low-cost screening of training data valuable for model fine-tuning operations.

[0037] In this embodiment, based on the consensus index values and the same strategy index values corresponding to the plurality of preference data samples respectively, target training samples corresponding to the model fine-tuning task are screened out from the plurality of preference data samples. Since the target training samples are high-quality data screened out based on the consensus index values and the same strategy index values, and the consensus degree is also relatively high, efficient and low-cost screening of training data valuable for model training operations is effectively realized. In this way, when the training operation of the language model to be trained is performed based on the target training samples, the model training quality and effect of the language model to be trained can be effectively guaranteed, and the practicability of the method is further guaranteed.

[0038] The technical solutions provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0039] Figure 2 A flowchart of a data screening method provided by an exemplary embodiment of the present application is shown. Reference is made to the accompanying drawings. Figure 2As shown, the embodiment provides a data screening method, the execution subject of the method is a data screening device, the data screening device can be implemented as software or a combination of software and hardware, when the data screening device is implemented as hardware, it can be various electronic devices that can implement data screening operations, in some examples, the data screening device can be implemented as an application end, a server, a cloud server, etc., when the data screening device is implemented as software, it can be installed in the above-mentioned electronic devices. Specifically, the data screening method provided in the embodiment can include: Step S201: obtaining a plurality of preference data samples, wherein the preference data sample includes input information, positive output samples and negative output samples under at least one preference dimension parameter.

[0040] Step S202: respectively determining the consensus index value of the plurality of preference data samples under different preference dimension parameters, wherein the preference data sample corresponds to an original preference dimension parameter, and the consensus index value is used to identify the recognition degree of the preference data sample under a non-original preference dimension parameter.

[0041] Step S203: respectively determining the same strategy index value corresponding to the plurality of preference data samples, and the same strategy index value is used to identify the adaptation degree between the preference data sample and the language model to be trained.

[0042] Step S204: based on the consensus index value and the same strategy index value corresponding to the plurality of preference data samples respectively, screening out target training samples corresponding to the model fine-tuning task from the plurality of preference data samples.

[0043] The specific implementation and implementation principle of each step are described in detail as follows: Step S201: obtaining a plurality of preference data samples, wherein the preference data sample includes input information, positive output samples and negative output samples under at least one preference dimension parameter.

[0044] When there is a data screening requirement, the data screening device can obtain a plurality of preference data samples, wherein the plurality of preference data samples can correspond to a plurality of preference dimension parameters, different application scenarios can correspond to different preference dimension parameters, for example: in the application scenario of hand card data, the plurality of preference dimension parameters can include at least one of the following: explanation skill, information authenticity, oralization, information comprehensiveness, etc.; in the application scenario of generating review information, the plurality of preference dimension parameters can include at least one of the following: accuracy, relevance, information amount, integrity, etc.

[0045] For any one preference data sample, it can include: input information, positive output samples and negative output samples under at least one preference dimension parameter, for example, the preference data sample is The above-mentioned For representing the preference data sample, As input information, in the application scenario of hand cards, the input information can include commodity information and copy generation instructions. The positive sample reply information can be determined as more preferred under the preset preference dimension parameter, The negative sample reply information can be determined as not preferred under the preset preference dimension parameter.

[0046] For the preference data sample, under different preference dimension parameters, the same input information can correspond to different positive output samples and negative output samples, for example: under the limitation condition of the preference dimension parameter A, the preference data sample can be The above-mentioned For representing the preference data sample, As input information, The positive sample reply information can be determined as more preferred under the preference dimension parameter A, The negative sample reply information can be determined as not preferred under the preference dimension parameter A. Under the limitation condition of the preference dimension parameter B, the preference data sample can be The above-mentioned For representing the preference data sample, As input information, The positive sample reply information can be determined as more preferred under the preference dimension parameter B, The negative sample reply information can be determined as not preferred under the preference dimension parameter B. The above-mentioned positive output sample is different from the positive output sample The negative output sample is different from the negative output sample .

[0047] In addition, for the preference data sample, the preference data sample can include any one of the positive output sample and the negative output sample, for example: under the limitation condition of the preference dimension parameter C, the preference data sample can be The above-mentioned For representing the preference data sample, As input information, The positive sample reply information can be determined as more preferred under the preference dimension parameter C, and at this time, the preference data sample does not include the negative sample reply information, that is, the negative sample reply information is "null".

[0048] In addition, the embodiment is not limited to the specific acquisition manner of the plurality of preference data samples. In some examples, the plurality of preference data samples can be obtained through interaction between the data filtering device and a preset server (for example, a preset database or a data server, etc.). At this time, obtaining the plurality of preference data samples can include determining a preset server in communication connection with the data filtering device, the preset server including the plurality of preference data samples. Then, the plurality of preference data samples can be actively or passively obtained through the preset server, thereby effectively ensuring the accuracy and reliability of obtaining the plurality of preference data samples.

[0049] Step S202: respectively determining consensus index values of the plurality of preference data samples under different preference dimension parameters, wherein the preference data sample corresponds to an original preference dimension parameter, and the consensus index value is used to identify the recognition degree of the preference data sample under a non-original preference dimension parameter.

[0050] Since the plurality of preference data samples can correspond to a plurality of preference dimension parameters, and the same preference data sample can correspond to different recognition degrees under different preference dimension parameters, for example, under the limitation of the preference dimension parameter T, the preference data sample a can correspond to a higher recognition degree; under the limitation of the preference dimension parameter K, the same preference data sample a can correspond to a lower recognition degree. Therefore, in order to efficiently and low-costly filter out training data valuable for model training operation, after obtaining the plurality of preference data samples, the plurality of preference data samples can be analyzed and processed to respectively determine consensus index values of the plurality of preference data samples under different preference dimension parameters, the consensus index value being used to identify the recognition degree of the preference data sample under a non-original preference dimension parameter, and the same preference data sample can correspond to different consensus index values under different preference dimension parameters.

[0051] For the consensus index values of the plurality of preference data samples under different preference dimension parameters, the plurality of preference data samples can be analyzed and processed by a pre-trained consensus network model to determine the consensus index values, at this time, respectively determining the consensus index values of the plurality of preference data samples under different preference dimension parameters can include: respectively determining a set of preference dimension parameters corresponding to the plurality of preference data samples, the set of preference dimension parameters including a plurality of preference dimension parameters; calling the pre-trained consensus network model to analyze and process the plurality of preference data samples and the set of preference dimension parameters, and obtaining the consensus index values of the plurality of preference data samples under different preference dimension parameters output by the consensus network model, thereby effectively ensuring the accuracy and reliability of determining the consensus index values of the plurality of preference data samples under different preference dimension parameters.

[0052] The consensus network model can be a pre-trained network model, which can be a language model (LM) based on artificial intelligence or a multimodal large language model (MLLM), etc. The number of model parameters supported by the model is not limited in the embodiments of the present application, and the actual demand is targeted. If the number of model parameters is relatively large, the size of the model will be relatively large, the performance of the model will be relatively better, and of course, more time and resources will be consumed in the inference or training process. If the number of model parameters is relatively small, the size of the model will be relatively small, and the model will be more lightweight in the case of meeting the performance requirements, and the time and resources consumed in the inference or training process will be relatively small. The consensus network model can be a deep learning model for processing and generating natural language text or multimodal data, which can be implemented based on a neural network architecture and can be pre-trained on a large amount of data. In an optional implementation manner, the consensus network model can include an encoder, a decoder, a self-attention layer, and a feed-forward neural network, etc. The encoder is mainly used to convert input data (usually in sequence form) into vector representation, which can capture the semantic features of the input data. The decoder is responsible for converting the intermediate representation generated by the encoder into output data (usually in sequence form). The self-attention layer is a mechanism that allows the model to focus on other positions in the sequence to better encode the current position information. The feed-forward neural network can perform nonlinear transformation on the output of the self-attention layer to enhance the expression ability of the model. Each part works together to make the model based on them perform well in various complex processing tasks, such as natural language processing, computer vision, speech recognition, machine translation, text summarization, and intelligent question answering, etc.

[0053] Step S203: respectively determining same-strategy indicator values corresponding to the plurality of preference data samples, the same-strategy indicator values being used to identify the adaptation degree between the preference data samples and the to-be-trained language model.

[0054] In order to efficiently and at low cost screen out training data valuable for model training operation, in addition to determining the consensus indicator values of the plurality of preference data samples under different preference dimension parameters, it is also necessary to determine the same strategy indicator values corresponding to the plurality of preference data samples, the same strategy indicator values being used to identify the adaptation degree between the preference data samples and the language model to be trained. Different application scenarios correspond to different language models to be trained, for example, in the application scenario of e-commerce, the language model to be trained can be a hand card model used to generate hand card data; in the application scenario of article generation, the language model to be trained can be an article generation model used to generate article information. Moreover, the model fine-tuning task is used to adjust and optimize the language model to be trained, and different preference data samples and model fine-tuning tasks can correspond to different adaptation degrees.

[0055] For the same strategy indicator value, it can be determined by analyzing and processing the plurality of preference data samples through the pre-trained same strategy network model. At this time, determining the same strategy indicator values corresponding to the plurality of preference data samples respectively can include: determining the model fine-tuning task of the language model to be trained (including: model type information, model use description information, model fine-tuning algorithm, etc. of the language model to be trained); calling the pre-trained same strategy network model to analyze and process the plurality of preference data samples and the model fine-tuning task respectively, and obtaining the same strategy indicator values corresponding to the plurality of preference data samples respectively output by the same strategy network model, which effectively ensures the accuracy and reliability of determining the same strategy indicator values of the plurality of preference data samples.

[0056] Step S204: Based on the consensus indicator values and the same strategy indicator values corresponding to the plurality of preference data samples respectively, screening out target training samples corresponding to the model fine-tuning task from the plurality of preference data samples.

[0057] Since the consensus indicator value can represent the recognition degree of the plurality of preference data samples under different preference dimension parameters, and the same strategy indicator value can represent the adaptation degree between the plurality of preference data samples and the language model to be trained under the limitation of different preference dimension parameters, that is, the consensus indicator value and the same strategy indicator value can evaluate the value information of the plurality of preference data samples for the model training operation of the language model to be trained from different perspectives. Therefore, after determining the consensus indicator values and the same strategy indicator values corresponding to the plurality of preference data samples respectively, data screening operation can be performed on the plurality of preference data samples based on the consensus indicator values and the same strategy indicator values corresponding to the plurality of preference data samples respectively, so as to screen out target training samples corresponding to the model fine-tuning task, the target training samples being the training data valuable for the language model to be trained.

[0058] In some examples, the screening operation of the target training sample can be implemented by using a pre-trained sample screening model. In this case, based on the consensus indicator values and the same-strategy indicator values respectively corresponding to the plurality of preference data samples, screening the target training sample corresponding to the model fine-tuning task from the plurality of preference data samples can include: determining the pre-trained sample screening model; inputting the consensus indicator values, the same-strategy indicator values, the plurality of preference data, and the model fine-tuning task respectively corresponding to the plurality of preference data samples into the sample screening model to perform the sample screening operation, and obtaining the target training sample corresponding to the model fine-tuning task. In this way, the accuracy and reliability of screening the target training sample are effectively ensured.

[0059] In other examples, the screening operation of the target training sample can not only be implemented by using a pre-trained sample screening model, but also be implemented by determining the screening indicator values respectively corresponding to the plurality of preference data samples. In this case, based on the consensus indicator values and the same-strategy indicator values respectively corresponding to the plurality of preference data samples, screening the target training sample corresponding to the model fine-tuning task from the plurality of preference data samples can include: determining the screening indicator values respectively corresponding to the plurality of preference data samples based on the consensus indicator values and the same-strategy indicator values respectively corresponding to the plurality of preference data samples; and screening the target training sample corresponding to the model fine-tuning task from the plurality of preference data samples based on the screening indicator values respectively corresponding to the plurality of preference data samples.

[0060] After determining the consensus indicator values and the same-strategy indicator values respectively corresponding to the plurality of preference data samples, the consensus indicator values and the same-strategy indicator values can be analyzed and processed to determine the screening indicator values respectively corresponding to the plurality of preference data samples. In some examples, the screening indicator values can be determined based on the sum of the consensus indicator values and the same-strategy indicator values. In this case, based on the consensus indicator values and the same-strategy indicator values respectively corresponding to the plurality of preference data samples, determining the screening indicator values respectively corresponding to the plurality of preference data samples can include: determining the indicator sum between the consensus indicator values and the same-strategy indicator values; and determining the screening indicator values respectively corresponding to the plurality of preference data samples based on the indicator sum respectively corresponding to the plurality of preference data samples. Specifically, the indicator sum can be directly determined as the screening indicator value, or the product value between the indicator sum and a preset coefficient can be determined as the screening indicator value. In this way, the accuracy and reliability of determining the screening indicator value are effectively ensured. It should be noted that, for the screening indicator value and the preference data sample, the higher the screening indicator value, the more suitable the corresponding preference data sample is as the training sample of the model fine-tuning task; the lower the screening indicator value, the less suitable the corresponding preference data sample is as the training sample of the model fine-tuning task.

[0061] In other instances, the screening index value can be obtained by performing a weighted summation operation on the consensus index value and the same-strategy index value corresponding to multiple preference data samples respectively. In this case, determining the screening index value corresponding to multiple preference data samples based on the consensus index value and the same-strategy index value corresponding to multiple preference data samples may include: determining the first weight corresponding to the consensus index value and the second weight corresponding to the same-strategy index value; and performing a weighted summation operation on the consensus index value and the same-strategy index value corresponding to multiple preference data samples based on the first weight and the first weight to determine the screening index value corresponding to multiple preference data samples.

[0062] For example, the consensus metric value corresponding to multiple preference data samples is The same strategy indicator value is At that time, the consensus indicator value can be determined. The corresponding first weight and the same strategy indicator value The corresponding second weight ( Then, the consensus index values ​​corresponding to multiple preference data samples can be obtained. Same strategy indicator value The weighted summation value is determined as the screening index value. ,Right now This also ensures the accuracy of the selection criteria values. To ensure accurate and reliable determination.

[0063] After determining the screening index values ​​corresponding to multiple preference data samples, target training samples corresponding to the model fine-tuning task can be selected from the multiple preference data samples based on these screening index values. In some instances, the target training samples can be determined based on parameter information used to limit the number of samples. In this case, determining the target training samples corresponding to the model fine-tuning task from the multiple preference data samples based on the screening index values ​​can include: determining a sample selection ratio to limit the number of target training samples; and determining the target training samples corresponding to the model fine-tuning task from the multiple preference data samples based on the sample selection ratio and the screening index values ​​corresponding to the multiple preference data samples.

[0064] In order to be able to realize efficient training operation on the to-be-trained language model, the sample selection ratio can be determined first, which can be pre-configured parameter information, or the sample selection ratio can be determined based on human-computer interaction operation, and the sample selection ratio is used to limit the number of target training samples. After determining the sample selection ratio for limiting the number of target training samples, sample selection operation can be performed based on the sample selection ratio and the screening index value corresponding to the plurality of preference data samples, so that the target training sample corresponding to the model fine-tuning task can be stably determined. For example, in the case of , the of the plurality of preference data samples can be screened out based on the screening index value, and determined as the target training sample corresponding to the model fine-tuning task; or in the case of , the of the plurality of preference data samples can be screened out based on the screening index value, and determined as the target training sample corresponding to the model fine-tuning task, which effectively ensures the accuracy and reliability of determining the target training sample.

[0065] After screening out the target training sample corresponding to the model fine-tuning task from the plurality of preference data samples, the target training sample can be used to perform model training operation on the to-be-trained language model, that is, the target training sample can be used to perform model fine-tuning task of the to-be-trained language model. Since the target training sample is determined by screening operation based on the consensus index value and the same strategy index value, when the target training sample is used to perform model training operation on the to-be-trained language model, the quality and effect of the model training operation can be effectively ensured, which is beneficial to improve the inference accuracy of the to-be-trained language model.

[0066] The data screening method provided by the embodiment, by obtaining a plurality of preference data samples, respectively determining the consensus index value of the plurality of preference data samples under different preference dimension parameters and the same strategy index value corresponding to the plurality of preference data samples, and then based on the consensus index value and the same strategy index value corresponding to the plurality of preference data samples, screening out the target training sample corresponding to the model fine-tuning task from the plurality of preference data samples, the target training sample is high-quality data screened out based on the consensus index value and the same strategy index value, and the consensus degree is also relatively high, thereby effectively realizing efficient and low-cost screening of training data valuable for model training operation. Therefore, when the target training sample is used to perform training operation on the to-be-trained language model, the model training quality and effect of the to-be-trained language model can be effectively improved, and the practicability of the method is further ensured.

[0067] Figure 3This application provides a schematic flowchart for determining consensus index values ​​of multiple preference data samples under different preference dimension parameters, as an exemplary embodiment. Based on the above embodiment, refer to the appendix... Figure 3 As shown, for the consensus index values ​​of multiple preference data samples under different preference dimension parameters, they can be determined not only by analyzing and processing the multiple preference data samples through a pre-trained consensus network model, but also by the sub-preference spacing of the multiple preference data samples under different preference dimension parameters. Therefore, determining the consensus index values ​​of multiple preference data samples under different preference dimension parameters in this embodiment can include: Step S301: Determine the sub-preference intervals of multiple preference data samples under different preference dimension parameters. The sub-preference intervals are used to identify the probability difference between positive and negative output samples of preference data samples under different preference dimension parameters.

[0068] In order to determine the consensus index values ​​of multiple preference data samples under different preference dimension parameters, the multiple preference data samples can be analyzed and processed from different preference dimension parameters to determine the sub-preference interval of the multiple preference data samples under different preference dimension parameters. The sub-preference interval is used to identify the probability difference between positive output samples and negative output samples under the constraints of different preference dimension parameters.

[0069] In some instances, the sub-preference spacing can be determined based on the preference prediction model corresponding to each preference dimension parameter. In this case, determining the sub-preference spacing of multiple preference data samples under different preference dimension parameters may include: determining the preference prediction model corresponding to each preference dimension parameter based on multiple preference data samples; and processing the multiple preference data samples separately using the preference prediction models corresponding to different preference dimension parameters to determine the sub-preference spacing of multiple preference data samples under different preference dimension parameters.

[0070] The process of determining the preference prediction model corresponding to each preference dimension parameter based on multiple preference data samples may include: determining multiple standard preference samples corresponding to preset preference dimension parameters based on multiple preference data samples, wherein any standard preference sample includes: input information, standard positive response samples under preset preference dimension parameters, and standard negative response samples under preset preference dimension parameters; training the model based on multiple standard preference samples to obtain the preference prediction model corresponding to the preset preference dimension parameters, wherein the preference prediction model is trained to identify the sub-preference spacing of preference data samples under preset preference dimension parameters.

[0071] With reference to the above model training operation, the model training operation can be performed using multiple standard preference samples corresponding to multiple preference dimension parameters respectively, so that the preference prediction model corresponding to each preference dimension parameter can be stably determined, and then the preference prediction model corresponding to different preference dimension parameters can be used to analyze and process multiple preference data samples respectively, i.e., multiple preference data samples are input into the preference prediction model corresponding to each preference dimension parameter for analysis and processing, so that the sub-preference interval of multiple preference data samples under different preference dimension parameters directly output by each preference prediction model can be obtained, which effectively ensures the accuracy and reliability of determining the sub-preference interval.

[0072] In some examples, when the preference prediction model is trained to determine the positive sample output probability and the negative sample output probability of the preference data sample under the limitation of different preference dimension parameters, the sub-preference interval can be determined based on the positive sample output probability and the negative sample output probability corresponding to multiple preference data samples. At this time, the processing of multiple preference data samples by the preference prediction model corresponding to different preference dimension parameters to determine the sub-preference interval of multiple preference data samples under different preference dimension parameters can include: processing multiple preference data samples by the preference prediction model corresponding to different preference dimension parameters to obtain the positive sample output probability and the negative sample output probability corresponding to multiple preference data samples respectively; and determining the sub-preference interval of multiple preference data samples under different preference dimension parameters based on the positive sample output probability and the negative sample output probability corresponding to multiple preference data samples.

[0073] For example, the preference data sample is , the input information is the positive sample reply information that can be determined as more preferred under the preference dimension parameter A is the negative sample reply information that can be determined as not preferred under the preference dimension parameter A is obtained After that, the preference prediction model corresponding to other preference dimension parameters (e.g., the preference dimension parameter B, the preference dimension parameter C, the preference dimension parameter D, etc.) can be determined first, and then the preference data sample is input into each preference prediction model for processing, so that the positive sample output probability and the negative sample output probability corresponding to multiple preference data samples output by the preference prediction model can be obtained. The positive sample output probability can be , and the negative sample output probability can be Then, the positive sample output probability and the negative sample output probability corresponding to multiple preference data samples can be analyzed and processed to determine the sub-preference interval of multiple preference data samples under different preference dimension parameters .

[0074] In some examples, the sub-preference distance may be It can be understood that different preference dimension parameters can obtain different sub-preference distances, and the number of other preference dimension parameters is the same as the number of sub-preference distances, that is, under the limitation of the preference dimension parameter B, the preference data sample corresponds to ; under the limitation of the preference dimension parameter C, the preference data sample corresponds to ; under the limitation of the preference dimension parameter D, the preference data sample corresponds to .

[0075] In addition, in order to ensure that the preference mode learned by the to-be-trained language model focuses on the internal quality of the script content rather than the surface features such as the length of the text, a penalty term related to the length of the text can be added in the sub-preference distance to offset the influence of the length difference. At this time, based on the positive sample output probability and the negative sample output probability corresponding to the plurality of preference data samples respectively, the determination of the sub-preference distance of the plurality of preference data samples under different preference dimension parameters can include: determining the first sample length of the positive output sample and the second sample length of the negative output sample in the plurality of preference data samples; based on the positive sample output probability, the negative sample output probability, the first sample length and the second sample length corresponding to the plurality of preference data samples respectively, determining the sub-preference distance of the plurality of preference data samples under different preference dimension parameters.

[0076] Among them, after obtaining the plurality of preference data samples, the first sample length of the positive output sample and the second sample length of the negative output sample in the plurality of preference data samples can be determined first, and then the positive sample output probability, the negative sample output probability, the first sample length and the second sample length corresponding to the plurality of preference data samples respectively can be analyzed and processed to determine the sub-preference distance of the plurality of preference data samples under different preference dimension parameters.

[0077] In some examples, the sub-preference distance can be determined based on the pre-trained distance identification network model analyzing and processing the positive sample output probability, the negative sample output probability, the first sample length, and the second sample length corresponding to the plurality of preference data samples respectively. In this case, determining the sub-preference distance of the plurality of preference data samples under different preference dimension parameters based on the positive sample output probability, the negative sample output probability, the first sample length, and the second sample length corresponding to the plurality of preference data samples respectively can include: determining the pre-trained distance identification network model; inputting the positive sample output probability, the negative sample output probability, the first sample length, and the second sample length corresponding to the plurality of preference data samples respectively into the distance identification network model to obtain the sub-preference distance of the plurality of preference data samples under different preference dimension parameters output by the distance identification network model, which effectively ensures the accuracy and reliability of determining the sub-preference distance.

[0078] In other examples, the sub-preference distance can be determined not only based on the pre-trained distance identification network model analyzing and processing the positive sample output probability, the negative sample output probability, the first sample length, and the second sample length corresponding to the plurality of preference data samples respectively, but also based on the length deviation between the first sample length and the second sample length. In this case, determining the sub-preference distance of the plurality of preference data samples under different preference dimension parameters based on the positive sample output probability, the negative sample output probability, the first sample length, and the second sample length corresponding to the plurality of preference data samples respectively can include: determining the length deviation between the first sample length and the second sample length; determining the output probability deviation between the positive sample output probability and the negative sample output probability based on the positive sample output probability and the negative sample output probability corresponding to the plurality of preference data samples respectively; and determining the sub-preference distance of the plurality of preference data samples under different preference dimension parameters based on the output probability deviation and the length deviation.

[0079] For example, when the preference data sample is , the first sample length of the positive output sample is , the second sample length of the negative output sample is , the positive sample output probability is , and the negative sample output probability is , the length deviation between the first sample length and the second sample length can be determined as follows: , the output probability deviation between the positive sample output probability and the negative sample output probability can be determined as follows: , and the sub-preference distance of the plurality of preference data samples under different preference dimension parameters can be determined based on the output probability deviation ( ) and length deviation ( To determine the sub-preference spacing of multiple preference data samples under different preference dimension parameters. Sub-preference spacing The output probability bias can be ( ) and length deviation ( The difference between ) is .

[0080] Or, based on the length deviation ( Corresponding preset hyperparameters To determine the sub-preference spacing of multiple preference data samples under different preference dimension parameters. At this point, the length deviation can be determined first. ) and preset hyperparameters The product value between ; output probability bias ( ) and product value The difference between them is determined as the sub-preference spacing. ,Right now This effectively ensures the accuracy and reliability of determining the sub-preference spacing.

[0081] Step S302: Based on the sub-preference spacing of multiple preference data samples under different preference dimension parameters, determine the consensus index values ​​of multiple preference data samples under different preference dimension parameters.

[0082] After obtaining the sub-preference intervals of multiple preference data samples under different preference dimension parameters, these intervals can be analyzed to determine the consensus index values ​​for each preference data sample under different preference dimension parameters. In some instances, the consensus index value can be determined by querying a preset mapping relationship. In this case, determining the consensus index value for multiple preference data samples under different preference dimension parameters based on their sub-preference intervals can include: determining a pre-configured preset mapping relationship table, which includes multiple one-to-one mapping relationships between sub-preference intervals and consensus index values; querying the preset mapping relationship table based on the sub-preference intervals of multiple preference data samples under different preference dimension parameters to determine the consensus index value for each preference data sample under different preference dimension parameters. This ensures the accuracy and reliability of determining the consensus index value.

[0083] In some examples, the consensus indicator value can also be determined by performing a normalization operation on the sub-preference intervals of the plurality of preference data samples under different preference dimension parameters. In this case, determining the consensus indicator value of the plurality of preference data samples under different preference dimension parameters based on the sub-preference intervals of the plurality of preference data samples under different preference dimension parameters can include: performing a normalization operation on the sub-preference intervals of the plurality of preference data samples under different preference dimension parameters to obtain a plurality of normalized intervals of the plurality of preference data samples under different preference dimension parameters; and determining the consensus indicator value of the plurality of preference data samples under different preference dimension parameters based on the plurality of normalized intervals of the plurality of preference data samples under different preference dimension parameters.

[0084] Since the number of different preference dimension parameters is multiple, and the number of sub-preference intervals of the plurality of preference data samples under different preference dimension parameters is multiple, in order to accurately determine the consensus indicator value of the plurality of preference data samples under different preference dimension parameters, a normalization operation can be performed on the sub-preference intervals of the plurality of preference data samples under different preference dimension parameters, so as to obtain a plurality of normalized intervals of the plurality of preference data samples under different preference dimension parameters.

[0085] In this case, performing a normalization operation on the sub-preference intervals of the plurality of preference data samples under different preference dimension parameters to obtain a plurality of normalized intervals of the plurality of preference data samples under different preference dimension parameters can include: determining a preference interval average value of the preference data sample under each preference dimension parameter based on the sub-preference intervals of the plurality of preference data samples under different preference dimension parameters and a preference interval variance of the preference data sample under each preference dimension parameter. Based on the preference interval average value and the preference interval variance

[0086] , the normalized interval is determined. For example, under the limitation of the preference dimension parameter A, the preference data sample is Under the limitation of the preference dimension parameter B, the sub-preference interval of the preference data sample under the preference dimension parameter B can be determined based on the preference data sample The preference interval average value of the preference data sample under the preference dimension parameter B and the preference interval variance , wherein the preference interval average value can be determined by the following formula: wherein, is an expectation function, the preference interval average value and the preference interval variance ​, is a preferred distance variance, is a desired function.

[0087] In determining the preferred distance average value of the preference data sample under each preference dimension parameter and the preferred distance variance Afterwards, the preferred distance average value and the preferred distance variance can be analyzed and processed to determine the normalized distance. In some examples, the normalized distance can be determined by presetting a normalized network model to analyze and process the preferred distance average value and the preferred distance variance . Alternatively, the normalized distance under the preference dimension parameter B can be determined by the following formula: , which effectively ensures the accuracy and reliability of determining the normalized distance .

[0088] For the preference data sample, the number of different preference dimension parameters can be multiple, and each different preference dimension parameter can determine a normalized distance. Therefore, the number of determined normalized distances is multiple. After determining multiple normalized distances of multiple preference data samples on different preference dimension parameters, the multiple normalized distances of the multiple preference data samples on different preference dimension parameters can be analyzed and processed to determine the consensus index value of the multiple preference data samples under different preference dimension parameters. In some examples, the consensus index value can be determined by a pre-trained network model analyzing and processing the multiple normalized distances of the multiple preference data samples on different preference dimension parameters.

[0089] In other examples, the consensus index value can be determined by accumulating the multiple normalized distances of the same preference data sample on different preference dimension parameters. At this time, based on the multiple normalized distances of the multiple preference data samples on different preference dimension parameters, determining the consensus index value of the multiple preference data samples under different preference dimension parameters can include: accumulating the multiple normalized distances of the same preference data sample on different preference dimension parameters to obtain a distance accumulation value corresponding to the preference data sample; and determining the consensus index value of the multiple preference data samples under different preference dimension parameters based on the distance accumulation value.

[0090] After obtaining the multiple normalized distances of the same preference data sample on different preference dimension parameters, the multiple normalized distances can be accumulated. For example, the multiple normalized distances include , , In the case that the plurality of normalized distances are accumulated, the accumulated value of the plurality of normalized distances corresponding to the preference data sample is obtained, i.e. The consensus index value of the plurality of preference data samples under different preference dimension parameters is determined based on the obtained accumulated value of the plurality of normalized distances. In some examples, the accumulated value of the plurality of normalized distances is directly determined as the consensus index value of the plurality of preference data samples under different preference dimension parameters. The consensus index value of the plurality of preference data samples under different preference dimension parameters is determined based on the obtained accumulated value of the plurality of normalized distances. In some examples, the accumulated value of the plurality of normalized distances is directly determined as the consensus index value of the plurality of preference data samples under different preference dimension parameters. i.e. Or, the product value between the accumulated value of the plurality of normalized distances and a preset coefficient is determined as the consensus index value, i.e. Or, the sum value between the accumulated value of the plurality of normalized distances and a preset amplitude parameter is determined as the consensus index value, i.e. This effectively ensures the flexibility and reliability of determining the consensus index value. In this embodiment, the sub-preference distance of the plurality of preference data samples under different preference dimension parameters is determined, and then the consensus index value of the plurality of preference data samples under different preference dimension parameters is determined based on the sub-preference distance of the plurality of preference data samples under different preference dimension parameters. This effectively ensures the accuracy and reliability of determining the consensus index value, and then improves the accuracy of determining the target training sample based on the consensus index value, further ensuring the stable and reliable operation of the data screening method. The flowchart for determining the same-strategy index value corresponding to the plurality of preference data samples is provided in an exemplary embodiment of the present application. Based on any one of the above embodiments, referring to FIG. 8, for the same-strategy index value corresponding to the plurality of preference data samples, it can not only be determined by analyzing and processing the plurality of preference data samples through the pre-trained same-strategy network model, but also be determined by the first prediction probability of the positive output sample and the second prediction probability of the negative output sample in the plurality of preference data samples. At this time, the determination of the same-strategy index value corresponding to the plurality of preference data samples in the present embodiment can include:

[0091] Step S401: processing the plurality of preference data samples by using the language model to be trained to determine the first prediction probability of the positive output sample and the second prediction probability of the negative output sample in the plurality of preference data samples.

[0092] Figure 4 The flowchart for determining the same-strategy index value corresponding to the plurality of preference data samples is provided in an exemplary embodiment of the present application. Based on any one of the above embodiments, referring to FIG. 8, for the same-strategy index value corresponding to the plurality of preference data samples, it can not only be determined by analyzing and processing the plurality of preference data samples through the pre-trained same-strategy network model, but also be determined by the first prediction probability of the positive output sample and the second prediction probability of the negative output sample in the plurality of preference data samples. At this time, the determination of the same-strategy index value corresponding to the plurality of preference data samples in the present embodiment can include: Figure 4 Step S401: processing the plurality of preference data samples by using the language model to be trained to determine the first prediction probability of the positive output sample and the second prediction probability of the negative output sample in the plurality of preference data samples.

[0093] ​​​​​The plurality of preference data samples can be processed by using the to-be-trained language model, that is, the plurality of preference data samples can be directly input into the to-be-trained language model for analysis and processing, so that the first prediction probability of the positive output sample and the second prediction probability of the negative output sample output by the to-be-trained language model can be obtained, which effectively ensures the accuracy and reliability of determining the first prediction probability of the positive output sample and the second prediction probability of the negative output sample.

[0094] In some examples, the first prediction probability of the positive output sample and the second prediction probability of the negative output sample in the plurality of preference data samples can also be determined in combination with the vocabulary size corresponding to the to-be-trained language model and the token information included in the vocabulary. At this time, the processing of the plurality of preference data samples by using the to-be-trained language model to determine the first prediction probability of the positive output sample and the second prediction probability of the negative output sample can include: determining the vocabulary size of the knowledge base corresponding to the to-be-trained language model and the token information included in the knowledge base; processing the preference data sample by using the to-be-trained language model, the vocabulary size, and the token information to determine the first prediction probability of the positive output sample and the second prediction probability of the negative output sample.

[0095] Specifically, the to-be-trained language model corresponds to a knowledge base for implementing data processing operations, and the knowledge base includes a plurality of tokens for being understood / recognized by the to-be-trained language model. The token is the smallest unit of text before being processed by a computer. Different tokenization methods will result in different token division methods. At this time, the token can have different forms of expression, for example, each Chinese character or word in Chinese can be a token, and the word separated by spaces in English can be a token, and the like.

[0096] The first prediction probability of the positive output sample is related to the vocabulary included in the positive output sample and the knowledge base, and the second prediction probability of the negative output sample is related to the vocabulary included in the negative output sample and the knowledge base. Therefore, in order to accurately determine the first prediction probability of the positive output sample and the second prediction probability of the negative output sample, the vocabulary size of the knowledge base corresponding to the to-be-trained language model and the token information included in the knowledge base can be determined first. The vocabulary size is used to identify the number of tokens that can be understood or recognized by the to-be-trained language model. The vocabulary size and the token information included in the knowledge base can be pre-configured and stored in the data in the preset area. When the vocabulary size and the token information are needed for corresponding data processing operations, the vocabulary size of the knowledge base corresponding to the to-be-trained language model and the token information included in the knowledge base can be determined by accessing the preset area.

[0097] After obtaining the lexicon size and lexical information, within the limits defined by these factors, the language model to be trained is used to process the preference data samples. This allows for the direct determination of the first predicted probability of positive output samples and the second predicted probability of negative output samples from the language model. For example, when the preference data samples are... , These can be positive sample responses that are judged as more preferred under preset preference dimension parameters. This can be negative sample responses that are judged as unpreferenced under preset preference dimension parameters. The vocabulary size of the knowledge base is... The lexical information in the knowledge base is When the language model to be trained is at that time, the first predicted probability of a positive output sample can be determined using the following formula: ) in, This represents the first predicted probability for a positive output sample. Positive output sample The first sample length, For the language model to be trained, When input information The first t words (meta-information) When input into the language model to be trained, the output vocabulary is... The probability of.

[0098] Similarly, the language model to be trained can use the following formula to determine the second predicted probability of negative output samples: ) in, negative output sample The second predicted probability, negative output sample The second sample length, For the language model to be trained, When input information The first t words (meta-information) When input into the language model to be trained, the output vocabulary is... The probability of.

[0099] Step S402: Based on the first prediction probability and the second prediction probability corresponding to the multiple preference data samples respectively, determine the same strategy index value corresponding to the multiple preference data samples.

[0100] After determining the first and second predicted probabilities corresponding to multiple preference data samples, these probabilities can be analyzed to determine the corresponding policy index values. In some instances, the policy index values ​​can be determined by querying a pre-defined mapping table. In this case, determining the policy index values ​​based on the first and second predicted probabilities of multiple preference data samples can include: determining the pre-defined mapping table, which contains multiple mapping relationships, such as the association between the first and second predicted probabilities and the policy index values; and querying the pre-defined mapping table using the first and second predicted probabilities to determine the policy index values ​​corresponding to them. This effectively ensures the accuracy and reliability of determining the policy index values.

[0101] In other instances, the same-policy index value can be determined not only by querying a preset mapping table, but also by the average probability of multiple preference data samples. In this case, determining the same-policy index value corresponding to multiple preference data samples based on the first and second prediction probabilities corresponding to the multiple preference data samples can include: determining the average probability of the multiple preference data samples based on the first and second prediction probabilities corresponding to the multiple preference data samples; and determining the same-policy index value based on the average probability.

[0102] For example, the first predicted probability corresponding to multiple preference data samples is: The second predicted probability is Then, based on the first prediction probability corresponding to multiple preference data samples, it is possible to... Second prediction probability To determine the average probability of multiple preference data samples. Then, the value of the same strategy indicator can be determined based on the probability average. In some instances, the probability average can be directly used as the value of the same strategy indicator. = Alternatively, the probability average can be compared with a preset coefficient. The product of the two values ​​is determined as the index value of the same strategy, that is... = This effectively ensures the flexibility and reliability of determining the values ​​of the same strategy indicators.

[0103] In this embodiment, the first prediction probability of the positive output sample and the second prediction probability of the negative output sample in the plurality of preference data samples are determined by processing the plurality of preference data samples by the language model to be trained, and then the same strategy index values corresponding to the plurality of preference data samples are determined based on the first prediction probability and the second prediction probability corresponding to the plurality of preference data samples respectively, so as to effectively ensure the accuracy and reliability of determining the same strategy index values corresponding to the plurality of preference data samples respectively.

[0104] Figure 5 The flowchart for screening the target training sample corresponding to the model fine-tuning task from the plurality of preference data samples is provided for an exemplary embodiment of the present application. Based on the above-mentioned embodiment, referring to the accompanying Figure 5 As shown in the figure, the target training sample can be determined not only by one data screening operation, but also by multiple rounds of data screening operation, at this time, the target training sample can include different batches of training samples. Based on the screening index values corresponding to the plurality of preference data samples, the target training sample corresponding to the model fine-tuning task can be screened from the plurality of preference data samples, which can include: Step S501: determining the sample batch for defining the target training sample and the sample quantity corresponding to the sample batch.

[0105] In order to stably screen the target training sample for training the language model to be trained, the target training sample can be determined by multiple rounds of dynamic data screening operation, at this time, the sample batch for defining the target training sample and the sample quantity corresponding to the sample batch can be determined first, and the above-mentioned sample batch and the sample quantity corresponding to the sample batch can be determined based on the human-computer interaction operation. Alternatively, the sample batch and the sample quantity corresponding to the sample batch can be default parameters stored in a preset area, at this time, by accessing the preset area, the sample batch for defining the target training sample and the sample quantity corresponding to the sample batch can be stably determined.

[0106] Step S502: determining the current batch training sample corresponding to the model fine-tuning task from the plurality of preference data samples based on the sample batch, the sample quantity, and the screening index values corresponding to the plurality of preference data samples respectively.

[0107] After obtaining the sample batch and the sample quantity corresponding to the sample batch, the data screening operation can be performed based on the sample batch, the sample quantity, and the screening index values corresponding to the plurality of preference data samples, so that the current batch training sample corresponding to the model fine-tuning task and matching the sample batch and the sample quantity can be obtained. For example, in the case that the current sample batch is the first batch, the sample quantity is 100, and the plurality of preference data samples is 10,000, the plurality of preference data samples can be sorted in descending order of the screening index values, and then the first 100 preference data samples are selected as the first batch training sample based on the sorting information, which effectively ensures the stability and reliability of determining the first batch training sample.

[0108] Step S503: Fine-tuning the to-be-trained language model based on the current batch training sample to determine a fine-tuned language model.

[0109] After screening the current batch training sample, the to-be-trained language model can be fine-tuned based on the current batch training sample. Specifically, the Direct Preference Optimization (DPO) algorithm can be used to fine-tune the model hyperparameters of the to-be-trained language model, thereby completing the model training operation and obtaining the fine-tuned language model after the model training operation.

[0110] Step S504: Using the fine-tuned language model to dynamically determine the next batch training sample corresponding to the model fine-tuning task from the plurality of preference data samples.

[0111] Since the model hyperparameters of the fine-tuned language model are different from those of the to-be-trained language model, and the same strategy index values corresponding to the plurality of preference data samples are determined based on the fine-tuned language model, when the plurality of preference data samples are analyzed and processed again based on the fine-tuned language model, new same strategy index values corresponding to the plurality of preference data samples can be obtained. The new same strategy index values will dynamically change the screening index values of the plurality of preference data samples, and then the data screening operation can be performed again based on the dynamically changed screening index values, so that the next batch training sample corresponding to the model fine-tuning task, such as the second batch training sample, the third batch training sample, etc., can be accurately and dynamically determined. Then, all batch training samples can be determined as target training samples corresponding to the model fine-tuning task.

[0112] In this embodiment, by determining the sample batch used to limit the target training samples and the number of samples corresponding to the sample batch, and based on the sample batch, the number of samples, and the screening index values ​​corresponding to multiple preference data samples, the current batch of training samples corresponding to the model fine-tuning task is determined from multiple preference data samples. Then, the language model to be trained is fine-tuned based on the current batch of training samples to determine the fine-tuned language model. Using the fine-tuned language model, the next batch of training samples corresponding to the model fine-tuning task is dynamically determined from multiple preference data samples. This effectively realizes the dynamic determination of target training samples through dynamic analysis operations, thereby ensuring the flexibility and reliability of the determination of target training samples.

[0113] In practical applications, taking the handcard model as an example in e-commerce scenarios, this paper illustrates how the handcard model is used to generate or filter high-quality handcard preference data by integrating multiple fine-grained preference dimension parameters. (See attached document.) Figure 6 As shown, this application embodiment provides a method for selecting card preference data based on consensus index values ​​and similar strategy index values. The "consensus index value" measures the value of card data by the degree of consensus or conflict between different preference dimensions. Simultaneously, the "similar strategy index value" is introduced to help identify card samples with better fit for the model fine-tuning task of the card model. The selected card samples can then be used to train the card model, thereby obtaining the target card model after training. Specifically, this method for selecting card preference data may include the following steps: Step 1: Collect multiple preference annotation data with fine granularity.

[0114] To select suitable sample data for fine-tuning the flashcard model, an aggregated dataset D containing labeled data with multiple fine-grained preference dimension parameters can be collected. For the flashcard model, these multiple preference dimension parameters may include "explanation skills parameters," "information authenticity parameters," "conversational style parameters," "fluency parameters (used to identify the logical fluency of the flashcard data and the fluency of the text when read aloud)," and "stylization parameters (pre-configured flashcard data style, such as an enthusiastic and cheerful style or a professional style, etc.)." The aggregated dataset D may include preference labeled data with multiple different preference dimension parameters. Specifically, each preference labeled data may include: input information, positive output samples and negative output samples under at least one preference dimension parameter. The positive and negative output samples mentioned above can be determined by manual or automatic labeling operations.

[0115] The technical solution does not perform overall labeling of data in a single overall dimension, but performs preference labeling of sample data from multiple fine-grained and more easily judged preference dimension parameters. For example, for the product explanation script generated for the host, the two sample data can be labeled from multiple fine-grained preference dimension parameters such as "explanation skill parameter", "information authenticity parameter", and "colloquial parameter", so as to obtain the "advantages and disadvantages" labels of the two sample data in each fine-grained preference dimension parameter.

[0116] Specifically, for the plurality of preference labeled data included in the aggregated data set , each fine-grained preference dimension parameter k corresponds to a sub-data set , the aggregated data set is the union of all the sub-data sets , and for any preference labeled data included in each sub-data set , it can be implemented as , wherein is used to represent a preference data sample, is input information; may be a positive sample reply information that is determined to be more preferred under the preset preference dimension parameter, may be a negative sample reply information that is determined to be not preferred under the preset preference dimension parameter, wherein the preset preference dimension parameter can be any fine-grained preference dimension parameter. By collecting the plurality of fine-grained preference labeled data in the above manner, the reliability and scalability of the collected plurality of preference labeled data are effectively ensured, so that the accuracy of the screening sample can be improved to some extent when data screening is performed based on the plurality of preference labeled data.

[0117] S2: training a sub-preference prediction model corresponding to each of the plurality of preference dimension parameters.

[0118] For each fine-grained preference dimension parameter k, a lightweight sub-preference prediction model can be trained using the preference labeled data under the preference dimension parameter, wherein the number of preference labeled data is multiple, each preference labeled data can include input information, positive sample reply information determined to be more preferred under the preference dimension parameter , and negative sample reply information determined to be not preferred under the preference dimension parameter , and the sub-preference prediction model can learn and imitate the preference judgment mode under the preference dimension parameter .

[0119] In the process of training the sub-preference prediction model, in order to ensure that the preference pattern learned by the sub-preference prediction model is the intrinsic quality of the script content rather than the surface features such as the length of the text, a penalty term can be added to the loss function of the sub-preference prediction model to offset the influence caused by the difference in the length of the text, so that the sub-preference prediction model can focus on the intrinsic quality of the script content itself. Specifically, when the hyperparameter of the sub-preference prediction model is , the sub-preference prediction model can perform fine-tuning operations through the following formula: , wherein is the hyperparameter of the sub-preference prediction model, is the expectation function, is the probability of outputting the positive sample reply information under the limitation of the input information , is the probability of outputting the negative sample reply information under the limitation of the input information , and and respectively represent the length of the positive sample reply information and the length of the negative sample reply information , is a preset penalty term hyperparameter, is an activation function used to implement the model training operation, and in some examples, is a normalized sigmoid function, can identify the sensitivity of the sample length to the above fine-tuning function. When the penalty term hyperparameter is 0, it indicates that the fine-tuning function does not pay attention to the length of the influence; when the penalty term hyperparameter is greater, the greater the influence of the length on the fine-tuning function.

[0120] S3: Estimate the consensus index value of each preference annotation data under different preference dimension parameters, wherein the preference annotation data corresponds to the original preference dimension parameter, and the consensus index value is used to identify the recognition degree of the preference annotation data under the non-original preference dimension parameter.

[0121] For any one preference annotation data in the aggregated data set D (assuming it is annotation data from the preference dimension k), the sub-preference prediction model trained using all other preference dimension parameters can be used to predict the sub-preference of the preference annotation data . Specifically, the sub-preference distance of the preference annotation data under different preference dimension parameters can be calculated through the following formula: ​ in, For parameters different from the preference dimension Other different preference dimension parameters Sub-preference spacing For different preference dimension parameters and input information Output positive sample response information in the case of The probability, For different preference dimension parameters and input information Output negative sample response information in the case of The probability, and These represent the positive sample response information. Length, negative sample response information Length, This is a preset penalty hyperparameter.

[0122] Different preference dimension parameters can correspond to different sub-preference intervals. After obtaining the sub-preference intervals corresponding to multiple preference dimension parameters, we can analyze and process the sub-preference intervals of the same preference-labeled data under all different preference dimension parameters to determine the consensus index values ​​of each preference-labeled data under different preference dimension parameters. In some instances, consensus metric values It can be determined using the following formula: ; ; ; ; in, Let be the expected function. Label data for preferences In different from preference dimension parameters Other different preference dimension parameters Sub-preference spacing Label data for preferences Mean information under different preference dimension parameters Label data for preferences Variance information under different preference dimension parameters This is a consensus metric value, which can be the sum of the normalized sub-preference intervals, for example: This consensus metric measures the value of the preference-labeled data. This approach effectively ensures the accuracy and reliability of determining consensus index values ​​by considering the degree of consensus from the perspective of different fine-grained preference dimension parameters.

[0123] when When it is large, it means that the original preference dimension parameter Preference labeling data The preference labeling direction is also influenced by most other preference dimension parameters. The acceptance of the sub-preference prediction model indicates that the preference-labeled data... It is a high-consensus, high-quality sample, and the preference information it contains is universal and reliable. Conversely, it indicates the preference dimension parameters. Sub-preference prediction model for positive sample response information and negative sample response information There is significant disagreement regarding prediction strategies for preference relationships, implying that preference-labeled data... These are low-quality or harmful samples, which may only perform well under certain specific preference dimension parameters.

[0124] S4: Determine the corresponding policy index value for each preference labeling data. The policy index value is used to identify the degree of fit between the preference data sample and the model fine-tuning task of the hand card model.

[0125] For the hand card model, in addition to evaluating the quality of the preference labeling data itself based on consensus index values, it is also necessary to determine the same-policy index value corresponding to each preference labeling data. This same-policy index value is used to measure the degree of same-policy between the preference labeling data and the current hand card model. The higher the same-policy index value, the higher the same-policy between the current preference labeling data and the current hand card model. This can avoid selecting sample data that differs too much from the current hand card model, thereby reducing the negative impact of sample data with excessively large distribution differences on the fine-tuning of the hand card model, thus ensuring the effectiveness of the fine-tuning of the hand card model.

[0126] Among them, the same strategy index value (used to identify the degree of same strategy) can be calculated by comparing the preference annotation data with the card model. The corrected likelihood operation determines that, under the same strategy, a larger indicator value indicates a preference for labeled data and the hand card model. The closer the inference preferences between them, the better. Specifically, to determine the same strategy index value, the following formula can be used to first determine the positive output samples in each preference-labeled data. First prediction probability and negative output samples Second prediction probability Then, based on the positive output samples in each preference-labeled data... First prediction probability and negative output samples Second prediction probability To determine the corresponding strategy indicator values ​​for the preference-labeled data: Where V represents the hand card model. The corresponding dictionary size, For all possible lexical units in the lexicon, In order to and before each word element After inputting into the flashcard model, the output is the vocabulary included in the preference annotation data. The probability, It is a logarithmic function. To determine the length of the response information in the biased labeled data, Used to identify exponential functions with the natural constant e as the base. Used to identify the hand card model during input. In the case of output The probability of [the outcome]. Using the above method to determine [the outcome]. The formula is used to determine the positive output samples in each preference-labeled data. First prediction probability and negative output samples Second prediction probability Then, based on the first predicted probability, and the second prediction probability To determine the value of the same strategy indicator, the following formula can be used: This effectively ensures the accuracy and reliability of determining the values ​​of the same strategy indicators.

[0127] S5: Based on consensus metrics Same strategy indicator value Data selection was performed to determine the set of hand card samples.

[0128] After obtaining the consensus index values ​​of various preference labeling data Same strategy indicator value Then, based on the consensus indicator value Same strategy indicator value To determine the screening index value corresponding to each preference data sample, the screening index value can be determined using the following formula: in, These are the screening index values ​​corresponding to the preference data samples. To align with consensus indicator values corresponding to the strategy indicator value corresponding to the strategy indicator value corresponding to the strategy indicator value.

[0129] After obtaining the screening indicator values corresponding to each preference data sample, the plurality of preference data samples can be sorted according to the high and low of the screening indicator values, and then a hand card sample set with a higher score can be selected based on the sorted preference data samples. Specifically, the hand card sample can be determined by the following formula: wherein, is a pre-specified sample selection ratio, denotes the size of the aggregated data set D, denotes the size of the aggregated data set , , is the selected hand card sample set, is used to identify that the hand card sample set is part of the aggregated data set.

[0130] S6: Based on the hand card sample set, the hand card model is iteratively optimized, and the training sample is dynamically selected based on the optimized hand card model.

[0131] After determining the hand card sample set , a high-quality data subset with high consensus and high same-strategy degree is obtained , and then the DPO algorithm can be applied to fine-tune the hand card model, so as to obtain an optimized hand card model. Since the quality of the hand card samples in the hand card sample set is relatively high and the conflicts are small, the hand card model can more efficiently learn the expected behavior when fine-tuning the hand card model based on the hand card sample set.

[0132] In order to further improve the quality and effect of the optimization operation of the hand card model, an iteration mechanism can be introduced to determine the training sample corresponding to the hand card model. Specifically, after determining the optimized hand card model, it can be identified whether the inference performance of the hand card model can meet the expected effect. When the inference performance of the hand card model does not meet the expected effect, the hand card model can be continuously dynamically adjusted and optimized. When the inference performance of the hand card model meets the expected effect, the final hand card model can be obtained.

[0133] Wherein, when continuing to perform dynamic adjustment and optimization operation on the hand card model, the optimized hand card model can be used to perform evaluation operation on the same strategy index value of the next round, and then a new round of hand card sample set can be dynamically selected based on the same strategy index value of the next round, and the dynamically selected hand card sample set can be used to perform training operation on the subsequent hand card model. In this way, the optimized hand card model obtained through iterative operation can dynamically select multiple rounds of hand card sample sets, and all rounds of hand card samples are training samples for dynamically training the hand card model. In this way, when the training sample is used to perform adjustment operation on the hand card model, the training quality and effect of the hand card model are effectively ensured.

[0134] The technical scheme provided by the application embodiment determines the consensus index value and the same strategy index value of each preference annotation data by training the sub-preference prediction model corresponding to each fine-grained preference dimension parameter. The consensus index value is used as a main index for measuring multi-dimensional preference consistency, which can provide a theoretical basis for screening high-quality hand card samples in complex data sets. The same strategy index value can help identify hand card samples with better adaptability to model fine-tuning based on strategy matching. Then, the consensus index value and the same strategy index value can be used to select a high-quality hand card sample set with high value from the preference annotation data of multiple fine-grained preference dimension parameters by using an iterative algorithm process. In this way, the shortcomings of "full data fine-tuning" are effectively overcome, that is, all data is not blindly used, but the quantitative indexes of "consensus index value and same strategy index value" are introduced to accurately select valuable hand card samples from a large amount of noisy data pool. Moreover, this scheme cleverly avoids the difficult "overall" judgment from the perspective of multi-dimensional fine-grained preference dimension parameters. Compared with relying on unreliable single overall preference for data screening operation, this scheme has the advantages of being more scientific, efficient and scalable, while reducing the dependence of the training cost of the hand card model on manual annotation operation and the difficulty of hand card preference data annotation.

[0135] Then, the hand card model can be fine-tuned based on the obtained hand card sample set, that is, trained on the high-quality and high-consensus data selected, which effectively ensures the model performance of the hand card model and significantly improves the application effect of sample selection. In addition, only the selected hand card training data (about 30% of all collected data) is used for model training operation, and the data amount is relatively small, which greatly shortens the model training time, saves expensive computing resources, and further improves the practicality of the method.

[0136] In addition, in some of the processes described in the above embodiments and the accompanying drawings, a plurality of operations are included in a specific order, but it should be clear that the operations can be executed in the order in which they appear in this document or in parallel, and the serial numbers of the operations such as 11, 12, etc. are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes can include more or fewer operations, and the operations can be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this document are used to distinguish different messages, devices, modules, etc. and do not represent the order of precedence. Also, "first" and "second" are not of different types.

[0137] Figure 7 A structural schematic diagram of a data screening device provided for an exemplary embodiment of the present application is shown in FIG. 1. As shown in the figure, the embodiment provides a data screening device for executing the data screening method described above. Figure 7 The data screening device can include the following components. Figure 2 The data screening device can include the following components. The first acquisition module 11 is configured to acquire a plurality of preference data samples, wherein the preference data samples include input information, positive output samples and negative output samples under at least one preference dimension parameter. The first determination module 12 is configured to determine a consensus index value of each of the plurality of preference data samples under different preference dimension parameters, wherein the preference data samples correspond to original preference dimension parameters, and the consensus index value is used to identify the degree of recognition of the preference data samples under non-original preference dimension parameters. The first determination module 12 is further configured to determine a same-strategy index value corresponding to each of the plurality of preference data samples, wherein the same-strategy index value is used to identify the degree of adaptation between the preference data samples and the language model to be trained. The first processing module 13 is configured to screen target training samples corresponding to the model fine-tuning task from the plurality of preference data samples based on the consensus index values and the same-strategy index values corresponding to the plurality of preference data samples, respectively.

[0138] The data screening device in the embodiment can also perform the description of the above Figures 1-6 embodiments, and specific reference can be made to the detailed description of the above embodiments, which will not be described in detail here.

[0139] As shown in FIG. 2, the embodiment provides an electronic device for executing the data screening method described above. Figure 8 The electronic device can include a memory 24 and a processor 25. Figure 2

[0140] ​The memory 24 is configured to store computer programs and can be configured to store various data to support operations on the electronic device. Examples of the data include instructions, data structures, contact data, phonebook data, messages, pictures, videos, etc. of any application or method for operating on the electronic device.

[0141] The processor 25 is coupled to the memory 24 and is configured to execute the computer programs in the memory 24 to: obtain a plurality of preference data samples, wherein the preference data sample includes input information, positive output samples and negative output samples under at least one preference dimension parameter; determine a consensus index value of the plurality of preference data samples under different preference dimension parameters respectively, wherein the preference data sample corresponds to an original preference dimension parameter, and the consensus index value is used to identify the recognition degree of the preference data sample under a non-original preference dimension parameter; determine a same-strategy index value corresponding to the plurality of preference data samples respectively, wherein the same-strategy index value is used to identify the adaptation degree between the preference data sample and the language model to be trained; and filter out target training samples corresponding to the model fine-tuning task from the plurality of preference data samples based on the consensus index value and the same-strategy index value corresponding to the plurality of preference data samples respectively.

[0142] Further, as shown in Figure 8 , the electronic device further includes a communication component 26, a display 27, a power supply component 28, an audio component 29, and other components. Figure 8 The partial components shown in Figure 8 do not mean that the electronic device only includes the components shown in Figure 8 . In addition, Figure 8 the components within the dashed line in Figure 8 are optional components rather than mandatory components, and the specific product form of the working node can be determined according to the actual situation. The working node of the present embodiment can be implemented as a terminal device such as a desktop computer, a notebook computer, a smart phone or an IOT device, or a server device such as a conventional server, a cloud server or a server array. If the working node of the present embodiment is implemented as a terminal device such as a desktop computer, a notebook computer or a smart phone, it can include the components within the dashed line in Figure 8 ; if the working node of the present embodiment is implemented as a server device such as a conventional server, a cloud server or a server array, it can not include the components within the dashed line in Figure 8 .

[0143] The above-described memory can be implemented by any type of volatile or nonvolatile memory devices or a combination thereof, such as a Static Random-Access Memory (SRAM), an Electrically Erasable Programmable Read Only Memory (EEPROM), an Erasable Programmable Read Only Memory (EPROM), a Programmable Read-Only Memory (PROM), a Read-Only Memory (ROM), a magnetic memory, a flash memory, a magnetic disk, or a compact disk.

[0144] The above-described communication component is configured to facilitate wired or wireless communication between the device in which the communication component is located and other devices. The device in which the communication component is located can access a wireless network based on a communication standard, such as a 2G, 3G, 4G / LTE, 5G, or the like mobile communication network, or a combination thereof. In an example embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast managing system via a broadcast channel.

[0145] The above-described display includes a screen, which can include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense a touch, a slide, and a gesture on the touch panel. The touch sensor can not only sense a boundary of a touching or a sliding action, but also detect a duration and a pressure associated with the touching or the sliding action.

[0146] The above-described power component provides power to various components of the device in which the power component is located. The power component can include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device in which the power component is located.

[0147] The above-described audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) that is configured to receive an external audio signal when the device in which the audio component is located is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory or transmitted via the communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0148] Accordingly, the embodiments of the present application also provide a computer readable storage medium storing a computer program, which, when executed by a processor, causes the processor to implement each step in the above method embodiments. The computer readable storage medium includes volatile or non-volatile or a combination thereof, and can be removable or non-removable. Examples of the computer readable storage medium include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital video disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store program code in the form of instructions or data structures and that can be accessed by a general purpose or special purpose computer, or both. Accordingly, the embodiments of the present application also provide a computer program product, which includes a computer program or instructions, and when the computer program or instructions are executed by a processor, causes the processor to implement each step in the above method embodiments. It should be understood that each process or a combination of multiple processes in the above method flow can be implemented by the computer program or instructions. In addition, these computer programs or instructions can be applied to the processor of a general purpose computer, a special purpose computer, an embedded processor or other programmable data processing devices, so that the processor of the general purpose computer, the special purpose computer, the embedded processor or other programmable data processing devices can be implemented as a device for implementing the corresponding functions in the above method embodiments.

[0149] It should also be noted that the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, so that processes, methods, articles or devices including a series of elements not only include those elements, but also include other elements not explicitly listed, or further include elements inherent in such processes, methods, articles or devices. Without more limitations, the element defined by the statement "including a" does not exclude the presence of additional identical elements in the process, method, article or device including the element.

[0150] The above merely provides an example of the present application, and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall fall into the scope of claims of the present application.

Claims

1. A data filtering method, characterized in that, include: Acquire multiple preference data samples, wherein the preference data samples include input information, positive output samples and negative output samples under at least one preference dimension parameter; Consensus index values ​​are determined for the multiple preference data samples under different preference dimension parameters, wherein the preference data samples correspond to original preference dimension parameters, and the consensus index values ​​are used to identify the degree of acceptance of the preference data samples under non-original preference dimension parameters. Determine the same strategy index value corresponding to each of the multiple preference data samples. The same strategy index value is used to identify the degree of fit between the preference data sample and the language model to be trained. Based on the consensus index value and the same strategy index value corresponding to the multiple preference data samples respectively, target training samples corresponding to the model fine-tuning task are selected from the multiple preference data samples.

2. The method according to claim 1, characterized in that, Determine the consensus index values ​​of the multiple preference data samples under different preference dimension parameters, including: The sub-preference intervals of the multiple preference data samples under different preference dimension parameters are determined respectively. The sub-preference intervals are used to identify the probability difference between outputting positive and negative output samples of the preference data samples under the constraints of different preference dimension parameters. Based on the sub-preference spacing of the multiple preference data samples under different preference dimension parameters, the consensus index values ​​of the multiple preference data samples under different preference dimension parameters are determined respectively.

3. The method according to claim 2, characterized in that, Determine the sub-preference spacing of the multiple preference data samples under different preference dimension parameters, including: Based on multiple preference data samples, determine the preference prediction model corresponding to each preference dimension parameter; The multiple preference data samples are processed using preference prediction models corresponding to different preference dimension parameters to determine the sub-preference spacing of the multiple preference data samples under different preference dimension parameters.

4. The method according to claim 3, characterized in that, The multiple preference data samples are processed using preference prediction models corresponding to different preference dimension parameters to determine the sub-preference intervals of the multiple preference data samples under different preference dimension parameters, including: The multiple preference data samples are processed using preference prediction models corresponding to different preference dimension parameters to obtain the positive sample output probability and negative sample output probability corresponding to the multiple preference data samples respectively; Based on the positive and negative output probabilities of the multiple preference data samples, the sub-preference spacing of the multiple preference data samples under different preference dimension parameters is determined.

5. The method according to claim 4, characterized in that, Based on the positive and negative output probabilities corresponding to the multiple preference data samples, the sub-preference intervals of the multiple preference data samples under different preference dimension parameters are determined, including: Determine the first sample length of the positive output sample and the second sample length of the negative output sample among the plurality of preference data samples; Based on the positive sample output probability, negative sample output probability, first sample length, and second sample length corresponding to the multiple preference data samples, the sub-preference spacing of the multiple preference data samples under different preference dimension parameters is determined.

6. The method according to claim 5, characterized in that, Based on the positive sample output probability, negative sample output probability, first sample length, and second sample length corresponding to the multiple preference data samples, the sub-preference spacing of the multiple preference data samples under different preference dimension parameters is determined, including: Determine the length deviation between the first sample length and the second sample length; Based on the positive sample output probability and negative sample output probability corresponding to the multiple preference data samples respectively, the output probability deviation between the positive sample output probability and the negative sample output probability is determined; Based on the output probability deviation and the length deviation, the sub-preference spacing of the multiple preference data samples under different preference dimension parameters is determined.

7. The method according to claim 2, characterized in that, Based on the sub-preference intervals of the multiple preference data samples under different preference dimension parameters, the consensus index values ​​of the multiple preference data samples under different preference dimension parameters are determined respectively, including: The sub-preference spacing of the multiple preference data samples under different preference dimension parameters is normalized to obtain multiple normalized spacings of the multiple preference data samples under different preference dimension parameters; Based on the multiple normalized intervals of the multiple preference data samples under different preference dimension parameters, the consensus index values ​​of the multiple preference data samples under different preference dimension parameters are determined respectively.

8. The method according to claim 7, characterized in that, Based on the multiple normalized intervals of the multiple preference data samples under different preference dimension parameters, the consensus index values ​​of the multiple preference data samples under different preference dimension parameters are determined respectively, including: The normalized intervals of the same preference data sample on different preference dimension parameters are accumulated to obtain the accumulated interval value corresponding to the preference data sample. Based on the accumulated interval value, the consensus index value of the multiple preference data samples under different preference dimension parameters is determined.

9. The method according to any one of claims 1-8, characterized in that, Determine the corresponding strategy index values ​​for the multiple preference data samples, including: The language model to be trained is used to process the multiple preference data samples to determine the first prediction probability of the positive output sample and the second prediction probability of the negative output sample among the multiple preference data samples. Based on the first prediction probability and the second prediction probability corresponding to the multiple preference data samples respectively, the same strategy index value corresponding to the multiple preference data samples is determined.

10. The method according to claim 9, characterized in that, Based on the first prediction probability and the second prediction probability corresponding to the multiple preference data samples, the same strategy index value corresponding to the multiple preference data samples is determined, including: Based on the first predicted probability and the second predicted probability corresponding to the multiple preference data samples respectively, the average probability of the multiple preference data samples is determined; The value of the same strategy index is determined based on the average probability.

11. The method according to claim 9, characterized in that, The language model to be trained is used to process the plurality of preference data samples to determine the first predicted probability of the positive output sample and the second predicted probability of the negative output sample among the plurality of preference data samples, including: Determine the size of the vocabulary corresponding to the language model to be trained and the lexical information included in the knowledge base; The multiple preference data samples are processed using the language model to be trained, the lexicon size, and the lexical information to determine the first prediction probability of the positive output sample and the second prediction probability of the negative output sample.

12. The method according to any one of claims 1-8, characterized in that, Based on the consensus index value and the same strategy index value corresponding to the multiple preference data samples, target training samples corresponding to the model fine-tuning task are selected from the multiple preference data samples, including: Based on the consensus index value and the same strategy index value corresponding to the multiple preference data samples respectively, the screening index value corresponding to the multiple preference data samples is determined respectively; Based on the screening index values ​​corresponding to the multiple preference data samples, target training samples corresponding to the model fine-tuning task are selected from the multiple preference data samples.

13. The method according to claim 12, characterized in that, The target training samples include training samples from different batches; Based on the screening index values ​​corresponding to the multiple preference data samples, target training samples corresponding to the model fine-tuning task are selected from the multiple preference data samples, including: Determine the sample batches used to limit the target training samples and the number of samples corresponding to the sample batches; Based on the sample batch, the sample quantity, and the screening index values ​​corresponding to the multiple preference data samples, determine the current batch of training samples corresponding to the model fine-tuning task from the multiple preference data samples; Based on the current batch of training samples, fine-tune the language model to be trained, and determine the fine-tuned language model; Using the fine-tuned language model, the next batch of training samples corresponding to the model fine-tuning task is dynamically determined from multiple preference data samples.

14. An electronic device, characterized in that, include: A memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions, when executed by the processor, implement the method of any one of claims 1-13.

15. A computer storage medium, characterized in that, Used to store a computer program that, when executed by a computer, implements the method of any one of claims 1-13.

16. A computer program product, characterized in that, include: A computer program, when executed by a processor of an electronic device, causes the processor to perform the steps of the method of any one of claims 1-13.

Images