Data evaluation method and device, equipment and storage medium

Abstract

The invention provides a data evaluation method and device, equipment and a storage medium, which can be applied to various fields such as model training and data processing. The method comprises the steps of obtaining a to-be-evaluated data set, wherein the data set comprises N pieces of sample data; performing similarity calculation on every two pieces of sample data in the N pieces of sample data to obtain the similarity between every two pieces of sample data; based on the similarity between every two pieces of sample data, determining a probability value that each piece of sample data in the N pieces of sample data belongs to each category in N categories, the N categories being real categories of each piece of sample data in the N pieces of sample data; and determining a diversity evaluation result of the N pieces of sample data based on the probability value that each piece of sample data in the N pieces of sample data belongs to each type in the N types. The diversity of the data set can be evaluated simply and efficiently, the training effect and the training speed of the model can be improved, and computing resources can be saved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a data evaluation method, apparatus, device and storage medium. Background Art

[0002] With the rapid development of artificial intelligence technology, various neural network models play an important role in various fields. Before using the neural network model, it is necessary to train it on the data set. The training effect of the model is closely related to the quality of the data set. Data diversity, as an important part of the quality of the data set, directly affects the performance of the trained model.

[0003] However, the current data set quality assessment methods mainly focus on the overall assessment of data quality. For example, a pre-trained quality assessment model is used to assess the quality level of a data sample, and then the quality assessment level of the entire data set is determined based on the quality level of the data sample. It can be seen that the current data assessment methods focus on the overall assessment of the data set and ignore the assessment of data diversity. If a data set with poor diversity is used for model training, the model training speed will be slow and the training effect will be unsatisfactory. Summary of the invention

[0004] The present application provides a data evaluation method, apparatus, device and storage medium for accurately evaluating multiple samples of a data set, so that a model can be trained using a data set that meets the diversity requirements, which can improve the training effect and training speed of the model and save computing resources.

[0005] In a first aspect, the present application provides a data evaluation method, comprising:

[0006] Obtain a data set to be evaluated, the data set comprising N sample data, where N is a positive integer greater than 1;

[0007] Calculating the similarity between every two sample data among the N sample data to obtain the similarity between every two sample data;

[0008] Based on the similarity between each two sample data, determine the probability value that each sample data in the N sample data belongs to each category in N categories, where the N categories are the true categories of each sample data in the N sample data;

[0009] Based on the probability value that each of the N sample data belongs to each of the N categories, a diversity evaluation result of the N sample data is determined.

[0010] In a second aspect, the present application provides a data evaluation device, comprising:

[0011] An acquisition unit, used to acquire a data set to be evaluated, wherein the data set includes N sample data, where N is a positive integer greater than 1;

[0012] A similarity calculation unit, used to perform similarity calculation on every two sample data among the N sample data to obtain the similarity between every two sample data;

[0013] A classification unit, used for determining, based on the similarity between each two sample data, a probability value that each of the N sample data belongs to each of N categories, where the N categories are the true categories of each of the N sample data;

[0014] An evaluation unit is used to determine a diversity evaluation result of the N sample data based on a probability value that each of the N sample data belongs to each of the N categories.

[0015] In some embodiments, the classification unit is specifically used to determine, for the i-th sample data among the N sample data, the probability value of the i-th sample data belonging to each of the N categories based on the similarity between the i-th sample data and each of the N sample data, where i is a positive integer from 1 to N.

[0016] In some embodiments, the classification unit is specifically used to determine, for the j-th category among the N categories, the probability value of the i-th sample data belonging to the j-th category based on the similarity between the i-th sample data and the j-th sample data, and the similarity between the i-th sample data and each of the N sample data, the j-th category is the true category of the j-th sample data, and j is a positive integer from 1 to N.

[0017] In some embodiments, the classification unit is specifically used to obtain a first value based on the similarity between the i-th sample data and the j-th sample data; obtain a second value based on the similarity between the i-th sample data and each of the N sample data; and determine the probability value that the i-th sample data belongs to the j-th category based on the first value and the second value.

[0018] In some embodiments, the classification unit is specifically used to perform an exponential operation on the similarity between the i-th sample data and the j-th sample data to obtain the first value; perform an exponential operation on the similarity between the i-th sample data and each of the N sample data, and add the results of the exponential operation to obtain the second value.

[0019] In some embodiments, the evaluation unit is specifically used to select, for the i-th sample data among the N sample data, from the probability values ​​of the i-th sample data belonging to each of the N categories, the probability value of the i-th sample data belonging to the i-th category, where the i-th category is the true category of the i-th sample data; determine the diversity evaluation value of the N sample data based on the probability value that each sample data among the N sample data belongs to the corresponding true category; and determine the diversity evaluation result of the N sample data based on the diversity evaluation value.

[0020] In some embodiments, the evaluation unit is specifically configured to add the probability value that each of the N sample data belongs to the corresponding true category to obtain the diversity evaluation value of the N sample data.

[0021] In some embodiments, the evaluation unit is specifically used to determine that the N data samples have diversity if the diversity evaluation value is greater than or equal to a preset threshold; if the diversity evaluation value is less than the preset threshold, determine that the N sample data do not have diversity.

[0022] In some embodiments, the sample data includes first data and second data, the first data is input data of a language model, and the second data is data obtained after the language model processes the first data.

[0023] In some embodiments, the similarity calculation unit is specifically used to obtain the downstream task type corresponding to the N sample data; for each sample data among the N sample data, based on the downstream task type, determine the diversity sensitive data corresponding to the sample data from the first data and the second data included in the sample data; perform similarity calculation on the diversity sensitive data corresponding to each two sample data among the N sample data, to obtain the similarity between the each two sample data.

[0024] In some embodiments, the similarity calculation unit is specifically used to extract the feature information of the diversity sensitive data corresponding to the kth sample data and the pth sample data among the N sample data, and extract the feature information of the diversity sensitive data corresponding to the pth sample data, where both k and p are positive integers less than or equal to N; perform similarity calculation on the feature information of the diversity sensitive data corresponding to the kth sample data and the feature information of the diversity sensitive data corresponding to the pth sample data to obtain the similarity between the kth sample data and the pth sample data.

[0025] In some embodiments, the similarity calculation unit is specifically used to select a feature extractor that matches the downstream task type based on the downstream task type; through the feature extractor, the feature information of the diversity sensitive data corresponding to the kth sample data and the feature information of the diversity sensitive data corresponding to the pth sample data are extracted.

[0026] In a third aspect, the present application provides an electronic device, comprising a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the method of the first aspect.

[0027] In a fourth aspect, a chip is provided for implementing the method in each implementation of the first aspect. Specifically, the chip includes: a processor for calling and running a computer program from a memory so that a device equipped with the chip executes the method in the first aspect.

[0028] In a fifth aspect, a computer-readable storage medium is provided for storing a computer program, wherein the computer program enables a computer to execute the method of the first aspect.

[0029] In a sixth aspect, a computer program product is provided, comprising computer program instructions, wherein the computer program instructions enable a computer to execute the method of the first aspect.

[0030] In a seventh aspect, a computer program is provided, which, when executed on a computer, enables the computer to execute the method of the first aspect.

[0031] In summary, the present application obtains a data set to be evaluated, which includes N sample data, where N is a positive integer greater than 1; performs similarity calculation on every two sample data in the N sample data to obtain the similarity between every two sample data; based on the similarity between every two sample data, determines the probability value that each sample data in the N sample data belongs to each category in the N categories, and the N categories are the true categories of each sample data in the N sample data; based on the probability value that each sample data in the N sample data belongs to each category in the N categories, determines the diversity evaluation result of the N sample data. It can be seen from this that the embodiment of the present application determines the probability value that each sample data belongs to N categories by determining the similarity between every two sample data in the data set, and then realizes the overall evaluation of the diversity of the data set from the perspective of sample classification. The obtained diversity value can be understood as the number of valid samples in the data set, so it has physical meaning. In addition, the embodiment of the present application determines the diversity evaluation result of the N sample data based on the probability value that the N sample data belong to N categories respectively, and its calculation speed is fast and efficient, and can realize the rapid and efficient evaluation of large data sets. During model training, the method of the embodiment of the present application can be used to quickly and accurately evaluate the diversity of the data set, and then the model can be trained using the data set whose diversity meets the requirements, which can improve the training effect and training speed of the model and save computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0033] Figure 1 A schematic diagram of an implementation environment for providing a data evaluation method for an embodiment of the present application;

[0034] Figure 2 A flowchart of a data evaluation method provided in one embodiment of the present application;

[0035] Figure 3 An example diagram of an implementation process for selecting diverse sensitive data of N sample data and determining characteristic information of the diverse sensitive data;

[0036] Figure 4 A schematic diagram of a classification probability matrix involved in an embodiment of the present application;

[0037] Figure 5 A flowchart of a data evaluation method provided in one embodiment of the present application;

[0038] Figure 6 A schematic diagram of a data evaluation method provided in an embodiment of the present application;

[0039] Figure 7 A schematic diagram of a model training method flow chart provided in one embodiment of the present application;

[0040] Figure 8 is a schematic block diagram of a data evaluation device provided in one embodiment of the present application;

[0041] Fig. 9 It is a schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0043] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In an embodiment of the present invention, "B corresponding to A" means that B is associated with A. In one implementation, B can be determined according to A. However, it should also be understood that determining B according to A does not mean that B is determined only according to A, but B can also be determined according to A and / or other information. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server that includes a series of steps or units does not have to be limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. In the description of the present application, unless otherwise specified, "multiple" refers to two or more than two.

[0044] The data evaluation method provided in the embodiment of the present application can be applied to various fields such as model training and data processing, and can evaluate the diversity of data sets, thereby improving the training effect of the model.

[0045] Large Language Models (LLMs): A language model consisting of an artificial neural network with many parameters (typically billions of weights or more), trained on large amounts of unlabeled text using self-supervised or semi-supervised learning.

[0046] Datasets Diversity: refers to the richness and differences of samples in the training data set of the neural network model. The better the diversity, the higher the richness of the samples.

[0047] With the rapid development of artificial intelligence technology, various neural network models play an important role in various fields. Before using the neural network model, it is necessary to train it on the data set. The training effect of the model is closely related to the quality of the data set. As an important part of the data set quality, data diversity directly affects the performance of the trained model. However, the current data set quality evaluation method mainly focuses on the overall evaluation of data quality. For example, the quality level of the data sample is evaluated using a pre-trained quality evaluation model, and then the quality evaluation level of the entire data set is comprehensively determined based on the quality level of the data sample. It can be seen that the current data evaluation method focuses on the overall evaluation of the data set and ignores the evaluation of data diversity.

[0048] In order to solve the above technical problems, the embodiment of the present application proposes a data evaluation method, by obtaining a data set to be evaluated, the data set includes N sample data, N is a positive integer greater than 1; similarity calculation is performed on each two sample data in the N sample data to obtain the similarity between each two sample data; based on the similarity between each two sample data, the probability value of each sample data in the N sample data belonging to each category in the N categories is determined, and the N categories are the true categories of each sample data in the N sample data; based on the probability value of each sample data in the N sample data belonging to each category in the N categories, the diversity evaluation result of the N sample data is determined. It can be seen that the embodiment of the present application determines the probability value of each sample data belonging to N categories by determining the similarity between each two sample data in the data set, and then realizes the overall evaluation of the diversity of the data set from the perspective of sample classification. The obtained diversity value can be understood as the number of valid samples in the data set, so it has physical meaning. In addition, the embodiment of the present application determines the diversity evaluation result of the N sample data based on the probability value that the N sample data belong to the N categories respectively, and its calculation speed is fast and efficient, and it can realize the fast and efficient evaluation of large data sets. During model training, the diversity of the data set is quickly and accurately evaluated using the method of the embodiment of the present application, and then the model is trained using the data set whose diversity meets the requirements. This can improve the training effect and training speed of the model and save computing resources.

[0049] The following is an introduction to the implementation environment of the data evaluation method provided in the embodiment of the present application.

[0050] Figure 1 A schematic diagram of an implementation environment of a data evaluation method is provided for an embodiment of the present application, such as Figure 1 As shown, the implementation environment includes: the implementation environment includes: a terminal device 101 and a server 102.

[0051] The terminal device 101 is connected to the server 102 via a wired or wireless connection.

[0052] In some embodiments, the terminal device 101 is installed with a client of the data evaluation platform, and the server 102 can be understood as the server or backend of the data evaluation platform. The data evaluation platform can evaluate the diversity of the data set. In an embodiment of the present application, an object (such as a user) can interact with the client through the terminal device 101, for example, inputting a prompt word of the text to be generated on the client. The language model can be called in the server 102 to generate text.

[0053] In some embodiments, the data set of the embodiment of the present application can be collected by an object. In some embodiments, the data set of the embodiment of the present application is generated by a neural network model, for example, a large language model.

[0054] In some embodiments, during the data evaluation process, the object can send the data set to be evaluated to the server 102 through the terminal device 101. For example, the object starts the client of the data evaluation platform on the terminal device 101, and the client displays the input box of the data set to be evaluated. The object enters the data set to be evaluated in the input box, and the data set includes N sample data. The terminal device 101 sends the data set to be evaluated input by the object to the server 102. After the server 102 obtains the data set to be evaluated, it calculates the similarity of each two sample data in the N sample data included in the data set to obtain the similarity between each two sample data. Then, based on the similarity between each two sample data, the server 102 determines the probability value that each sample data in the N sample data belongs to each category in the N categories, where the N categories are the true categories of each sample data in the N sample data. In this way, the server 102 determines the diversity evaluation results of the N sample data based on the probability value that each sample data in the N sample data belongs to each category in the N categories.

[0055] In some embodiments, the data evaluation method of the embodiment of the present application can be completed by the terminal device 101. For example, the object starts the client of the data evaluation platform on the terminal device 101, and the client displays the input box of the data set to be evaluated. The object enters the data set to be evaluated in the input box, and the data set includes N sample data. After the terminal device 101 obtains the data set to be evaluated, it calculates the similarity of every two sample data in the N sample data included in the data set to obtain the similarity between every two sample data. Then, based on the similarity between every two sample data, the terminal device 101 determines the probability value that each sample data in the N sample data belongs to each category in the N categories, where the N categories are the true categories of each sample data in the N sample data. In this way, the terminal device 101 determines the diversity evaluation results of the N sample data based on the probability value that each sample data in the N sample data belongs to each category in the N categories.

[0056] It can be seen from this that the embodiment of the present application determines the probability value that each sample data belongs to N categories by determining the similarity between every two sample data in the data set, and then realizes the overall evaluation of the diversity of the data set from the perspective of sample classification. The obtained diversity value can be understood as the number of valid samples in the data set, so it has a physical meaning. In addition, the embodiment of the present application determines the diversity evaluation result of N sample data based on the probability value that N sample data belong to N categories respectively. It has fast calculation speed and high efficiency, and can realize fast and efficient evaluation of large data sets. During model training, the diversity of the data set is quickly and accurately evaluated using the method of the embodiment of the present application, and then the model is trained using a data set whose diversity meets the requirements, which can improve the training effect and training speed of the model, and can save computing resources.

[0057] In some embodiments, the terminal device 101 includes but is not limited to: desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. Terminal devices are often equipped with display devices, which can also be displays, display screens, touch screens, etc., and touch screens can also be touch screens, touch panels, etc.

[0058] In some embodiments, the server 102 may be one or more servers. When there are multiple servers, there are at least two servers for providing different services, and / or there are at least two servers for providing the same service, such as providing the same service in a load balancing manner, which is not limited in the embodiments of the present application. Among them, the server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The server can also become a node of the blockchain.

[0059] It should be noted that the implementation environment of the embodiments of the present application includes but is not limited to Figure 1 shown.

[0060] The technical solutions of the embodiments of the present application are described in detail below through some embodiments. The following embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0061] Figure 2 The flowchart of the data evaluation method provided in one embodiment of the present application is shown in FIG. The execution subject of the embodiment of the present application may be a device having a data evaluation function, such as a data evaluation device. The data evaluation device may be the above-mentioned Figure 1 The server 102 or terminal device 101 shown may also be Figure 1 The server 102 and the terminal device 101 shown in the figure are composed of a system. For the convenience of description, the following embodiment takes an electronic device as an example to illustrate the method of the embodiment of the present application.

[0062] like Figure 2 As shown, the data evaluation method of the embodiment of the present application includes:

[0063] S101: Obtain a data set to be evaluated.

[0064] The data set includes N sample data, where N is a positive integer greater than 1.

[0065] It should be noted that the method of obtaining data during the data evaluation process in the embodiments of the present application and the data obtained are in compliance with the provisions of relevant laws and regulations.

[0066] With the use of data sets in model training, some recent work has pointed out the impact of the diversity of generated data sets on model performance or expressiveness. The data evaluation method provided in the embodiment of the present application is used to evaluate the data diversity of a data set.

[0067] The embodiment of the present application does not limit the specific data type of the sample data included in the data set, for example, it can be image data, text data, video data, etc.

[0068] The embodiment of the present application does not limit the specific method of obtaining the data set.

[0069] In some embodiments, the data set of the embodiments of the present application can be any existing data set. In some embodiments, the data set can also be called a training set.

[0070] In some embodiments, part or all of the N sample data included in the data set of the embodiments of the present application are sample data collected from the object.

[0071] In some embodiments, some or all of the N sample data included in the data set of the embodiment of the present application are generated by the neural network model. For example, with the excellent content generation ability of the large language model, the large language model can be used as a data generator or data enhancer to construct synthetic sample data (such as text data). This type of sample data is also called a large language model (LLMs) data set.

[0072] In some embodiments, among the N sample data included in the data set of the embodiment of the present application, some of the sample data are sample data collected from the object, and some of the sample data are sample data generated by the large language model.

[0073] In some embodiments, the sample data included in the data set of the embodiment of the present application is unlabeled sample data. Exemplarily, the sample data can be represented as Among them, x i Input data to the model during model training.

[0074] In some embodiments, the sample data included in the data set of the embodiment of the present application is labeled sample data. Exemplarily, the sample data can be represented as Among them, x i The model input data for model training, y i For the input data x i The label data of the model, or the output data of the model. In a possible example, if the sample data is sample data generated by a large language model, then the dataset to be evaluated is N sample data (or input-output pairs) Composition, of which x i is the input data of the large language model, y i For x iThe output data of the corresponding large language model, that is, when using the large language model to generate a data set, the data x i Input into the large language model, the large language model performs i Process and output data y i , and then the data x i and data y i As a sample data.

[0075] S102: Calculate the similarity between every two sample data among the N sample data to obtain the similarity between every two sample data.

[0076] In the embodiment of the present application, the electronic device obtains the data set to be evaluated based on the above steps, and then calculates the similarity between every two sample data in the N sample data included in the data set to obtain the similarity between every two sample data in the N sample data.

[0077] For example, for each sample data in N sample data, such as the i-th sample data, the similarity between the i-th sample data and each sample data in the N sample data is calculated. Specifically, the similarity between the i-th sample data and itself, and the similarity between the i-th sample data and other sample data in the N sample data except the i-th sample data are calculated.

[0078] The embodiment of the present application does not limit the specific manner in which the electronic device calculates the similarity between each two sample data.

[0079] In some embodiments, the electronic device may calculate the similarity between each two sample data by calculating the distance between each two sample data, wherein the distance between each two sample data may be calculated by any distance such as cosine distance and Euclidean distance.

[0080] In some embodiments, the electronic device may also predict the similarity between every two sample data using a trained similarity prediction model.

[0081] In some embodiments, for each sample data of the N sample data, the electronic device first extracts feature information of each sample data before calculating the similarity between each two sample data. The feature information is a vector, so that the similarity between each two sample data can be determined based on the feature information of each two sample data.

[0082] Example 1: The electronic device calculates the inner product between the feature information of each two sample data, and then determines the similarity between each two sample data based on the inner product between the feature information of each two sample data. For example, the inner product between the feature information of each two sample data is determined as the similarity between each two sample data.

[0083] Exemplarily, the electronic device calculates the inner product between the feature information of each two sample data by the following formula (1):

[0084] a·b=a 1 b 1 +a 2 b 2 +…+a n b n (1)

[0085] Among them, a=(a 1 ,a 2 ,…,a n ) is the characteristic information of a sample data among N sample data, b=(b 1 ,b 2 ,…,b n ) is the feature information of another sample data in N sample data. a·b is the dot product between the feature information of two sample data.

[0086] Example 2: The electronic device calculates the distance (e.g., cosine distance) between the feature information of each two sample data, and then determines the similarity between each two sample data based on the distance between the feature information of each two sample data. For example, the cosine distance between the feature information of each two sample data is determined as the similarity between each two sample data.

[0087] Exemplarily, the electronic device calculates the cosine distance between the feature information of each pair of sample data by the following formula (2):

[0088] cos(a,b)=(a·b) / (|a||b|)(2)

[0089] Among them, cos(a,b) is the cosine distance between the feature information of two sample data, (a·b) is the dot product between the feature information of two sample data, |.| is the calculation method of vector length, and |a||b| is the product of the lengths of the feature information of two sample data.

[0090] Example 3: The electronic device calculates the similarity between two pairs of sample data through a kernel function. Specifically, for any two sample data of N sample data, the electronic device maps the feature information of the two sample data from low-dimensional data to a high-dimensional space through a kernel function, so that the data that was originally not linearly separable in the high-dimensional space becomes linearly separable. In the high-dimensional space, the inner product between the mapped data is calculated, and this inner product value reflects the similarity between the two sample data.

[0091] Exemplarily, the electronic device calculates the similarity between every two sample data using the following formula (3):

[0092] k=Kernel(a,b) (3)

[0093] Wherein, Kernel (,) is a kernel function, and the embodiment of the present application does not limit the specific type of the kernel function, and can be selected according to actual needs. K is the similarity between two sample data.

[0094] It can be seen from the above that the sample data included in the data set of the embodiment of the present application can be unlabeled sample data or labeled sample data.

[0095] In some embodiments, if the sample data of the embodiment of the present application is a labeled sample data generated by a language model, the sample data includes first data x and second data y, wherein the first data is the input data of the language model, and the second data is the data obtained after the language model processes the first data. In some embodiments, the second data can be called the label data of the first data. At this time, the data set to be evaluated obtained by the electronic device in the embodiment of the present application can be expressed as N sample data (or input-output pairs) Composition, of which x i is the first data, i.e., the input data of the input language model, y i is the second data, that is, the language model pair x i The data output after processing. For example, x i For a review of a movie, i is the sentiment implied in the movie review (good or bad).

[0096] In an embodiment of the present application, if the sample data included in the data set includes first data and second data, the electronic device can use the above method of calculating the similarity between every two sample data to calculate the similarity between every two sample data in N sample data.

[0097] In some embodiments, if the sample data included in the data set includes first data and second data, in order to reduce the amount of data processing, the electronic device may select part of the sample data to calculate the similarity. Based on this, in a possible implementation, the electronic device may determine the similarity between each two sample data through the following steps S102-A to S102-C:

[0098] S102-A, obtaining downstream task types corresponding to N sample data;

[0099] S102-B, for each sample data among the N sample data, based on the downstream task type, determining the diversity sensitive data corresponding to the sample data from the first data and the second data included in the sample data;

[0100] S102-C: perform similarity calculation on the diversity sensitive data corresponding to every two sample data among the N sample data to obtain the similarity between every two sample data.

[0101] In the embodiment of the present application, a data set is used to train the model of the downstream task. For example, if the downstream task is to classify text, the data set of the embodiment of the present application needs to be used to train the text classification model. However, before training the text classification model, it is first necessary to determine the data diversity of the data set. If the diversity of the data set meets the requirements, the data set is used to train the text classification model, so that the trained text classification model has strong robustness and can accurately classify different types of text. If the diversity of the data set does not meet the requirements, other data sets are used to train the text classification model.

[0102] Based on this, in this implementation, the electronic device obtains the downstream task type corresponding to the N sample data included in the data set, and then based on the downstream task type, determines the diversity sensitive data corresponding to each sample data from the first data and the second data included in each sample data in the N sample data. Among them, the diversity sensitive data can be understood as the data that is more sensitive to the downstream task in the first data and the second data included in the sample data. For example, in the text classification downstream task movie review scenario, x i For a review of a movie, i is the implicit emotion (good or bad) in the movie review. In this scenario, y is not important. i The diversity of x i That is the diversity sensitive component.

[0103] Based on the above steps, the electronic device determines the diversity sensitive data corresponding to each sample data in the N sample data. When the diversity is subsequently calculated, the input of the method is Diversity sensitive components Specifically, similarity calculation is performed on the diversity sensitive data corresponding to every two sample data among the N sample data to obtain the similarity between every two sample data.

[0104] The embodiment of the present application does not limit the specific manner in which the electronic device calculates the similarity between the diversity sensitive data corresponding to every two sample data.

[0105] In some embodiments, the electronic device may calculate the similarity between the diversity sensitive data corresponding to each two sample data by calculating the distance between the diversity sensitive data corresponding to each two sample data. The distance between the diversity sensitive data corresponding to each two sample data may be calculated by any distance such as cosine distance and Euclidean distance.

[0106] In some embodiments, the electronic device may also predict the similarity between the diversity sensitive data corresponding to every two sample data using a trained similarity prediction model.

[0107] In some embodiments, the electronic device determines the similarity between every two sample data through the following steps S102-C1 and S102-C2:

[0108] S102-C1. For the k-th sample data and the p-th sample data among the N sample data, extract feature information of the diversity sensitive data corresponding to the k-th sample data, and extract feature information of the diversity sensitive data corresponding to the p-th sample data, where k and p are both positive integers less than or equal to N.

[0109] S102-C2, performing similarity calculation on feature information of the diversity sensitive data corresponding to the k-th sample data and feature information of the diversity sensitive data corresponding to the p-th sample data, to obtain the similarity between the k-th sample data and the p-th sample data.

[0110] In this implementation, when determining the similarity between the diversity sensitive data corresponding to every two sample data among the N sample data, the electronic device first extracts the feature information of the diversity sensitive data corresponding to each sample data, and then determines the similarity between the diversity sensitive data corresponding to every two sample data based on the feature information of the diversity sensitive data corresponding to each sample data. Then, based on the similarity between the diversity sensitive data corresponding to every two sample data, the similarity between every two sample data is determined. For example, the similarity between the diversity sensitive data corresponding to every two sample data is determined as the similarity between every two sample data.

[0111] In the embodiment of the present application, the method of determining the similarity between every two sample data in N sample data is basically the same. For the convenience of description, the kth sample data and the pth sample data in the N sample data are taken as an example for explanation.

[0112] In this implementation, for the kth sample data and the pth sample data among N sample data, the electronic device first uses a feature extractor to extract feature information of the diversity sensitive data corresponding to the kth sample data and feature information of the diversity sensitive data corresponding to the pth sample data.

[0113] The embodiment of the present application does not limit the specific type of the feature extractor, and can be any trained feature extractor used to extract feature information of text.

[0114] In some embodiments, in the feature extraction stage, the present application embodiment is committed to obtaining feature information of sample data through a feature extractor that can comprehensively consider multiple factors, where the multiple factors include semantics, style, theme, style, etc. between samples. Due to the comprehensive feature information of multiple factors, the diversity evaluation value calculated in the subsequent steps also refers to multiple factors.

[0115] In some embodiments, the electronic device selects a feature extractor that matches the downstream task type based on the downstream task type corresponding to the data set, and then extracts feature information of the diversity sensitive data corresponding to each sample data through the feature extractor. For example, the feature extractor extracts feature information of the diversity sensitive data corresponding to the kth sample data, and extracts feature information of the diversity sensitive data corresponding to the pth sample data.

[0116] For example, if the downstream task type of the embodiment of the present application needs to focus on the differences between the semantics of the samples, a model that can better capture the semantic information can be selected as the feature extractor. For example, Sentence Transformer such as Sentence-Bert and SimCSE can be selected as the feature extractor of the embodiment of the present application. Figure 3 As shown, the electronic device obtains N sample data Determine the diversity sensitive data corresponding to each sample data The diverse sensitive data corresponding to the N sample data The input is input into the feature extractor for feature extraction to obtain feature information of diverse sensitive data corresponding to N sample data.

[0117] Exemplarily, the electronic device obtains feature information H of the diverse sensitive data corresponding to N sample data. This process can be summarized as the following formula (4):

[0118]

[0119] in, are the diverse sensitive data corresponding to the N sample data respectively, and the feature extractor Φ is a feature extractor matching the downstream task type of the embodiment of the present application.

[0120] In the embodiment of the present application, the electronic device extracts the feature information of the diversity sensitive data corresponding to each sample data in the N sample data based on the above steps, and then calculates the similarity between the feature information of the diversity sensitive data corresponding to every two sample data. For example, the feature information of the diversity sensitive data corresponding to the kth sample data and the feature information of the diversity sensitive data corresponding to the pth sample data are similarly calculated to obtain the similarity between the kth sample data and the pth sample data.

[0121] In the embodiment of the present application, the electronic device performs similarity calculation on feature information of diversity sensitive data corresponding to the kth sample data and feature information of diversity sensitive data corresponding to the pth sample data, and obtains the similarity between the kth sample data and the pth sample data without limiting the specific method.

[0122] In one example, the electronic device calculates the inner product between the feature information of the diversity sensitive data corresponding to the kth sample data and the feature information of the diversity sensitive data corresponding to the pth sample data, and then determines the similarity between the kth sample data and the pth sample data based on the inner product between the feature information of the diversity sensitive data corresponding to the kth sample data and the feature information of the diversity sensitive data corresponding to the pth sample data. For example, the inner product between the feature information of the diversity sensitive data corresponding to the kth sample data and the feature information of the diversity sensitive data corresponding to the pth sample data is determined as the similarity between the kth sample data and the pth sample data.

[0123] In one example, the electronic device calculates the cosine distance between the feature information of the diversity sensitive data corresponding to the kth sample data and the feature information of the diversity sensitive data corresponding to the pth sample data, and then determines the similarity between the kth sample data and the pth sample data based on the cosine distance between the feature information of the diversity sensitive data corresponding to the kth sample data and the feature information of the diversity sensitive data corresponding to the pth sample data. For example, the cosine distance between the feature information of the diversity sensitive data corresponding to the kth sample data and the feature information of the diversity sensitive data corresponding to the pth sample data is determined as the similarity between the kth sample data and the pth sample data.

[0124] In one example, the electronic device processes the feature information of the diversity sensitive data corresponding to the kth sample data and the feature information of the diversity sensitive data corresponding to the pth sample data through a kernel function to obtain the similarity between the kth sample data and the pth sample data. Specifically, the electronic device maps the feature information of the diversity sensitive data corresponding to the kth sample data and the feature information of the diversity sensitive data corresponding to the pth sample data from low-dimensional data to a high-dimensional space through a kernel function, and calculates the inner product between the mapped data in the high-dimensional space to obtain the similarity between the kth sample data and the pth sample data.

[0125] Exemplarily, the process of using a kernel function to process the feature information H of the diversity sensitive data corresponding to N sample data to obtain the similarity between every two sample data in the N sample data can be expressed as shown in the following formula (5):

[0126] K=Kernel(H) (5)

[0127] Among them, Kernel represents the kernel function used, and the kernel function Kernel can be selected according to needs. K is a pairwise similarity matrix formed by the similarity between every two sample data in N sample data. The size of the pairwise similarity matrix is ​​N*N, and the value of the element ij represents the similarity between the i-th template data and the j-th sample data.

[0128] After the electronic device determines the similarity between every two sample data based on the above steps, it executes the following step S103.

[0129] S103 : Based on the similarity between every two sample data, determine the probability value that each sample data in the N sample data belongs to each category in the N categories.

[0130] Among them, the N categories are the true categories of each sample data in the N sample data.

[0131] In the embodiment of the present application, each of the N sample data has a real category, thereby forming N categories.

[0132] The embodiment of the present application does not limit the specific method of determining the true category of each sample data in the N sample data.

[0133] In a possible implementation, the true category of each sample data in the N sample data may be labeled by manual labeling.

[0134] In a possible implementation, the true category of each sample data among the N sample data may be determined by using a pre-trained classification model.

[0135] In a possible implementation, the N sample data themselves are determined as their respective true categories. For example, the true category of the i-th sample data is the i-th sample data itself.

[0136] In an embodiment of the present application, the electronic device determines the similarity between every two sample data in the N sample data based on the above steps, and then determines the probability value of each sample data in the N sample data belonging to each of the N categories based on the similarity between every two sample data.

[0137] The embodiment of the present application does not limit the specific manner in which the electronic device determines the probability value of each of N sample data belonging to each of N categories based on the similarity between each two sample data.

[0138] In some embodiments, the electronic device inputs the similarity between each two sample data determined above, that is, the pairwise similarity matrix K, into a pre-trained classification model for separate predictions to obtain the probability value that each of the N sample data belongs to each of the N categories.

[0139] In some embodiments, the above S103 includes the following step S103-A:

[0140] S103-A. For the i-th sample data among N sample data, based on the similarity between the i-th sample data and each sample data among the N sample data, determine the probability value of the i-th sample data belonging to each category among the N categories, where i is a positive integer from 1 to N.

[0141] In this implementation, for each sample data in the N sample data, for example, the i-th sample data, the electronic device determines the probability value that the i-th sample data belongs to each of the N categories based on the similarity between the i-th sample data and each of the N sample data. In this way, the electronic device can determine the probability value that each sample data in the N sample data belongs to each of the N categories.

[0142] The embodiment of the present application does not limit the specific manner in which the electronic device determines the probability value of the i-th sample data belonging to each of the N categories based on the similarity between the i-th sample data and each of the N sample data. The greater the similarity between the i-th sample data and the j-th sample data, the greater the probability value that the i-th sample data belongs to the category corresponding to the j-th sample data.

[0143] In some embodiments, the electronic device determines the similarity between the i-th sample data and each of the N sample data as a probability value that the i-th sample data belongs to the true category of the N sample data. For example, the similarity between the i-th sample data and the j-th sample data is determined as a probability value that the i-th sample data belongs to the j-th category, where the j-th category is the true category of the i-th sample data among the N categories.

[0144] In some embodiments, the above S103-A includes the following step S103-A1:

[0145] S103-A1. For the j-th category among N categories, based on the similarity between the i-th sample data and the j-th sample data, and the similarity between the i-th sample data and each of the N sample data, determine the probability value that the i-th sample data belongs to the j-th category, the j-th category is the true category of the j-th sample data, and j is a positive integer from 1 to N.

[0146] In this implementation, the electronic device determines the probability value that the i-th sample data belongs to each of the N categories in a basically consistent manner, and the j-th category among the N categories is used as an example for explanation, where the j-th category is the true category of the j-th sample data. When the electronic device determines that the i-th sample data belongs to the j-th category, it determines the probability value that the i-th sample data belongs to the j-th category based on the similarity between the i-th sample data and the j-th sample data, and the similarity between the i-th sample data and each of the N sample data.

[0147] The embodiment of the present application does not limit the specific manner in which the electronic device determines the probability value that the i-th sample data belongs to the j-th category based on the similarity between the i-th sample data and the j-th sample data, and the similarity between the i-th sample data and each of the N sample data.

[0148] In one possible implementation, the electronic device determines the sum of the similarities between the i-th sample data and each of the N sample data, and then determines the ratio of the similarity between the i-th sample data and the j-th sample data to the sum as the probability value that the i-th sample data belongs to the j-th category.

[0149] In a possible implementation, the electronic device obtains a first value based on the similarity between the i-th sample data and the j-th sample data; obtains a second value based on the similarity between the i-th sample data and each of the N sample data; and determines the probability value that the i-th sample data belongs to the j-th category based on the first value and the second value.

[0150] For example, the electronic device performs a preset linear operation or nonlinear operation on the similarity between the i-th sample data and the j-th sample data to obtain a first value, performs a preset linear operation or nonlinear operation on the similarity between the i-th sample data and each of the N sample data to obtain a second value, and then determines the probability value that the i-th sample data belongs to the j-th category based on the first value and the second value.

[0151] In one example, the electronic device performs an exponential operation on the similarity between the ith sample data and the jth sample data to obtain a first value.

[0152] For example, the electronic device determines the first value D1 by the following formula (6):

[0153] D1=exp(K[i,j] / τ)(6)

[0154] Among them, K[i,j] represents the similarity between the i-th sample data and the j-th sample data, and τ is the temperature hyperparameter that controls the classification resolution. A smaller τ will amplify the difference between the sample data, which means a lower classification resolution. exp is an exponential operation with base e.

[0155] In one example, the electronic device performs an exponential operation on the similarity between the ith sample data and each of the N sample data, and adds the exponential operation results to obtain a second value.

[0156] For example, the electronic device determines the first value D2 by the following formula (7):

[0157] D1=∑ k exp(K[i,k] / τ)(7)

[0158] Wherein, K[i,k] represents the similarity between the i-th sample data and the k-th sample data, and the value of k ranges from 1 to N. exp is an exponential operation with base e.

[0159] After the electronic device determines the first value and the second value based on the above steps, it determines the probability value that the i-th sample data belongs to the j-th category based on the first value and the second value. For example, the electronic device determines the ratio of the first value and the second value as the probability value that the i-th sample data belongs to the j-th category. For another example, the electronic device multiplies the ratio of the first value and the second value by a preset parameter to obtain the probability value that the i-th sample data belongs to the j-th category.

[0160] In one example, the electronic device determines the probability value that the i-th sample data belongs to the j-th category by using the following formula (8):

[0161]

[0162] Among them, c j is the jth category among N categories, P[i,j] is the probability value that the i-th sample data belongs to the j-th category, K[i,j] is the similarity between the i-th sample data and the j-th sample data, f K (·) is the classification function, exp(K[i,j] / τ) is the first value corresponding to the i-th sample data, ∑ k exp(K[i,k] / τ) is the second value corresponding to the i-th sample data.

[0163] The above is an introduction to the specific process of an electronic device determining the probability value of the i-th sample data belonging to the j-th category. Referring to this method, the electronic device can determine the probability value of each sample data in N sample data belonging to each category in N categories, and obtain a classification probability matrix P. The size of the classification probability matrix P is N*N, where the ij-th element represents the probability value of the i-th sample data belonging to the j-th category.

[0164] S104: Determine a diversity evaluation result of the N sample data based on a probability value that each of the N sample data belongs to each of the N categories.

[0165] In an embodiment of the present application, if the data set to be evaluated exhibits a high diversity, the probability that the i-th sample data is classified into the i-th category corresponding to the i-th sample data is much greater than the probability of being classified into other categories. Based on this, the embodiment of the present application can obtain the diversity evaluation result of the data set by counting the probability value of the i-th sample data being classified into the i-th category.

[0166] In some embodiments, the electronic device determines the diversity evaluation results of N sample data through the following steps S104-A to S104-C:

[0167] S104-A, for the i-th sample data among the N sample data, select the probability value of the i-th sample data belonging to the i-th category from the probability values ​​of the i-th sample data belonging to each category among the N categories, and the i-th category is the true category of the i-th sample data;

[0168] S104-B, determining a diversity evaluation value of the N sample data based on a probability value that each of the N sample data belongs to a corresponding true category;

[0169] S104-C. Based on the diversity evaluation value, determine the diversity evaluation results of the N sample data.

[0170] In this implementation, for each sample data among the N sample data, for example, the i-th sample data, the electronic device selects the probability value of the i-th sample data belonging to the i-th category from the probability values ​​of the i-th sample data belonging to each category among the N categories determined above. In this way, the electronic device can determine the probability value of each sample data among the N sample data belonging to the corresponding true category. Then, the electronic device determines the diversity evaluation value of the N sample data based on the probability value of each sample data among the N sample data belonging to the corresponding true category.

[0171] The embodiment of the present application does not limit the manner in which the electronic device determines the diversity evaluation value of N sample data based on the probability value that each sample data in the N sample data belongs to the corresponding real category.

[0172] In some embodiments, the electronic device may determine the product of the probability values ​​that each of the N sample data belongs to the corresponding true category as the diversity evaluation value of the N sample data.

[0173] In some embodiments, the electronic device may add up the probability values ​​that each of the N sample data belongs to the corresponding true category to obtain the diversity evaluation values ​​of the N sample data.

[0174] As can be seen from the above, in the embodiment of the present application, the probability value of each sample data in the N sample data belonging to each category in the N categories constitutes a classification probability matrix P, such as Figure 4 As shown, the size of the classification probability matrix P is N*N, where the ijth element represents the probability value that the ith sample data belongs to the jth category. The elements on the diagonal of the classification probability matrix P represent the probability value that the ith sample data belongs to the ith category. For example, the probability value of the first sample data of the N sample data belonging to the first category is 0.88, the probability value of the second sample data belonging to the second category is 0.98, and the probability value of the Nth sample data belonging to the Nth category is 0.77. In this way, the electronic device calculates the trace of the classification probability matrix P, that is, adds the elements on the diagonal of the classification probability matrix P to obtain the diversity evaluation value of the N sample data.

[0175] Exemplarily, the electronic device obtains the diversity evaluation value of N sample data based on the following formula (9):

[0176] Among them, P[i,i] represents the probability value that the i-th sample data belongs to the i-th category, and tr(P) is the trace of the classification probability matrix.

[0177] After the electronic device determines the diversity evaluation value of the N sample data based on the above steps, it determines the diversity evaluation result of the N sample data based on the diversity evaluation value of the N sample data. For example, the determined diversity evaluation value of the N sample data is compared with a preset threshold to determine the diversity evaluation result of the N sample data.

[0178] In one example, if the diversity evaluation values ​​of the N data samples are greater than or equal to a preset threshold, it is determined that the N data samples have diversity.

[0179] In another example, if the diversity evaluation value of the N data samples is less than a preset threshold, it is determined that the N sample data do not have diversity.

[0180] The embodiments of the present application do not limit the specific value of the preset threshold. In one example, if the diversity evaluation value of N sample data is obtained by adding the probability value of each sample data in the N sample data belonging to the corresponding true category, and the probability value of each sample data in the N sample data belonging to the corresponding true category does not exceed the value 1, based on this, the diversity evaluation value of the N sample data is a value not exceeding N, and therefore, the preset threshold is a threshold value less than or equal to N, such as 0.98, 0.9 or 0.85, etc.

[0181] In some embodiments, the electronic device may determine whether to select the data set for training the downstream task model based on the diversity evaluation result of the data set. For example, if the diversity evaluation result of the data set is that the data set has diversity, the data set may be used to train the downstream task model. For another example, if the diversity evaluation result of the data set is that the data set does not have diversity, the data set may not be used to train the downstream task model.

[0182] Furthermore, due to the powerful content generation capabilities of large language models, they will be used on a larger scale to generate high-quality training data in the future. The data evaluation method provided in the embodiments of the present application can promote the efficiency and quality of constructing data sets for large language models, thereby providing reliable guarantees for model training.

[0183] The data evaluation method proposed in the embodiment of the present application obtains a data set to be evaluated, and the data set includes N sample data, where N is a positive integer greater than 1; performs similarity calculation on each two sample data in the N sample data to obtain the similarity between each two sample data; based on the similarity between each two sample data, determines the probability value that each sample data in the N sample data belongs to each category in the N categories, and the N categories are the true categories of each sample data in the N sample data; based on the probability value that each sample data in the N sample data belongs to each category in the N categories, determines the diversity evaluation result of the N sample data. It can be seen that the embodiment of the present application determines the probability value that each sample data belongs to N categories by determining the similarity between each two sample data in the data set, and then realizes the overall evaluation of the diversity of the data set from the perspective of sample classification. The obtained diversity value can be understood as the number of valid samples in the data set, so it has physical meaning. In addition, the embodiment of the present application determines the diversity evaluation result of the N sample data based on the probability value that the N sample data belong to N categories respectively, and its calculation speed is fast and efficient, and can realize the rapid and efficient evaluation of large data sets. During model training, the method of the embodiment of the present application can be used to quickly and accurately evaluate the diversity of the data set, and then the model can be trained using the data set whose diversity meets the requirements, which can improve the training effect and training speed of the model and save computing resources.

[0184] The above is an overall introduction to the data evaluation method. The following takes the data set generated by the model as an example to further introduce the data evaluation method provided in the embodiment of the present application.

[0185] Figure 5 A flowchart of a data evaluation method provided in accordance with an embodiment of the present application. Figure 6 The framework diagram of the data evaluation method provided in the embodiment of the present application. The execution subject of the embodiment of the present application can be a device with a data evaluation function, such as a data evaluation device. The data evaluation device can be the above-mentioned Figure 1 The server 102 or terminal device 101 shown may also be Figure 1 The server 102 and the terminal device 101 shown in the figure are composed of a system. For the convenience of description, the following embodiment takes an electronic device as an example to illustrate the method of the embodiment of the present application.

[0186] like Figure 5 and Figure 6 As shown, the data evaluation method of the embodiment of the present application includes:

[0187] S201. Obtain a data set to be evaluated, where the data set includes N sample data, and the sample data includes first data and second data. The first data is input data of a language model, and the second data is data obtained after the language model processes the first data.

[0188] Wherein, N is a positive integer greater than 1.

[0189] As language models (such as large language models) demonstrate superior content generation performance, language models can be used to generate or enhance data sets to alleviate the problem of lack of high-quality labeled data in certain vertical fields. However, with the increasing application of language model-generated data sets, relevant studies have also pointed out the impact of generated data sets on model performance or expressiveness. As an important aspect of data quality judgment, diversity is often overlooked in the quality assessment of generated data sets. At the same time, current studies have pointed out the impact of data diversity on trained models. The data evaluation method provided in the embodiment of the present application can effectively evaluate the diversity of data sets generated by the model.

[0190] Specifically, the data set to be evaluated generated by the electronic device acquisition model can be expressed as The data set consists of N sample data (or input-output pairs) Composition, of which x i is the first data, i.e., the input data of the input language model, y i is the second data, that is, the language model pair x i The data output after processing, in some embodiments, the second data can be called the label data of the first data. For example, x i For a review of a movie, i is the sentiment implied in the movie review (good or bad).

[0191] It should be noted that the specific implementation process of the above S201 can refer to the relevant description of the above S101, which will not be repeated here.

[0192] S202: Obtain downstream task types corresponding to N sample data.

[0193] If the sample data included in the data set includes the first data and the second data, in order to reduce the amount of data processing, the electronic device can select part of the sample data to calculate the similarity. When selecting part of the sample data, it is necessary to select according to the downstream task type corresponding to the data set. Based on this, the electronic device needs to obtain the downstream task type corresponding to N sample data.

[0194] It should be noted that the embodiments of the present application do not limit the specific implementation order of the above S201 and S202. In some embodiments, the above S202 can be executed before the above S201, in some embodiments, the above S202 can be executed after the above S202, and in some embodiments, the above S202 can also be executed synchronously with the above S201, and the embodiments of the present application do not limit this.

[0195] In an embodiment of the present application, the downstream task type corresponding to the data set can be input to the object, or determined based on the output result of the model to be trained.

[0196] S203: For each sample data among the N sample data, based on the downstream task type, determine the diversity sensitive data corresponding to the sample data from the first data and the second data included in the sample data.

[0197] The diversity sensitive data can be understood as the data that is more sensitive to the downstream task in the first data and the second data included in the sample data. For example, in the movie review scenario of the text classification downstream task, x i For a review of a movie, i is the implicit emotion (good or bad) in the movie review. In this scenario, y is not important. i The diversity of x i That is the diversity sensitive component.

[0198] Based on the above steps, the electronic device determines the diversity sensitive data corresponding to each sample data in the N sample data. When the diversity is subsequently calculated, the input of the method is Diversity sensitive components

[0199] S204: Extract feature information of the diverse sensitive data corresponding to each sample data in the N sample data.

[0200] In this implementation, when determining the similarity between the diversity sensitive data corresponding to every two sample data among N sample data, the electronic device first extracts the feature information of the diversity sensitive data corresponding to each sample data.

[0201] In the embodiment of the present application, the method of determining the similarity between every two sample data in N sample data is basically the same. For the convenience of description, the kth sample data and the pth sample data in the N sample data are taken as an example for explanation.

[0202] In this implementation, if Figure 6As shown, for the kth sample data and the pth sample data among the N sample data, the electronic device first uses a feature extractor to extract feature information of the diversity sensitive data corresponding to the kth sample data and feature information of the diversity sensitive data corresponding to the pth sample data.

[0203] The embodiment of the present application does not limit the specific type of the feature extractor, and can be any trained feature extractor used to extract feature information of text.

[0204] In some embodiments, the electronic device selects a feature extractor that matches the downstream task type based on the downstream task type corresponding to the data set. For example, if the downstream task type of the embodiment of the present application needs to focus on the difference between the semantics of the samples, a model that can better capture the semantic information can be selected as the feature extractor. Then, through the feature extractor, the feature information of the diversity sensitive data corresponding to each sample data is extracted. For example, the feature extractor extracts the feature information of the diversity sensitive data corresponding to the kth sample data, and extracts the feature information of the diversity sensitive data corresponding to the pth sample data.

[0205] It should be noted that the specific implementation process of the above S204 can refer to the relevant description of the above S102, which will not be repeated here.

[0206] S205 . Calculate the similarity between every two sample data based on the feature information of the diversity sensitive data corresponding to every two sample data.

[0207] For example, for the kth sample data and the pth sample data among N sample data, the electronic device performs similarity calculation on the feature information of the diversity sensitive data corresponding to the kth sample data and the feature information of the diversity sensitive data corresponding to the pth sample data to obtain the similarity between the kth sample data and the pth sample data.

[0208] In the embodiment of the present application, the electronic device performs similarity calculation on feature information of diversity sensitive data corresponding to the kth sample data and feature information of diversity sensitive data corresponding to the pth sample data, and obtains the similarity between the kth sample data and the pth sample data without limiting the specific method.

[0209] In one example, if Figure 6 As shown, the electronic device processes the feature information of the diversity sensitive data corresponding to the kth sample data and the feature information of the diversity sensitive data corresponding to the pth sample data through a kernel function to obtain the similarity between the kth sample data and the pth sample data.

[0210] It should be noted that the specific implementation process of the above S205 can refer to the relevant description of the above S102, which will not be repeated here.

[0211] S206 . For the ith sample data among the N sample data, based on the similarity between the ith sample data and each sample data among the N sample data, determine the probability value that the ith sample data belongs to each category among the N categories.

[0212] Wherein, i is a positive integer from 1 to N.

[0213] In the embodiment of the present application, each of the N sample data has a real category, thereby forming N categories.

[0214] Based on the above steps, the electronic device can calculate the similarity between every two sample data in the N sample data, and then calculate the probability value of each sample data belonging to each type in the N categories based on the similarity between every two sample data.

[0215] In the embodiment of the present application, for each sample data in the N sample data, for example, the i-th sample data, the electronic device determines the probability value that the i-th sample data belongs to each of the N categories based on the similarity between the i-th sample data and each of the N sample data. In this way, the electronic device can determine the probability value that each sample data in the N sample data belongs to each of the N categories.

[0216] The embodiment of the present application does not limit the specific manner in which the electronic device determines the probability value of the i-th sample data belonging to each of the N categories based on the similarity between the i-th sample data and each of the N sample data. The greater the similarity between the i-th sample data and the j-th sample data, the greater the probability value that the i-th sample data belongs to the category corresponding to the j-th sample data.

[0217] In some embodiments, for the jth category among N categories, based on the similarity between the i-th sample data and the j-th sample data, and the similarity between the i-th sample data and each of the N sample data, the probability value of the i-th sample data belonging to the j-th category is determined, the j-th category is the true category of the j-th sample data, and j is a positive integer from 1 to N.

[0218] In some embodiments, based on the similarity between the i-th sample data and the j-th sample data, and the similarity between the i-th sample data and each of the N sample data, the probability value of the i-th sample data belonging to the j-th category is determined, including: obtaining a first value based on the similarity between the i-th sample data and the j-th sample data; obtaining a second value based on the similarity between the i-th sample data and each of the N sample data; and determining the probability value of the i-th sample data belonging to the j-th category based on the first value and the second value.

[0219] In one example, an exponential operation is performed on the similarity between the i-th sample data and the j-th sample data to obtain a first value; an exponential operation is performed on the similarity between the i-th sample data and each of the N sample data, and the exponential operation results are added to obtain a second value. The ratio of the first value to the second value is determined as the probability value that the i-th sample data belongs to the j-th category.

[0220] Based on the above method, the electronic device can determine the probability value that the i-th sample data belongs to each of the N categories, and further determine the probability value that each of the N sample data belongs to each of the N categories.

[0221] It should be noted that the specific implementation process of the above S206 can refer to the relevant description of the above S103, which will not be repeated here.

[0222] S207 . Select the probability value of the i-th sample data belonging to the i-th category from the probability values ​​of the i-th sample data belonging to each category of the N categories.

[0223] Among them, the i-th category is the true category of the i-th sample data.

[0224] According to this method, the electronic device can select the probability value of each sample data belonging to the corresponding true category from the probability value of each sample data belonging to each category in the N categories, and then execute the following step S208.

[0225] S208 . Determine a diversity evaluation value of the N sample data based on a probability value that each of the N sample data belongs to a corresponding true category.

[0226] For example, Figure 6 As shown, the probability value of each sample data in N sample data belonging to the corresponding real category is added to obtain the diversity evaluation value of the N sample data.

[0227] S209: Determine the diversity evaluation results of the N sample data based on the diversity evaluation values.

[0228] For example, if the diversity evaluation value is greater than or equal to a preset threshold, it is determined that the N data samples have diversity.

[0229] For another example, if the diversity evaluation value is less than a preset threshold, it is determined that the N sample data do not have diversity.

[0230] As can be seen from the above, the embodiment of the present application evaluates multiple samples of the data set generated by the model, and the process is simple and the computational cost is low, which provides an effective solution for the evaluation of large-scale generated data sets in industrial scenarios. At the same time, since the method proposed in the embodiment of the present application has the characteristics of simple calculation and differentiability, diversity evaluation can also be used as a related model optimization target, which can further improve the efficiency of model training. In addition, the embodiment of the present application realizes an overall evaluation of the diversity of the data set from the perspective of sample classification, and the obtained diversity value can be understood as the number of valid samples in the data set, so it has a physical meaning that can be understood by humans. Due to the powerful content generation ability of the large language model, it will be used on a larger scale in the future to generate high-quality training data. The diversity evaluation ideas provided in the embodiment of the present application can facilitate humans to understand the diversity of the data set, thereby promoting the efficiency and quality assurance of the construction of large-scale generated data sets.

[0231] The data evaluation method provided in the embodiment of the present application obtains a data set to be evaluated and a downstream task type, wherein the data set includes N sample data, the sample data includes first data and second data, the first data is the input data of the language model, and the second data is the data obtained after the language model processes the first data. For each sample data in the N sample data, based on the downstream task type, the diversity sensitive data corresponding to the sample data is determined from the first data and the second data included in the sample data. Then, the feature information of the diversity sensitive data corresponding to each sample data in the N sample data is extracted. Then, based on the feature information of the diversity sensitive data corresponding to each two sample data, the similarity between each two sample data is calculated. For the i-th sample data in the N sample data, based on the similarity between the i-th sample data and each sample data in the N sample data, the probability value of the i-th sample data belonging to each category in the N categories is determined, and then the probability value of the i-th sample data belonging to the i-th category is selected from the probability value of the i-th sample data belonging to each category in the N categories. In this way, the electronic device can determine the probability value of each sample data in the N sample data belonging to the corresponding real category. Finally, the electronic device determines the diversity evaluation value of the N sample data based on the probability value that each sample data in the N sample data belongs to the corresponding real category, and then determines the diversity evaluation result of the N sample data based on the diversity evaluation value. It can be seen that, for the sample data generated by the model, the embodiment of the present application selects diversity sensitive data from the first data and the second data included in the sample data, and then calculates the similarity between every two sample data based on the diversity sensitive data, which can not only achieve the accuracy of the calculation of the similarity, but also reduce the amount of data processing. Then, by determining the similarity between every two sample data in the data set, the probability value that each sample data belongs to N categories is determined, and then the overall evaluation of the diversity of the data set is achieved from the perspective of sample classification. The obtained diversity value can be understood as the number of valid samples in the data set, so it has a physical meaning. In addition, the embodiment of the present application determines the diversity evaluation result of the N sample data based on the probability value that the N sample data belong to N categories respectively, and its calculation speed is fast and efficient, and it can achieve fast and efficient evaluation of large data sets. During model training, the method of the embodiment of the present application can be used to quickly and accurately evaluate the diversity of the data set, and then the model can be trained using the data set whose diversity meets the requirements, which can improve the training effect and training speed of the model and save computing resources.

[0232] The following introduces the process of applying the data evaluation method of an embodiment of the present application to model training.

[0233] Figure 7 A flowchart of a model training method provided in one embodiment of the present application.

[0234] like Figure 7 As shown, the model training method of the embodiment of the present application includes:

[0235] S301. Obtain a data set corresponding to the model to be trained.

[0236] The data set is used to train the model to be trained, and the data set includes N text data, the text data includes first data and second data, wherein the first data can be called an independent variable x, and the second data can be called a dependent variable y. That is, the i-th text data is represented as (xi, yi).

[0237] In some embodiments, if the model to be trained is a text classification model, the xi in the above text data can be understood as the text to be classified, and yi is the text classification result label of the text to be classified. The embodiment of the present application does not limit the specific content of the text data, and can be set according to the specific actual situation, for example, it can be network information text, academic documents, comment text, etc.

[0238] In one example, for example, if the model to be trained is a film review classification model, then xi in the text data can be a text review of a movie, and yi is the classification result label of the movie review, that is, the emotion (good or bad) implied in the movie review.

[0239] In some embodiments, the above text data is generated by a language model, that is, the first data xi in the text data is the input data of the language model. The second data yi is the data obtained after the language model processes the first data. In other words, the sample label can be generated by the language model.

[0240] Wherein, N is a positive integer greater than 1.

[0241] The specific implementation process of the above S301 can refer to the relevant description of the above S201, which will not be repeated here.

[0242] S302: Obtain downstream task types corresponding to N text data.

[0243] The downstream task type corresponding to the N text data can be understood as the task type of the model to be trained. For example, when the model to be trained is a text classification model, it can be determined that the downstream task type corresponding to the N text data is text classification.

[0244] The specific implementation process of the above S302 can refer to the relevant description of the above S202, which will not be repeated here.

[0245] S303: For each of the N text data, based on the downstream task type, determine the diversity sensitive data corresponding to the text data from the first data and the second data included in the text data.

[0246] The specific implementation process of the above S303 can refer to the relevant description of the above S203, which will not be repeated here.

[0247] S304: Extract feature information of the diverse sensitive data corresponding to each text data in the N text data.

[0248] The specific implementation process of the above S304 can refer to the relevant description of the above S204, which will not be repeated here.

[0249] S305 . Calculate the similarity between every two text data based on the feature information of the diversity sensitive data corresponding to every two text data.

[0250] The specific implementation process of the above S305 can refer to the relevant description of the above S205, which will not be repeated here.

[0251] S306 . For the ith text data among the N text data, based on the similarity between the ith text data and each of the N text data, determine the probability value that the ith text data belongs to each of the N categories.

[0252] The specific implementation process of the above S306 can refer to the relevant description of the above S206, which will not be repeated here.

[0253] S307 . Select the probability value that the i-th text data belongs to the i-th category from the probability values ​​that the i-th text data belongs to each category of the N categories.

[0254] The specific implementation process of the above S307 can refer to the relevant description of the above S207, which will not be repeated here.

[0255] S308 : Determine the diversity evaluation value of the N text data based on the probability value that each text data in the N text data belongs to the corresponding real category.

[0256] The specific implementation process of the above S308 can refer to the relevant description of the above S208, which will not be repeated here.

[0257] S309: If the diversity evaluation value of the N text data is greater than or equal to a preset threshold, the data set is used to train the model to be trained to obtain a trained model.

[0258] Exemplarily, if the model to be trained is a text classification model, the text classification model is trained using the N text data included in the data set whose diversity meets the preset requirements to obtain a trained text classification model.

[0259] For example, if the above-mentioned model to be trained is a film review classification model, the film review classification model can be trained using text data included in a data set whose diversity meets preset requirements. For example, for each text data, the text data includes a text review of a movie and a classification result label of the movie review. The movie review text included in the text data is input into the film review classification model to be trained, and the classification result corresponding to the review text predicted by the film review classification model is obtained. The classification result corresponding to the review text predicted by the model is then compared with the classification result label included in the text data to determine the model loss, and then the film review classification model to be trained is trained based on the model loss, and iterated multiple times to obtain a trained film review classification model.

[0260] In some embodiments, if the diversity evaluation value of a data set is less than a preset threshold, the data set is not used to train the model to be trained, and another data set with diversity can be selected, or the data set can be adjusted so that the data set meets the diversity requirement, and then the model is trained using the diverse data set.

[0261] The model training method provided in the embodiment of the present application calculates the similarity between every two text data in the N text data included in the data set of the training model, and obtains the similarity between every two text data. Then, based on the similarity between every two text data, the probability value of each text data in the N text data belonging to each category in the N categories is determined, and based on the probability value of each text data in the N text data belonging to each category in the N categories, the diversity evaluation value of the data set is determined. If the diversity evaluation value of the data set is greater than or equal to the preset threshold, the data set is used to train the model to be trained, which can improve the training effect and training speed of the model and save computing resources.

[0262] Combination of the above Figures 2 to 7 , describes in detail the method embodiment of the present application, and the following is combined with Figure 8 , describe in detail the device embodiments of the present application.

[0263] Figure 8 It is a schematic block diagram of a data evaluation device provided in one embodiment of the present application.

[0264] like Figure 8 As shown, the data evaluation device 10 comprises:

[0265] An acquisition unit 11 is used to acquire a data set to be evaluated, wherein the data set includes N sample data, where N is a positive integer greater than 1;

[0266] A similarity calculation unit 12 is used to perform similarity calculation on every two sample data among the N sample data to obtain the similarity between every two sample data;

[0267] A classification unit 13, configured to determine, based on the similarity between each two sample data, a probability value that each of the N sample data belongs to each of N categories, wherein the N categories are the true categories of each of the N sample data;

[0268] The evaluation unit 14 is configured to determine a diversity evaluation result of the N sample data based on a probability value that each of the N sample data belongs to each of the N categories.

[0269] In some embodiments, the classification unit 13 is specifically used to determine, for the i-th sample data among the N sample data, the probability value of the i-th sample data belonging to each of the N categories based on the similarity between the i-th sample data and each of the N sample data, where i is a positive integer from 1 to N.

[0270] In some embodiments, the classification unit 13 is specifically used to determine, for the j-th category among the N categories, the probability value that the i-th sample data belongs to the j-th category based on the similarity between the i-th sample data and the j-th sample data, and the similarity between the i-th sample data and each of the N sample data, the j-th category is the true category of the j-th sample data, and j is a positive integer from 1 to N.

[0271] In some embodiments, the classification unit 13 is specifically used to obtain a first value based on the similarity between the i-th sample data and the j-th sample data; obtain a second value based on the similarity between the i-th sample data and each of the N sample data; and determine the probability value that the i-th sample data belongs to the j-th category based on the first value and the second value.

[0272] In some embodiments, the classification unit 13 is specifically used to perform an exponential operation on the similarity between the i-th sample data and the j-th sample data to obtain the first value; perform an exponential operation on the similarity between the i-th sample data and each of the N sample data, and add the results of the exponential operation to obtain the second value.

[0273] In some embodiments, the evaluation unit 14 is specifically used to select, for the i-th sample data among the N sample data, from the probability values ​​of the i-th sample data belonging to each of the N categories, a probability value that the i-th sample data belongs to the i-th category, where the i-th category is the true category of the i-th sample data; determine a diversity evaluation value of the N sample data based on the probability value that each sample data among the N sample data belongs to the corresponding true category; and determine a diversity evaluation result of the N sample data based on the diversity evaluation value.

[0274] In some embodiments, the evaluation unit 14 is specifically configured to add the probability value that each of the N sample data belongs to the corresponding true category to obtain the diversity evaluation value of the N sample data.

[0275] In some embodiments, the evaluation unit 14 is specifically configured to determine that the N data samples have diversity if the diversity evaluation value is greater than or equal to a preset threshold; and to determine that the N sample data do not have diversity if the diversity evaluation value is less than the preset threshold.

[0276] In some embodiments, the sample data includes first data and second data, the first data is input data of a language model, and the second data is data obtained after the language model processes the first data.

[0277] In some embodiments, the similarity calculation unit 12 is specifically used to obtain the downstream task type corresponding to the N sample data; for each sample data among the N sample data, based on the downstream task type, determine the diversity sensitive data corresponding to the sample data from the first data and the second data included in the sample data; perform similarity calculation on the diversity sensitive data corresponding to each two sample data among the N sample data, to obtain the similarity between the each two sample data.

[0278] In some embodiments, the similarity calculation unit 12 is specifically used to extract the feature information of the diversity sensitive data corresponding to the kth sample data and the pth sample data among the N sample data, and extract the feature information of the diversity sensitive data corresponding to the pth sample data, where both k and p are positive integers less than or equal to N; perform similarity calculation on the feature information of the diversity sensitive data corresponding to the kth sample data and the feature information of the diversity sensitive data corresponding to the pth sample data to obtain the similarity between the kth sample data and the pth sample data.

[0279] In some embodiments, the similarity calculation unit 12 is specifically used to select a feature extractor that matches the downstream task type based on the downstream task type; through the feature extractor, the feature information of the diversity sensitive data corresponding to the kth sample data and the feature information of the diversity sensitive data corresponding to the pth sample data are extracted.

[0280] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, Figure 8 The device shown can execute the embodiments of the above method, and the aforementioned and other operations and / or functions of each module in the device are respectively for realizing the embodiments of the above method, which will not be described in detail here for the sake of brevity.

[0281] The above describes the device of the embodiment of the present application from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or a combination of hardware and software modules in the decoding processor to perform. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above method embodiment in conjunction with its hardware.

[0282] Fig. 9 It is a schematic block diagram of an electronic device provided in an embodiment of the present application, and the electronic device may be the above-mentioned terminal device or server.

[0283] like Fig. 9 As shown, the electronic device 40 may include:

[0284] The memory 41 and the processor 42, the memory 41 is used to store the computer program 43 and transmit the program code 43 to the processor 42. In other words, the processor 42 can call and run the computer program 43 from the memory 41 to implement the method in the embodiment of the present application.

[0285] For example, the processor 42 may be configured to execute the steps in the above method according to the instructions in the computer program 43 .

[0286] In some embodiments of the present application, the processor 42 may include but is not limited to:

[0287] General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0288] In some embodiments of the present application, the memory 41 includes but is not limited to:

[0289] Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0290] In some embodiments of the present application, the computer program 43 may be divided into one or more modules, which are stored in the memory 41 and executed by the processor 42 to complete the method for recording pages provided by the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program 43 in the electronic device.

[0291] like Fig. 9 As shown, the electronic device 40 may further include:

[0292] The transceiver 44 may be connected to the processor 42 or the memory 41 .

[0293] The processor 42 may control the transceiver 44 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 44 may include a transmitter and a receiver. The transceiver 44 may further include an antenna, and the number of antennas may be one or more.

[0294] It should be understood that the various components in the electronic device 40 are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0295] According to one aspect of the present application, a computer storage medium is provided, on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. In other words, the present application embodiment also provides a computer program product containing instructions, and when the instructions are executed by a computer, the computer can perform the method of the above method embodiment.

[0296] According to another aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method of the above method embodiment.

[0297] In other words, when implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (digital video disc, DVD)), or a semiconductor medium (e.g., a solid state drive (solid state disk, SSD)), etc.

[0298] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0299] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0300] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0301] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data evaluation method, characterized in that: include: Obtain a data set to be evaluated, the data set comprising N sample data, where N is a positive integer greater than 1; Calculating the similarity between every two sample data among the N sample data to obtain the similarity between every two sample data; Based on the similarity between each two sample data, determine the probability value that each sample data in the N sample data belongs to each category in N categories, where the N categories are the true categories of each sample data in the N sample data; Based on the probability value that each of the N sample data belongs to each of the N categories, a diversity evaluation result of the N sample data is determined.

2. The method according to claim 1, characterized in that: The determining, based on the similarity between each two sample data, a probability value that each of the N sample data belongs to each of the N categories comprises: For the i-th sample data among the N sample data, based on the similarity between the i-th sample data and each of the N sample data, determine the probability value of the i-th sample data belonging to each of the N categories, where i is a positive integer from 1 to N.

3. The method according to claim 2, characterized in that The determining, based on the similarity between the i-th sample data and each of the N sample data, a probability value that the i-th sample data belongs to each of the N categories comprises: For the j-th category among the N categories, based on the similarity between the i-th sample data and the j-th sample data, and the similarity between the i-th sample data and each of the N sample data, determine the probability value of the i-th sample data belonging to the j-th category, the j-th category is the true category of the j-th sample data, and j is a positive integer from 1 to N.

4. The method according to claim 3, characterized in that The determining, based on the similarity between the i-th sample data and the j-th sample data, and the similarity between the i-th sample data and each of the N sample data, a probability value that the i-th sample data belongs to the j-th category includes: Obtaining a first value based on the similarity between the i-th sample data and the j-th sample data; Obtaining a second value based on the similarity between the i-th sample data and each of the N sample data; Based on the first value and the second value, a probability value that the i-th sample data belongs to the j-th category is determined.

5. The method according to claim 4, characterized in that The obtaining a first value based on the similarity between the i-th sample data and the j-th sample data includes: Performing an exponential operation on the similarity between the i-th sample data and the j-th sample data to obtain the first value; The obtaining a second value based on the similarity between the i-th sample data and each of the N sample data includes: An exponential operation is performed on the similarity between the i-th sample data and each of the N sample data, and the exponential operation results are added to obtain the second value.

6. The method according to any one of claims 1 to 5, characterized in that: The determining the diversity evaluation result of the N sample data based on the probability value that each sample data in the N sample data belongs to each category in the N categories includes: For the i-th sample data among the N sample data, the probability value of the i-th sample data belonging to the i-th category is selected from the probability values ​​of the i-th sample data belonging to each of the N categories, and the i-th category is the true category of the i-th sample data; Determining a diversity evaluation value of the N sample data based on a probability value that each of the N sample data belongs to a corresponding true category; Based on the diversity evaluation value, a diversity evaluation result of the N sample data is determined.

7. The method according to claim 6, characterized in that The determining the diversity evaluation value of the N sample data based on the probability value that each sample data in the N sample data belongs to the corresponding real category includes: The probability value that each of the N sample data belongs to the corresponding true category is added to obtain the diversity evaluation value of the N sample data.

8. The method according to claim 7, characterized in that The determining, based on the diversity evaluation value, a diversity evaluation result of the N sample data includes: If the diversity evaluation value is greater than or equal to a preset threshold, it is determined that the N data samples have diversity; If the diversity evaluation value is less than the preset threshold, it is determined that the N sample data do not have diversity.

9. The method according to any one of claims 1 to 5, characterized in that: The sample data includes first data and second data, the first data is input data of a language model, and the second data is data obtained after the language model processes the first data.

10. The method according to claim 9, characterized in that The performing similarity calculation on every two sample data among the N sample data to obtain the similarity between every two sample data includes: Obtain downstream task types corresponding to the N sample data; For each sample data of the N sample data, based on the downstream task type, determining diversity sensitive data corresponding to the sample data from first data and second data included in the sample data; Similarity calculation is performed on the diversity sensitive data corresponding to every two sample data in the N sample data to obtain the similarity between the every two sample data.

11. The method according to claim 10, characterized in that The performing similarity calculation on the diversity sensitive data corresponding to each two sample data in the N sample data to obtain the similarity between each two sample data includes: For the k-th sample data and the p-th sample data among the N sample data, extract the feature information of the diversity sensitive data corresponding to the k-th sample data, and extract the feature information of the diversity sensitive data corresponding to the p-th sample data, where both k and p are positive integers less than or equal to N; A similarity calculation is performed on the feature information of the diversity sensitive data corresponding to the k-th sample data and the feature information of the diversity sensitive data corresponding to the p-th sample data to obtain the similarity between the k-th sample data and the p-th sample data.

12. The method according to claim 11, characterized in that The extracting the feature information of the diverse sensitive data corresponding to the k-th sample data, and extracting the feature information of the diverse sensitive data corresponding to the p-th sample data, includes: Based on the downstream task type, selecting a feature extractor matching the downstream task type; The feature extractor is used to extract feature information of the diversity sensitive data corresponding to the k-th sample data and feature information of the diversity sensitive data corresponding to the p-th sample data.

13. A data evaluation device, characterized in that: include: An acquisition unit, used to acquire a data set to be evaluated, wherein the data set includes N sample data, where N is a positive integer greater than 1; A similarity calculation unit, used for performing similarity calculation on every two sample data among the N sample data to obtain the similarity between every two sample data; A classification unit, used for determining, based on the similarity between each two sample data, a probability value that each of the N sample data belongs to each of N categories, where the N categories are the true categories of each of the N sample data; An evaluation unit is used to determine a diversity evaluation result of the N sample data based on a probability value that each of the N sample data belongs to each of the N categories.

14. An electronic device, characterized in that: including a processor and a memory; The memory is used to store computer programs; The processor is used to execute the computer program to implement the method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: For storing computer programs; The computer program enables a computer to execute the method according to any one of claims 1 to 12.

Images