Data set construction method, device and electronic device

Abstract

The embodiments of the present application disclose a method, apparatus, and electronic device for constructing a dataset. The method comprises: obtaining data samples to be processed; grouping data samples that meet the same preset correlation coefficient into the same example package to obtain multiple example packages; grouping the multiple example packages into multiple subsets based on pre-set grouping parameters; and constructing a dataset based on the multiple subsets. By using the above method, data samples that meet the same preset correlation coefficient are grouped into the same example package to obtain multiple example packages, thereby ensuring that samples in the dataset obtained based on the multiple example packages also have a correlation relationship, thereby improving the mineability of sample information in the dataset and thereby improving the accuracy of the training task corresponding to the dataset.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to a data set construction method, device, and electronic device. Background Art

[0002] Deep learning has been widely applied in fields such as images, text, and speech. Because deep learning's superior performance is closely related to the data samples required for deep learning, more and more people are beginning to pay attention to methods for dividing the data samples required for deep learning. In a related method, data sampling can be used to divide data samples into training sets, validation sets, and test sets. However, this method does not consider the correlation between data samples. Summary of the Invention

[0003] In view of the above problems, the present application proposes a data set construction method, device and electronic device to improve the above problems.

[0004] In a first aspect, the present application provides a method for constructing a dataset, the method comprising: obtaining data samples to be processed; dividing data samples that satisfy the same preset correlation relationship into the same example package to obtain multiple example packages; dividing the multiple example packages into multiple subsets based on pre-set division parameters; and constructing a dataset based on the multiple subsets.

[0005] In the second aspect, the present application provides a data set construction device, which includes: a data sample acquisition unit, used to obtain data samples to be processed; an example package acquisition unit, used to divide data samples that meet the same preset correlation coefficient into the same example package to obtain multiple example packages; a subset division unit, used to divide the multiple example packages into multiple subsets based on pre-set division parameters; and a data set construction unit, used to construct a data set based on the multiple subsets.

[0006] In a third aspect, the present application provides an electronic device comprising one or more processors and a memory; one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the above-mentioned method.

[0007] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores program code, wherein the above method is executed when the program code is run.

[0008] The present application provides a data set construction method, device, electronic device, and storage medium. After obtaining data samples to be processed, data samples that meet the same preset correlation relationship are divided into the same example package to obtain multiple example packages. Based on pre-set division parameters, the multiple example packages are divided into multiple subsets, and a data set is constructed based on the multiple subsets. Through the above method, data samples that meet the same preset correlation coefficient are divided into the same example package to obtain multiple example packages, so that the samples in the data set obtained based on the multiple example packages also have a correlation relationship, thereby improving the mineability of the sample information in the data set and thereby improving the accuracy of the training task corresponding to the data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0010] Figure 1 A flowchart of a data set construction method proposed in an embodiment of the present application is shown;

[0011] Figure 2 A flowchart of a data set construction method proposed in another embodiment of the present application is shown;

[0012] Figure 3 The following is a schematic diagram showing the basic process of the dataset construction method proposed in this application;

[0013] Figure 4 A structural block diagram of a data set construction device proposed in an embodiment of the present application is shown;

[0014] Figure 5 Shows a structural block diagram of an electronic device proposed in this application;

[0015] Figure 6 It is a storage unit for storing or carrying program codes for implementing the data set construction method according to the embodiment of the present application. DETAILED DESCRIPTION

[0016] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0017] Because the superior performance of deep learning is closely linked to the data samples required for deep learning, increasing interest has emerged in methods for partitioning these data samples. Data sampling methods can be used to partition data into training, validation, and test sets. For example, data can be partitioned using the classic random stratified sampling method. Another example is the partitioning method based on distribution space regeneration.

[0018] The inventors found in related research that related methods all divide data based on the assumption that data samples are independent of each other, which results in a problem of poor correlation between samples in the data set.

[0019] Therefore, the inventors have proposed a data set construction method, device, and electronic device in this application. After obtaining the data samples to be processed, the data samples that meet the same preset correlation relationship are divided into the same example package to obtain multiple example packages. Based on pre-set division parameters, the multiple example packages are divided into multiple subsets, and a data set is constructed based on the multiple subsets. Through the above method, data samples that meet the same preset correlation coefficient are divided into the same example package to obtain multiple example packages, so that the samples in the data set obtained based on the multiple example packages also have a correlation relationship, which improves the mineability of the sample information in the data set, and further improves the accuracy of the training task corresponding to the data set.

[0020] In order to better understand the solutions of the embodiments of the present application, the technical terms used in the embodiments of the present application are explained below.

[0021] Multiple Instance Learning (MIL): A method derived from supervised learning algorithms, MIL involves learning problems using multiple bags as training units. In MIL, the training set can consist of a set of bags with classification labels, each of which can contain several instances. An instance can be a data sample.

[0022] The embodiments of the present application will be described below with reference to the accompanying drawings.

[0023] See also Figure 1 , the present application provides a method for constructing a data set, the method comprising:

[0024] S110: Obtain data samples to be processed.

[0025] Data samples refer to samples that can be used for AI training tasks, and samples can be images, videos, audio, etc. Data samples can be labeled data.

[0026] As a method, data samples to be processed can be obtained based on the purpose of the training task. For example, when the purpose of the training task is to classify cats and dogs, images of cats and dogs can be obtained as data samples to be processed.

[0027] Optionally, the data samples to be processed can be obtained from public data sets, can be obtained from the Internet through crawlers and other technologies, and can also be collected through relevant equipment (such as cameras, recorders, etc.).

[0028] S120: Divide data samples that meet the same preset correlation relationship into the same example package to obtain multiple example packages.

[0029] The correlation relationship may refer to a mapping relationship between data samples and example packages, the mapping relationship may represent an association between data samples included in the same example package, and the association may refer to a common attribute of data samples included in the same example package.

[0030] As a method, we can pre-set the correlation relationship based on the training task, and then divide the data samples that meet the same preset correlation relationship into the same example package to obtain multiple example packages. The correlation relationship can be expressed as follows:

[0031] B=f(E)

[0032] Here, E can represent a data sample, f(E) can represent the mapping relationship between a data sample and an example package, and B can be a package affiliation mapping vector, which can be used to characterize the affiliation between a data sample and an example package. For example, the package affiliation mapping vector can be a vector of size N×1, and the data sample corresponding to the current example package can be determined based on the elements in the vector. For another example, when there is no correlation between data samples, the example package and the data sample can have a one-to-one correspondence, i.e., B=E, and N in the package affiliation mapping vector can be 1. For example, the data samples can be video frames from multiple videos, and the mapping relationship can be such that the video frames in the same video are grouped into one example package. Then, the video frames in the same video can be grouped into the same example package to obtain multiple example packages. For another example, the data samples can be physical examination reports of different people at different time periods. The mapping relationship can be such that the physical examination reports of the same person at different time periods are grouped into one example package. Then, the physical examination reports obtained by the same person at different examination times can be grouped into the same example package. The data samples included in the same example package can be physical examination reports obtained by the same person at different examination times. For another example, the data sample may be a feature vector, and the mapping relationship may be dividing the clusters obtained by the clustering algorithm into an example package. Then, the K-means algorithm may be used to cluster the data sample, and a clustering result may be used as an example package.

[0033] Optionally, when the package membership mapping vector is known and the sample data needs to be represented as tree data, a union-find method can be used to quickly generate a merged tree to obtain multiple sample packages.

[0034] In an embodiment of the present application, each data sample may correspond to a category label, and each of the multiple example packages may correspond to a category label. As a method, after obtaining multiple example packages, the category labels corresponding to the multiple example packages may be determined based on the category labels of the data samples corresponding to the multiple example packages.

[0035] Optionally, the data samples may have classification labels, and the classification labels corresponding to the multiple example packages may be determined based on the classification labels of the data samples corresponding to the multiple example packages. For example, if the classification label of the data samples included in the example package may be "dog," then the category label corresponding to the example package is also "dog."

[0036] Optionally, the data samples may be regression labels, and the category labels corresponding to the multiple example packages may be determined based on at least one of the mode, mean, quantile, etc. of the regression labels of the data samples corresponding to the multiple example packages.

[0037] For example, the data samples in the example package can be the temperature values of XX city in the same week, and the regression labels of the data samples can be: 21℃, 22℃, 21℃, 25℃, 21℃, 26℃, 21℃. Then, based on the mode of the regression labels, the category label of the example package can be 21℃; based on the mean of the regression labels, the category label of the example package can be 22.43℃; based on the quantile of the regression labels, the category label of the example package can be 21℃; the category label obtained based on the mode of the regression labels and the category label obtained based on the mean of the regression labels are weighted to obtain the category label 21×0.8+22.43×0.2=21.3℃.

[0038] S130: Divide the plurality of example packets into a plurality of subsets based on pre-set division parameters.

[0039] The division parameter may include a preset number of divisions, which may refer to the total number of sets into which the multiple example packages corresponding to each category label are evenly divided, or may represent the number of subsets. The multiple example packages may each correspond to a category label.

[0040] As a method, based on a preset number of divisions, the multiple example packages corresponding to each category label can be evenly divided into multiple groups of sets to obtain multiple groups of sets corresponding to each category label; the multiple groups of sets corresponding to each category label can be merged into multiple subsets, each subset containing a group of sets corresponding to each category label.

[0041] Optionally, a set can be randomly selected from the multiple sets corresponding to each category label, and the selected sets can be merged into a subset. A set can be randomly selected from the unselected sets corresponding to each category label, and the selected sets can be merged into a subset again, until all sets are merged into subsets, so as to obtain multiple subsets.

[0042] For example, the preset number of divisions may be K, and the category labels corresponding to the example packages may be A, B, C, or D. Taking category label C as an example, the example packages corresponding to category label C may be M. c Where c only represents the sample package category label, that is, the number of sample packages corresponding to the category label C is M, then the multiple sample packages corresponding to the category label C can be evenly divided into K groups of sets, each of which can contain example packages, so we can get the K group sets corresponding to categories A, B, C, and D. The K group set corresponding to category A can be expressed as: A , 1 A , 2 A , ..., (K-1) A ; ...; The set of K groups corresponding to category D can be expressed as: 0 D , 1 D , 2 D , ..., (K-1) D .

[0043] Randomly extracting a set of sets from each of the K sets corresponding to categories A, B, C, and D can yield four sets of sets from different category labels (1 A , 3 B , 2 C , 0 D ), then merge the four sets into a subset, and continue to randomly extract a set from each of the unselected sets corresponding to categories A, B, C, and D (0 A , 2 B , 3 C , 1 D ), and merge the extracted sets into a subset again until all sets are merged into subsets.

[0044] Optionally, set numbers may be set for each of the multiple sets corresponding to each category label, and then all sets of category labels with the same set number may be merged into one subset to obtain multiple subsets.

[0045] For example, the preset number of divisions may be K, and the category labels corresponding to the example packages may be A, B, C, or D. Taking category label C as an example, the example packages corresponding to category label C may be M. cWhere c only represents the sample package category label, that is, the number of sample packages corresponding to the category label C is M, then the multiple sample packages corresponding to the category label C can be evenly divided into K groups of sets, each of which can contain example packages, so we can get the K group sets corresponding to categories A, B, C, and D. The K group set corresponding to category A can be expressed as: A , 1 A , 2 A , ..., (K-1) A ; ...; The set of K groups corresponding to category D can be expressed as: 0 D , 1 D , 2 D , ..., (K-1) D The set of all category labels with set number 0 can be merged into subsets, and the resulting multiple subsets can be: (0 A , 0 B , 0 C , 0 D ), (1 A , 1 B , 1 C , 1 D ),...,((K-1) A , (K-1) B , (K-1) C , (K-1) D ).

[0046] In the embodiments of the present application, by randomly merging the multiple sets corresponding to each category label into multiple subsets, the randomness of the dataset can be enhanced, thereby ensuring good generalization of the training task corresponding to the dataset. Furthermore, by evenly dividing the multiple example packages corresponding to each category label into multiple sets based on a preset number of divisions, the resulting multiple sets corresponding to each category label can ensure that the distribution of the example packages corresponding to each set is consistent with the distribution of the example packages corresponding to the data samples to be processed, thereby improving the accuracy and generalization of the training task corresponding to the dataset.

[0047] S140: Constructing a data set based on the multiple subsets.

[0048] The data set may include multiple data sets, for example, the data set may include a training set, a validation set, and a test set.

[0049] As one approach, multiple subsets can be divided into multiple data sets based on a preset ratio. For example, there can be 10 subsets, and the preset ratio can be training set: validation set: test set = 8:1:1. Then, 8 subsets can be randomly selected from the 10 subsets to form the training set, 1 subset to form the validation set, and 1 subset to form the test set.

[0050] Optionally, multiple subsets may be divided into training sets and validation sets based on a preset ratio, and then other data samples may be obtained to construct a test set.

[0051] Optionally, when the data samples in the training set are imbalanced in categories or have a long-tail distribution, the example packages in the training set can be resampled to ensure that the distribution of each training sample in the training task is uniform. The resampling method may include, but is not limited to, undersampling the majority class example packages (the number of all data samples of multiple example packages corresponding to the same category label is too large), oversampling the minority class example packages (the number of all data samples of multiple example packages corresponding to the same category label is too small), undersampling the majority class example packages and oversampling the minority class example packages, or fixing the distribution of example packages in each training sample in accordance with OHEM (Online Hard Example Mining). More specifically, a first threshold and a second threshold can be set. The first threshold can be used to determine whether the example package is a majority class example package, and the second threshold can be used to determine whether the example package is a minority class example package. If the number of all data samples of multiple example packages corresponding to a certain category label is greater than the first threshold, it can be indicated that the multiple example packages corresponding to the category label are majority class example packages, and undersampling can be performed; if the number of all data samples of multiple example packages corresponding to a certain category label is less than the second threshold, it can be indicated that the multiple example packages corresponding to the category label are minority class example packages, and oversampling can be performed.

[0052] This embodiment provides a dataset construction method. After obtaining data samples to be processed, data samples that meet the same preset correlation relationship are grouped into the same example package to obtain multiple example packages. Based on pre-set division parameters, the multiple example packages are divided into multiple subsets, and a dataset is constructed based on the multiple subsets. This method allows data samples that meet the same preset correlation coefficient to be grouped into the same example package to obtain multiple example packages. As a result, samples in the dataset obtained based on the multiple example packages also have a correlation relationship, thereby improving the mineability of sample information in the dataset and, in turn, improving the accuracy of the training task corresponding to the dataset.

[0053] See also Figure 2 , the present application provides a method for constructing a data set, the method comprising:

[0054] S210: Obtain data samples to be processed.

[0055] S220: Divide the data samples that meet the same preset correlation relationship into the same example package to obtain multiple example packages.

[0056] S230: In the current division process corresponding to the category label currently being divided, a preset number of example packages are selected from multiple example packages to be divided corresponding to the current division process, where the preset number is obtained based on the number of example packages of the category label and the preset number of divisions.

[0057] In the embodiment of the present application, the calculation formula for the preset number is:

[0058]

[0059] Among them, c can represent the current classification category label C, M c It can be said that the number of example packages corresponding to the category label C is M, K can represent the preset number of divisions, X c It can represent the preset number corresponding to the category label C.

[0060] As a method, the multiple example packages to be divided corresponding to the current division process can each correspond to an example package number, and the order of the multiple example packages to be divided corresponding to the current division process can be randomly rearranged based on the example package number, and the first preset number of example packages can be selected from the randomly rearranged multiple example packages to be divided to obtain the preset number of example packages corresponding to the current division process.

[0061] For example, the preset number of divisions may be K, and the category labels corresponding to the example packages may be A, B, C, or D. Taking category label C as an example, the example packages corresponding to category label C may be M. c Where c only represents the sample package category label, that is, the number of sample packages corresponding to the category label C is M, then the multiple sample packages corresponding to the category label C can be evenly divided into K groups of sets, each of which can contain If there are sample packages, a set of random seeds can be set. Each random seed can be the same as the sample package number. Then each random seed can be matched with the sample package number to obtain a preset number of sample packages corresponding to the current division process. Taking category label C as an example, the sample package number corresponding to category label C can be: {0 C , 1 C , 2 C , ..., (M-1) C}, a set of random seeds can be: {0 C , 8 C , ..., (M-1) C , 5 C}, among which, the front A random seed can be: {0 C , 8 C ,...,6 C}, it means that the number of the preset number of example packages corresponding to the current division process can be {0 C , 8 C ,...,6 C}.

[0062] In an embodiment of the present application, by randomly rearranging the order of multiple example packages to be divided corresponding to the current division process, the randomness of the data set can be enhanced, so that the training task corresponding to the data set can have good generalization ability.

[0063] S240: If the number of data samples included in the preset number of example packages meets the target condition, the preset number of example packages are divided into a group of sets, and the next division process corresponding to the category label currently being divided is entered, and the example packages that have not been divided in the example packages corresponding to the category label currently being divided are used as multiple example packages to be divided corresponding to the next division process, until the number of divided sets is the same as the preset number of divisions, so as to obtain multiple groups of sets corresponding to each of the category labels.

[0064] The partitioning parameters may further include a fluctuation coefficient, which represents the fluctuation range of the data samples included in a set. The larger the fluctuation coefficient, the larger the fluctuation range of the data samples included in a set. The relationship between the fluctuation coefficient and the fluctuation range of the data samples included in a set may be as follows:

[0065]

[0066] Among them, N can represent the total number of data samples included in the category label currently being divided, K can represent the preset number of divisions, α can represent the fluctuation coefficient, and N x It can represent the number of data samples included in any set of category labels currently being divided.

[0067] As a method, the total number of data samples included in the category label currently being divided can be obtained, and the target interval can be obtained based on the fluctuation coefficient, the preset number of divisions, and the total number of data samples. If the number of data samples included in the preset number of sample packages is within the target interval, it can be determined that the number of samples meets the target condition. The calculation formula for the target interval is as follows:

[0068]

[0069] Wherein, Z may represent the number of data samples included in the preset number of example packages obtained in the current division process, N may represent the total number of data samples included in the category labels currently being divided, K may represent the preset number of divisions, and β may represent the target interval control coefficient, which may be calculated based on the following formula:

[0070]

[0071] Where K can represent the preset number of divisions, and α can represent the fluctuation coefficient. The calculation formulas of β and α are as follows:

[0072] Since the number of data samples included in each example package can be different, the number of data samples included in the multiple sets corresponding to the category labels currently being divided can also be different. In order to ensure that the number of data samples in each set of the category labels currently being divided is within the data sample fluctuation range, the target interval control coefficient β can be introduced. If the solution space corresponding to the category labels currently being divided intersects with the receptive domain determined based on the target interval control coefficient β, it can be indicated that the category labels currently being divided are correctly divided into multiple sets.

[0073] In an embodiment of the present application, the solution space may represent the data sample quantity intervals corresponding to each of the multiple sets of category labels currently being divided.

[0074] Taking the category label C as an example, the solution space corresponding to the category label C can be in, It can be defined as the number of data samples in the set with the least number of examples in the class label C. It is defined as the number of data samples in the set with the least number of examples in the class label C. The receptive field can be Among them, N c It can represent the total number of data samples included in the category label C, K can represent the preset number of divisions, and β can represent the target interval control coefficient. This indicates that the category labels C are correctly divided into multiple groups.

[0075] Furthermore, according to the fluctuation coefficient, any set of category labels C must satisfy the following Assuming that K-1 partitions are performed, the last remaining set must also satisfy

[0076] And, for any set, when When , the solution space intersects with the receptive domain, then the sum of the data samples of the first K-1 divisions can be:

[0077]

[0078] Then, the last set corresponding to the category label C can satisfy:

[0079]

[0080] Right now

[0081] therefore:

[0082] and Calculated

[0083] For example, if the number of data samples included in the preset number of sample packages is within the target range A preset number of example packages can be divided into a group of sets, and the next division process corresponding to the category label currently being divided is entered, and the example packages that have not been divided in the example packages corresponding to the category label currently being divided are used as multiple example packages to be divided corresponding to the next division process, until the number of divided sets is the same as the preset number of divisions, so as to obtain multiple groups of sets corresponding to each category label.

[0084] Optionally, if the number of data samples included in the preset number of example packages does not meet the target condition, the division parameters are adjusted, and based on the adjusted division parameters, the first division process corresponding to the category label currently being divided is restarted.

[0085] Optionally, if the number of data samples included in the preset number of example packages does not meet the target condition, at least one parameter among the fluctuation coefficient and the preset number of divisions may be adjusted.

[0086] Among them, as a method, if the number of data samples included in the preset number of example packages does not meet the target conditions, the fluctuation coefficient can be increased, so that the range of the target interval is also increased, making it easier to obtain a set that meets the conditions. In this way, the set division of subsequent category labels can be started from the category label currently being divided, that is, there is no need to re-divide the set of category labels before the category label currently being divided, thereby saving computing resources and obtaining calculation results faster. Because the previous target range is smaller than the adjusted target range, the set that previously met the target conditions also meets the adjusted target conditions.

[0087] Alternatively, if the number of data samples included in the preset number of example packages does not meet the target criteria, the preset number of partitions can be reduced to expand the target interval, making it easier to obtain a set that meets the criteria. In this way, the fluctuation coefficient can be left unchanged, making the distribution of data samples in each set more similar to the distribution of the data samples to be processed, thereby improving the accuracy of the training task corresponding to the dataset.

[0088] As another approach, if the number of data samples included in the preset number of example packages does not meet the target conditions, the preset number of divisions can be reduced and the fluctuation coefficient increased to expand the range of the target interval, making it easier to obtain a set that meets the conditions. In this approach, the fluctuation coefficient and the preset number of divisions can be fine-tuned simultaneously, making the distribution of example packages in each set more similar to the distribution of all example packages corresponding to the data samples to be processed, and the distribution of data samples in each set more similar to the distribution of the data samples to be processed, thereby improving the accuracy of the training task corresponding to the dataset.

[0089] S250: Constructing a data set based on the multiple subsets.

[0090] This embodiment provides a dataset construction method that, through the aforementioned method, groups data samples that meet the same preset correlation coefficient into the same example package, thereby obtaining multiple example packages. This allows samples in the dataset obtained based on the multiple example packages to also be correlated, thereby improving the mineability of sample information in the dataset and, in turn, the accuracy of the training task corresponding to the dataset. Furthermore, in this embodiment, based on the fluctuation coefficient, the preset number of divisions, and the total number of data samples, it is determined whether the set obtained during the current division of the category label meets the target condition. This ensures that the distribution of data samples in each set is similar to the distribution of the data samples to be processed, thereby improving the accuracy and generalization ability of the training task corresponding to the dataset.

[0091] In order to better understand the solutions of all embodiments of the present application, a business process of the data set construction method of the present application is introduced below.

[0092] See also Figure 3, you can first execute step S310 to set the division parameters, then execute step S320 to obtain the data samples to be processed based on the training task, and then execute step S330 to divide the data samples that meet the same preset correlation relationship into the same example package to obtain multiple example packages, and each example package can have a category label, and then you can respectively perform set division on the example packages corresponding to each category label based on steps S340 to S370 to obtain multiple groups of sets corresponding to each category label, so that you can merge the multiple groups of sets corresponding to each category label into multiple subsets based on step S380, and execute step S390 to construct a data set based on the multiple subsets.

[0093] In the current division process corresponding to the category label currently being divided, a preset number of example packages can be selected from the multiple example packages to be divided corresponding to the current division process based on step S340. The preset number can be obtained based on the number of example packages of the category label and the preset number of divisions. Then, step S350 is performed to verify the result of the current division process by determining whether the number of data samples included in the preset number of example packages meets the target condition. If the target condition is met, it can be indicated that the verification is passed. Based on step S360, the preset number of example packages can be divided into a group of sets, and the next division process corresponding to the category label currently being divided is entered. The example packages in the example packages corresponding to the category label currently being divided that have not yet been divided are used as the multiple example packages to be divided corresponding to the next division process until the number of divided sets is the same as the preset number of divisions, so as to obtain multiple groups of sets corresponding to each category label. If the target condition is not met, it can be indicated that the verification is failed. The division parameters can be adjusted based on step S370, and the first division process corresponding to the category label currently being divided is restarted based on the adjusted division parameters.

[0094] See also Figure 4 The present application provides a data set construction device 400, which runs on an electronic device. The device 400 includes:

[0095] The data sample acquisition unit 410 is configured to acquire data samples to be processed.

[0096] The sample package acquisition unit 420 is configured to group data samples that meet the same preset correlation coefficient into the same sample package to obtain multiple sample packages.

[0097] The subset division unit 430 is configured to divide the plurality of example packets into a plurality of subsets based on preset division parameters.

[0098] The data set construction unit 440 is configured to construct a data set based on the multiple subsets.

[0099] As a method, each of the data samples corresponds to a category label, and each of the multiple example packages corresponds to a category label. The example package acquisition unit 420 is specifically used to determine the category labels corresponding to each of the multiple example packages based on the category labels of the data samples corresponding to each of the multiple example packages.

[0100] As a method, the division parameters include a preset number of divisions, and the multiple example packages each correspond to a category label. The subset division unit 430 is specifically used to divide the multiple example packages corresponding to each category label into multiple groups of sets based on the preset number of divisions, and obtain multiple groups of sets corresponding to each category label; merge the multiple groups of sets corresponding to each category label into the multiple subsets, and each subset contains a group of sets corresponding to each category label.

[0101] Among them, optionally, the subset division unit 430 is specifically used to select a preset number of example packages from multiple example packages to be divided corresponding to the current division process corresponding to the category label currently being divided, and the preset number is obtained based on the number of example packages of the category label and the preset number of divisions; if the number of data samples included in the preset number of example packages meets the target condition, the preset number of example packages are divided into a group of sets, and the next division process corresponding to the category label currently being divided is entered, and the example packages that have not been divided in the example packages corresponding to the category label currently being divided are used as multiple example packages to be divided corresponding to the next division process, until the number of divided sets is the same as the preset number of divisions, so as to obtain multiple groups of sets corresponding to each of the category labels.

[0102] Optionally, the division parameters also include a fluctuation coefficient, which represents the fluctuation range of the data samples included in a set. The subset division unit 430 is specifically used to obtain the total number of data samples included in the category label currently being divided; obtain the target interval based on the fluctuation coefficient, the preset number of divisions and the total number of data samples; if the number of data samples included in the preset number of example packages is within the target interval, it is determined that the number of examples meets the target condition.

[0103] Optionally, the subset partitioning unit 430 is specifically used to adjust the partitioning parameters if the number of data samples included in the preset number of example packages does not meet the target conditions, and restart the first partitioning process corresponding to the category label currently being partitioned based on the adjusted partitioning parameters.

[0104] Optionally, the division parameter further includes a fluctuation coefficient, and the subset division unit 430 is specifically configured to adjust at least one parameter among the fluctuation coefficient and the preset number of divisions.

[0105] As an approach, the data set includes multiple data sets, and the data set construction unit 440 is specifically configured to divide the multiple subsets into the multiple data sets based on a preset ratio.

[0106] The following will be combined Figure 5 An electronic device provided by this application is described.

[0107] See also Figure 5 Based on the above-mentioned dataset construction method and apparatus, the embodiments of the present application also provide another electronic device 100 that can execute the above-mentioned dataset construction method. The electronic device 100 includes one or more (only one is shown in the figure) processors 102 and a memory 104 that are coupled to each other. The memory 104 stores a program that can execute the content of the above-mentioned embodiments, and the processor 102 can execute the program stored in the memory 104.

[0108] The processor 102 may include one or more processing cores. The processor 102 utilizes various interfaces and circuits to connect various components within the electronic device 100. It executes instructions, programs, code sets, or instruction sets stored in the memory 104, and accesses data stored in the memory 104 to perform various functions and process data for the electronic device 100. Optionally, the processor 102 may be implemented using at least one of the following hardware forms: a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 102 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily processes the operating system, user interface, and application programs; the modem handles wireless communications. It is understood that the modem may not be integrated into the processor 102 and may be implemented separately via a communication chip.

[0109] The memory 104 may include a random access memory (RAM) or a read-only memory (ROM). The memory 104 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 104 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data created by the terminal 100 during use (such as a phone book, audio and video data, chat history data), etc.

[0110] Please refer to Figure 6 , which shows a block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable storage medium 800 stores program code, which can be called by a processor to execute the method described in the above method embodiment.

[0111] The computer-readable storage medium 1000 can be an electronic memory such as a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), an EPROM, a hard disk, or a ROM. Alternatively, the computer-readable storage medium 800 comprises a non-transitory computer-readable storage medium. The computer-readable storage medium 800 has storage space for program code 810 for executing any of the method steps described above. These program codes can be read from or written to one or more computer program products. The program code 810 can be compressed, for example, in a suitable form.

[0112] In summary, the present application provides a data set construction method, device, and electronic device. After obtaining data samples to be processed, data samples that meet the same preset correlation relationship are divided into the same example package to obtain multiple example packages. Based on pre-set division parameters, the multiple example packages are divided into multiple subsets, and a data set is constructed based on the multiple subsets. Through the above method, data samples that meet the same preset correlation coefficient are divided into the same example package to obtain multiple example packages, so that the samples in the data set obtained based on the multiple example packages also have a correlation relationship, thereby improving the mineability of the sample information in the data set, and thereby improving the accuracy of the training task corresponding to the data set.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for constructing a data set, characterized in that: Applied to electronic equipment, the method includes: Obtaining a data sample to be processed, wherein the data sample is a sample for an artificial intelligence training task, and the data sample is an image with a category label determined based on the purpose of the training task; Grouping data samples that satisfy the same preset correlation into the same example package to obtain multiple example packages, wherein the correlation includes a mapping relationship between data samples and example packages, the mapping relationship represents an association between data samples included in the same example package, and the association represents a common attribute of the data samples included in the same example package; Determining the category labels corresponding to the multiple example packages based on the category labels of the data samples corresponding to the multiple example packages; Based on the preset number of divisions included in the division parameter, the multiple example packages corresponding to each of the category labels are evenly divided into multiple groups of sets to obtain multiple groups of sets corresponding to each of the category labels; A set is randomly selected from each of the multiple sets corresponding to each category label, and the selected sets are merged into a subset; a set is randomly selected from each of the unselected sets corresponding to each category label, and the selected sets are merged into a subset again, until all sets are merged into subsets, thereby obtaining multiple subsets; A data set is constructed based on the multiple subsets to perform the training task using the constructed data set, wherein the purpose of the training task is to train an artificial intelligence model for image classification.

2. The method according to claim 1, characterized in that The method of evenly dividing the plurality of example packages corresponding to each of the category labels into a plurality of sets based on the preset number of divisions included in the division parameter, to obtain the plurality of sets corresponding to each of the category labels, comprises: In a current division process corresponding to a category label currently being divided, selecting a preset number of example packages from a plurality of example packages to be divided corresponding to the current division process, wherein the preset number is obtained based on the number of example packages of the category label and the preset number of divisions; If the number of data samples included in the preset number of example packages meets the target condition, the preset number of example packages are divided into a group of sets, and the next division process corresponding to the category label currently being divided is entered, and the example packages that have not been divided in the example packages corresponding to the category label currently being divided are used as multiple example packages to be divided corresponding to the next division process, until the number of divided sets is the same as the preset number of divisions, so as to obtain multiple groups of sets corresponding to each of the category labels.

3. The method according to claim 2, characterized in that The partitioning parameter further includes a fluctuation coefficient, wherein the fluctuation coefficient represents a fluctuation range of the data samples included in a set. The method further includes: Get the total number of data samples included in the category label currently being divided; Obtaining a target interval based on the fluctuation coefficient, the preset number of divisions, and the total number of data samples; If the number of data samples included in the preset number of example packets is within the target interval, it is determined that the number of data samples included in the preset number of example packets meets the target condition.

4. The method according to claim 2, characterized in that The method further comprises: If the number of data samples included in the preset number of example packages does not meet the target condition, the division parameter is adjusted, and based on the adjusted division parameter, the first division process corresponding to the category label currently being divided is restarted.

5. The method according to claim 4, wherein the partition parameter further comprises a fluctuation coefficient, and the adjusting the partition parameter comprises: Adjust at least one parameter of the fluctuation coefficient and the preset number of divisions.

6. The method according to claim 1, characterized in that The data set includes multiple data sets, and constructing the data set based on the multiple subsets includes: The plurality of subsets are divided into the plurality of data sets based on a preset ratio.

7. A data set construction device, characterized in that: Running on an electronic device, the device includes: A data sample acquisition unit, configured to acquire data samples to be processed, wherein the data samples are samples for artificial intelligence training tasks, and the data samples are images with category labels determined based on the purpose of the training tasks; an example package acquisition unit, configured to group data samples that satisfy the same preset correlation into the same example package to obtain multiple example packages, wherein the preset correlation includes a mapping relationship between data samples and example packages, wherein the mapping relationship represents an association between data samples included in the same example package, and wherein the association represents a common attribute of the data samples included in the same example package; a subset division unit, configured to determine, based on the category labels of the data samples corresponding to the multiple example packages, the category labels corresponding to each of the multiple example packages; divide the multiple example packages corresponding to each of the category labels into multiple sets evenly based on a preset number of divisions included in a division parameter, to obtain multiple sets corresponding to each of the category labels; randomly extract a set from the multiple sets corresponding to each of the category labels, and merge the extracted sets into a subset; and randomly extract a set from the unextracted sets corresponding to each of the category labels, and merge the extracted sets again into a subset, until all sets are merged into subsets, to obtain multiple subsets; A data set construction unit is used to construct a data set based on the multiple subsets, so as to perform the training task through the constructed data set, wherein the purpose of the training task is to train and obtain an artificial intelligence model for image classification.

8. An electronic device, characterized in that: including one or more processors and memory; One or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores program code, wherein when the program code is run, the method according to any one of claims 1 to 6 is executed.

Images