Data processing method, chip and related equipment

Abstract

The invention provides a data processing method and related equipment, and the method comprises the steps: carrying out the clustering of first sample data, and obtaining d clustering results; extracting centroid sample data in each clustering result in the d clustering results to obtain second sample data; sending a first request for requesting to label the second sample data; receiving a first label labeled for the second sample data; and training the first model based on the second sample data and the first label of the second sample data to obtain a trained first model. The performance of the model obtained by training can be improved while the model training efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, a chip and related equipment. Background Art

[0002] With the development of intelligent technology, people's demand for various intelligent products is also increasing, and a large number of intelligent products have emerged for corresponding processing. For example, the processing may include but is not limited to valuation, classification, etc. For example, as people's demand for materials increases, they are paying more and more attention to the value of materials. Therefore, it is necessary to value materials (for example, including but not limited to assets, real estate, vehicles, etc.). Valuation can be understood as an assessment of value. For another example, as the types of items increase, it is necessary to classify the items. Summary of the invention

[0003] The embodiments of the present application provide a data processing method, a chip and related equipment, which are beneficial to improving the efficiency of model training while improving the performance of the trained model.

[0004] In order to solve the above technical problems, this application is implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides a data processing method, including:

[0006] Cluster the first sample data to obtain d clustering results, where d is a positive integer;

[0007] Extracting centroid sample data from each clustering result in the d clustering results to obtain second sample data;

[0008] Sending a first request, where the first request is used to request labeling of the second sample data;

[0009] Receiving a first label annotated for the second sample data;

[0010] Based on the second sample data and the first label of the second sample data, the first model is trained to obtain a trained first model.

[0011] In a second aspect, the present application also provides a processing method, including:

[0012] Acquire feature data of an object to be processed in at least one of a valuation scenario and a classification scenario;

[0013] Inputting the feature data of the object to be processed into a trained first model, performing at least one of valuation and classification by the trained first model, and obtaining a processing result;

[0014] The trained first model is obtained by training using the method of the first aspect mentioned above.

[0015] In a third aspect, a chip is provided, the chip comprising: a controller, a processor, an input buffer and an output buffer, including:

[0016] The controller loads the first sample data into the input buffer;

[0017] The processor clusters the first sample data to obtain d clustering results, where d is a positive integer; extracts the centroid sample data in each of the d clustering results to obtain second sample data; sends a first request, where the first request is used to request labeling of the second sample data; receives a first label labeled for the second sample data; trains a first model based on the second sample data and the first label of the second sample data to obtain a trained first model, and stores the trained first model in the output cache.

[0018] In a fourth aspect, an embodiment of the present application further provides a data processing device, including:

[0019] A clustering module, used for clustering the first sample data to obtain d clustering results, where d is a positive integer;

[0020] A sample extraction module, used to extract centroid sample data from each of the d clustering results to obtain second sample data;

[0021] A sending module, used for sending a first request, where the first request is used for requesting to label the second sample data;

[0022] A receiving module, configured to receive a first label annotated for the second sample data;

[0023] A training module is used to train the first model based on the second sample data and the first label of the second sample data to obtain a trained first model.

[0024] In a fifth aspect, an embodiment of the present application further provides a processing device, including:

[0025] An object feature acquisition module, used to acquire feature data of an object to be processed in at least one of a valuation scenario and a classification scenario;

[0026] A first processing module, used for inputting the feature data of the object to be processed into a trained first model, performing at least one of valuation and classification through the trained first model, and obtaining a processing result;

[0027] The trained first model is obtained by training using the method of the first aspect mentioned above.

[0028] In a sixth aspect, an embodiment of the present application further provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the above method when executing the computer program.

[0029] In a seventh aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above method are implemented.

[0030] In an eighth aspect, an embodiment of the present application further provides a computer program product, comprising a computer program, which implements the steps in the above method when executed by a processor.

[0031] It can be seen that in an embodiment of the present application, the first sample data can be clustered first, and the centroid sample data can be selected from the d clustering results to obtain the second sample data, and then the first label of the second sample data can be obtained. The first model can be trained using the second sample data and the first label of the second sample data to obtain the trained first model, which can be used for valuation and / or classification, etc. The centroid sample data of the clustering result best represents the sample data of this type of clustering result, and is the sample data of higher quality in the clustering result. That is, in this embodiment, the first sample data can be firstly clustered, and the centroid sample data can be screened in each cluster of the d clustering results. While reducing the amount of sample data, the centroid sample data that best represents the cluster in each cluster can be obtained to form second sample data of higher quality. Afterwards, there is no need to label all the first sample data, and it is only necessary to request the labeling of the second sample data, and then receive the first label labeled for the second sample data, which can improve the efficiency of obtaining the label of the sample data. In this way, a small number of centroid sample data that can represent each type of cluster and its first label can be used for model training, which can improve the efficiency of model training and the performance of the trained model. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0033] Figure 1 It is one of the flow charts of a data processing method provided in an embodiment of the present application;

[0034] Figure 2This is the second flowchart of a data processing method provided in an embodiment of the present application;

[0035] Figure 3 is a schematic diagram of a data processing method provided in an embodiment of the present application;

[0036] Figure 4 This is the second flowchart of a data processing method provided in an embodiment of the present application;

[0037] Figure 5 is a structural schematic diagram of a data processing device provided in an embodiment of the present application;

[0038] Figure 6 is a structural schematic diagram of a data processing device provided in an embodiment of the present application;

[0039] Figure 7 This is one of the structural schematic diagrams of an electronic device provided in an embodiment of the present application;

[0040] Figure 8 This is a second structural diagram of an electronic device provided in an embodiment of the present application;

[0041] Fig. 9 This is one of the structural schematic diagrams of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0043] See also Figure 1 , Figure 1 1 is a flow chart of a data processing method provided by an embodiment of the present application. The data processing method of this embodiment can be applied to a first electronic device, which can be a mobile device or a non-mobile device. Figure 1 As shown, the following steps are included:

[0044] Step 101: cluster the first sample data to obtain d clustering results, where d is a positive integer.

[0045] The data processing method can be applied to multiple scenarios, for example, it can be applied to but not limited to at least one of the scenarios such as valuation scenarios and classification scenarios. The first sample data may include sample data in the scenario to which the method is applied. For example, the first sample data may include but not limited to sample data in at least one of the scenarios such as valuation scenarios and classification scenarios. It can be understood that the first sample data may include feature data of multiple samples in the scenario to which the method is applied. In addition, it should be noted that the valuation scenario can be understood as a scenario for estimating the value of an object, and the classification scenario can be understood as a scenario for classifying an object. In the first sample data, the feature data of the sample may be data that characterizes the features of the sample, which is not specifically limited in the embodiments of the present application. For example, as an example, in the valuation scenario, the first feature data of the sample in the first sample data may include the attribute features of the sample, for example, including but not limited to at least one of year, newness, size, location, material, etc. As an example, in the classification scenario, the feature data of the sample in the first sample data may include features of the sample that can be used to distinguish categories. For example, in the face classification scenario, it may include face feature data, for example, including but not limited to face contour, facial features, etc. In addition, the characteristic data of each sample can be used as a characteristic vector respectively. In the clustering process, clustering can be performed according to the distance between the characteristic data (ie, characteristic vectors) of each sample.

[0046] In this embodiment, the first sample data may be clustered to obtain d clustering results. There are many clustering methods, which are not specifically limited in this embodiment. For example, by way of example, a K-center clustering algorithm may be used to cluster the first sample data to obtain d clustering results. Each clustering result may include at least one sample data in the first sample data.

[0047] Step 102: extracting centroid sample data from each clustering result in d clustering results to obtain second sample data;

[0048] Step 103: Send a first request, where the first request is used to request labeling of the second sample data;

[0049] Step 104: Receive a first label annotated for second sample data.

[0050] It should be understood that clustering is to aggregate data with similarity together to obtain clustering results. For example, the first sample data can be clustered to obtain d clustering results, each of which has a centroid. In this embodiment, the centroid can be understood as the center of a clustering result. In this embodiment, the centroid sample data in the clustering result can be understood as the central sample in the clustering result. For example, it can be a sample data in the clustering result that is closest to the mean of the sample data in the clustering result. For example, a clustering result includes m sample data, and sample data X is a sample data that is closest to the mean of the m sample data in the clustering result. Then the sample data X is the central sample in the clustering result, that is, the centroid sample data in the clustering result. In this embodiment, the centroid sample data in each of the d clustering results can be extracted as the second sample data to be labeled, and then a first request is sent to request the second sample data to be labeled, and a first label labeled for the second sample data is received to obtain the first label of the second sample data. Exemplarily, a first request may be sent to the electronic device corresponding to the labeling personnel, and after receiving the first request, the electronic device corresponding to the labeling personnel displays request information for labeling the second sample data. The labeling personnel may input the first label for labeling through the electronic device, and the electronic device corresponding to the labeling personnel returns the first label and receives the first label for labeling the second sample data sent by the electronic device corresponding to the labeling personnel.

[0051] In addition, it should be noted that the first sample data may be sample data with a second label, or may be sample data without a label. The second label is a label obtained in a different manner from the first label. For example, the first label may be a label with a high accuracy for labeling professional objects in the scene field to which the method is applied. For the second label, third-party data may be retrieved to obtain the second label, or the second label may be obtained by labeling it through preset software or a preset model.

[0052] Step 105: Based on the second sample data and the first label of the second sample data, the first model is trained to obtain a trained first model.

[0053] It should be understood that the first model may be a model corresponding to the scenario to which the method of the present application is applied. For example, the application scenario may include but is not limited to at least one of a valuation scenario and a classification scenario, and the first model may include but is not limited to at least one of a valuation model and a classification model. The first label is a label under the scenario to which the method is applied, and the trained first model can be used to process the object to be processed corresponding to the scenario, that is, the object to be processed can be obtained later, and based on the trained first model, at least one of valuation and classification can be performed on the object to be processed, but not limited to. For example, for the valuation scenario, the first model includes a valuation model, the object to be processed may be an object to be valued, the first label is a first valuation label, and the valuation model can be used to value the object to be valued. For the classification scenario, the first model includes a classification model, the object to be processed may be an object to be classified, the first label is a first classification label, and the classification model can be used to classify the object to be classified.

[0054] In the embodiment of the present application, the first sample data may be clustered first, and the centroid sample data may be selected from the d clustering results to obtain the second sample data, and then the first label of the second sample data may be obtained, and the first model may be trained using the second sample data and the first label of the second sample data to obtain the trained first model, which may be used for valuation and / or classification, etc. The centroid sample data of the clustering result is the most representative of the sample data of the clustering result, and is the sample data of higher quality in the clustering result, that is, in the present embodiment, the first sample data may be clustered first, and the centroid sample data may be selected from each cluster of the d clustering results, and the centroid sample data of each cluster that best represents the cluster may be obtained while reducing the amount of sample data, thereby forming the second sample data of higher quality, and then, there is no need to label all the first sample data, and only the second sample data may be requested to be labeled, and then the first label labeled for the second sample data may be received, which may improve the efficiency of obtaining the label of the sample data, and thus, the centroid sample data and the first label of each cluster that are less in number and can represent the centroid sample data of each cluster and can be used for model training, and the performance of the trained model may be improved while improving the efficiency of model training, etc.

[0055] In some embodiments, the first label may be an expert label, which may be a label used by a professional object corresponding to the scenario field to which the method is applied (e.g., a professional or professional organization in the scenario field) to annotate the second sample data. The first label incorporates the experience of the professional object and has a high accuracy. The centroid sample data of each clustering result is the sample data that best represents this category in the clustering result. In this embodiment, the centroid sample data of each clustering result in d clustering results may be selected to form the second sample data. The number of centroid sample data in the second sample data is d, and then the first label annotated for each sample data in the second sample data is received for training. In this way, a small number of centroid sample data with more accurate labels and representative of each type of cluster may be used for model training, thereby improving the efficiency of model training and the performance of the trained model. In addition, it should be noted that, as an example, the labeler can also be an ordinary labeler. Since the centroid sample data of the clustering result can best represent the sample data of the clustering result, it is the better sample data in the clustering result, and its category is obvious and easy to determine. Therefore, after selecting the centroid sample data in each cluster of the d clustering results to obtain the second sample data, even an ordinary labeler can perform more accurate labeling. In this way, a smaller number of centroid sample data with more accurate labels and representative of each type of cluster can be used for model training, which can improve the efficiency of model training and improve the performance of the trained model.

[0056] Exemplarily, the first label may be a label annotated with a professional object corresponding to at least one of the valuation scenario and the classification scenario (for example, a person who has an expert identity corresponding to at least one of the valuation scenario and the classification scenario). For example, for the valuation scenario, the first label may be a label used by a professional valuation person for valuation, that is, a valuation label annotated by a valuation expert. For another example, for the classification scenario, the first label may be a label used by a professional classifier for classification, that is, a valuation label annotated by a classification expert. In this way, a small number of centroid sample data with more accurate labels and representative of each type of cluster can be used to train the first model, thereby improving the model training efficiency while improving the valuation and / or classification performance of the trained model. In addition, it should be noted that the first sample data has a corresponding second label, and the second label is a label obtained in a different manner from the first label.

[0057] In some embodiments, the number of first models is p, where p is an integer greater than 1;

[0058] Based on the second sample data and the first label of the second sample data, the first model is trained to obtain the trained first model, including:

[0059] According to the first sampling quantity, the second sample data is sampled p times to obtain p third sample data sets, the number of samples in each third sample data set is the first sampling quantity, and p is an integer greater than 1;

[0060] Based on the p third sample data sets and the first labels of the sample data in the p third sample data sets, the p first models are trained to obtain the trained p first models.

[0061] It should be understood that the first sampling number can be set in advance according to the needs without specific limitation. For example, sampling can be sampling with replacement, and the first sampling number can be d, that is, it can be the same as the number of clusters d of the first sample data. The size of the number of samples in the third sample data set of each sampling is d, that is, there are d sample data. It should be understood that the trained p first models correspond to the p third sample data sets one by one, that is, for each third sample data set in the p third sample data sets, a corresponding first model is obtained for training.

[0062] In this embodiment, it is only necessary to obtain the first label of the second sample data that can represent the d-class sample data, and then sample the second sample data p times to obtain p third sample data sets. That is, for training p first models, first select the centroid sample data from the d clustering results to obtain the second sample data, sample based on the second sample data that can represent the d-class sample data, and use the sampled p third sample data and the corresponding first label to perform model training, and then obtain the trained p first models. On the one hand, there is no need to obtain a large amount of labeled data during the training process, and only the first label of the third sample data set sampled from the centroid sample data of the d clusters needs to be obtained, and the third sample data set and its first label need to be used for training, thereby improving the model training efficiency; on the other hand, in the process of training to obtain P first models, the third sample data extracted from different batches are used, that is, the corresponding model training is performed by training the third sample data sampled from different batches, and p different trained first models can be obtained, so that learning can be done from different sample data, which can improve the generalization ability of the model and reduce the risk of overfitting of the model, thereby improving the overall performance of the model.

[0063] In some embodiments, based on the p third sample data sets and the first labels of the sample data in the p third sample data sets, the p first models are trained to obtain the trained p first models, including:

[0064] Training the p first models based on the p third sample data sets and their first labels to obtain p intermediate models;

[0065] When predicting a preset number of fourth sample data according to the p intermediate models to obtain p prediction results for each fourth sample data, based on the prediction results of the preset number of fourth sample data, fifth sample data is selected from the preset number of fourth sample data, and the first label of the fifth sample data is obtained, the fourth sample data is sample data having a second label and different from the first sample data;

[0066] Updating the fifth sample data into the second sample data;

[0067] Based on the updated second sample data and its first label, the p intermediate models are trained to obtain the trained p first models.

[0068] In some embodiments, the process of obtaining the first label of the fifth sample data is similar to the process of obtaining the first label described above. For example, a second request may be sent, the second request is used to request labeling of the fifth sample data, and the first label labeled for the fifth sample data is received. As an example, the first label labeled for the fifth sample data by a professional object in the scenario in which the method is applied may be, for example, in a valuation scenario, the professional object may be a valuation expert in the valuation scenario, etc., and in a classification scenario, the professional object may be a classification expert in the classification scenario, etc.

[0069] It should be understood that for each first model, the first model may include at least one of a valuation model and a classification model, and the trained p first models are models obtained after training the p first models, which are one-to-one corresponding. Each trained first model may correspondingly include at least one of a trained valuation model and a trained classification model. For example, if the first model is a valuation model, then the trained first model is a trained valuation model, and if the first model is a classification model, then the trained first model is a trained classification model.

[0070] In this embodiment, p first models may be trained by first using p third sample data sets and first labels of their sample data to obtain p intermediate models. After obtaining the p intermediate models, the p intermediate models may be used to predict the fourth sample data, and p prediction results of the fourth sample data may be obtained. The prediction here may be at least one of a valuation prediction and a classification prediction. For example, if the intermediate model is an intermediate valuation model, the prediction is a valuation prediction, and if the intermediate model is an intermediate classification model, the prediction is a classification prediction.

[0071] It should be understood that for each fourth sample data, there are p prediction results corresponding to the p intermediate models. When the prediction results of each fourth sample data are obtained by predicting a preset number (for example, k, k is a positive integer) of fourth sample data through p intermediate models, the fifth sample data can be screened from the preset number of fourth sample data according to the prediction results of the preset number of fourth sample data, and the first label of the fifth sample data can be obtained, and the screened fifth sample data can be added to the second sample data to update the second sample data, improve the diversity of the sample data, and use the updated second sample data and its first label to further train the p intermediate models to obtain the trained p first models, thereby improving the performance of the model. That is, in this embodiment, every time the prediction results of the intermediate model for the preset number of fourth sample data are accumulated, the sample data that needs to be labeled is screened, that is, the fifth sample data is screened, and then the first label of the fifth sample data is obtained and updated to the second sample data. After that, the p intermediate models can be further trained using the updated second sample data and its first label to obtain the trained p first models. In this way, not only the diversity of the second sample data used for further training is improved, but also on the basis of the p intermediate models trained based on the p third sample data sets and their first labels, further training is carried out using the second sample data with more diverse samples, thereby improving the performance of the trained first model.

[0072] Exemplarily, the first label of the fifth sample data can be an expert label of the five sample data, that is, the expert object annotates the fifth sample data, which can ensure the accuracy of the label. Subsequently, the fifth sample data is added to the second sample data, and the second sample data and its expert label are used for further training. In this way, the performance of the trained first model can be further ensured.

[0073] In addition, exemplarily, the second label can be a label different from the first label, for example, it can be called a common label / automatic label (automatic valuation label), the second label and the first label are labels obtained by different labeling methods, and the first label can be a label with higher accuracy labeled by a professional object (professional labeler) in the scene field to which the method is applied. For the second label, compared with the first label that requires a professional object to label, the efficiency of obtaining the second label is faster. For example, third-party data can be retrieved to obtain it. For a fourth sample data, its second label is successfully retrieved, and the fourth sample data can be accumulated; it can also be labeled by a preset software or a preset model to obtain the second label. When the fourth sample data is labeled by a preset software or a preset model to obtain the second label, the fourth sample data can be accumulated. When a preset number of fourth sample data with a second label are accumulated and the corresponding prediction results are obtained through the intermediate model, the fifth sample data can be screened from the preset number of fourth sample data.

[0074] In some embodiments, based on the updated second sample data and its first label, p intermediate models are trained to obtain p trained first models, including:

[0075] According to the second sampling quantity, sampling is performed p times on the updated second sample data to obtain p sixth sample data sets, and the number of samples in each sixth sample data set is the second sampling quantity;

[0076] Based on the p sixth sample data sets and the first labels of the sample data in the p sixth sample data, the p intermediate models are trained to obtain p first models.

[0077] In this embodiment, in the process of retraining the intermediate model using the updated second sample data and its first label, the updated second sample data can be sampled, and the sampling can be performed p times. Each sampling obtains a sixth sample data set, that is, p sixth sample data sets are obtained. The p sixth sample data sets obtained by sampling and the first labels of the sample data are used to adjust the p intermediate models to obtain the trained p first models, thereby realizing the training of the first model. It should be noted that the p sixth sample data sets correspond one-to-one to the p first models after training, and the p first models after training correspond one-to-one to the p intermediate models, that is, a trained first model is obtained by training an intermediate model using a sixth sample data set and its first label. In this way, on the one hand, for different intermediate models, the seventh sample data set obtained by sampling from different batches and its first label are used for training, so that different intermediate models can learn from different sample data, improve the generalization ability of the model and reduce model overfitting, thereby improving the overall performance of the model; on the other hand, in the process of obtaining the intermediate model by training the above-mentioned model, the third sample data set is obtained by sampling the second sample data for training. This embodiment is based on the trained intermediate model for retraining. This embodiment adds the screened fifth sample data to the second sample data to update the second sample data, and then resamples the updated second sample data to obtain p sixth sample data sets for training p intermediate models, thereby improving the difference between the third sample data set used for the intermediate model obtained by training and the sixth sample data set used for retraining the intermediate model, which can reduce the occurrence of overfitting of the initial model caused by using the same sample data set for initial model training, thereby improving the performance of the first model obtained by training.

[0078] In some embodiments, the second sampling number is greater than the first sampling number. Exemplarily, the second sampling number can be determined according to the first sampling number, for example, the second sampling number can be the sum of the first sampling number and a preset value, the preset value is a number greater than zero, for example, the preset value can be 1.

[0079] In this embodiment, the number of samples taken for the second sample data is the first sampling number. After the fifth sample data is obtained, it can be added to the second sample data, that is, the updated second sample data adds the fifth sample data. The amount of data available for sampling is increased. The sampling number can be adaptively increased to ensure the number of samples used to retrain the model and improve the model training effect.

[0080] In some embodiments, based on the prediction results of the preset number of fourth sample data, screening the fifth sample data from the preset number of fourth sample data includes:

[0081] For each fourth sample data in the preset number of fourth sample data, determining a prediction variance of the fourth sample data according to p prediction results of the fourth sample data;

[0082] The sample data whose prediction variances satisfy the preset conditions among the preset number of fourth sample data are determined as the fifth sample data.

[0083] Each time a fourth sample data is obtained, it can be input into p intermediate models for prediction to obtain corresponding p prediction results, which can be understood as the prediction results corresponding to the scenario to which the method is applied. The prediction results can also be understood as prediction labels. For example, in the valuation scenario, the valuation prediction is performed through the intermediate model to obtain the predicted valuation, which can also be called the predicted valuation label. In the classification scenario, the classification prediction is performed through the intermediate model to obtain the predicted classification, which can also be called the predicted classification label. Exemplarily, the fourth sample data can be the feature data of the actual sample (the object that actually needs to be processed, also called the newly added sample data) in the actual application after the intermediate model is obtained.

[0084] In addition, each time p prediction results of each sample data in the preset number of fourth sample data are obtained, the sample data whose prediction variance meets the preset conditions can be selected from the preset number of fourth sample data, and the fifth sample data can be determined once, and the second sample data can be updated once, and the latest model can be further retrained. For example, the fifth sample data is obtained for the first time, and the latest model is the intermediate model. The intermediate model is retrained using the second sample data after the first update to obtain the trained first model. The fifth sample data is obtained for the second time, and the latest model is the latest first model. The fifth sample data obtained for the second time is added to the latest second sample data, and the latest first model can be trained again using the second sample data after the second update, etc., and so on, and the first model can be iteratively updated. In addition, it should be noted that each time the fifth sample data is obtained, the latest fifth sample data can be added to the latest second sample data, and the second sample data can be updated once, and the second sampling quantity can be updated once, for example, the second sampling quantity is increased by a preset value, for example, the preset value can be 1, etc.

[0085] Variance is used to measure the degree of discreteness of data, that is, the degree of dispersion of data distribution. The larger the variance, the more dispersed the data is, and the smaller the contrast, the more concentrated the data is. Therefore, the stability of the data can be represented by the variance. In this embodiment, when p prediction results of each fourth sample data are obtained, for each fourth sample data, the variance of its p prediction results can be calculated. The larger the variance of the prediction result, the more dispersed the prediction result is, that is, the greater the fluctuation, the more unstable the result is, indicating that the fourth sample data is the more difficult to predict sample data. In this case, the fourth sample data whose prediction variance meets the preset conditions can be further determined as the fifth sample data, and its first label can be obtained and updated to the second sample data. After that, the updated second sample data and its first label can be used to further train the p intermediate models to obtain the trained p first models. In this way, on the basis of the second sample data set, combined with the samples whose variance of the prediction results in the fourth sample data meets the preset conditions and their first labels, the sample coverage is improved so that the updated second sample data can cover the sample data that is more difficult to predict. The updated second sample data and its first label are used to further adjust the intermediate model to improve the model training effect.

[0086] It should be understood that the above-mentioned preset conditions may include multiple variance conditions, which are not specifically limited in the present embodiment. For example, they may include the maximum variance, that is, the sample data with the largest predicted variance among a preset number of fourth sample data may be determined as the fifth sample data. Alternatively, they may include a variance greater than a preset variance threshold (a threshold that can be pre-set based on actual needs or historical experience, etc., without specific limitations), that is, the sample data with a predicted variance greater than the preset threshold among a preset number of fourth sample data may be determined as the fifth sample data, etc.

[0087] In some embodiments, the method is applied to at least one of a valuation scenario and a classification scenario;

[0088] When applied to a valuation scenario, the first sample data includes first feature data of a valuation sample in the valuation scenario. After obtaining the trained p first models, the method further includes: obtaining first feature data of an object to be valued in the valuation scenario, inputting the first feature data of the object to be valued into the trained p first models, performing valuation through the trained p first models, and obtaining p valuations; and determining a valuation result of the object to be valued based on the p valuations.

[0089] When applied in a classification scenario, the first sample data includes the second feature data of the classification sample in the classification scenario. After obtaining the trained p first models, it also includes: obtaining the second feature data of the object to be classified in the classification scenario, inputting the second feature data of the object to be classified into the trained p first models, classifying the object through the trained p first models, and obtaining p first classification results; determining the second classification result of the object to be classified based on the p first classification results.

[0090] It should be understood that the first feature data of the valuation sample / sample to be valued may be feature data that characterizes the valuation sample / sample to be valued, and the first feature data is varied, and the embodiments of the present application do not specifically limit it. For example, as an example, the first feature data of the classification sample / sample to be classified may include attribute features of the classification sample / sample to be classified, for example, including but not limited to at least one of year, newness, size, location, material, etc. The second feature data of the classification sample / sample to be classified may characterize feature data of the classification sample / sample to be classified, and the second feature data is varied, and the embodiments of the present application do not specifically limit it. For example, as an example, the second feature data of the classification sample / sample to be classified may include features of the classification sample / sample to be classified that can be used to distinguish categories, for example, in a face classification scenario, it may include facial feature data, for example, it may include but not limited to facial contours, facial features, etc.

[0091] In addition, since there are p first models, each first model performs corresponding processing based on the input feature data of each object, and p output results of the object can be obtained. For example, in the valuation scenario, p valuations of the object to be valued can be obtained, so that the final valuation result of the object to be valued can be determined based on the p valuations. In the classification scenario, p first classification results of the object to be classified can be obtained, so that the final second classification result of the object to be classified can be determined based on the p first classification results.

[0092] That is, in this embodiment, in the valuation scenario, the valuations of the p first models are comprehensively considered to obtain the final valuation result, thereby improving the accuracy of the valuation result of the object to be valued. In the classification scenario, the first classification results of the p first models are comprehensively considered to obtain the final second classification result, thereby improving the accuracy of the classification result of the object to be valued.

[0093] In addition, it should be noted that there are many ways to determine the valuation result of the object to be valued based on p valuations, which are not specifically limited in the embodiments of the present application. As an example, it can be a voting method (that is, there may be identical valuations among the p valuations, and the one with the largest number of identical valuations can be taken as the valuation result. For example, among 10 first models, 5 first models obtain the same valuation, and the number of identical valuations is 5. The other 5 first models obtain different valuations, then the identical valuations obtained by the 5 first models can be taken as the final valuation result), or taking the average, or selecting a median among the p valuations as the valuation result, or selecting the largest or smallest valuation as the valuation result, etc. There are many ways to determine the second classification result of the object to be classified based on p first classification results, which are not specifically limited in the embodiments of the present application. As an example, a voting method can be used, that is, according to the voting method, the minority obeys the majority mechanism, a classification result is selected from the p first classification results as the second classification result. For example, the number of classification results A in 10 classification results is 5 (that is, 5 first model classification results are the same, all of which are classification results A), which is the largest number, and classification result A can be used as the second classification result; or it can be selected from the p first classification results. The number of TOP N (the value of N can be preset, that is, the number is in the first N) classification results are selected, and a classification result is randomly selected from the number of TOP N classification results as the second classification result. For example, the number of classification results A is 5, and the number of classification results B is 4, which is the number TOP2 classification result, and then a classification result can be randomly selected from these two classification results as the target classification result, for example, 4 classification results B are randomly selected as the second classification result, etc.

[0094] See also Figure 2 , Figure 2 1 is a flowchart of a processing method provided by an embodiment of the present application. The processing method of this embodiment can be applied to a second electronic device, which can be a mobile device or a non-mobile device. The first electronic device and the second electronic device can be the same or different. Figure 2 As shown, the following steps are included:

[0095] Step 201: Acquire feature data of an object to be processed in at least one of a valuation scenario and a classification scenario;

[0096] Step 202: inputting the feature data of the object to be processed into the trained first model, performing at least one of valuation and classification by the trained first model, and obtaining a processing result;

[0097] The trained first model is obtained by training using the data processing method of the above embodiment.

[0098] It should be understood that, in a valuation scenario, the object to be processed may include the object to be valued, the feature data of the object to be processed includes the first feature data of the object to be processed, and the corresponding processing result may include the valuation result of the object to be valued; in a classification scenario, the object to be processed may include the object to be classified, the feature data of the object to be processed includes the second feature data of the object to be processed, and the corresponding processing result may include the classification result of the object to be classified.

[0099] The trained first model in the processing method in this embodiment corresponds one-to-one to the trained first model in the data processing method in the above embodiment, and the technical features correspond to the technical features in the data processing method in the above embodiment, and achieve the same effect, which will not be repeated.

[0100] The following is a specific example of the method process, taking the method applied to a valuation scenario as an example.

[0101] like Figure 3 As shown, it is a schematic diagram of the scheme of this embodiment. First, the data set (i.e., multiple first sample data) with automatic valuation labels are clustered, and then the centroid sample data in each clustering result is preliminarily obtained to obtain the second sample data), that is, the sample data that most needs expert labeling is screened out from the data set, and the corresponding expert valuation label (first label) is obtained. P sampling is performed to obtain p second sample data sets, and p expert models (first models) are trained. Subsequently, new samples (corresponding to the fourth sample data mentioned above, for example, collateral) can be added, and their automatic labels (second labels) can be retrieved. For the new samples, valuation can be performed by p expert models to obtain prediction results. For each additional preset number (k) of fourth sample data, the fourth sample data with the largest variance in the prediction results of the p expert models is screened and labeled by experts, that is, the expert valuation label marked by the expert is obtained, and added to the second sample data, and the sampling is returned to re-sample, and the p expert models are re-trained to obtain the trained p expert models.

[0102] like Figure 4 As shown, the process of the method of this embodiment is as follows:

[0103] Initialization: first extract sample S (i.e., the first sample data), set the initial number of expert labels d, the number of models in the expert model set p, and the expert label ratio (1 / k, for every k accumulated automatic labels, one expert valuation label is obtained);

[0104] Preliminary acquisition of expert labels: Use the K-medoids algorithm to cluster the sample S into d categories, obtain the centroid of each of the d clusters, that is, obtain the center of each of the d clusters to obtain the second sample data (that is, the centroid sample data of the d clustering results, that is, d sample data), and have the experts label them, that is, obtain the expert valuation label (expert label) of the second sample data;

[0105] Use the second sample data and its expert valuation labels to train p expert models:

[0106] The Bootstrapping self-sampling method is used for the second sample data obtained above, sampling p times, and d samples are drawn each time, that is, p third sample data sets are finally drawn, and each third sample data set has d sample data;

[0107] The p third sample data sets respectively use the expert valuation labels as target variables and as features to train p expert models (e.g., linear regression models), and these p expert models constitute an expert model set;

[0108] After each preset number (e.g., k) of fourth sample data with automatic labels are accumulated, one sample data (i.e., the fifth sample data) that most needs an expert valuation label can be selected according to the QBC (Query-by-Committee) method. For example, the following formula calculates the prediction variance (σ) of the prediction results of the expert model set for these k fourth sample data with automatic labels. n is the variance of the nth sample in the preset number of fourth sample data, is the predicted value (prediction result) of the p-th expert model for the n-th sample, is the mean of the predicted values ​​of all expert models), and the sample data (fifth sample data) with the largest corresponding prediction variance is selected from the preset number of fourth sample data:

[0109]

[0110] Obtain the expert valuation label of the fifth sample data, add the selected fifth sample data to the second sample data, update the second sample data, that is, update the number of the second sample data to d+1, and then return to the above to resample the second sample data (updated, the latest second sample data) p times, and further train the p expert models. As the samples accumulate, the expert model set is gradually iterated;

[0111] Each newly added collateral to be valued can first retrieve the automatic label, and accumulate a preset number of new collateral samples with automatic labels to be valued, then the collateral sample with the largest prediction method can be screened out and labeled by professionals, i.e. the first label. The collateral sample is added to the second sample data, and then the expert model set is further trained. During the training process, the expert model in the expert model set can output the corresponding final valuation prediction value.

[0112] The method of the embodiment of the present application uses active learning to actively screen samples to more effectively obtain expert labels, and uses an expert model set to output a final valuation, rather than simply using the average of automatically acquired labels. The valuation obtained through the embodiment of the present application is more reasonable, because of the model training characteristics, the expert model can be continuously iterated and optimized to improve model performance.

[0113] See also Fig. 9 , Fig. 9 It is a structural diagram of a chip 900 provided in an embodiment of the present application. In this embodiment, a chip 900 includes: a controller 901, a processor 902, an input cache 903 and an output cache 904, including: the controller 901 loads the first sample data into the input cache 903; the processor 902 clusters the first sample data to obtain d clustering results, d is a positive integer; extracts the centroid sample data in each of the d clustering results to obtain second sample data; sends a first request, the first request is used to request labeling of the second sample data; receives a first label labeled for the second sample data; based on the second sample data and the first label of the second sample data, trains the first model to obtain the trained first model, and stores the trained first model in the output cache 904.

[0114] The processor 902 first performs sample clustering on the first sample data, and selects the centroid sample data in each cluster of the d clustering results, so that while reducing the amount of sample data processing, the centroid sample data that best represents the cluster in each cluster can be obtained to form second sample data with better quality. After that, there is no need to label all the first sample data, and only a request to label the second sample data is required, and then the first label labeled for the second sample data is received, which can improve the efficiency of obtaining the label of the sample data. In this way, a small number of centroid sample data that can represent each type of cluster and its first label can be used for model training, which can improve the efficiency of model training and improve the performance of the trained model. That is, the technical solution enhances the performance of the trained model and reduces the computing resources of the computer. On the basis of not increasing the sample computing amount of the processor, the performance of the trained model can be improved, which effectively saves the computing resources for processing, and does not need to increase a large number of high-quality samples due to the need to improve the performance of the model, further saving the storage resources of the memory.

[0115] A chip 900 provided in an embodiment of the present application can implement each process of the data processing method in the above embodiment, and will not be described again here to avoid repetition.

[0116] See also Figure 5 , Figure 5 5 is a structural diagram of a data processing device 500 provided in an embodiment of the present application, which can implement the details of the data processing method in the above embodiment and achieve the same effect. Figure 5 As shown, the data processing device 500 includes:

[0117] The clustering module 501 is used to cluster the first sample data to obtain d clustering results, where d is a positive integer;

[0118] The sample extraction module 502 is used to extract the centroid sample data in each clustering result of the d clustering results to obtain the second sample data;

[0119] A sending module 503 is used to send a first request, where the first request is used to request to label the second sample data;

[0120] A receiving module 504 is used to receive a first label annotated for the second sample data;

[0121] The training module 505 is used to train the first model based on the second sample data and the first label of the second sample data to obtain a trained first model.

[0122] In some embodiments, the number of first models is p, where p is an integer greater than 1;

[0123] The training module 504 includes:

[0124] A first sampling unit is used to sample the second sample data p times according to the first sampling quantity to obtain p third sample data sets, where the number of samples in each third sample data set is the first sampling quantity;

[0125] The training unit is used to train the p first models based on the p third sample data sets and the first labels of the sample data in the p third sample data sets to obtain the trained p first models.

[0126] In some embodiments, the training unit comprises:

[0127] A first sub-training unit is used to train the p first models based on the p third sample data sets and their first labels to obtain p intermediate models;

[0128] an acquiring unit, configured to, when predicting a preset number of fourth sample data according to the p intermediate models to obtain p prediction results for each fourth sample data, screen fifth sample data from the preset number of fourth sample data based on the prediction results of the preset number of fourth sample data, and acquire a first label of the fifth sample data, the fourth sample data being sample data having a second label and being different from the first sample data;

[0129] An adding unit, used for updating the fifth sample data into the second sample data;

[0130] The second sub-training unit is used to train the p intermediate models based on the updated second sample data and the first label thereof to obtain the trained p first models.

[0131] In some embodiments, the second sub-training unit includes:

[0132] A second sampling unit is used to sample the updated second sample data p times according to the second sampling quantity to obtain p sixth sample data sets, where the number of samples in each sixth sample data set is the second sampling quantity;

[0133] The model adjustment unit is used to train the p intermediate models based on the p sixth sample data sets and the first labels of the sample data in the p sixth sample data to obtain p first models.

[0134] In some embodiments, the acquisition unit includes:

[0135] a variance determination unit, configured to determine, for each fourth sample data in a preset number of fourth sample data, a prediction variance of the fourth sample data according to p prediction results of the fourth sample data;

[0136] The sample determination unit is used to determine the sample data whose prediction variance meets the preset condition among the preset number of fourth sample data as the fifth sample data.

[0137] In some embodiments, the second sampling number is greater than the first sampling number.

[0138] In some embodiments, the device is applied to at least one of a valuation scenario and a classification scenario;

[0139] In the case of application to a valuation scenario, the first sample data includes first feature data of a valuation sample in the valuation scenario, and the device further includes: a first feature acquisition module, used to acquire first feature data of an object to be valued in the valuation scenario; a valuation module, used to input the first feature data of the object to be valued into p trained first models, and perform valuation through the trained p first models to obtain p valuations; and a valuation result determination module, used to determine a valuation result of the object to be valued based on the p valuations;

[0140] When applied in a classification scenario, the first sample data includes second feature data of the classification sample in the classification scenario, and the device also includes: a second feature acquisition module, used to acquire the second feature data of the object to be classified in the classification scenario; a classification module, used to input the second feature data of the object to be classified into p trained first models, and classify through the trained p first models to obtain p first classification results; a classification result determination module, used to determine the second classification result of the object to be classified based on the p first classification results.

[0141] The data processing device provided in the embodiment of the present application can implement each process of the data processing method in the above embodiment, and will not be described again here to avoid repetition.

[0142] See also Figure 6 , Figure 6 600 is a structural diagram of a processing device 600 provided in an embodiment of the present application, which can implement the details of the processing method in the above embodiment and achieve the same effect. Figure 6 As shown, the processing device 600 includes:

[0143] The object feature acquisition module 601 is used to acquire feature data of the object to be processed in at least one of the valuation scenario and the classification scenario;

[0144] A first processing module 602 is used to input the feature data of the object to be processed into a trained first model, and perform at least one of valuation and classification through the trained first model to obtain a processing result;

[0145] The trained first model is obtained by training using the method of the first aspect mentioned above.

[0146] The processing device provided in the embodiment of the present application can implement each process of the processing method in the above embodiment, and will not be described again here to avoid repetition.

[0147] like Figure 7 As shown, an embodiment of the present application also provides an electronic device, including a processor 710, a memory 709, and a computer program stored in the memory 709 and executable on the processor 710. When the computer program is executed by the processor 710, the various processes of the above-mentioned data processing method embodiment or the various processes of the above-mentioned processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0148] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above-mentioned data processing method embodiment or each process of the above-mentioned processing method embodiment can be implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. Among them, the computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0149] like Figure 8 As shown, an embodiment of the present application also provides an electronic device, including a processor 810, a memory 809, and a computer program stored in the memory 809 and executable on the processor 810. When the computer program is executed by the processor 810, the various processes of the above-mentioned processing method embodiment or the various processes of the above-mentioned processing method embodiment can be implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0150] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above-mentioned processing method embodiment or each process of the above-mentioned processing method embodiment can be implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. Among them, the computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0151] The embodiments of the present application also provide a computer program product, including a computer program. When the computer program is executed by a processor, it implements each process of the above-mentioned data processing method embodiment or each process of the above-mentioned processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0152] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0153] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0154] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A data processing method, characterized in that: include: Cluster the first sample data to obtain d clustering results, where d is a positive integer; Extracting centroid sample data from each clustering result in the d clustering results to obtain second sample data; Sending a first request, where the first request is used to request labeling of the second sample data; Receiving a first label annotated for the second sample data; Based on the second sample data and the first label of the second sample data, the first model is trained to obtain a trained first model.

2. The method according to claim 1, characterized in that The number of the first models is p, where p is an integer greater than 1; The step of training the first model based on the second sample data and the first label of the second sample data to obtain the trained first model includes: According to the first sampling quantity, sampling is performed p times on the second sample data to obtain p third sample data sets, and the number of samples in each third sample data set is the first sampling quantity; Based on the p third sample data sets and the first labels of the sample data in the p third sample data sets, the p first models are trained to obtain the trained p first models.

3. The method according to claim 2, characterized in that The step of training the p first models based on the p third sample data sets and the first labels of the sample data in the p third sample data sets to obtain the trained p first models comprises: Training the p first models based on the p third sample data sets and their first labels to obtain p intermediate models; When predicting a preset number of fourth sample data according to the p intermediate models to obtain p prediction results for each fourth sample data, based on the prediction results of the preset number of fourth sample data, fifth sample data is screened from the preset number of fourth sample data, and a first label of the fifth sample data is obtained, the fourth sample data being sample data having a second label and different from the first sample data; Updating the fifth sample data into the second sample data; Based on the updated second sample data and the first label thereof, the p intermediate models are trained to obtain the trained p first models.

4. The method according to claim 3, characterized in that: The step of training the p intermediate models based on the updated second sample data and the first label to obtain the trained p first models includes: According to the second sampling quantity, sampling is performed p times on the updated second sample data to obtain p sixth sample data sets, and the number of samples in each sixth sample data set is the second sampling quantity; Based on the p sixth sample data sets and the first labels of the sample data in the p sixth sample data, the p intermediate models are trained to obtain the p first models.

5. The method according to claim 3, characterized in that: The step of selecting fifth sample data from the preset number of fourth sample data based on the prediction results of the preset number of fourth sample data includes: For each fourth sample data in the preset number of fourth sample data, determining a prediction variance of the fourth sample data according to p prediction results of the fourth sample data; The sample data whose prediction variances satisfy a preset condition among the preset number of fourth sample data are determined as the fifth sample data.

6. The method according to claim 2, characterized in that The method is applied to at least one of a valuation scenario and a classification scenario; When applied to the valuation scenario, the first sample data includes first feature data of the valuation sample in the valuation scenario, and after obtaining the trained p first models, the method further includes: obtaining first feature data of the object to be valued in the valuation scenario, inputting the first feature data of the object to be valued into the trained p first models, performing valuation through the trained p first models to obtain p valuations; and determining a valuation result of the object to be valued based on the p valuations; When applied to the classification scenario, the first sample data includes second feature data of the classification samples in the classification scenario, and after obtaining the trained p first models, it also includes: acquiring the second feature data of the object to be classified in the classification scenario, inputting the second feature data of the object to be classified into the trained p first models, classifying through the trained p first models to obtain p first classification results; and determining the second classification result of the object to be classified based on the p first classification results.

7. A chip, characterized in that: The chip includes: a controller, a processor, an input buffer and an output buffer, including: The controller loads the first sample data into the input buffer; The processor clusters the first sample data to obtain d clustering results, where d is a positive integer; extracts the centroid sample data in each of the d clustering results to obtain second sample data; sends a first request, where the first request is used to request labeling of the second sample data; receives a first label labeled for the second sample data; trains a first model based on the second sample data and the first label of the second sample data to obtain a trained first model, and stores the trained first model in the output cache.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the data processing method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the data processing method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images