Sample data processing method and device based on machine learning
Through machine learning sample data processing methods, representative samples in medical images are selected, and the number of labels and redundant data is reduced, which solves the problem of wasted labeling resources and long training cycles in medical image recognition, and improves the recognition accuracy of the model and Iteration speed.
Patent Information
- Application Number
- CN202411098780.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-12
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-12
AI Technical Summary
The existing technology has a large number of invalid labels in medical image recognition, resulting in wasted labeling resources and prolonged model training cycle, and the inability to iterate quickly, affecting the model training efficiency.
Using machine learning-based sample data processing method, the most representative samples are selected through pre-built deep learning networks and sample selection models, reducing the number of labels, removing redundant data in the database, and improving model training efficiency.
Without affecting the recognition accuracy, the number of samples to be marked is greatly reduced, the limited annotation resources are made full use of the model identification accuracy and iteration speed, the problem of data imbalance is alleviated, and the loss of annotation resource and the overhead of model training resource.
Smart Images

Figure CN118747837B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a sample data processing method and device based on machine learning. Background Art
[0002] When processing medical images and other data, artificial intelligence recognition systems are usually used to improve the recognition efficiency and accuracy of medical images. When building an artificial intelligence recognition system, it is necessary to collect a large amount of target data, and the collected data must be accurately labeled by field experts or relevant labeling personnel to build a huge database. For the labeling of data in natural scenes, it can be distributed to data labelers, and a large number of labelers will label the data. Due to the large amount of data, the cost of human labeling is high, and for large-scale data to be labeled, it is easy to make labeling errors during labeling. In the medical image scenario, such data labeling requires a large number of experienced and senior experts and scholars to accurately determine the specific categories of the data. However, for the medical industry, which is short of talent, it is impossible to carry out large-scale data labeling with limited labeling resources. In addition, in a large-scale database, different samples may contain similar information, so information redundancy may exist.
[0003] In the existing technology, the artificial intelligence recognition system used for medical image recognition is usually a statistical learning recognition model based on supervised learning. During the construction process, it is necessary to carry out feature construction, data preprocessing, statistical learning model training and other processes. In this process, the deep neural network directly extracts features and automatically classifies, and annotates all collected data indiscriminately. However, the method used in the existing technology treats all samples in the training data set equally, without considering the information redundancy of the data set, annotates all data, generates a large number of invalid annotations (information redundancy), and seriously wastes scarce annotation resources; and a large amount of collected data is all used for model training, which prolongs the model training cycle, increases the training cost, and cannot quickly iterate the model, affecting the efficiency of model training.
[0004] In view of this, providing a sample data processing method and device based on machine learning, in order to process the collected raw data, reduce the amount of data used for training, eliminate redundant data in the database, reduce the amount of annotation, and thus improve the efficiency of model training, has become an urgent problem to be solved by technical personnel in this field. Summary of the invention
[0005] The present invention provides a sample data processing method and device based on machine learning, which are used to solve the problem of a large number of invalid annotations in the data set existing in the prior art, so as to reduce the amount of data used for training by processing the collected original data, eliminate redundant data in the database, reduce the amount of annotations, and thus improve the efficiency of model training.
[0006] The present invention provides a sample data processing method based on machine learning, the method comprising:
[0007] Acquire data to be processed and construct a first database;
[0008] Inputting the data in the first database into a pre-trained data processing model to obtain target sample categories and data corresponding to each target sample category output by the data processing model;
[0009] Count the number of data for each target sample category;
[0010] In which, the data processing model is based on a pre-built deep learning network and is trained using image samples and corresponding category labels in a second database, and the second database is constructed using image samples screened out from original data samples using a pre-built sample selection model.
[0011] In some embodiments, based on a pre-built deep learning network, training is performed using image samples and corresponding category labels in the second database to obtain the data processing model, specifically including:
[0012] Get the original data sample;
[0013] Dividing the collected original data samples into multiple subsets, and inputting the data in each subset into a pre-built sample selection model to obtain image samples output by the sample selection model;
[0014] Annotating the screened image samples to obtain a category label corresponding to each image sample;
[0015] constructing the second database using the image samples and the category labels;
[0016] Dividing the second database into a training set, a validation set and a test set;
[0017] Inputting the samples in the training set into a pre-built deep learning network for training to obtain an initial data processing model;
[0018] The validation set is used to select model parameters for the initial data processing model to obtain the data processing model.
[0019] In some embodiments, the data in each of the subsets are respectively input into a pre-built sample selection model to obtain image samples output by the sample selection model, specifically including:
[0020] The data in each of the subsets are respectively input into a pre-constructed sample selection model, the mutual information is calculated in the sample selection model, and the sample with the maximum mutual information in each subset is output as the image sample.
[0021] In some embodiments, during the construction of the sample selection model, parameter sampling is performed on the sample selection model.
[0022] In some embodiments, during the construction of the sample selection model, neurons in the last fully connected layer in the neural network are randomly deactivated.
[0023] In some embodiments, the mutual information is expressed as:
[0024] ;
[0025] Among them, I(X,Y) represents the mutual information of X and Y, and H(X|Y) represents the conditional entropy of X given Y.
[0026] The present invention also provides a sample data processing device based on machine learning, the device comprising:
[0027] A data acquisition unit, used to acquire data to be processed and construct a first database;
[0028] A data screening unit, used for inputting the data in the first database into a pre-trained data processing model to obtain target sample categories and data corresponding to each target sample category output by the data processing model;
[0029] A data statistics unit is used to count the data quantity of each target sample category;
[0030] In which, the data processing model is based on a pre-built deep learning network and is trained using image samples and corresponding category labels in a second database, and the second database is constructed using image samples screened out from original data samples using a pre-built sample selection model.
[0031] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described above when executing the program.
[0032] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method described above when executed by a processor.
[0033] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method described above is implemented.
[0034] The sample data processing method and device based on machine learning provided by the present invention obtain the data to be processed and construct a first database, input the data in the first database into a pre-trained data processing model to obtain the target sample category output by the data processing model and the data corresponding to each target sample category; and count the number of data in each target sample category; wherein the data processing model is based on a pre-constructed deep learning network, trained using image samples and corresponding category labels in a second database, and the second database is constructed using image samples screened out from the original data samples using a pre-constructed sample selection model.
[0035] In this way, in the training process of the data processing model, by training the samples and corresponding labels selected from the second database, the trained data processing model has the ability to screen samples. The model can select the most representative samples from the original data, and then these samples are labeled by experts. Under the premise of not affecting the recognition accuracy, the number of samples to be labeled can be greatly reduced, and limited labeling resources can be fully utilized; at the same time, in the case of limited labeled data, the samples with the largest amount of information are labeled, which will greatly improve the recognition accuracy of the model and reduce the experimental cycle, and effectively alleviate the common problems in practical applications such as data imbalance generated during data collection. The problem of more invalid labels in the data set existing in the prior art is solved. By processing the collected original data, the number of data used for training is reduced, the redundant data in the database is eliminated, and the labeling amount is reduced, thereby improving the efficiency of model training. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0037] Figure 1 It is one of the flow charts of the sample data processing method based on machine learning provided by the present invention;
[0038] Figure 2This is the second flow chart of the sample data processing method based on machine learning provided by the present invention;
[0039] Figure 3 This is the third flow chart of the sample data processing method based on machine learning provided by the present invention;
[0040] Figure 4 is a schematic diagram of an image in an embodiment;
[0041] Figure 5 is a network structure diagram in one embodiment;
[0042] Figure 6 This is the fourth flow chart of the sample data processing method based on machine learning provided by the present invention;
[0043] Figure 7 is a structural block diagram of a sample data processing device based on machine learning provided by the present invention;
[0044] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0045] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0046] In order to solve the problems existing in the prior art, the sample data processing method and device based on machine learning provided by the present invention enable the model to achieve the best recognition performance under the premise of annotating the data as little as possible, fully eliminate redundant samples in the data set, greatly reduce the time required for expert annotation, and actively avoid the generation of "long tail effect" when collecting data, thereby constructing a data processing model with faster iteration speed, better recognition performance and stronger generalization ability. The data processing model is applied to process the input data to be processed, and the screened samples are then annotated by experts, which significantly reduces the number of invalid samples and reduces the time cost of expert annotation.
[0047] In a specific embodiment, Figure 1 As shown, the sample data processing method based on machine learning provided by the present invention includes the following steps:
[0048] S110: Acquire data to be processed and construct a first database; in the process of data collection, different types of data have different collection methods; for example, for medical data, images can be acquired with the help of electron microscopes, phase contrast microscopes, and optical microscopes, and data can be collected using microscope cameras; for natural data, data can be collected using ordinary cameras or networks.
[0049] S120: inputting the data in the first database into a pre-trained data processing model to obtain target sample categories and data corresponding to each target sample category output by the data processing model;
[0050] S130: Counting the number of data of each target sample category and outputting the counted data;
[0051] In which, the data processing model is based on a pre-built deep learning network and is trained using image samples and corresponding category labels in a second database, and the second database is constructed using image samples screened out from original data samples using a pre-built sample selection model.
[0052] In some embodiments, Figure 2 and Figure 3 As shown, based on the pre-built deep learning network, the image samples in the second database and the corresponding category labels are used for training to obtain the data processing model, which specifically includes the following steps:
[0053] S210: Obtain raw data samples; the data sample collection method is the same as the aforementioned collection method for the raw data to be processed, and different types of data have different collection methods; for medical data, images can be acquired with the aid of an electron microscope, a phase contrast microscope, or an optical microscope, and data can be collected using a microscope camera; for natural data, data can be collected using an ordinary camera or the Internet.
[0054] S220: Divide the collected raw data samples into multiple subsets, input the data in each subset into a pre-constructed sample selection model, and obtain the image samples output by the sample selection model; specifically, input the data in each subset into a pre-constructed sample selection model, calculate the mutual information in the sample selection model, and output the sample with the largest mutual information in each subset as the image sample. For example, divide the collected raw data into N groups, input each group of data into the sample selection model, select the most representative samples with the largest amount of information in each group, and have them annotated by annotators and included in the training database to achieve active data selection.
[0055] S230: Annotate the screened image samples to obtain a category label corresponding to each image sample; for medical data, for example Figure 4 As shown, professional doctors can use specific labeling tools to label categories and other information. For natural data, experts in related fields or outsourced service teams can use labeling tools such as Label Me to label data. When the labeling volume reaches a certain scale, the labeled data will be divided into training sets, validation sets, and test sets in a certain proportion to prepare data for building the model.
[0056] S240: constructing the second database using the image samples and the category labels, and dividing the second database into a training set, a validation set, and a test set;
[0057] S250: Inputting the samples in the training set into a pre-built deep learning network for training to obtain an initial data processing model;
[0058] S260: Select model parameters for the initial data processing model using the validation set to obtain the data processing model.
[0059] It should be understood that the "first" and "second" mentioned in the text are only for the convenience of description and do not represent a certain order, nor should they be understood as any limitation. In this embodiment, the first database is a database used to store data to be processed during the model calling process, and the second database is a database used to store training sample data during the model training process.
[0060] like Figure 3 As shown, the left side is the training process of the data processing model, and the right side is the calling process of the data processing model. In the training process of the data processing model, data collection is first performed, and the original database is constructed. Samples in the original database are actively selected, and the selected samples are labeled. The labeled data is used to construct a training database, and the training database is divided into a training set, a validation set, and a test set. The training set, validation set, and test set are respectively input into the deep learning network for model training, and then the generated model is packaged and stored in the system. In the process of model calling, the collected raw data to be processed is used to construct a database through data collection, and the database is input into the data processing model to obtain the sample categories output by the data processing model, and the number of each type of samples is counted and then output.
[0061] It should be understood that the data processing model is a model based on intelligent recognition, such as Figure 3 As shown, the data processing model described in this article is an intelligent recognition model.
[0062] In step S220, the sample selection model is constructed based on information theory, such as Figure 6 As shown, the specific steps include:
[0063] Collect and accumulate original data, build an original database, and divide the original database into N groups, that is, form multiple subsets through subset division, such as subset 1, subset 2, ..., subset N; that is, collect data and build an original database with redundant data.
[0064] The collected data are screened, and each group of data is sent to the sample selection model respectively. The mutual information is calculated after model parameter sampling and random neuron inactivation. The samples with the largest mutual information in each group are screened out as the most valuable samples, and the screened samples are labeled. In other words, the data in the original database containing redundant data is input into the sample selection model, the samples with redundant information are eliminated, and the samples with the largest amount of information are retained.
[0065] The selected labeled samples are collected and handed over to professional labelers for labeling. After the labeling is completed, a second database for training is formed, and the second database is divided into training set, validation set and test set in the ratio of 7:2:1. The training set is used to train the intelligent recognition model, the validation set is used to select the optimal model parameters, and the test set is used to test the model performance. Finally, the category of each sample is output, and the number of samples in each category is counted to obtain the final statistical results and output.
[0066] In step S220, during the construction of the sample selection model, parameter sampling is performed on the sample selection model, and random inactivation processing is performed on the neurons of the last fully connected layer in the neural network.
[0067] Furthermore, in the specific construction process of the sample selection model, in principle, under information theory, a good sample with high information content should have the ability to quickly reduce the uncertainty of the model parameter space; after receiving the sample with high information content, the model further reduces the uncertainty of the model parameters, so that the model parameters further converge to the target parameters. Based on the above theory, the following optimization objectives are established:
[0068] ;
[0069] Among them, x represents a single sample, y represents the possible category of the sample, H represents the entropy of the random variable, and E represents the expectation of the random variable. The above optimization goal requires calculating the entropy of the parameter space, and at the same time calculating the change in the entropy of the parameter space after adding a certain sample, so as to select the sample that causes the largest change in the entropy of the parameter space, that is, the sample with the largest amount of information, and annotate the sample with professional annotators and include it in the training database. However, the above goal requires calculating the entropy of the parameter space. For a neural network with millions of parameters, calculating the entropy of the parameters is an NP-hard problem and cannot be effectively calculated in most cases.
[0070] Since it is impossible to calculate effectively, in this embodiment, the above-mentioned problem that cannot be calculated can be equivalently converted into a problem that can be calculated. The above-mentioned optimization target can be regarded as the mutual information between the category distribution of the sample and the distribution of the model parameters, and the mutual information is related to entropy.
[0071] Entropy is defined as follows:
[0072] ;
[0073] Where H represents the entropy of the random variable x, and p(x) is the distribution function of the random variable x.
[0074] The mutual information is defined as follows:
[0075] ;
[0076] Among them, I(X,Y) represents the mutual information of X and Y, and H(X|Y) represents the conditional entropy of X given Y.
[0077] The conditional entropy is defined as follows:
[0078] ;
[0079] Among them, H(X|Y) represents the conditional entropy of X under the condition of Y, P(Y) represents the density function of the random variable Y, and P(X|Y) represents the probability density function of the random variable X under given Y.
[0080] And because the initial optimization goal is:
[0081] ;
[0082] Rewrite it into the desired form:
[0083] ;
[0084] According to the definitions of mutual information and conditional entropy, the above forms represent H(θ|D) and H(θ|y,x,D) respectively. According to the definition of mutual information, the optimization goal is to maximize the mutual information between the model parameters θ and the sample category distribution y.
[0085] According to the symmetry of mutual information, that is, I(X,Y)=I(Y,X), the above optimization objectives can be transformed, that is,
[0086] ;
[0087] Therefore, the above optimization objective becomes:
[0088] ;
[0089] Where x is the sample to be selected, θ is the model parameter, D is the selected part of the data set, and y is the possible category of the sample; based on the symmetry of mutual information, a new optimization goal is obtained, which avoids calculating the entropy of the distribution obeyed by the model parameters, and converts an uncalculated optimization goal into a calculable optimization goal, so as to achieve the purpose of selecting the sample with the largest amount of information; , can be directly calculated according to the number of categories in actual application; , it is necessary to calculate the expectation of the class conditional entropy of the sample under the parameter θ under the parameter distribution; that is, it is necessary to perform Monte Carlo sampling on the model parameter θ and then estimate the expectation.
[0090] When sampling model parameters, for neural network models, such as Figure 5 As shown in the figure, the number of parameters is usually between millions and hundreds of billions. Accurate and efficient sampling of neural network parameters is an important step in completing the calculation of the expected results. The random deactivation technology is used for neurons to make the model parameters have a certain degree of randomness. At the same time, in order to reduce the complexity of the parameter space to be sampled, only the neurons in the last fully connected layer of the neural network are randomly deactivated. While ensuring the randomness of the parameters, the stability of the parameters is maintained, and the uncertainty of the parameter space to be sampled is reduced, so that the sampled model parameters can accurately, stably and efficiently calculate the expectations, estimate the size of the mutual information, and thus find the sample x that can maximize the mutual information, mark it and include it in the training set to assist in the training of the final intelligent recognition model.
[0091] When the training database is constructed, samples are selected based on the active sample selection system. The above method is used to select samples for each set of data, and finally the most valuable samples in each set of data are unified and handed over to the labeling personnel for labeling. After the labeling is completed, the training database is constructed. After the training database is constructed, the database is divided into training set, validation set, and test set according to the convention of 7:2:1, which is convenient for the subsequent intelligent recognition model training. All the data in the database are high-information, high-value, and low-redundancy samples, which make full use of limited labeling resources and greatly reduce the loss of labeling resources and the resource overhead of model training. This method has excellent practical value.
[0092] The method provided by the present invention uses information theory to screen out redundant information samples in the original database, greatly reducing the number of samples that need to be labeled, making full use of limited labeling resources, improving labeling efficiency, and building a diverse and accurate training database. The number of samples in the training database is greatly reduced, the resource overhead of model training is reduced, and the model experiment cycle is shortened, so that the model can be quickly iterated and verified. Less data to be labeled reduces the possibility of data labeling errors, making the labeling errors of all screened samples smaller. At the same time, diverse and high-information samples greatly improve the recognition and generalization capabilities of the model, so that the final recognition model has a higher application value.
[0093] In the above specific implementation, the sample data processing method based on machine learning provided by the present invention obtains the data to be processed and constructs a first database, inputs the data in the first database into a pre-trained data processing model to obtain the target sample category output by the data processing model and the data corresponding to each target sample category; and counts the number of data in each target sample category; wherein the data processing model is based on a pre-constructed deep learning network, trained using image samples and corresponding category labels in a second database, and the second database is constructed using image samples screened out from the original data samples using a pre-constructed sample selection model.
[0094] In this way, in the training process of the data processing model, by training the samples screened out from the second database and the corresponding labels, the trained data processing model has the ability to screen samples. The model can select the most representative samples from the original data, and then these samples are labeled by experts. This can greatly reduce the number of samples to be labeled without affecting the recognition accuracy, and make full use of limited labeling resources. At the same time, when the labeled data is limited, labeling the samples with the most information will greatly improve the recognition accuracy of the model and reduce the experimental cycle, while effectively alleviating common problems in practical applications such as data imbalance caused by data collection.
[0095] The method provided by the present invention enables the model to achieve optimal recognition performance while annotating the data as little as possible, fully eliminates redundant samples in the data set, greatly reduces the time required for expert annotation, and actively avoids the generation of the "long tail effect" when collecting data. A data processing model with faster iteration speed, better recognition performance, and stronger generalization ability is constructed, which solves the problem of a large number of invalid annotations in the data set existing in the prior art. By processing the collected original data, the amount of data used for training is reduced, redundant data in the database is eliminated, and the amount of annotation is reduced, thereby improving the efficiency of model training.
[0096] In addition to the above method, the present invention also provides a sample data processing device based on machine learning, such as Figure 7 As shown, the device comprises:
[0097] The data acquisition unit 710 is used to acquire data to be processed and construct a first database;
[0098] A data screening unit 720, configured to input the data in the first database into a pre-trained data processing model to obtain target sample categories and data corresponding to each target sample category output by the data processing model;
[0099] A data statistics unit 730 is used to count the data quantity of each target sample category;
[0100] In which, the data processing model is based on a pre-built deep learning network and is trained using image samples and corresponding category labels in a second database, and the second database is constructed using image samples screened out from original data samples using a pre-built sample selection model.
[0101] In some embodiments, based on a pre-built deep learning network, training is performed using image samples and corresponding category labels in the second database to obtain the data processing model, specifically including:
[0102] Get the original data sample;
[0103] Dividing the collected original data samples into multiple subsets, and inputting the data in each subset into a pre-built sample selection model to obtain image samples output by the sample selection model;
[0104] Annotating the screened image samples to obtain a category label corresponding to each image sample;
[0105] constructing the second database using the image samples and the category labels;
[0106] Dividing the second database into a training set, a validation set and a test set;
[0107] Inputting the samples in the training set into a pre-built deep learning network for training to obtain an initial data processing model;
[0108] The validation set is used to select model parameters for the initial data processing model to obtain the data processing model.
[0109] In some embodiments, the data in each of the subsets are respectively input into a pre-built sample selection model to obtain image samples output by the sample selection model, specifically including:
[0110] The data in each of the subsets are respectively input into a pre-constructed sample selection model, the mutual information is calculated in the sample selection model, and the sample with the maximum mutual information in each subset is output as the image sample.
[0111] In some embodiments, during the construction of the sample selection model, parameter sampling is performed on the sample selection model.
[0112] In some embodiments, during the construction of the sample selection model, neurons in the last fully connected layer in the neural network are randomly deactivated.
[0113] In some embodiments, the mutual information is expressed as:
[0114] ;
[0115] Among them, I(X,Y) represents the mutual information of X and Y, and H(X|Y) represents the conditional entropy of X given Y.
[0116] In the above specific implementation, the sample data processing device based on machine learning provided by the present invention obtains the data to be processed and constructs a first database, and inputs the data in the first database into a pre-trained data processing model to obtain the target sample category output by the data processing model and the data corresponding to each target sample category; the number of data in each target sample category is counted; wherein, the data processing model is based on a pre-constructed deep learning network, and is trained using image samples and corresponding category labels in a second database, and the second database is constructed using image samples screened out from the original data samples using a pre-constructed sample selection model.
[0117] In this way, in the training process of the data processing model, by training the samples screened out from the second database and the corresponding labels, the trained data processing model has the ability to screen samples. The model can select the most representative samples from the original data, and then these samples are labeled by experts. This can greatly reduce the number of samples to be labeled without affecting the recognition accuracy, and make full use of limited labeling resources. At the same time, when the labeled data is limited, labeling the samples with the most information will greatly improve the recognition accuracy of the model and reduce the experimental cycle, while effectively alleviating common problems in practical applications such as data imbalance caused by data collection.
[0118] The device provided by the present invention enables the model to achieve optimal recognition performance while annotating the data as little as possible, fully eliminates redundant samples in the data set, greatly reduces the time required for expert annotation, and actively avoids the generation of the "long tail effect" when collecting data, thereby constructing a data processing model with faster iteration speed, better recognition performance, and stronger generalization ability, solving the problem of a large number of invalid annotations in the data set existing in the prior art, and reducing the amount of data used for training by processing the collected raw data, eliminating redundant data in the database, and reducing the amount of annotations, thereby improving the efficiency of model training.
[0119] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the above method.
[0120] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0121] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the above method.
[0122] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0123] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A sample data processing method based on machine learning, characterized in that: The method comprises: Acquire data to be processed and construct a first database; Inputting the data in the first database into a pre-trained data processing model to obtain target sample categories and data corresponding to each target sample category output by the data processing model; Count the number of data for each target sample category; The data processing model is obtained by training based on a pre-built deep learning network using image samples and corresponding category labels in a second database, and the second database is constructed by using image samples screened from original data samples using a pre-built sample selection model; Based on the pre-built deep learning network, the data processing model is obtained by training using the image samples and corresponding category labels in the second database, which specifically includes: Get the original data sample; Dividing the collected original data samples into multiple subsets, and inputting the data in each subset into a pre-built sample selection model respectively, so as to obtain image samples output by the sample selection model; Annotating the screened image samples to obtain a category label corresponding to each image sample; constructing the second database using the image samples and the category labels; Dividing the second database into a training set, a validation set and a test set; Inputting the samples in the training set into a pre-built deep learning network for training to obtain an initial data processing model; Using the validation set to select network parameters for the initial data processing model to obtain the data processing model; Inputting the data in each of the subsets into a pre-built sample selection model to obtain image samples output by the sample selection model specifically includes: Inputting the data in each of the subsets into a pre-built sample selection model, respectively, calculating mutual information in the sample selection model, and outputting the sample with the largest mutual information in each subset as the image sample; In the process of constructing the sample selection model, the optimization objective is The expression is: ; Among them, x is the sample to be selected, θ is the model parameter, D is the selected part of the data set, y is the possible category of the sample, P (X|Y) represents the probability density function of the random variable X under given Y, H (y|x) represents the conditional entropy of y given x, is the expectation of the class conditional entropy of the sample under parameter θ under the parameter distribution.
2. The sample data processing method based on machine learning according to claim 1, characterized in that: During the construction of the sample selection model, parameter sampling is performed on the sample selection model.
3. The sample data processing method based on machine learning according to claim 1, characterized in that: In the process of constructing the sample selection model, the neurons in the last fully connected layer of the neural network are randomly inactivated.
4. The sample data processing method based on machine learning according to claim 1, characterized in that: The expression of the mutual information is: ; Among them, I(X,Y) represents the mutual information of X and Y, and H(X|Y) represents the conditional entropy of X given Y.
5. A sample data processing device based on machine learning, based on the method according to any one of claims 1 to 4, characterized in that: The device comprises: A data acquisition unit, used to acquire data to be processed and construct a first database; A data screening unit, used for inputting the data in the first database into a pre-trained data processing model to obtain target sample categories and data corresponding to each target sample category output by the data processing model; A data statistics unit is used to count the data quantity of each target sample category; In which, the data processing model is based on a pre-built deep learning network and is trained using image samples and corresponding category labels in a second database, and the second database is constructed using image samples screened out from original data samples using a pre-built sample selection model.
6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 4 is implemented.
7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.
8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.
Citation Information
Patent Citations
Risk event prediction method and system and risk event prediction system generation method
CN112489803A
Data screening method and device, electronic equipment and storage medium
CN112560993A
MSI typing determination method and device, electronic equipment and storage medium
CN116344037A
Image classification method, system and device based on active domain self-adaption and medium
CN116630708A
Dynamic Gradient Deception Against Adversarial Examples in Machine Learning Models
US20220180242A1
