Target classification method based on neural network mutual information estimation
By introducing mutual information estimation methods into neural networks, and using multiple neural network models to calculate the boundaries to estimate mutual information, the problem of low training efficiency caused by the simple neural network structure in the prior art is solved, and the accuracy of feature selection and target classification is improved.
Patent Information
- Application Number
- CN202510226659.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-27
- Publication Date
- 2025-06-20
AI Technical Summary
The neural network structure adopted by the neural network-based mutual information estimation method in the prior art is too simple, resulting in low efficiency in training process, affecting the efficiency of feature selection and target classification.
By obtaining the feature vectors and category labels of the samples, a uniform distribution is generated as a reference distribution, and the neural network models XY_net, X_net and Y_net are used to calculate the boundaries, thereby estimating mutual information, and selecting the features that contribute the most to the classification for target classification.
It improves the efficiency of mutual information estimation, enhances the accuracy of feature selection, and thus improves the accuracy of target classification.
Smart Images
Figure CN120180182A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of data processing, and particularly relates to an object classification method based on neural network mutual information estimation. Background Art
[0002] Feature selection is an important issue in object classification, aiming to reduce the number of features and eliminate irrelevant features. Mutual information is a basic quantity for measuring the mutual dependence between random variables, so mutual information can be used for feature selection. However, since the self-mutual information was proposed, how to obtain the value of mutual information has been a difficult problem. Only discrete variables with exact summation or problems of a finite family with known probabilities can calculate the mutual information accurately. For continuous variables, the accurate calculation of mutual information requires integrating the joint probability density function of the variables, and the integration dimension of the joint distribution is too large to calculate.
[0003] In recent years, neural networks have shown great potential in processing large-scale and high-dimensional data. Those skilled in the art have begun to pay attention to the application of neural networks in mutual information estimation. How to quickly estimate the value of mutual information is still an important challenge. However, the neural network structures adopted by the existing neural network-based mutual information estimation methods are too simple, and each neural network is trained independently, and the data distribution and network structure characteristics are not fully utilized. In the calculation of high-dimensional data such as mutual information estimation, it will lead to low efficiency in the training process of the neural network, further affecting the efficiency of feature selection and object classification. Summary of the Invention
[0004] In order to solve the above problems existing in the prior art, the present invention provides an object classification method based on neural network mutual information estimation. The technical problems to be solved by the present invention are realized through the following technical solutions:
[0005] The present invention provides an object classification method based on neural network mutual information estimation, including:
[0006] Obtain the feature vector X and class label Y of each sample, and generate a set of uniform distributions of the feature vectors and class labels of each sample as the first random reference distribution. Each feature vector X and each feature vector X' in the first random reference distribution include the feature values of multiple features of the sample;
[0007] Based on the feature values of each sample under each feature, the feature values of each sample in the first random reference distribution, the class label Y of each sample, and the class label Y' of each sample in the first random reference distribution, use the neural network model XY_net to calculate the first supremum;
[0008] Based on the feature values of each sample under each feature and the feature values of each sample in the first random reference distribution, use the neural network model X_net to calculate the second supremum;
[0009] Based on the class labels Y of each sample and the class labels Y' of each sample in the first random reference distribution, use the neural network model Y_net to calculate the third supremum;
[0010] Based on the first supremum, the second supremum, and the third supremum, estimate the mutual information between each feature and the class label Y;
[0011] Select the first feature that contributes the most to classification based on the mutual information;
[0012] Use the first features of multiple samples to train the target classification model for target classification.
[0013] In an embodiment of the present invention, the step of calculating the first supremum based on the feature values of each sample under each feature and the feature values of each sample in the first random reference distribution, the class labels Y of each sample and the class labels Y' of each sample in the first random reference distribution, and using the neural network model XY_net includes:
[0014] Input the class label Y of each sample and the feature values of each sample under each feature into the neural network model XY_net to obtain the parameter θ1;
[0015] Input the feature values of each sample in the first random reference distribution and the class labels Y' of each sample in the first random reference distribution into the neural network model XY_net to obtain the parameter θ2;
[0016] Calculate the first supremum according to the parameter θ1 and the parameter θ2.
[0017] In an embodiment of the present invention, the first supremum is expressed as:
[0018]
[0019] In the formula, sup 1,j represents the first supremum corresponding to feature j, N represents the number of samples, x ij represents the eigenvalue corresponding to feature j in the feature vector X of the i-th sample i Y i represents the class label of the i-th sample, N' represents the number of samples in the first random reference distribution, x' ij represents the eigenvalue corresponding to feature j in the feature vector X' of the i-th sample in the first random reference distribution i Y i′ represents the class label of the i-th sample in the first random reference distribution, denotes inputting x ij , Y i into the neural network model XY_net to obtain the parameter θ1, denotes inputting x′ ij , Y i ′ into the neural network model XY_net to obtain the parameter θ2.
[0020] The second supremum is expressed as:
[0021]
[0022] where sup 2,j represents the second supremum corresponding to feature j, denotes inputting x ij into the neural network model X_net to obtain the parameter θ3, denotes inputting x′ ij into the neural network model X_net to obtain the parameter θ4.
[0023] The third supremum is expressed as:
[0024]
[0025] where sup 3,j represents the third supremum corresponding to feature j, denotes the parameter θ5 obtained by inputting Y i into the neural network model Y_net, denotes the parameter θ6 obtained by inputting Y i ′ into the neural network model Y_net.
[0026] The mutual information between feature j and the class label Y is expressed as:
[0027]
[0028] In an embodiment of the present invention, the neural network models XY_net, X_net, and Y_net have the same network structure, all including an input layer, a first hidden layer, a second hidden layer, and an output layer.
[0029] In an embodiment of the present invention, the neural network model X_net is trained based on the following first preset loss function:
[0030]
[0031] where x pqDenotes the eigenvalue corresponding to feature q in the feature vector of the p-th training sample, x′ pq Denotes the eigenvalue corresponding to feature q in the feature vector of the p-th sample in the generated second random reference distribution, where the second random reference distribution is a set of uniform distributions regarding the feature vectors and class labels of each training sample Denotes during the t-th round of training, when x pq Is input into the first neural network to be trained, the parameter θ is obtained 3,t , Denotes during the t-th round of training, when x′ pq Is input into the first neural network to be trained, the parameter θ is obtained 4,t , E[] represents the expectation
[0032] In an embodiment of the present invention, the neural network model Y_net is trained based on the second preset loss function shown as follows
[0033]
[0034] In the formula, Y p Denotes the class label of the p-th training sample, Y′ p Denotes the class label of the p-th sample in the generated second random reference distribution Denotes during the t-th round of training, when Y p Is input into the second neural network to be trained, the parameter θ is obtained 5,t , Denotes during the t-th round of training, when Y′ p Is input into the second neural network to be trained, the parameter θ is obtained 6,t
[0035] In an embodiment of the present invention, the parameters of the first hidden layer in the neural network model XY_net are obtained by merging and migrating the parameters of the first hidden layer in the neural network model X_net and the neural network model Y_net, and the remaining parameters are trained based on the third preset loss function shown as follows
[0036]
[0037] In the formula Denotes during the t-th round of training, when x pq And Y p Are input into the third neural network to be trained, the parameter θ is obtained 1,t , Denotes during the t-th round of training, when x′ pq And Y′ p Are input into the third neural network to be trained, the parameter θ is obtained 2,t
[0038] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0039] The present invention provides a target classification method based on neural network mutual information estimation. By introducing mutual information to select the features of training samples used in the training process of the target classification model, it is beneficial to improve the accuracy of subsequent target classification.
[0040] Secondly, the present invention uses neural network models XY_net, X_net, and Y_net to determine the first supremum, the second supremum, and the third supremum, and then estimates the mutual information based on the first supremum, the second supremum, and the third supremum. Since the above neural network model XY_net adopts a transfer learning strategy during the training process, that is, the parameters of the first hidden layer in XY_net are obtained by merging and transferring the parameters of the first hidden layer in neural network models X_net and Y_net, thus the similarity of data distribution can be more fully utilized. And since the role of the first hidden layer is to extract low-level features of the input data, the statistical characteristics of the data distribution can be fully utilized, improving the efficiency of mutual information estimation.
[0041] The following will further elaborate on the present invention in conjunction with the drawings and embodiments. Brief Description of the Drawings
[0042] Figure 1 is a flowchart of a target classification method based on neural network mutual information estimation provided by an embodiment of the present invention;
[0043] Figure 2 is a schematic structural diagram of neural network models XY_net, Y_net, and X_net provided by an embodiment of the present invention;
[0044] Figures 3a to 3c is a curve graph showing the change of mutual information estimation results of different methods with the number of iterations of the neural network model. Detailed Embodiment
[0045] The following further describes the present invention in detail with specific embodiments, but the embodiments of the present invention are not limited thereto.
[0046] Figure 1 is a flowchart of a target classification method based on neural network mutual information estimation provided by an embodiment of the present invention. Please refer to Figure 1 , an embodiment of the present invention provides a target classification method based on neural network mutual information estimation, including:
[0047] S1. Obtain the feature vectors X and class labels Y of each sample, and generate a set of uniformly distributed feature vectors and class labels of each sample as the first random reference distribution. Each feature vector X and each feature vector X' in the first random reference distribution include the eigenvalues of various features of the sample.
[0048] Specifically, taking the iris dataset in the target classification field as an example, the features of each iris flower sample include: sepal length, sepal width, petal length, and petal width. Then the feature vector X includes the specific values of these four features (unit: cm). In addition, the class label Y includes 0, 1, and 2, representing Iris setosa, Iris versicolor, and Iris virginica respectively.
[0049] S2. Based on the feature values of each sample under each feature and the feature values of each sample in the first random reference distribution, the class labels Y of each sample and the class labels Y' of each sample in the first random reference distribution, use the neural network model XY_net to calculate the first supremum.
[0050] It should be understood that in this embodiment, mutual information is introduced to measure the correlation and dependence between features and class labels. The greater the mutual information, the higher the degree of mutual dependence between the feature and the class label, and the stronger the correlation between the two, so as to perform feature selection for target classification.
[0051] Based on the MINE (Mutual Information Neural Estimator) algorithm, by using the reference distribution to replace the product of marginal distributions in MINE, the calculation formula of mutual information can be derived as follows:
[0052]
[0053] Furthermore, in this embodiment, the neural network models XY_net, X_net, and Y_net are respectively used to estimate three suprema.
[0054] Optionally, step S2 includes:
[0055] S201. Input the class label Y of each sample and the feature values of each sample under each feature into the neural network model XY_net to obtain the parameter θ1;
[0056] S202. Input the feature values of each sample in the first random reference distribution and the class labels Y' of each sample in the first random reference distribution into the neural network model XY_net to obtain the parameter θ2;
[0057] S203. Calculate the first supremum according to the parameter θ1 and the parameter θ2.
[0058] In this embodiment, the first supremum is expressed as:
[0059]
[0060] Wherein, sup 1,j represents the first supremum corresponding to feature j, N represents the number of samples, and x ij represents the feature vector X of the i-th sample i the eigenvalue corresponding to feature j in, and Y i represents the class label of the i-th sample, N' represents the number of samples in the first random reference distribution, and x' ij represents the feature vector X' of the i-th sample in the first random reference distribution i the eigenvalue corresponding to feature j in, and Y i ' represents the class label of the i-th sample in the first random reference distribution, represents that after inputting x ij , Y i into the neural network model XY_net, the parameter θ1 is obtained, represents that after inputting x' ij , Y i ' into the neural network model XY_net, the parameter θ2 is obtained.
[0061] S3. Based on the eigenvalues of each sample under each feature and the eigenvalues of each sample in the first random reference distribution, use the neural network model X_net to calculate the second supremum.
[0062] In this embodiment, the second supremum is expressed as:
[0063]
[0064] Wherein, sup 2,j represents the second supremum corresponding to feature j, represents that after inputting x ij into the neural network model X_net, the parameter θ3 is obtained, represents that after inputting x' ij into the neural network model X_net, the parameter θ4 is obtained.
[0065] S4. Based on the class labels Y of each sample and the class labels Y' of each sample in the first random reference distribution, use the neural network model Y_net to calculate the third supremum.
[0066] Optionally, the third supremum is expressed as:
[0067]
[0068] Wherein, sup 3,j represents the third supremum corresponding to feature j, represents that after inputting Yi The parameter θ5 obtained after inputting into the neural network model Y_net, indicates that after inputting Y i ′ into the neural network model Y_net, the parameter θ6 is obtained.
[0069] Figure 2 is a schematic structural diagram of the neural network models XY_net, Y_net, and X_net provided by the embodiments of the present invention. It should be noted that in this embodiment, the network structures of the neural network model XY_net, the neural network model X_net, and the neural network model Y_net are the same. As Figure 2 shown, all include an input layer, a first hidden layer, a second hidden layer, and an output layer.
[0070] In this embodiment, during the training process of the neural network models XY_net, X_net, and Y_net, the parameters of the neural network to be trained, such as weights and biases, are gradually adjusted by minimizing the loss function to make the predicted output results closer to the true values. Specifically, first, the class labels of multiple training samples and the feature vectors of each training sample are obtained, and a set of uniform distributions of the feature vectors and class labels of each training sample are generated as the second random reference distribution. Since the feature vectors of each training sample include the eigenvalue of multiple features, when inputting into the neural network to be trained, the corresponding feature vectors also need to be input in sequence according to the features.
[0071] Optionally, the neural network model X_net is trained based on the first preset loss function shown as follows:
[0072]
[0073] In the formula, x pq represents the eigenvalue corresponding to the feature q in the feature vector of the p-th training sample, and x′ pq represents the eigenvalue corresponding to the feature q in the feature vector of the p-th sample in the generated second random reference distribution. The second random reference distribution is a set of uniform distributions of the feature vectors and class labels of each training sample. represents the parameter θ obtained after inputting x pq into the first neural network to be trained during the t-th round of training process 3,t , represents the parameter θ obtained after inputting x′ pq into the first neural network to be trained during the t-th round of training process 4,t , and E[] represents the expectation.
[0074] The neural network model Y_net is trained based on the second preset loss function shown as follows:
[0075]
[0076] Wherein, Y p represents the class label of the p-th training sample, and Y' p represents the class label of the p-th sample in the generated second random reference distribution. represents the parameter θ obtained after inputting Y p into the second neural network to be trained during the t-th round of training. 5,t , represents the parameter θ obtained after inputting Y' p into the second neural network to be trained during the t-th round of training. 6,t .
[0077] In addition, the difference between the neural network model Y_net and XY_net, X_net during training is that the neural network model Y_net adopts transfer learning during training, that is, the parameters of the first hidden layer in the neural network model Y_net are obtained by merging and transferring the parameters of the first hidden layer in the neural network models XY_net and X_net, and the remaining parameters are trained based on the third preset loss function shown below:
[0078]
[0079] Wherein, represents the parameter θ obtained after inputting x pq and Y p into the third neural network to be trained during the t-th round of training. 1,t , represents the parameter θ obtained after inputting x' pq and Y' p into the third neural network to be trained during the t-th round of training. 2,t .
[0080] S5. Estimate the mutual information between each feature and each class label based on the first supremum, the second supremum, and the third supremum.
[0081] Specifically, the mutual information between feature j and each class label is expressed as:
[0082]
[0083] S6. Select the first feature that contributes the most to classification based on the mutual information.
[0084] Specifically, still taking the iris dataset as an example, the mutual information between the four features of sepal length, sepal width, petal length, and petal width and the class label is sorted in descending order, and the first feature with the largest mutual information with the class label is selected as the best feature for iris classification.
[0085] S7. Train a target classification model using the first features of multiple samples for target classification.
[0086] It should be noted that in this step, the first features of multiple samples can be used to selectively train an existing neural network to obtain a target classification model.
[0087] Figures 3a to 3c They are respectively the curves of the mutual information estimation results of the MI-NEE method, the MINE method, and the present invention with the number of iterations of the neural network model. Among them, the vertical axis is the mutual information estimation result, and the horizontal axis represents the number of iterations of the adopted neural network model. It can be seen that Figure 3c the curve corresponding to the present invention in [reference] reaches 90% reached, and the required number of iterations is much lower than Figure 3a and Figure 3b the MI-NEE and MINE methods shown in [reference], and the training efficiency is significantly improved.
[0088] As can be seen from the above embodiments, the beneficial effects of the present invention are as follows:
[0089] The present invention provides a target classification method based on neural network mutual information estimation. By introducing mutual information to select the features of training samples used in the training process of the target classification model, it is beneficial to improve the accuracy of subsequent target classification.
[0090] Secondly, the present invention uses the neural network models XY_net, X_net, and Y_net to determine the first supremum, the second supremum, and the third supremum, and then estimates the mutual information according to the first supremum, the second supremum, and the third supremum. Since the above neural network model XY_net adopts a transfer learning strategy during the training process, that is, the parameters of the first hidden layer in XY_net are obtained by merging and transferring the parameters of the first hidden layer in the neural network models X_net and Y_net, the similarity of data distribution can be more fully utilized. And since the role of the first hidden layer is to extract the low-level features of the input data, the statistical characteristics of the data distribution can be fully utilized, improving the efficiency of mutual information estimation.
[0091] In the description of the present invention, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the present invention, "a plurality" means two or more unless otherwise specifically defined.
[0092] In the description of this specification, the description referring to terms such as "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification.
[0093] Although the present application has been described herein in connection with various embodiments, however, in the process of implementing the claimed present application, those skilled in the art can understand and achieve other variations of the disclosed embodiments by viewing the accompanying drawings, the disclosure, and the appended claims.
[0094] The above content is a further detailed description of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is only limited to these descriptions. For those of ordinary skill in the technical field to which the present invention pertains, without departing from the concept of the present invention, several simple deductions or substitutions can still be made, and all should be regarded as belonging to the protection scope of the present invention.
Claims
1. A target classification method based on neural network mutual information estimation, characterized in that: include: Obtain a feature vector X and a category label Y of each sample, and generate a set of uniform distributions of the feature vectors and category labels of each sample as a first random reference distribution, wherein each feature vector X and each feature vector X′ in the first random reference distribution includes feature values of multiple features of the sample; Based on the feature value of each sample under each feature and the feature value of each sample in the first random reference distribution, the category label Y of each sample and the category label Y′ of each sample in the first random reference distribution, the first supremum is calculated using the neural network model XY_net; Based on the eigenvalues of each sample under each feature and the eigenvalues of each sample in the first random reference distribution, a second supremum is calculated using a neural network model X_net; Based on the category label Y of each sample and the category label Y′ of each sample in the first random reference distribution, a third supremum is calculated using a neural network model Y_net; Based on the first supremum, the second supremum and the third supremum, estimating the mutual information between each feature and the category label Y; Selecting a first feature that contributes most to classification based on the mutual information; The target classification model is trained using the first features of the plurality of samples to perform target classification.
2. The target classification method based on neural network mutual information estimation according to claim 1 is characterized in that: The step of calculating the first supremum using the neural network model XY_net based on the feature value of each sample under each feature and the feature value of each sample in the first random reference distribution, the category label Y of each sample and the category label Y′ of each sample in the first random reference distribution includes: Input the category label Y of each sample and the feature value of each sample under each feature into the neural network model XY_net to obtain the parameter θ1; Input the characteristic value of each sample in the first random reference distribution and the category label Y′ of each sample in the first random reference distribution into the neural network model XY_net to obtain parameter θ2; The first supremum is calculated according to the parameter θ1 and the parameter θ2.
3. The target classification method based on neural network mutual information estimation according to claim 2 is characterized in that: The first supremum is expressed as: In the formula, sup 1,j represents the first supremum corresponding to feature j, N represents the number of samples, x ij Represents the feature vector X of the i-th sample i The eigenvalue corresponding to feature j in Y i represents the category label of the i-th sample, N ′ represents the number of samples in the first random reference distribution, x i ′ j represents the feature vector X of the i-th sample in the first random reference distribution i ′ The eigenvalue corresponding to feature j in Y i ′ represents the category label of the i-th sample in the first random reference distribution, Indicates that x ij , Y i After inputting the neural network model XY_net, the parameter θ1 is obtained. Indicates that x i ′ j , Y i ′ After inputting the neural network model XY_net, the parameter θ2 is obtained.
4. The target classification method based on neural network mutual information estimation according to claim 3 is characterized in that: The second supremum is expressed as: In the formula, sup 2,j represents the second supremum corresponding to feature j, Indicates that x ij After inputting the neural network model X_net, the parameter θ3 is obtained. Indicates that x i ′ j After inputting the neural network model X_net, the parameter θ4 is obtained.
5. The target classification method based on neural network mutual information estimation according to claim 4 is characterized in that: The third supremum is expressed as: In the formula, sup 3,j represents the third supremum corresponding to feature j, Indicates that Y i The parameter θ5 obtained after inputting the neural network model Y_net, Indicates that Y i ′ After inputting the neural network model Y_net, the parameter θ6 is obtained.
6. The target classification method based on neural network mutual information estimation according to claim 5, characterized in that: The mutual information between feature j and the category label Y is expressed as:
7. The target classification method based on neural network mutual information estimation according to claim 1, characterized in that: The neural network model XY_net, the neural network model X_net and the neural network model Y_net have the same network structure, and all include an input layer, a first hidden layer, a second hidden layer and an output layer.
8. The target classification method based on neural network mutual information estimation according to claim 7 is characterized in that: The neural network model X_net is trained based on the first preset loss function as shown below: In the formula, x pq represents the eigenvalue corresponding to feature q in the feature vector of the pth training sample, x ′ pq represents the eigenvalue corresponding to the feature q in the feature vector of the p-th sample in the generated second random reference distribution, where the second random reference distribution is a set of uniform distributions of the feature vectors and category labels of each training sample, Indicates that during the tth round of training, x pq After inputting the first neural network to be trained, the parameter θ is obtained 3,t , Indicates that during the tth round of training, x ′ pq After inputting the first neural network to be trained, the parameter θ is obtained 4,t , E[] represents expectation.
9. The target classification method based on neural network mutual information estimation according to claim 8, characterized in that: The neural network model Y_net is trained based on the second preset loss function as shown below: Where Y p represents the category label of the pth training sample, Y p ′ represents the category label of the pth sample in the second random reference distribution generated, Indicates that Y p After inputting the second neural network to be trained, the parameter θ is obtained 5,t , Indicates that Y p ′ After inputting the second neural network to be trained, the parameter θ is obtained 6,t .
10. The target classification method based on neural network mutual information estimation according to claim 9, characterized in that: The parameters of the first hidden layer in the neural network model XY_net are obtained by merging and migrating the parameters of the first hidden layer in the neural network model X_net and the neural network model Y_net, and the remaining parameters are obtained by training based on the third preset loss function as shown below: In the formula, Indicates that during the tth round of training, x pq and Y p After inputting the third neural network to be trained, the parameter θ is obtained 1,t , Indicates that during the tth round of training, x ′ pq and Y p ′ After inputting the third neural network to be trained, the parameter θ is obtained 2,t .