Unbalanced spectral data detection method, device, electronic device and storage medium
The unbalanced spectral data is processed through the K-mean clustering algorithm and the variational EM algorithm, which solves the problem of difficulty in selecting representative spectral data points in the existing technology, and realizes the accurate classification and identification and detection efficiency of unbalanced spectral data. promote.
Patent Information
- Application Number
- CN202410996293.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-24
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-07-24
AI Technical Summary
Existing spectral data detection methods are difficult to select the most representative spectral data points from unbalanced spectral data, which makes it impossible to accurately analyze the internal laws of unbalanced spectral data.
The K-mean clustering algorithm is used to sample the unbalanced spectral data by dimensionality reduction and variational EM algorithm. After obtaining the spectral sampling data, it is input into the preset spectral detection model to calculate the category results of the unbalanced spectral data.
Accurate and rapid classification and identification of unbalanced spectral data is achieved, and detection efficiency is improved.
Smart Images

Figure CN118551280B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of detecting spectral data, and in particular to a method, device, electronic device and storage medium for detecting unbalanced spectral data. Background Art
[0002] In spectral detection, detecting unbalanced spectral data is a common and challenging problem. Unbalanced spectral data refers to spectral data in which the number of spectral data points of a certain category is much larger than that of other categories, or the distribution of spectral data points of some categories in the feature space is extremely sparse.
[0003] In order to process unbalanced spectral data, sampling technology is applied to the field of machine learning algorithms. By sampling a portion of representative spectral data points from the majority class spectral data point samples through machine learning algorithms, the unbalanced spectral data can be balanced, which can effectively improve the model's recognition ability for minority class spectral data point samples and improve classification accuracy. However, the existing spectral detection methods based on machine learning algorithms are difficult to accurately analyze unbalanced spectral data, and cannot effectively select the most representative spectral data points, thus failing to accurately characterize the inherent laws of unbalanced spectral data.
[0004] Therefore, in order to solve the technical problem that the existing spectral data detection method cannot effectively select the most representative spectral data points from the unbalanced spectral data and is difficult to accurately analyze the unbalanced spectral data, an unbalanced spectral data detection method, device, electronic device and storage medium are urgently needed. Summary of the invention
[0005] The purpose of the present application is to provide an unbalanced spectral data detection method, device, electronic device and storage medium. By inputting the spectral sampling data obtained based on the K-means clustering algorithm and the variational EM algorithm into a preset spectral detection model, the category result of the unbalanced spectral data is calculated, so as to solve the problem that the existing spectral data detection method cannot effectively select the most representative spectral data points from the unbalanced spectral data and is difficult to accurately analyze the unbalanced spectral data. The unbalanced spectral data can be accurately and quickly classified and identified, thereby improving the detection efficiency of the unbalanced spectral data.
[0006] In a first aspect, the present application provides a method for detecting unbalanced spectral data, comprising:
[0007] Acquiring unbalanced spectral data;
[0008] After reducing the dimension of the unbalanced spectral data, clustering the unbalanced spectral data after the dimension reduction is performed using a K-means clustering algorithm to obtain dimension-reduced spectral data;
[0009] Sampling the reduced-dimensional spectral data by a variational EM algorithm to obtain spectral sampling data;
[0010] The spectral sampling data is input into a preset spectral detection model to calculate and obtain the classification result of the unbalanced spectral data.
[0011] The unbalanced spectral data detection method provided in the present application can realize the detection of unbalanced spectral data. By inputting the spectral sampling data obtained based on the K-means clustering algorithm and the variational EM algorithm into a preset spectral detection model, the category result of the unbalanced spectral data is calculated, which solves the problem that the existing spectral data detection method cannot effectively select the most representative spectral data points from the unbalanced spectral data and is difficult to accurately analyze the unbalanced spectral data. The unbalanced spectral data can be accurately and quickly classified and identified, thereby improving the detection efficiency of the unbalanced spectral data.
[0012] Optionally, the unbalanced spectral data includes spectral data points; after reducing the dimension of the unbalanced spectral data, a K-means clustering algorithm is used to cluster the unbalanced spectral data after dimension reduction to obtain dimension-reduced spectral data, including:
[0013] Performing dimension reduction on the unbalanced spectral data to obtain unbalanced spectral data after dimension reduction;
[0014] Selecting a plurality of the spectral data points from the unbalanced spectral data after dimension reduction as a plurality of initial clustering centers;
[0015] Based on the initial cluster centers, all the spectral data points are divided into a plurality of spectral data point groups by using the K-means clustering algorithm to obtain dimension-reduced spectral data.
[0016] The unbalanced spectral data detection method provided in the present application can realize the detection of unbalanced spectral data. Through the K-means clustering algorithm, the spectral data points in the unbalanced spectral data are divided into multiple spectral data point groups to obtain reduced-dimensional spectral data. By sampling the reduced-dimensional spectral data, it is beneficial to improve the detection efficiency of the unbalanced spectral data.
[0017] Optionally, based on the initial cluster center, all the spectral data points are divided into a plurality of spectral data point groups by the K-means clustering algorithm to obtain dimension-reduced spectral data, including:
[0018] Step A1, calculating the distances between other spectral data points among the spectral data points except the initial cluster center and the multiple initial cluster centers;
[0019] Step A2, allocating the other spectral data points to the initial cluster center that is closest to the other spectral data points to form a spectral data point group;
[0020] Step A3, recalculating the cluster center in the spectral data point grouping, and reallocating all the other spectral data points outside the cluster center according to the distance between the cluster center and all the other spectral data points outside the cluster center;
[0021] Step A4, repeatedly executing step A3 until the cluster center no longer changes, and then grouping the multiple spectral data points obtained by division into reduced-dimensional spectral data.
[0022] Optionally, the dimension-reduced spectral data is sampled by a variational EM algorithm to obtain spectral sampling data, including:
[0023] Based on a preset energy functional function, while eliminating abnormal data in the reduced-dimensionality spectral data, a corresponding conditional probability distribution is constructed to calculate the variational probability distribution of the reduced-dimensionality spectral data through a sampling method;
[0024] Adopting the gradient descent method, iteratively updating the reduced-dimensionality spectral data, and iterating the conditional probability distribution corresponding to the iteratively updated reduced-dimensionality spectral data, to obtain a target conditional probability distribution with the minimum KL divergence with the variational probability distribution;
[0025] The iterated dimension-reduced spectral data corresponding to the target conditional probability distribution is determined as the spectral sampling data.
[0026] The unbalanced spectral data detection method provided in the present application can realize the detection of unbalanced spectral data. Through the variational EM algorithm, after eliminating abnormal data, the target conditional probability distribution with the smallest KL divergence with the variational probability distribution is calculated, and the reduced-dimensional spectral data corresponding to the target conditional probability distribution is obtained as spectral sampling data. The spectral sampling data is detected by the spectral detection model, and the category of the unbalanced spectral data can be obtained, thereby improving the detection efficiency of the unbalanced spectral data.
[0027] Optionally, based on a preset energy functional function, while eliminating abnormal data in the reduced-dimensionality spectral data, a corresponding conditional probability distribution is constructed to calculate the variational probability distribution of the reduced-dimensionality spectral data by a sampling method, including:
[0028] Setting the conditional probability distribution of the reduced-dimensional spectrum data to a normal distribution having the preset energy functional function;
[0029] Calculating the envelope of the preset energy functional function, and removing the spectral data points outside the envelope in the dimension-reduced spectral data;
[0030] Taking the sampling number of spectral data points of each spectral data point group as a constraint and maximizing the normal distribution as a goal, sampling is performed in the reduced-dimensional spectral data to calculate the variational probability distribution of the reduced-dimensional spectral data.
[0031] Optionally, the reduced-dimensionality spectral data is iteratively updated using a gradient descent method, and the conditional probability distribution corresponding to the iteratively updated reduced-dimensionality spectral data is iterated to obtain a target conditional probability distribution with the minimum KL divergence with the variational probability distribution, including:
[0032] Adopting a gradient descent method to iteratively update the dimension-reduced spectral data, recording the dimension-reduced spectral data after each iteration, and obtaining the dimension-reduced spectral data after multiple iterations;
[0033] Calculating a plurality of conditional probability distributions corresponding to the iterative dimension-reduced spectral data, and calculating KL divergences between the plurality of conditional probability distributions and the variational probability distribution to obtain a plurality of KL divergences;
[0034] Comparing the magnitude relationships between the multiple KL divergences to extract the minimum value among the multiple KL divergences;
[0035] The conditional probability distribution corresponding to the minimum value is determined as the target conditional probability distribution with the minimum KL divergence with the variational probability distribution.
[0036] Optionally, the preset spectral detection model is a random forest model.
[0037] In a second aspect, the present application provides an unbalanced spectral data detection device, comprising:
[0038] An acquisition module, used for acquiring unbalanced spectral data;
[0039] A dimension reduction module, used for performing dimension reduction on the unbalanced spectral data, and clustering the unbalanced spectral data after dimension reduction using a K-means clustering algorithm to obtain dimension reduced spectral data;
[0040] A sampling module, used for sampling the dimension-reduced spectral data by using a variational EM algorithm to obtain spectral sampling data;
[0041] The calculation module is used to input the spectrum sampling data into a preset spectrum detection model to calculate and obtain the classification result of the unbalanced spectrum data.
[0042] The unbalanced spectral data detection device inputs spectral sampling data obtained based on the K-means clustering algorithm and the variational EM algorithm into a preset spectral detection model to calculate the classification result of the unbalanced spectral data, thereby solving the problem that the existing spectral data detection method cannot effectively select the most representative spectral data points from the unbalanced spectral data and is difficult to accurately analyze the unbalanced spectral data. The unbalanced spectral data can be accurately and quickly classified and identified, thereby improving the detection efficiency of the unbalanced spectral data.
[0043] In a third aspect, the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores a computer program executable by the processor, and when the processor executes the computer program, it runs the steps in the unbalanced spectral data detection method described above.
[0044] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in the unbalanced spectral data detection method described above are executed.
[0045] Beneficial effects: The unbalanced spectral data detection method, device, electronic device and storage medium provided in the present application, by inputting the spectral sampling data obtained based on the K-means clustering algorithm and the variational EM algorithm into a preset spectral detection model, calculates the category result of the unbalanced spectral data, and solves the problem that the existing spectral data detection method cannot effectively select the most representative spectral data points from the unbalanced spectral data and is difficult to accurately analyze the unbalanced spectral data. It can accurately and quickly classify and identify the unbalanced spectral data, thereby improving the detection efficiency of the unbalanced spectral data. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 A flow chart of a method for detecting unbalanced spectral data provided in an embodiment of the present application.
[0047] Figure 2 A schematic diagram of the structure of an unbalanced spectral data detection device provided in an embodiment of the present application.
[0048] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.
[0049] Explanation of reference numerals: 1. Acquisition module; 2. Dimensionality reduction module; 3. Sampling module; 4. Calculation module; 301. Processor; 302. Memory; 303. Communication bus. DETAILED DESCRIPTION
[0050] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.
[0051] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.
[0052] Please refer to Figure 1 , Figure 1 A method for detecting unbalanced spectral data in some embodiments of the present application is used to detect unbalanced spectral data, comprising the steps of:
[0053] Step S101, obtaining unbalanced spectrum data;
[0054] Step S102, after reducing the dimension of the unbalanced spectral data, clustering the unbalanced spectral data after the dimension reduction is performed using a K-means clustering algorithm to obtain dimension-reduced spectral data;
[0055] Step S103, sampling the dimension-reduced spectral data by using a variational EM algorithm to obtain spectral sampling data;
[0056] Step S104: input the spectrum sampling data into a preset spectrum detection model to calculate and obtain the classification result of the unbalanced spectrum data.
[0057] The unbalanced spectral data detection method, by inputting the spectral sampling data obtained based on the K-means clustering algorithm and the variational EM algorithm into a preset spectral detection model, calculates the category result of the unbalanced spectral data, solves the problem that the existing spectral data detection method cannot effectively select the most representative spectral data points from the unbalanced spectral data and is difficult to accurately analyze the unbalanced spectral data, can accurately and quickly classify and identify the unbalanced spectral data, and improves the detection efficiency of the unbalanced spectral data.
[0058] Specifically, in step S101, unbalanced spectral data is obtained. The unbalanced spectral data consists of spectral data points. The unbalanced spectral data refers to spectral data in which the number of spectral data points of a certain category is much greater than that of spectral data of other categories, or spectral data points of certain categories are extremely sparsely distributed in the feature space.
[0059] Specifically, in step S102, after the unbalanced spectral data is reduced in dimension, the K-means clustering algorithm is used to cluster the unbalanced spectral data after the dimension reduction to obtain the reduced dimension spectral data, including:
[0060] Performing dimension reduction on the unbalanced spectral data to obtain unbalanced spectral data after dimension reduction;
[0061] Selecting a plurality of spectral data points from the unbalanced spectral data after dimension reduction as a plurality of initial clustering centers;
[0062] Based on the initial cluster centers, all spectral data points are divided into multiple spectral data point groups through the K-means clustering algorithm to obtain dimension-reduced spectral data.
[0063] In step S102, redundant data in the unbalanced spectral data is removed by existing dimensionality reduction methods such as principal component analysis (PCA) method or linear discriminant analysis method (LDA) method, and the number of spectral data points with original features is reduced to a certain number of spectral data points with the most important features, thereby obtaining unbalanced spectral data after dimensionality reduction. Among them, existing dimensionality reduction methods such as principal component analysis method or linear discriminant analysis method are prior arts and will not be described in detail here.
[0064] The K-means clustering algorithm is used to reduce the dimension of the unbalanced spectral data after dimensionality reduction. Multiple spectral data points are randomly selected from the unbalanced spectral data after dimensionality reduction as multiple initial clustering centers, and other spectral data points are assigned to the initial clustering center with the smallest distance therefrom to form a spectral data point grouping based on the initial clustering center (that is, the clustering center of the spectral data point grouping is the initial clustering center). After each grouping is completed, the clustering center of each spectral data point grouping is re-divided (the division method is to select the spectral data point closest to the grouping center in the spectral data point grouping as the new clustering center), and based on the divided clustering center, the other spectral data points in each spectral data point grouping are reallocated (the allocation method is to assign other spectral data points to the clustering center with the smallest distance therefrom), until all spectral data points in the reallocated spectral data point grouping are no longer reallocated, forming multiple spectral data point groups, and obtaining reduced-dimensional spectral data.
[0065] Specifically, in step S102, based on the initial cluster center, all spectral data points are divided into a plurality of spectral data point groups by using the K-means clustering algorithm to obtain dimension-reduced spectral data, including:
[0066] Step A1, calculating the distances between other spectral data points in the spectral data points except the initial cluster center and the multiple initial cluster centers;
[0067] Step A2, assigning other spectral data points to the nearest initial cluster center to form spectral data point groups;
[0068] Step A3, recalculating the cluster center in the spectral data point grouping, and reallocating all other spectral data points outside the cluster center according to the distance between the cluster center and all other spectral data points outside the cluster center;
[0069] Step A4, repeatedly executing step A3 until the cluster center no longer changes, and then grouping the multiple spectrum data points obtained by division into reduced-dimensional spectrum data.
[0070] In step S102, the distances between other spectral data points and the initial clustering center in the unbalanced spectral data after dimensionality reduction are calculated, and the initial clustering center is used as the central node for grouping. The initial clustering center with the smallest distance to other spectral data points is extracted from all the initial clustering centers, and the other spectral data points are assigned to the group where the initial clustering center with the smallest distance to the other spectral data points is located. After each assignment is completed, it is necessary to recalculate the spectral data point closest to the center point position as the clustering center of the spectral data point group according to the positions of each spectral data point in each spectral data point group, recalculate the distances between other spectral data points except the clustering center and multiple clustering centers, and assign the other spectral data points to the group where the clustering center with the smallest distance to the other spectral data points is located, and repeat the steps of calculating the clustering center and assigning the spectral data points until the clustering center no longer changes, or the spectral data points of each spectral data point group no longer change, and the spectral data point group finally calculated is determined as the reduced dimensionality spectral data.
[0071] Specifically, in step S103, the dimension-reduced spectral data is sampled by using a variational EM algorithm to obtain spectral sampling data, including:
[0072] Based on the preset energy functional function, while eliminating abnormal data in the reduced-dimensionality spectral data, the corresponding conditional probability distribution is constructed to calculate the variational probability distribution of the reduced-dimensionality spectral data through the sampling method;
[0073] The gradient descent method is used to iteratively update the reduced-dimensional spectral data, and the conditional probability distribution corresponding to the iteratively updated reduced-dimensional spectral data is iterated to obtain the target conditional probability distribution with the minimum KL divergence with the variational probability distribution;
[0074] The iterative dimension-reduced spectral data corresponding to the target conditional probability distribution is determined as spectral sampling data.
[0075] Specifically, in step S103, based on a preset energy functional function, while removing abnormal data in the reduced-dimensionality spectral data, a corresponding conditional probability distribution is constructed to calculate the variational probability distribution of the reduced-dimensionality spectral data by a sampling method, including:
[0076] The conditional probability distribution of the dimension-reduced spectral data is set to a normal distribution with a preset energy functional function;
[0077] Calculate the envelope of a preset energy functional function, and remove spectral data points outside the envelope in the dimension-reduced spectral data;
[0078] Taking the equal sampling number of spectral data points in each spectral data point group as a constraint and maximizing the normal distribution as the goal, sampling is performed in the reduced-dimensional spectral data, and the variational probability distribution of the reduced-dimensional spectral data is calculated.
[0079] In step S103, when sampling the reduced-dimensionality spectral data, it is necessary to model the reduced-dimensionality spectral data and solve the probability distribution p(z|x). However, due to the imbalance of the unbalanced spectral data itself, the probability distribution p(z|x) is difficult to calculate directly. Therefore, the variational EM algorithm is used during sampling to calculate the variational probability distribution q(z) of the reduced-dimensionality spectral data to infer the distribution of the probability distribution p(z|x). The purpose of the variational EM algorithm is to make q(z) and p(z|x) as similar as possible, that is, their KL (Kullback-Leibler) divergence is as small as possible.
[0080] Among them, the KL divergence calculation formula is specifically:
[0081] ;
[0082] in, is the KL divergence between the probability distribution p(z|x) and the variational probability distribution q(z); is the expected value under the variational probability distribution q(z).
[0083] Because p(z|x)=p(x|z)p(z) / p(x) (p(x|z) is the probability of extracting a spectral data point of the same category in the unbalanced spectral data x under the condition of extracting a spectral data point of any category in the dimension-reduced spectral data z, and p(x) is the probability of extracting a spectral data point of any category in the unbalanced spectral data x), and after clustering, it can be considered that , according to Jenson's inequality (Jenson's inequality, Jenson's inequality is a prior art and will not be described in detail here), the KL divergence calculation formula is optimized to obtain:
[0084] ;
[0085] in, This is the conditional probability distribution.
[0086] Therefore, when the conditional probability distribution With variational probability distribution When the KL divergence is the smallest, the divergence between the variational probability distribution q(z) and the probability distribution p(z|x) is the smallest, that is, the variational probability distribution q(z) and the probability distribution p(z|x) are closest.
[0087] In step S103, it is assumed that the conditional probability distribution p(z) of the reduced-dimensional spectrum data satisfies a normal distribution with a mean of m and a variance of S, and an energy functional function is set in the normal distribution. Specifically, the normal distribution is:
[0088] ; (1)
[0089] in, is a normal distribution (i.e., conditional probability distribution), is the variance of the spectral data point in the reduced spectral data, z is the spectral data point in the reduced spectral data, m is the mean of the spectral data point in the reduced spectral data, and the superscript T is the matrix transpose. is the natural constant e The power, where ,but is the preset energy functional function; K is the number of spectral data point groups.
[0090] The envelope of the preset energy functional function is calculated by the mean m and variance S , envelope The specific calculation formula is:
[0091] ;
[0092] in, For the envelope.
[0093] From the above formula, we can see that with envelope = 0 as the target, the center of the envelope is determined to be m, and in the reduced-dimensional spectral data, the radius in each dimension is the standard deviation in this dimension.
[0094] At the same time, it will be outside the envelope (i.e. satisfy ) are considered as abnormal data and are removed.
[0095] The sampling number of spectral data points in each spectral data point group is constrained to be equal. For the dimension-reduced spectral data after removing abnormal data, assume that K is the number of spectral data point groups, m i represents the mean of the ith spectral data point group, S i represents the variance of the ith spectral data point group. In the process of sampling the dimension-reduced spectral data, the number of spectral data points in each spectral data point group is constrained to be equal, that is, the number of spectral data points in the ith spectral data point group before sampling is n i , the number of spectral data points in each spectral data point group obtained after sampling is n.
[0096] The goal is to maximize the normal distribution, that is, to maximize the conditional probability distribution p(z) corresponding to the normal distribution, that is, to minimize the energy functional function, that is, to calculate the minimum value of the energy functional function .
[0097] In summary, the minimum value of the energy functional function Specifically:
[0098] ;
[0099] in, is the minimum value of the energy functional, The spectral data points grouped for the i-th spectral data point, is the mean of the spectral data points grouped in the i-th spectral data point, is the variance of the spectral data points grouped by the i-th spectral data point, It is a constrained symbol, indicating that the minimum value of the previous expression needs to meet the following specific conditions; The number of spectral data points grouped for each spectral data point obtained after sampling, The number of spectral data points to group for the ith spectral data point before sampling.
[0100] The conditional probability distribution p(z) corresponding to the minimum value of the energy functional function is calculated, which is the variational probability distribution of the reduced-dimensional spectral data.
[0101] Specifically, in step S103, the reduced-dimensionality spectral data is iteratively updated using the gradient descent method, and the conditional probability distribution corresponding to the iterated reduced-dimensionality spectral data is iterated to obtain a target conditional probability distribution with the minimum KL divergence with the variational probability distribution, including:
[0102] Adopting the gradient descent method, iteratively updating the dimension-reduced spectral data, recording the dimension-reduced spectral data after each iteration, and obtaining the dimension-reduced spectral data after multiple iterations;
[0103] Calculate and obtain multiple conditional probability distributions corresponding to the dimension-reduced spectral data after each iteration, and calculate the KL divergence between the multiple conditional probability distributions and the variational probability distribution to obtain multiple KL divergences;
[0104] Compare the size relationships between multiple KL divergences to extract the minimum value among multiple KL divergences;
[0105] The conditional probability distribution corresponding to the minimum value is determined as the target conditional probability distribution with the minimum KL divergence with the variational probability distribution.
[0106] In step S103, the reduced-dimensionality spectral data is iteratively updated using the gradient descent method so that the conditional probability distribution and the variational probability distribution corresponding to the iteratively updated reduced-dimensionality spectral data are as close as possible, that is, the KL divergence between the conditional probability distribution and the variational probability distribution is minimized.
[0107] Among them, the iterative formula for iteratively updating the dimension-reduced spectral data is specifically:
[0108] ;
[0109] in, is the dimension-reduced spectral data after the t+1th iteration; is the dimension-reduced spectral data after the tth iteration; t≥0, t is an integer, when t is equal to 0, is the initial dimension-reduced spectral data; is the step length, is the gradient operator, is the energy functional function corresponding to the dimension-reduced spectral data after the tth iteration.
[0110] Using the normal distribution of the above steps (i.e., using formula (1)), the conditional probability distribution corresponding to the dimension-reduced spectral data after each iteration is calculated to obtain multiple conditional probability distributions. The KL divergence between the multiple conditional probability distributions and the variational probability distribution is calculated respectively. The calculation formula of the KL divergence is as follows:
[0111] ;
[0112] in, is the conditional probability distribution With variational probability distribution The KL divergence of .
[0113] Compare the size relationships between multiple KL divergences, extract the minimum value among the multiple KL divergences, and determine the conditional probability distribution corresponding to the minimum value as the target conditional probability distribution with the minimum KL divergence with the variational probability distribution.
[0114] The iterative reduced-dimensionality spectral data corresponding to the target conditional probability distribution is obtained, and the iterative reduced-dimensionality spectral data is determined as the spectral sampling data.
[0115] Specifically, in step S104, the spectrum sampling data is input into a preset spectrum detection model, and after calculation by the spectrum detection model, the category result of the spectrum sampling data (i.e., all categories included in the spectrum data points in the spectrum sampling data) is obtained, and the category result of the spectrum sampling data is determined as the category result of the unbalanced spectrum data to represent all categories included in the spectrum data points in the unbalanced spectrum data. The preset spectrum detection model is specifically a random forest model, which is a prior art and will not be described in detail herein.
[0116] For example, using the unbalanced spectral data detection method of the present application, Raman spectroscopy is performed on some urine samples from a hospital for routine urine tests to predict the negative or positive urine protein or urine sugar. First, 3060 pieces of spectral data are collected, and the spectral sampling data are obtained through the variational EM algorithm through the unbalanced spectral data detection method, and the spectral sampling data are input into the random forest model for detection, achieving an accuracy of 88% in urine protein detection and 91% in urine sugar detection. Compared with the unbalanced spectral data detection method without using the present application, the accuracy of the result obtained by directly inputting the spectral data into the random forest model is less than the detection result using the unbalanced spectral data detection method of the present application. The accuracy of the detection results is shown in the following table:
[0117] Detection indicators Urine protein Urine sugar diagnosis Occult blood leukocyte Direct Input 0.842 0.829 0.820 0.761 0.744 Unbalanced spectral data detection method of the present application 0.881 0.909 0.896 0.825 0.826
[0118] It can be seen from the above table that the unbalanced spectral data detection method of the present application can accurately detect the category result of each spectral data point in the spectral data.
[0119] As can be seen from the above, the unbalanced spectral data detection method obtains unbalanced spectral data, uses the K-means clustering algorithm to reduce the dimension of the unbalanced spectral data, obtains reduced-dimensional spectral data, samples the reduced-dimensional spectral data through the variational EM algorithm, obtains spectral sampling data, inputs the spectral sampling data into a preset spectral detection model, and calculates the category result of the unbalanced spectral data; thus, by inputting the spectral sampling data obtained based on the K-means clustering algorithm and the variational EM algorithm into the preset spectral detection model, the category result of the unbalanced spectral data is calculated, which solves the problem that the existing spectral data detection method cannot effectively select the most representative spectral data points from the unbalanced spectral data and is difficult to accurately analyze the unbalanced spectral data, and can accurately and quickly classify and identify the unbalanced spectral data, thereby improving the detection efficiency of the unbalanced spectral data.
[0120] refer to Figure 2The present application provides an unbalanced spectral data detection device for detecting unbalanced spectral data, comprising:
[0121] An acquisition module 1 is used to acquire unbalanced spectral data;
[0122] Dimensionality reduction module 2, used for reducing the dimension of unbalanced spectral data using K-means clustering algorithm to obtain dimension-reduced spectral data;
[0123] The sampling module 3 is used to sample the reduced-dimensional spectral data by using a variational EM algorithm to obtain spectral sampling data;
[0124] The calculation module 4 is used to input the spectrum sampling data into a preset spectrum detection model to calculate the classification result of the unbalanced spectrum data.
[0125] The unbalanced spectral data detection device inputs spectral sampling data obtained based on the K-means clustering algorithm and the variational EM algorithm into a preset spectral detection model to calculate the classification result of the unbalanced spectral data, thereby solving the problem that the existing spectral data detection method cannot effectively select the most representative spectral data points from the unbalanced spectral data and is difficult to accurately analyze the unbalanced spectral data. The unbalanced spectral data can be accurately and quickly classified and identified, thereby improving the detection efficiency of the unbalanced spectral data.
[0126] Specifically, when the acquisition module 1 is executed, unbalanced spectral data is acquired. The unbalanced spectral data is composed of spectral data points. The unbalanced spectral data refers to spectral data in which the number of spectral data points of a certain category is far greater than that of spectral data of other categories, or spectral data points of certain categories are extremely sparsely distributed in the feature space.
[0127] Specifically, after reducing the dimension of the unbalanced spectral data, the dimension reduction module 2 uses the K-means clustering algorithm to cluster the unbalanced spectral data after dimension reduction, and when obtaining the dimension reduced spectral data, executes:
[0128] Performing dimension reduction on the unbalanced spectral data to obtain unbalanced spectral data after dimension reduction;
[0129] Selecting a plurality of spectral data points from the unbalanced spectral data after dimension reduction as a plurality of initial clustering centers;
[0130] Based on the initial cluster centers, all spectral data points are divided into multiple spectral data point groups through the K-means clustering algorithm to obtain dimension-reduced spectral data.
[0131] When the dimension reduction module 2 is executed, redundant data in the unbalanced spectral data is removed by existing dimensionality reduction methods such as principal component analysis (PCA) method or linear discriminant analysis method (LDA), and the number of spectral data points with original features is reduced to a certain number of spectral data points with the most important features, thereby obtaining unbalanced spectral data after dimensionality reduction. Among them, existing dimensionality reduction methods such as principal component analysis method or linear discriminant analysis method are prior arts and will not be described in detail here.
[0132] The K-means clustering algorithm is used to reduce the dimension of the unbalanced spectral data after dimensionality reduction. Multiple spectral data points are randomly selected from the unbalanced spectral data after dimensionality reduction as multiple initial clustering centers, and other spectral data points are assigned to the initial clustering center with the smallest distance therefrom to form a spectral data point grouping based on the initial clustering center (that is, the clustering center of the spectral data point grouping is the initial clustering center). After each grouping is completed, the clustering center of each spectral data point grouping is re-divided (the division method is to select the spectral data point closest to the grouping center in the spectral data point grouping as the new clustering center), and based on the divided clustering center, the other spectral data points in each spectral data point grouping are reallocated (the allocation method is to assign other spectral data points to the clustering center with the smallest distance therefrom), until all spectral data points in the reallocated spectral data point grouping are no longer reallocated, forming multiple spectral data point groups, and obtaining reduced-dimensional spectral data.
[0133] Specifically, when the dimension reduction module 2 divides all spectral data points into multiple spectral data point groups based on the initial cluster center and obtains the dimension-reduced spectral data by the K-means clustering algorithm, the following is executed:
[0134] Step A1, calculating the distances between other spectral data points in the spectral data points except the initial cluster center and the multiple initial cluster centers;
[0135] Step A2, assigning other spectral data points to the nearest initial cluster center to form spectral data point groups;
[0136] Step A3, recalculating the cluster center in the spectral data point grouping, and reallocating all other spectral data points outside the cluster center according to the distance between the cluster center and all other spectral data points outside the cluster center;
[0137] Step A4, repeatedly executing step A3 until the cluster center no longer changes, and then grouping the multiple spectrum data points obtained by division into reduced-dimensional spectrum data.
[0138] In step S102, the distances between other spectral data points and the initial clustering center in the unbalanced spectral data after dimensionality reduction are calculated, and the initial clustering center is used as the central node for grouping. The initial clustering center with the smallest distance to other spectral data points is extracted from all the initial clustering centers, and the other spectral data points are assigned to the group where the initial clustering center with the smallest distance to the other spectral data points is located. After each assignment is completed, it is necessary to recalculate the spectral data point closest to the center point position as the clustering center of the spectral data point group according to the positions of each spectral data point in each spectral data point group, recalculate the distances between other spectral data points except the clustering center and multiple clustering centers, and assign the other spectral data points to the group where the clustering center with the smallest distance to the other spectral data points is located, and repeat the steps of calculating the clustering center and assigning the spectral data points until the clustering center no longer changes, or the spectral data points of each spectral data point group no longer change, and the spectral data point group finally calculated is determined as the reduced dimensionality spectral data.
[0139] Specifically, when the sampling module 3 samples the reduced-dimensional spectral data through the variational EM algorithm to obtain spectral sampling data, it executes:
[0140] Based on the preset energy functional function, while eliminating abnormal data in the reduced-dimensionality spectral data, the corresponding conditional probability distribution is constructed to calculate the variational probability distribution of the reduced-dimensionality spectral data through the sampling method;
[0141] The gradient descent method is used to iteratively update the reduced-dimensional spectral data, and the conditional probability distribution corresponding to the iteratively updated reduced-dimensional spectral data is iterated to obtain the target conditional probability distribution with the minimum KL divergence with the variational probability distribution;
[0142] The iterative dimension-reduced spectral data corresponding to the target conditional probability distribution is determined as spectral sampling data.
[0143] Specifically, the sampling module 3 constructs a corresponding conditional probability distribution based on a preset energy functional function while eliminating abnormal data in the reduced-dimensionality spectral data, so as to calculate the variational probability distribution of the reduced-dimensionality spectral data by the sampling method, and executes:
[0144] The conditional probability distribution of the dimension-reduced spectral data is set to a normal distribution with a preset energy functional function;
[0145] Calculate the envelope of a preset energy functional function, and remove spectral data points outside the envelope in the dimension-reduced spectral data;
[0146] Taking the equal sampling number of spectral data points in each spectral data point group as a constraint and maximizing the normal distribution as the goal, sampling is performed in the reduced-dimensional spectral data, and the variational probability distribution of the reduced-dimensional spectral data is calculated.
[0147] When sampling module 3 is executed, when sampling the reduced-dimensionality spectral data, it is necessary to model the reduced-dimensionality spectral data and solve the probability distribution p(z|x). However, due to the imbalance of the unbalanced spectral data itself, the probability distribution p(z|x) is difficult to calculate directly. Therefore, the variational EM algorithm is used during sampling to calculate the variational probability distribution q(z) of the reduced-dimensionality spectral data to infer the distribution of the probability distribution p(z|x). The purpose of the variational EM algorithm is to make q(z) and p(z|x) as similar as possible, that is, their KL (Kullback-Leibler) divergence is as small as possible.
[0148] Among them, the KL divergence calculation formula is specifically:
[0149] ;
[0150] in, is the KL divergence between the probability distribution p(z|x) and the variational probability distribution q(z); is the expected value under the variational probability distribution q(z).
[0151] Because p(z|x)=p(x|z)p(z) / p(x) (p(x|z) is the probability of extracting a spectral data point of the same category in the unbalanced spectral data x under the condition of extracting a spectral data point of any category in the dimension-reduced spectral data z, and p(x) is the probability of extracting a spectral data point of any category in the unbalanced spectral data x), and after clustering, it can be considered that , according to Jenson's inequality (Jenson's inequality, Jenson's inequality is a prior art and will not be described in detail here), the KL divergence calculation formula is optimized to obtain:
[0152] ;
[0153] in, This is the conditional probability distribution.
[0154] Therefore, when the conditional probability distribution With variational probability distribution When the KL divergence is the smallest, the divergence between the variational probability distribution q(z) and the probability distribution p(z|x) is the smallest, that is, the variational probability distribution q(z) and the probability distribution p(z|x) are closest.
[0155] When the sampling module 3 is executed, it is assumed that the conditional probability distribution p(z) of the dimension-reduced spectral data satisfies a normal distribution with a mean of m and a variance of S, and an energy functional function is set in the normal distribution. The normal distribution is specifically:
[0156] ; (1)
[0157] in, is a normal distribution (i.e., conditional probability distribution), is the variance of the spectral data point in the reduced spectral data, z is the spectral data point in the reduced spectral data, m is the mean of the spectral data point in the reduced spectral data, and the superscript T is the matrix transpose. is the natural constant e The power, where ,but is the preset energy functional function; K is the number of spectral data point groups.
[0158] The envelope of the preset energy functional function is calculated by the mean m and variance S , envelope The specific calculation formula is:
[0159] ;
[0160] in, For the envelope.
[0161] From the above formula, we can see that with envelope = 0 as the target, the center of the envelope is determined to be m, and in the reduced-dimensional spectral data, the radius in each dimension is the standard deviation in this dimension.
[0162] At the same time, it will be outside the envelope (i.e. satisfy ) are considered as abnormal data and are removed.
[0163] The sampling number of spectral data points in each spectral data point group is constrained to be equal. For the dimension-reduced spectral data after removing abnormal data, assume that K is the number of spectral data point groups, m i represents the mean of the ith spectral data point group, S i represents the variance of the ith spectral data point group. In the process of sampling the dimension-reduced spectral data, the number of spectral data points in each spectral data point group is constrained to be equal, that is, the number of spectral data points in the ith spectral data point group before sampling is n i , the number of spectral data points in each spectral data point group obtained after sampling is n.
[0164] The goal is to maximize the normal distribution, that is, to maximize the conditional probability distribution p(z) corresponding to the normal distribution, that is, to minimize the energy functional function, that is, to calculate the minimum value of the energy functional function .
[0165] In summary, the minimum value of the energy functional function Specifically:
[0166] ;
[0167] in, is the minimum value of the energy functional, The spectral data points grouped for the i-th spectral data point, is the mean of the spectral data points grouped in the i-th spectral data point, is the variance of the spectral data points grouped by the i-th spectral data point, It is a constrained symbol, indicating that the minimum value of the previous expression needs to meet the following specific conditions; The number of spectral data points grouped for each spectral data point obtained after sampling, The number of spectral data points to group for the ith spectral data point before sampling.
[0168] The conditional probability distribution p(z) corresponding to the minimum value of the energy functional function is calculated, which is the variational probability distribution of the reduced-dimensional spectral data.
[0169] Specifically, the sampling module 3 iteratively updates the reduced-dimensionality spectral data using the gradient descent method, and iterates the conditional probability distribution corresponding to the iteratively updated reduced-dimensionality spectral data to obtain the target conditional probability distribution with the minimum KL divergence with the variational probability distribution, and executes:
[0170] Adopting the gradient descent method, iteratively updating the dimension-reduced spectral data, recording the dimension-reduced spectral data after each iteration, and obtaining the dimension-reduced spectral data after multiple iterations;
[0171] Calculate and obtain multiple conditional probability distributions corresponding to the dimension-reduced spectral data after each iteration, and calculate the KL divergence between the multiple conditional probability distributions and the variational probability distribution to obtain multiple KL divergences;
[0172] Compare the size relationships between multiple KL divergences to extract the minimum value among multiple KL divergences;
[0173] The conditional probability distribution corresponding to the minimum value is determined as the target conditional probability distribution with the minimum KL divergence with the variational probability distribution.
[0174] When the sampling module 3 is executed, the gradient descent method is used to iteratively update the reduced-dimensionality spectral data so that the conditional probability distribution and the variational probability distribution corresponding to the iterative reduced-dimensionality spectral data are as close as possible, that is, the KL divergence between the conditional probability distribution and the variational probability distribution is minimized.
[0175] Among them, the iterative formula for iteratively updating the dimension-reduced spectral data is specifically:
[0176] ;
[0177] in, is the dimension-reduced spectral data after the t+1th iteration; is the dimension-reduced spectral data after the tth iteration; t≥0, t is an integer, when t is equal to 0, is the initial dimension-reduced spectral data; is the step length, is the gradient operator, is the energy functional function corresponding to the dimension-reduced spectrum data after the tth iteration. Step size The number of iterations can be set as needed, and the step size It is generally set to 0.01, and the number of iterations is generally set to 50 times.
[0178] Using the normal distribution obtained previously (i.e., using formula (1)), the conditional probability distribution corresponding to the dimension-reduced spectral data after each iteration is calculated to obtain multiple conditional probability distributions. The KL divergence between the multiple conditional probability distributions and the variational probability distribution is calculated respectively. The calculation formula of the KL divergence is as follows: ;
[0179] in, is the conditional probability distribution With variational probability distribution The KL divergence of .
[0180] Compare the size relationships between multiple KL divergences, extract the minimum value among the multiple KL divergences, and determine the conditional probability distribution corresponding to the minimum value as the target conditional probability distribution with the minimum KL divergence with the variational probability distribution.
[0181] The iterative reduced-dimensionality spectral data corresponding to the target conditional probability distribution is obtained, and the iterative reduced-dimensionality spectral data is determined as the spectral sampling data.
[0182] Specifically, when the calculation module 4 is executed, the spectrum sampling data is input into the preset spectrum detection model, and after the spectrum detection model is calculated, the category result of the spectrum sampling data (i.e., all categories included in the spectrum data points in the spectrum sampling data) is obtained, and the category result of the spectrum sampling data is determined as the category result of the unbalanced spectrum data to represent all categories included in the spectrum data points in the unbalanced spectrum data. Among them, the preset spectrum detection model is specifically a random forest model, which is a prior art and will not be described in detail here.
[0183] For example, the unbalanced spectral data detection device of the present application is used to perform Raman spectroscopy on some urine samples from a hospital for routine urine tests to predict the negative or positive results of urine protein or urine sugar. First, 3060 pieces of spectral data are collected, and the spectral sampling data are obtained through the unbalanced spectral data detection device through the variational EM algorithm, and the spectral sampling data are input into the random forest model for detection, achieving an accuracy of 88% in urine protein detection and 91% in urine sugar detection. Compared with the unbalanced spectral data detection device of the present application, the accuracy of the result obtained by directly inputting the spectral data into the random forest model is less than the detection result using the unbalanced spectral data detection device of the present application. The accuracy of the detection results is shown in the following table:
[0184] Detection indicators Urine protein Urine sugar diagnosis Occult blood leukocyte Direct Input 0.842 0.829 0.820 0.761 0.744 Unbalanced spectral data detection device of the present application 0.881 0.909 0.896 0.825 0.826
[0185] It can be seen from the above table that the unbalanced spectral data detection device of the present application can accurately detect the category result of each spectral data point in the spectral data.
[0186] As can be seen from the above, the unbalanced spectral data detection device obtains unbalanced spectral data, uses the K-means clustering algorithm to reduce the dimension of the unbalanced spectral data, obtains reduced-dimensional spectral data, samples the reduced-dimensional spectral data through the variational EM algorithm, obtains spectral sampling data, inputs the spectral sampling data into a preset spectral detection model, and calculates the category result of the unbalanced spectral data; thus, by inputting the spectral sampling data obtained based on the K-means clustering algorithm and the variational EM algorithm into the preset spectral detection model, the category result of the unbalanced spectral data is calculated, which solves the problem that the existing spectral data detection method cannot effectively select the most representative spectral data points from the unbalanced spectral data and is difficult to accurately analyze the unbalanced spectral data, and can accurately and quickly classify and identify the unbalanced spectral data, thereby improving the detection efficiency of the unbalanced spectral data.
[0187] Please refer to Figure 3 , Figure 3This is a structural schematic diagram of an electronic device provided in an embodiment of the present application. The present application provides an electronic device, including: a processor 301 and a memory 302. The processor 301 and the memory 302 are interconnected and communicate with each other through a communication bus 303 and / or other forms of connection mechanisms (not shown). The memory 302 stores a computer program executable by the processor 301. When the electronic device is running, the processor 301 executes the computer program to execute the unbalanced spectral data detection method in any optional implementation of the above embodiments to achieve the following functions: obtaining unbalanced spectral data, using a K-means clustering algorithm to reduce the dimension of the unbalanced spectral data to obtain reduced-dimensional spectral data, sampling the reduced-dimensional spectral data through a variational EM algorithm to obtain spectral sampling data, inputting the spectral sampling data into a preset spectral detection model, and calculating the category result of the unbalanced spectral data.
[0188] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the unbalanced spectral data detection method in any optional implementation of the above embodiment is executed to achieve the following functions: obtaining unbalanced spectral data, using a K-means clustering algorithm to reduce the dimension of the unbalanced spectral data to obtain reduced-dimensional spectral data, sampling the reduced-dimensional spectral data through a variational EM algorithm to obtain spectral sampling data, inputting the spectral sampling data into a preset spectral detection model, and calculating to obtain a category result of the unbalanced spectral data. Among them, the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable red-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.
[0189] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.
[0190] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0191] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
[0192] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.
[0193] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. A method for detecting unbalanced spectral data, for detecting unbalanced spectral data, characterized in that: Includes steps: Acquiring unbalanced spectral data; After reducing the dimension of the unbalanced spectral data, clustering the unbalanced spectral data after the dimension reduction is performed using a K-means clustering algorithm to obtain dimension-reduced spectral data; Sampling the reduced-dimensional spectral data by a variational EM algorithm to obtain spectral sampling data; Inputting the spectrum sampling data into a preset spectrum detection model to calculate and obtain the classification result of the unbalanced spectrum data; The reduced-dimensional spectral data is sampled by a variational EM algorithm to obtain spectral sampling data, including: Based on a preset energy functional function, while eliminating abnormal data in the reduced-dimensionality spectral data, a corresponding conditional probability distribution is constructed to calculate the variational probability distribution of the reduced-dimensionality spectral data through a sampling method; Adopting the gradient descent method, iteratively updating the reduced-dimensionality spectral data, and iterating the conditional probability distribution corresponding to the iteratively updated reduced-dimensionality spectral data, to obtain a target conditional probability distribution with the minimum KL divergence with the variational probability distribution; The iterated dimension-reduced spectral data corresponding to the target conditional probability distribution is determined as the spectral sampling data.
2. The unbalanced spectral data detection method according to claim 1, characterized in that: The unbalanced spectral data includes spectral data points; after reducing the dimension of the unbalanced spectral data, a K-means clustering algorithm is used to cluster the unbalanced spectral data after dimension reduction to obtain dimension-reduced spectral data, including: Performing dimension reduction on the unbalanced spectral data to obtain unbalanced spectral data after dimension reduction; Selecting a plurality of the spectral data points from the unbalanced spectral data after dimension reduction as a plurality of initial clustering centers; Based on the initial cluster centers, all the spectral data points are divided into a plurality of spectral data point groups by using the K-means clustering algorithm to obtain dimension-reduced spectral data.
3. The unbalanced spectral data detection method according to claim 2, characterized in that: Based on the initial cluster center, all the spectral data points are divided into a plurality of spectral data point groups by the K-means clustering algorithm to obtain dimension-reduced spectral data, including: Step A1, calculating the distances between other spectral data points among the spectral data points except the initial cluster center and the multiple initial cluster centers; Step A2, allocating the other spectral data points to the initial cluster center that is closest to the other spectral data points to form a spectral data point group; Step A3, recalculating the cluster center in the spectral data point grouping, and reallocating all the other spectral data points outside the cluster center according to the distance between the cluster center and all the other spectral data points outside the cluster center; Step A4, repeatedly executing step A3 until the cluster center no longer changes, and then grouping the multiple spectral data points obtained by division into reduced-dimensional spectral data.
4. The unbalanced spectral data detection method according to claim 1, characterized in that: Based on a preset energy functional function, while eliminating abnormal data in the reduced-dimensionality spectral data, a corresponding conditional probability distribution is constructed to calculate the variational probability distribution of the reduced-dimensionality spectral data through a sampling method, including: Setting the conditional probability distribution of the reduced-dimensional spectrum data to a normal distribution having the preset energy functional function; Calculating the envelope of the preset energy functional function, and removing the spectral data points outside the envelope in the dimension-reduced spectral data; Taking the sampling number of spectral data points of each spectral data point group as a constraint and maximizing the normal distribution as a goal, sampling is performed in the reduced-dimensional spectral data to calculate the variational probability distribution of the reduced-dimensional spectral data.
5. The unbalanced spectral data detection method according to claim 1, characterized in that: The reduced-dimensional spectral data is iteratively updated using a gradient descent method, and the conditional probability distribution corresponding to the iteratively updated reduced-dimensional spectral data is iteratively updated to obtain a target conditional probability distribution with the minimum KL divergence with the variational probability distribution, including: Adopting a gradient descent method to iteratively update the dimension-reduced spectral data, recording the dimension-reduced spectral data after each iteration, and obtaining the dimension-reduced spectral data after multiple iterations; Calculating a plurality of conditional probability distributions corresponding to the iterative dimension-reduced spectral data, and calculating KL divergences between the plurality of conditional probability distributions and the variational probability distribution to obtain a plurality of KL divergences; Comparing the magnitude relationships between the multiple KL divergences to extract the minimum value among the multiple KL divergences; The conditional probability distribution corresponding to the minimum value is determined as the target conditional probability distribution with the minimum KL divergence with the variational probability distribution.
6. The unbalanced spectral data detection method according to claim 1, characterized in that: The preset spectral detection model is a random forest model.
7. An unbalanced spectrum data detection device, used for detecting unbalanced spectrum data, characterized in that: include: An acquisition module, used for acquiring unbalanced spectral data; A dimension reduction module, used for performing dimension reduction on the unbalanced spectral data, and clustering the unbalanced spectral data after dimension reduction using a K-means clustering algorithm to obtain dimension reduced spectral data; A sampling module, used for sampling the dimension-reduced spectral data by using a variational EM algorithm to obtain spectral sampling data; A calculation module, used for inputting the spectrum sampling data into a preset spectrum detection model to calculate and obtain the classification result of the unbalanced spectrum data; The sampling module is used to sample the dimension-reduced spectral data by using a variational EM algorithm to obtain spectral sampling data, including: Based on a preset energy functional function, while eliminating abnormal data in the reduced-dimensionality spectral data, a corresponding conditional probability distribution is constructed to calculate the variational probability distribution of the reduced-dimensionality spectral data through a sampling method; Adopting the gradient descent method, iteratively updating the reduced-dimensionality spectral data, and iterating the conditional probability distribution corresponding to the iteratively updated reduced-dimensionality spectral data, to obtain a target conditional probability distribution with the minimum KL divergence with the variational probability distribution; The iterated dimension-reduced spectral data corresponding to the target conditional probability distribution is determined as the spectral sampling data.
8. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program executable by the processor, and when the processor executes the computer program, the steps in the method for detecting unbalanced spectral data according to any one of claims 1 to 6 are executed.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for detecting imbalanced spectral data according to any one of claims 1 to 6 are executed.
