Object classification method and device based on deep reinforcement learning under complex noise conditions

Abstract

The present disclosure relates to the technical field of computers, comprising an object classification method and device based on deep reinforcement learning under complex noise conditions. By obtaining data to be classified; based on the noise classification model obtained by deep reinforcement learning, the noise classification of the data to be classified is performed to obtain the noise classification corresponding to the data to be classified; the noise reduction algorithm corresponding to the noise classification is called to perform noise reduction processing on the data to be classified to obtain the data after noise reduction; based on the object classification model obtained by deep reinforcement learning, the data after noise reduction is subjected to object classification to obtain the object classification result; the problem of poor noise removal effect of the traditional denoising algorithm can be solved; in the case of many noise types, the denoising algorithm can be adaptively switched for noise reduction processing, thereby improving the noise reduction effect of the data to be classified under complex noise conditions.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of computers, and particularly relates to an object classification method and device based on deep reinforcement learning under complex noise conditions. BACKGROUND

[0002] Deep reinforcement learning has a wide range of applications in fields such as games and robot control, and is also widely used in object classification scenarios, such as device fault detection and diagnosis scenarios. In object classification based on deep reinforcement learning, in order to improve the performance of object classification, the signal to be classified needs to be denoised.

[0003] Traditional denoising methods include using a denoising algorithm to denoise the collected signal, effectively removing noise and interference signals, and then performing object classification based on deep reinforcement learning. Common denoising algorithms include, but are not limited to, wavelet denoising and deep learning denoising, such as analyzing the signal through three steps of parameter offline setting based on wavelet threshold denoising, Hilbert transform method, and detection method based on principal component analysis, achieving fault detection of a turbine; or wavelet threshold denoising optimized based on a genetic algorithm, achieving rotating machinery fault diagnosis by combining variational mode decomposition; or using a multi-layer convolutional neural network to process different types of noise data, improving the purity of vibration signals. Or, through a denoising fault perception wavelet network, wavelet convolution and dynamic hard threshold are fused to eliminate noise interference, achieving fault diagnosis under high noise conditions. Or, a graph wavelet denoising convolutional neural network based on a discrete graph wavelet frame, through multi-scale feature extraction of graph structure data, achieving signal denoising and fault diagnosis.

[0004] However, the above denoising algorithms mostly only filter out single noise, while in the complex environment of actual production, the noise in the signal to be classified often exists in the form of mixed noise, and the noise removal effect of the traditional denoising algorithm is not good. SUMMARY

[0005] Therefore, the present disclosure proposes an object classification method and device based on deep reinforcement learning under complex noise conditions, which can adaptively switch denoising algorithms for denoising processing in the case of multiple noise types, and improve the denoising effect of data to be classified under complex noise conditions.

[0006] According to an aspect of the present disclosure, an object classification method based on deep reinforcement learning under complex noise conditions is provided, the method comprising:

[0007] acquiring data to be classified;

[0008] classifying the data to be classified based on a noise classification model obtained by deep reinforcement learning, to obtain a noise classification corresponding to the data to be classified.

[0009] invoke a denoising algorithm corresponding to the noise classification to perform denoising processing on the to-be-classified data, to obtain denoised data; different noise classifications correspond to different denoising algorithms.

[0010] perform object classification on the denoised data based on an object classification model obtained through deep reinforcement learning, to obtain an object classification result.

[0011] In a possible implementation, the noise classification model is obtained through deep reinforcement learning based on unlabeled sample data, and the sample data used to train the noise classification model is sample data with noise; and / or, the object classification model is obtained through deep reinforcement learning based on unlabeled sample data, and the sample data used to train the object classification model is denoised sample data.

[0012] The deep reinforcement learning based on the unlabeled sample data includes:

[0013] determine the number A of class cluster centers based on the number of classifications, and initialize the class cluster centers as A sample data with the farthest sample distances; A is an integer greater than 1.

[0014] For each sample data, determine the sample distance between the sample data and each class cluster center, and determine the index of the class cluster center with the smallest sample distance to the sample data as the class cluster label of the sample data.

[0015] construct a data environment based on each sample data and the class cluster label corresponding to the sample data.

[0016] obtain a trained reinforcement learning model through deep reinforcement learning between an agent and the data environment.

[0017] After completion of the current round of training, update the class cluster centers based on the mean of the sample data corresponding to each class cluster center, trigger the step of determining the sample distance between each sample data and each class cluster center, and determining the index of the class cluster center with the smallest sample distance to the sample data as the class cluster label of the sample data, and the subsequent steps, until the training round reaches a preset requirement.

[0018] In a possible implementation, the deep reinforcement learning between the agent and the data environment to obtain the trained reinforcement learning model includes:

[0019] returning the i-th sample data to the agent in sequence through the data environment; the agent comprises a reinforcement learning model, the reinforcement learning model comprises a deep Q network and a target network, and network parameters of the deep Q network and network parameters of the target network are initialized to be the same; i is a positive integer sequentially taking values from 1 to n, and n is a total number of the sample data;

[0020] determining an i-th classification action corresponding to the i-th sample data through the deep Q network; wherein the i-th classification action indicates one of the respective class cluster labels;

[0021] returning the i-th classification action to the data environment through the agent, so that the data environment determines an i-th reward of the i-th classification action based on a comparison result between the i-th classification action and a class cluster label corresponding to the i-th sample data, and returns the i-th reward and an (i+1)-th sample data to the agent; in a case where the comparison result is the same, the reward is in a case where the comparison result is different, the reward is -1; wherein, denotes a sample distance between the i-th sample data and a corresponding class cluster center;

[0022] storing the i-th experience data in an experience pool in the agent, and updating i=i+1 to trigger execution of the step of returning the i-th sample data to the agent in sequence through the data environment and subsequent steps; the i-th experience data comprises the i-th sample data, the i-th classification action, the i-th reward and the (i+1)-th sample data;

[0023] after n sample data are traversed, randomly extracting K groups of experience data from the experience pool; the K is a positive integer;

[0024] inputting each group of experience data into the deep Q network and the target network respectively, to update the deep Q network based on a difference between a first network result of the deep Q network and a second network result of the target network;

[0025] after each group of experience data is traversed, synchronizing network parameters of the deep Q network to the target network, until each group of experience data is traversed, to obtain network parameters of the deep Q network obtained through this round of training, and the W is a positive integer.

[0026] In a possible implementation, in a case where the number of classifications corresponding to the noise classification model and / or the object classification model is less than or equal to a preset threshold, the deep reinforcement learning is obtained based on the unlabeled sample data.

[0027] In one possible implementation, the noise classification model is obtained through deep reinforcement learning based on labeled sample data, and the labeled sample data includes sample data for a first noise classification and sample data for a second noise classification, wherein the number of sample data for the first noise classification is less than the number of sample data for the second noise classification; accordingly,

[0028] In deep reinforcement learning based on labeled sample data, the reward for interaction between the data environment and the agent is determined by the following formula:

[0029]

[0030] Among them, s t This represents the current sample data returned by the data environment to the agent, a. t Indicates based on the current sample data s t The classification action performed, R(s) t ,a t ) indicates based on the current sample data s t Rewards for performing categorization actions, l t Indicates the current state s t The corresponding noise classification label, D minority D represents the dataset consisting of sample data for the first noise classification. majority ρ represents the dataset consisting of sample data for the second noise category, and ρ represents the ratio of the number of sample data for the first noise category to the number of sample data for the second noise category.

[0031] In one possible implementation, the deep reinforcement learning process of the object classification model includes:

[0032] Obtain a training set, which includes noisy sample data;

[0033] Based on the noise classification model, the noisy sample data is classified to obtain the noise classification of each noisy sample data.

[0034] The noise reduction algorithm corresponding to the noise classification is called to perform noise reduction processing on the noisy sample data to obtain the noise-reduced sample data;

[0035] The object classification model is obtained by performing deep reinforcement learning on the denoised sample data.

[0036] In one possible implementation, the step of calling the denoising algorithm corresponding to the noise classification to denoise the data to be classified, and obtaining the denoised data, includes:

[0037] Obtain the correspondence between noise classification and the optimal wavelet threshold denoising algorithm;

[0038] According to the optimal wavelet threshold denoising algorithm corresponding to the noise classification, wavelet threshold denoising is performed on the to-be-classified data, to obtain the denoised data.

[0039] In a possible implementation, the obtaining of the correspondence between the noise classification and the optimal wavelet threshold denoising algorithm comprises:

[0040] For each noise classification, different wavelet bases are respectively used to perform wavelet threshold denoising on the same noise data belonging to the noise classification, and a wavelet base with the optimal denoising effect is determined from denoising results corresponding to different wavelet bases, to obtain the optimal wavelet threshold denoising algorithm corresponding to the noise classification.

[0041] In a possible implementation, the object classification model is used for fault diagnosis of a device, and accordingly, the to-be-classified signal includes a device signal with noise, and the object classification includes fault signal classification and normal signal classification.

[0042] According to another aspect of the present disclosure, there is provided an object classification apparatus based on deep reinforcement learning under complex noise conditions, comprising a processor and a memory for storing processor-executable instructions, wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0043] According to another aspect of the present disclosure, there is provided a non-volatile computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions are executed by a processor to implement the above method.

[0044] According to another aspect of the present disclosure, there is provided a computer program product comprising computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device executes the above method.

[0045] The method comprises the following steps: obtaining to-be-classified data; performing noise classification on the to-be-classified data based on a noise classification model obtained through deep reinforcement learning, to obtain a noise classification corresponding to the to-be-classified data; calling a noise reduction algorithm corresponding to the noise classification to perform noise reduction processing on the to-be-classified data, to obtain noise-reduced data; and performing object classification on the noise-reduced data based on an object classification model obtained through deep reinforcement learning, to obtain an object classification result. The method can solve the problem that the noise removal effect of a traditional noise reduction algorithm is poor. Before the to-be-classified data is subjected to object classification, the noise classification of the to-be-classified data is determined first, and then a noise reduction algorithm corresponding to the noise classification is called to perform noise reduction processing. Therefore, in the case of a large number of noise types, the noise reduction algorithm can be adaptively switched to perform noise reduction processing, and thus the noise reduction effect of the to-be-classified data in a complex noise scenario is improved.

[0046] Meanwhile, the noise classification model obtained through deep reinforcement learning is used to perform noise classification on the to-be-classified data, which can avoid the problem that the performance of the noise classification model is poor due to unbalanced noise samples. The object classification model obtained through deep reinforcement learning is used to perform object classification on the noise-reduced data, which can avoid the problem that the performance of the object classification model is poor due to unbalanced object classification samples. Therefore, the accuracy of noise classification and object classification in a sample imbalance scenario is improved.

[0047] Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0048] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the present disclosure and serve to explain the principles of the present disclosure.

[0049] Figure 1 A schematic diagram showing the principle of a Markov decision framework according to an embodiment of the present disclosure;

[0050] Figure 2 A flowchart showing a method of object classification based on deep reinforcement learning in a complex noise condition according to an embodiment of the present disclosure;

[0051] Figure 3 A schematic diagram showing a training process of an object classification model according to an embodiment of the present disclosure;

[0052] Figure 4 A schematic diagram showing augmented sample data according to an embodiment of the present disclosure;

[0053] Figure 5 A schematic diagram showing a change curve of a total reward obtained by an agent in a training process according to an embodiment of the present disclosure;

[0054] Figure 6Fig. 1 shows a plot of accuracy during training according to an embodiment of the present disclosure;

[0055] Figure 7 Fig. 2 shows a block diagram of a deep reinforcement learning based object classification apparatus under complex noise conditions according to an embodiment of the present disclosure;

[0056] Figure 8 Fig. 3 shows a block diagram of a deep reinforcement learning based object classification apparatus under complex noise conditions according to another embodiment of the present disclosure. DETAILED DESCRIPTION

[0057] Various exemplary embodiments, features, and aspects of the present disclosure will be described below in detail with reference to the accompanying drawings. The same reference numbers in different drawings denote the same or similar elements. Although various aspects of embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

[0058] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations.

[0059] In addition, for the purpose of convenience and brevity, detailed descriptions of well-known functions and structures incorporated in the present disclosure can be omitted. It will be appreciated by those skilled in the art that the present disclosure can be practiced without these specific details. In some instances, well-known methods, procedures, components, and circuits have not been described in detail since it can obscure the pertinent aspects of the subject matter.

[0060] First, several terms related to the present application are introduced.

[0061] Deep reinforcement learning (DRL) is a combination of reinforcement learning (RL) and deep learning (DL). Specifically, it can include reinforcement learning and neural networks. Reinforcement learning is a machine learning method in which an agent learns an optimal behavior policy through interaction with an environment. Neural networks are used to model the mapping between states and actions, enabling the processing of high-dimensional data. In reinforcement learning, an agent explores the environment by trying different actions and adjusts the policy based on the feedback (reward) from the environment to maximize the long-term cumulative reward. The agent can be a robot, a software program, or any entity capable of interacting with the environment and taking actions.

[0062] Leveraging the powerful advantages of deep reinforcement learning algorithms in board games, object classification problems can be viewed as a guessing game: At each time step, the agent receives an environment state s represented by sample data, and then performs action a to classify the object under the guidance of a policy. The environment returns a reward r and the next sample s' to the agent. The experience data consisting of the environment state s, action a, reward r, and the next sample s' is stored in the agent's experience pool. After traversing all sample data, multiple sets of experience data are randomly selected from the experience pool to train the deep Q-network in the agent. The interaction process between the agent and the environment is illustrated in the following figure. Figure 1 As shown, the agent receives a reward r when performing an action. When the agent learns the optimal policy from its interaction with the environment to obtain the maximum cumulative reward, the deep Q-network in the agent can correctly classify the samples.

[0063] Will Figure 1 The Markov decision framework shown can be transformed into a sequential decision problem, leading to classification models based on deep Q-networks (including the noise classification model and object classification model discussed below). Assume the training dataset is D = {x1, x2, ..., x...} n}, where x i Let represent the i-th sample in the training dataset. Then, in a classification model based on deep reinforcement learning:

[0064] State S: Determined by the sample. At the start of training, the AI ​​receives the first sample x1 as its initial state s1, and the environment's state s at each time step... t It will correspond to sample x t When a new episode begins, the environment randomly shuffles the order of samples in the training dataset. This means that in different training episodes, the order in which the environment feeds back samples to the agent varies. An episode is the transition trajectory from the initial state to the final state.

[0065] Action A: Related to the categories in the training dataset. In classification problems, the action taken by the agent is to determine the category. For example, for a noise classification problem, A = (0, 1, 2), where 0, 1, and 2 represent the three noise categories respectively; another example is a fault detection problem, A = (0, 1), where 0 represents normal and 1 represents fault.

[0066] Reward R: Feedback from the environment. The environment measures the success or failure of the agent's behavior through feedback rewards.

[0067] Strategy π θ The mapping function π: S→A indicates that the mapping function π is a function that maps state S to action A; where π θ (a t |st represents the probability that the agent executes action a t at state s t .

[0068] In the classification model, the mapping function π is a function representing the probability of returning all categories A according to the sample S. The mapping function π can be represented by the following formula:

[0069] π(a|s)=P(a t =a|s t =s)

[0070] where s t represents the sample returned by the environment to the agent at the t-th time step, a t represents the action executed by the agent according to the sample s t , π(a|s) represents the probability of executing action a based on the current sample s, and t takes values from 1 to n in turn, where n is the total number of samples.

[0071] The environment feeds back a reward to the agent when the agent correctly classifies the sample, which can achieve the goal of correctly classifying the sample by maximizing the cumulative return g t function, and the g t function can be represented by the following formula:

[0072]

[0073] where the discount factor γ ∈ [0, 1] is used to balance the current reward and the future reward, r t+k represents the reward corresponding to the t+k-th time step, t represents the t-th time step, and t takes values from 1 to n in turn, where n is the total number of samples.

[0074] According to the Bellman equation, the action value function (i.e., the Q function) Q π (s,a)=E π [g t |s t =s,a t =a] can be represented as:

[0075] Q π (s,a)=E π [r t +γQ π (s t+1 ,a t+1 )|s t =s,a t =a];

[0076] where Q π(s,a) represents the action value (i.e., Q-value) of the mapping function π based on sample s when performing action a. t Let a represent the sample returned by the environment to the agent at time step t. t This indicates that the agent, based on sample s t The action to be performed, g t For the cumulative reward function, r t Q represents the reward given by the environment to the agent at the t-th time step, γ represents the discount factor, and Q represents the reward. π (s t+1 ,a t+1 ) represents the mapping function π based on sample s t+1 Perform action a t+1 Q value, E π [] indicates that the expected value of the data within the brackets is calculated.

[0077] Based on the definitions and notation above, the classification problem can be defined as finding an optimal policy π. * The strategy S→A maximizes the cumulative reward. The agent maximizes the cumulative reward by solving for the optimal policy, π, under the optimal Q-function. * This is the optimal strategy for the classification model, strategy π. * It can be expressed by the following formula:

[0078]

[0079] Q * (s,a)=E π [r t +γmax a Q * (s t+1 ,a t+1 )|s t =s,a t =a];

[0080] Where, π * (a|s) represents the probability that the classification model will perform different actions a based on the current sample s, and argmax a Q * (s,a) represents the search for action a that maximizes Q in sample s. That is, if action a maximizes Q... * If (s,a) is maximized, then strategy π * The probability value corresponding to action a is determined to be 1; if action a cannot make Q * If (s,a) is maximized, then strategy π * The probability value corresponding to action 'a' is determined to be 0; Q * (s,a) is the action value function for calculating the Q value, E π [] indicates calculating the expected value of the data within the brackets; Q* (s t+1 ,a t+1 ) represents a policy function π * based on sample s t+1 , action a t+1 , and Q value of performing action a t at the t-th time step, s t represents a sample returned by the environment to the agent at the t-th time step, a t represents an action performed by the agent according to sample s t , r k represents a reward corresponding to the t-th time step fed back by the environment to the agent, and γ represents a discount factor.

[0081] In the deep reinforcement learning classification algorithm of the present application, a deep neural network is used to fit a Q function, and a deep Q network is obtained. According to the experience replay mechanism, a plurality of sets of experience data are stored in an experience pool B, and each set of experience data can be represented as (s, a, r, s’), wherein s represents a sample corresponding to a time step, a represents an action performed based on the sample at the time step, r represents a reward obtained by performing the action a at the time step, and s’ represents a sample corresponding to a next time step. Then, gradient descent is performed on the deep Q network according to a loss function as follows:

[0082] L(θ k ) = ∑ (s,a,r,s′)∈B (y - Q(s, a; θ k )) 2 ;

[0083] wherein L(θ k ) represents a loss function of the deep Q network with model parameters θ k ; Q(s, a; θ k ) represents a Q value determined by the deep neural network based on sample s and action a with model parameters θ k ; y represents a true value of the Q function, and the value of y can be determined based on the value of the target network, which can be represented by the following formula:

[0084] y = r + γmax a′ Q(s’, a’; θ k-1 )

[0085] wherein r represents a reward of performing action a according to sample s, γ represents a discount factor, s’ represents a next sample, a’ represents an action performed according to the next sample s’, and max a′ Q(s’, a’; θ k-1 ) represents a maximum Q value determined by the target network based on the next sample s’ and action a’ with model parameters θ k-1 .

[0086] The derivation of the loss function with respect to θ can determine the adjustment direction of the model parameters θ in the deep Q network to minimize the loss function, thereby optimizing the prediction performance of the deep Q network. The derivation of the loss function with respect to θ can be represented by the following formula:

[0087]

[0088] wherein, represents the gradient of the loss function with respect to θ, and the gradient is used to indicate the direction in which the loss function value increases fastest. When updating the model parameters θ in the deep Q network, the model parameters θ are updated in the opposite direction of the gradient (i.e., the negative gradient direction).

[0089] In the present application, the optimal Q function Q k can be obtained by minimizing the loss function L(θ * , thereby obtaining the optimal strategy and the maximum cumulative return, and realizing the object classification based on deep reinforcement learning.

[0090] Wavelet threshold denoising: a signal denoising method, which decomposes noise data into sub-signals of different frequencies through wavelet transform, then performs threshold processing on wavelet coefficients, and finally performs inverse wavelet transform to obtain the denoised signal. Different wavelet bases will affect the denoising effect. Common wavelet bases include but are not limited to: Haar wavelet, Daubechies (dbN) wavelet, Symlets (symN) wavelet, Coiflets (coifN) wavelet, Biorthogonal (biorNr.Nd) wavelet, etc. The present application will not list them one by one here.

[0091] Next, the object classification method based on deep reinforcement learning under complex noise conditions provided by the present application will be described in detail. The application scenarios of the object classification method provided by the present application include but are not limited to: fault diagnosis scenarios (i.e., classifying whether the signal to be classified is a fault signal and the fault type), text classification scenarios (i.e., determining which text classification the signal to be classified belongs to), etc. The application scenarios of the object classification method are not limited in this embodiment.

[0092] Taking the fault diagnosis scenario as an example, rotating machinery is the most common mechanical component in the industrial field, and is widely used in many key fields such as aviation, aerospace, transportation, intelligent manufacturing, etc. If the rotating machinery has a serious fault, it will cause a series of chain reactions, causing huge economic losses and even casualties. Therefore, it is of great significance to establish a fast and effective fault detection mechanism, to discover the fault of the machinery as soon as possible, and to handle the fault in time, to ensure the safe operation of the rotating machinery, and to enhance the reliability and stability of the rotating machinery.

[0093] In practical work, fault detection technology faces great challenges. On the one hand, most of the time, rotating machinery is in normal operation state, and the fault data is much less than the normal data, so the fault detection faces serious imbalance problem. On the other hand, in the complex environment of actual production, the running state of mechanical equipment will inevitably be disturbed by the external environment, generating vibration signals containing various noises, so that the fault characteristics are easy to be disturbed and even covered.

[0094] For object classification methods in other scenarios, there are also the above problems of complex noise signals and unbalanced sample data, which will not be described one by one in this embodiment.

[0095] In recent years, deep reinforcement learning has shown strong decision-making ability and intelligent application prospect in the fields of games and robot control. For the problem of deep reinforcement learning fault detection and diagnosis, the agent can extract data features by using deep network, and through the interaction with the environment, it can carry out self-motivated reinforcement learning to optimize the fault detection and diagnosis strategy. In this process, the learning performance is evaluated by the overall reward of the data, in other words, the agent cannot distinguish from the input signal which samples have not been well learned in the learning process. This way forces the agent to find the internal differences between fault patterns, so that deep reinforcement learning has great potential in fault detection and diagnosis. For example, the existing deep feature enhancement reinforcement learning method for rolling bearing fault diagnosis breaks the dependence of the previous fault diagnosis method on the model structure, and successfully obtains higher recognition accuracy. Or, the deep reinforcement learning fault diagnosis algorithm based on Gated Recurrent Unit (GRU) is successfully applied to nuclear power plant fault diagnosis. Or, the hybrid combination of reinforcement learning and multi-agent system successfully detects local and global abnormal behaviors in micro-service architecture. Or, the unbalanced classification algorithm model of deep reinforcement learning is proposed by converting the classification problem into a sequential decision problem and constructing a reward function according to the imbalance ratio. Or, the reward function is improved according to the principle of K-Means algorithm, and the experimental verification shows that it has good effect in rolling bearing fault diagnosis. Or, a double experience pool deep reinforcement learning model solves the imbalance problem in fault diagnosis by balancing cross sampling. The above existing fault diagnosis methods reflect the advantages of deep reinforcement learning in fault detection and diagnosis from the aspects of solving complex working conditions and data imbalance problems.

[0096] Based on this, in the present application, deep reinforcement learning is applied to noise classification and object classification, using a noise classification model based on deep reinforcement learning to classify the data to be classified can avoid the problem of poor performance of the noise classification model caused by unbalanced noise samples, using an object classification model based on deep reinforcement learning to classify the data after noise reduction can avoid the problem of poor performance of the object classification model caused by unbalanced object classification samples, and improve the accuracy of noise classification and object classification in the sample imbalance scenario. At the same time, by determining the noise classification of the data to be classified before classifying the data to be classified, and then calling the noise reduction algorithm corresponding to the noise classification to perform noise reduction processing, the noise reduction algorithm can be adaptively switched to perform noise reduction processing in the case of multiple noise types, and the noise reduction effect of the data to be classified in a complex noise scenario can be improved.

[0097] Figure 2 A flowchart of an object classification method based on deep reinforcement learning in a complex noise condition according to an embodiment of the present disclosure is shown. In this embodiment, the method is used in an electronic device with processing capability such as a user terminal or a server, which includes but is not limited to a computer, a tablet computer, a mobile phone, etc. The embodiment does not limit the implementation of the electronic device. As shown in Figure 2 The method includes:

[0098] Step 201, obtaining data to be classified.

[0099] The data to be classified refers to data to be classified. For different object classification scenarios, the data content of the data to be classified can be different, such as: in the fault diagnosis scenario, the data to be classified can be device signals (such as motor vibration signals, etc.); in the text classification scenario, the data to be classified can be text to be classified. Generally, the data to be classified includes noise data. The noise data refers to data that will be disturbed by noise when the data to be classified is classified. Optionally, when the data to be classified is collected, the noise data can include multiple types, such as Gaussian noise, gamma noise, etc. The embodiment does not limit the classification of noise data.

[0100] The data to be classified can be collected by an electronic device or sent by another electronic device. The embodiment does not limit the acquisition method of the data to be classified.

[0101] Step 202, based on a noise classification model based on deep reinforcement learning, classifying the data to be classified to obtain the noise classification corresponding to the data to be classified.

[0102] The noise classification model is used to determine the noise classification of the to-be-classified data. In this embodiment, the noise classification model is obtained based on deep reinforcement learning, so that the noise classification performance of the noise classification model can be ensured in the case that the number of noise samples corresponding to each noise classification is unbalanced. For the specific model training process, refer to the following embodiments, which will not be described here in detail.

[0103] In step 203, a noise reduction algorithm corresponding to the noise classification is called to perform noise reduction processing on the to-be-classified data, to obtain noise-reduced data.

[0104] Different noise reduction algorithms correspond to different noise classifications.

[0105] In one example, calling a noise reduction algorithm corresponding to the noise classification to perform noise reduction processing on the to-be-classified data to obtain noise-reduced data includes: obtaining a correspondence between the noise classification and the optimal wavelet threshold noise reduction algorithm; performing wavelet threshold noise reduction on the to-be-classified data according to the optimal wavelet threshold noise reduction algorithm corresponding to the noise classification, to obtain noise-reduced data.

[0106] The correspondence between the noise classification and the optimal wavelet threshold noise reduction algorithm is pre-stored in the electronic device. The correspondence can be obtained in the following manner: for each noise classification, the same noise data belonging to the noise classification is subjected to wavelet threshold noise reduction using different wavelet bases respectively, and the wavelet base with the optimal noise reduction effect is determined from the noise reduction results corresponding to different wavelet bases, to obtain the optimal wavelet threshold noise reduction algorithm corresponding to the noise classification. Then, the correspondence stored in the electronic device is read to obtain the optimal wavelet threshold noise reduction algorithm corresponding to the noise classification determined by the noise classification model this time. The to-be-classified data is subjected to noise reduction processing by using the optimal wavelet threshold noise reduction algorithm, so that the data noise reduction effect can be improved.

[0107] Optionally, the different wavelet bases include but are not limited to Haar wavelet, dbN wavelet, symN wavelet, coifN wavelet, biorNr.Nd wavelet, etc. The implementation of different wavelet bases is not limited in this embodiment.

[0108] In other embodiments, the different noise reduction algorithms can also include singular value decomposition (SVD) noise reduction, deep learning noise reduction, etc. The implementation of the noise reduction algorithm is not limited in this embodiment.

[0109] In step 204, the object classification model obtained based on deep reinforcement learning is used to perform object classification on the noise-reduced data, to obtain an object classification result.

[0110] The object classification model is used to judge the noise classification of the denoised data. In this embodiment, the object classification model is obtained based on deep reinforcement learning, so that the object classification performance of the object classification model can be ensured in the case that the sample quantity corresponding to each object classification is unbalanced. For specific model training process, refer to the following embodiments, which will not be described here in this embodiment.

[0111] In summary, the object classification method based on deep reinforcement learning under complex noise conditions provided in this embodiment comprises the following steps: obtaining to-be-classified data; performing noise classification on the to-be-classified data based on a noise classification model obtained based on deep reinforcement learning to obtain the noise classification corresponding to the to-be-classified data; calling a denoising algorithm corresponding to the noise classification to perform denoising processing on the to-be-classified data to obtain denoised data; performing object classification on the denoised data based on an object classification model obtained based on deep reinforcement learning to obtain an object classification result; the problem of poor noise removal effect of a traditional denoising algorithm can be solved; since the noise classification of the to-be-classified data is determined before the object classification of the to-be-classified data is performed, and then the denoising algorithm corresponding to the noise classification is called to perform denoising processing, the denoising algorithm can be adaptively switched to perform denoising processing in the case that there are many noise types, and therefore the denoising effect of the to-be-classified data under a complex noise scene is improved.

[0112] At the same time, the noise classification model obtained based on deep reinforcement learning is used to perform noise classification on the to-be-classified data, which can avoid the problem of poor performance of the noise classification model caused by unbalanced noise samples, and the object classification model obtained based on deep reinforcement learning is used to perform object classification on the denoised data, which can avoid the problem of poor performance of the object classification model caused by unbalanced object classification samples, and improve the accuracy of noise classification and object classification in a sample unbalanced scene.

[0113] In some scenarios, the label information of the sample data used to train the noise classification model and / or the object classification model may be unknown. For example, the label information of the machine running state is mostly unknown, and the acquisition of the fault type label depends on expert experience and is cumbersome. At this time, the cost of supervised model training is often very high. Therefore, the object classification method directly facing the unlabeled, unbalanced and multi-noise data has greater application value and broader application prospect.

[0114] However, in the traditional deep reinforcement learning method, the agent mostly obtains rewards by whether the action is consistent with the sample label, and cannot get rid of the dependence on the label.

[0115] Based on this, in some conventional methods, fault detection and diagnosis can be performed based on unsupervised training methods. Traditional unsupervised fault detection and diagnosis methods mainly perform fault identification through cluster analysis. For example, a k-means clustering algorithm is used to realize fault identification of bearings and centrifugal compressors respectively. Alternatively, a sequential two-stage clustering method is used to diagnose faults of a fan tube disc unit, and good results are achieved. Alternatively, principal component analysis is combined with a K-Means clustering algorithm to automatically detect and diagnose faults of unlabeled data. Alternatively, a fuzzy logic controller and a fuzzy mean clustering method are used to detect and diagnose early faults of a transformer in combination with an expert system. Although the above-mentioned traditional unsupervised training methods have achieved good results, most of them do not consider the influence of unbalanced class clusters on clustering results and cannot be directly used to analyze unbalanced fault detection and diagnosis data. Therefore, improving the clustering performance of the algorithm when processing unbalanced data sets is an indispensable step to enhance the efficiency of unlabeled fault detection.

[0116] Based on this, in the present application, the noise classification model and / or the object classification model are obtained based on deep reinforcement learning of unlabeled sample data, and the relationship between the sample and the class cluster center is used to feedback reward information to improve the performance of unsupervised model training.

[0117] Specifically, deep reinforcement learning based on unlabeled sample data includes the following steps:

[0118] Step 1, determine the number A of class cluster centers based on the number of classifications, and initialize the class cluster centers as A sample data with the farthest distance; A is an integer greater than 1.

[0119] The number of classifications refers to the total number of classifications corresponding to the model (the noise classification model and / or the object classification model), and the value of A is equal to the number of classifications. For example, the noise classification corresponding to the noise classification model has 3 types, which are Gaussian noise, gamma noise and Rayleigh noise, and the number of classifications is 3, and accordingly A = 3. For another example, the object classification corresponding to the object classification model has 2 types, which are normal and fault, and the number of classifications is 2, and accordingly A = 2.

[0120] The sample distance is used to indicate the similarity between different sample data. Since the similarity between sample data of different classifications is small, by initializing A sample data with the farthest distance as the class cluster centers, the accuracy of initializing the class cluster centers can be improved as much as possible, and the number of updates of the class cluster centers can be reduced.

[0121] Optionally, the sample distance can be Manhattan distance, or Euclidean distance, etc., and the present embodiment does not limit the acquisition method of the sample distance.

[0122] The initial cluster centers are set as A sample data with the farthest sample distance, including: determining the sample distance between each sample data and other sample data; and determining the initial cluster centers based on the sample data with the farthest sample distance.

[0123] Since the Manhattan distance is calculated faster, in the embodiment, the sample distance is taken as the Manhattan distance as an example, assuming that each sample data is an n-dimensional vector, the sample distance between the sample data x (x1, x2, … x n ) and other sample data y (y1, y2, … y n ) can be expressed by the following formula:

[0124]

[0125] wherein d xy represents the Manhattan distance between the sample data x and the sample data y, x k represents the k-dimensional vector in the sample data x, and y k represents the k-dimensional vector in the sample data y.

[0126] Illustratively, in the case of the number of categories being 2 a , the initial cluster centers are determined based on the sample data with the farthest sample distance, including: determining the sample data corresponding to the first a largest sample distances as the initial cluster centers. According to the above, each sample distance is determined according to two sample data, that is, the sample data corresponding to each sample distance includes two, at this time, determining the sample data corresponding to the sample distance as the initial cluster center means: determining both of the two sample data corresponding to the sample distance as the initial cluster center.

[0127] In the case of the number of categories being 2 a +1, the initial cluster centers are determined based on the sample data with the farthest sample distance, including: determining the sample data corresponding to the first a largest sample distances as the initial cluster centers, and determining one sample data with the largest sample distance between the non-cluster center sample data and the cluster center, and determining the sample data as the initial cluster center.

[0128] The determining of the sample data with the largest sample distance from the class cluster center includes: determining, for each sample data of the non-class cluster center, a maximum distance value of the sample distance between the sample data and each class cluster center; determining, from the maximum distance values corresponding to each sample data of the non-class cluster center, a maximum value of the maximum distance values, and the sample data corresponding to the maximum value of the maximum distance values is the sample data with the largest sample distance from the class cluster center. For example, a is 1, and the class cluster centers determined based on the sample data corresponding to the maximum a sample distances include two, which are 1 and 2. For each sample data x of the non-class cluster center, the sample distances between x and the class cluster center 1 and the class cluster center 2 are determined, and the maximum distance value is selected. Then, the maximum value of the maximum distance values corresponding to each sample data x is selected, and the sample data x corresponding to the maximum value is also determined as the initial class cluster center.

[0129] Step 2, for each sample data, the sample distance between the sample data and each class cluster center is determined, and the index of the class cluster center with the smallest sample distance from the sample data is determined as the class cluster label of the sample data.

[0130] The determination manner of the sample distance between the sample data and the class cluster center is the same as the determination manner of the sample distance between different sample data, and the related description is referred to step 1, which will not be repeated here.

[0131] The index of the class cluster center is used to identify different class cluster centers. For example, A is 2, and the index of the class cluster center includes {0, 1}, which respectively indicates different class cluster centers. In other embodiments, A is 2, and the index of the class cluster center can also include {1, 2}, and the embodiment does not limit the setting manner of the index of the class cluster center.

[0132] Suppose all class cluster centers are represented as C i , i = 0, … |A| - 1, and the index i of the class cluster center to which the sample data belongs can be represented by the following formula:

[0133]

[0134] Wherein, i represents the index of the class cluster center, represents the sample distance between the sample data s and the class cluster center C i , represents the minimum value of the sample distance between the sample data s and each class cluster center C i .

[0135] Step 3, constructing a data environment based on each sample data and the class cluster label corresponding to the sample data.

[0136] Step 4, deep reinforcement learning is performed through interaction between the agent and the data environment to obtain a trained reinforcement learning model.

[0137] In the present application, the agent refers to an entity software program interacting with the data environment, which can be a process running the agent.

[0138] In one example, deep reinforcement learning is performed through interaction between the agent and the data environment to obtain a trained reinforcement learning model, including the following steps:

[0139] Step 41, the data environment returns the i-th sample data to the agent in turn; the agent includes a reinforcement learning model, the reinforcement learning model includes a deep Q network and a target network, the network parameters of the deep Q network and the network parameters of the target network are initialized to be the same; i is a positive integer from 1 to n in turn, and n is the total number of sample data.

[0140] Suppose the training set includes n unlabeled sample data, through step 2, the class cluster label corresponding to each sample data can be determined, and the correspondence between each sample data and the class cluster label is obtained. At this time, the data environment randomly selects a sample data as the first sample data and returns it to the agent for interaction. Then, a sample data that has not been traversed is selected as the second sample data and returned to the agent for interaction. This cycle continues until all sample data in the training set is traversed, and the interaction between the agent and the agent is completed.

[0141] Step 42, determine the i-th classification action corresponding to the i-th sample data through the deep Q network.

[0142] The deep Q network is based on a neural network, which includes an input layer, a hidden layer and an output layer. The dimension of the input layer is the same as that of the sample data, and the dimension of the output layer is the same as the number of classifications of the model.

[0143] The i-th classification action indicates one of the class cluster labels. For example, if the class cluster label includes {0, 1}, the i-th classification action is one of {0, 1}.

[0144] Step 43, the agent returns the i-th classification action to the data environment, so that the data environment determines the i-th reward of the i-th classification action based on the comparison result between the i-th classification action and the class cluster label corresponding to the i-th sample data, and returns the i-th reward and the i+1-th sample data to the agent; in the case of the same comparison result, the reward is In the case of different comparison results, the reward is -1. Specifically, the reward function in the data environment can be represented by the following formula:

[0145]

[0146] wherein, denotes the sample distance between the ith sample data and the corresponding class cluster center; s t denotes the ith sample data, a t denotes the ith classification action, R(s t , a t ) denotes the ith reward, c t denotes the class cluster label corresponding to the ith sample data.

[0147] Step 44, store the ith experience data in the experience pool in the agent, and update i=i+1 to trigger the execution of step 41; wherein the ith experience data includes the ith sample data, the ith classification action, the ith reward and the ith+1 sample data.

[0148] Step 45, after the traversal of the n sample data is completed, randomly extract K groups of experience data from the experience pool; K is a positive integer.

[0149] Step 46, input each group of experience data into the deep Q network and the target network respectively, to update the deep Q network based on the difference between the first network result of the deep Q network and the second network result of the target network.

[0150] Specifically, the random gradient descent method is used to train the weight and bias value parameters of the deep Q network, so as to minimize the difference between the first network result and the second network result of the target network, thereby maximizing the Q function value, and obtaining the optimal strategy of the deep Q network. Wherein, the difference between the first network result and the second network result of the target network is calculated by a preset loss function, which can be established in the manner of L(θ k ) in the foregoing, in other embodiments, the loss function can also be other implementation manners, and the embodiment does not limit the implementation manner of the loss function. The updating direction of the deep Q network is determined by the gradient determined by the derivation of L(θ k ) with respect to θ, and when updating the model parameter θ in the deep Q network, the model parameter θ is updated in the opposite direction of the gradient (i.e. the negative gradient direction).

[0151] Step 47, synchronize the network parameters of the deep Q network to the target network every W groups of experience data are traversed, until the traversal of each group of experience data is completed, and the network parameters of the deep Q network obtained by the current round of training are obtained, and W is a positive integer.

[0152] Step 5, after the current round of training is completed, update the class cluster center based on the mean value of the sample data corresponding to each class cluster center, and trigger the execution of step 2 until the training round reaches the preset requirement.

[0153] The deep Q network obtained after the last round of training is determined as the classification model obtained by unlabeled training.

[0154] An episode of training refers to a process in which interactions between the agent and the data environment traverse each sample data in the training set, and K sets of experience data in the experience pool are used to train the deep Q network. In different training processes, the order in which the data environment returns each sample data to the agent is different.

[0155] The preset requirement for the training episode includes but is not limited to: the training episode is equal to a preset threshold T of the training episode; and / or, the change amount of the loss function value after x consecutive training ends is less than a preset change threshold. The embodiment does not limit the implementation manner of the preset condition.

[0156] In this embodiment, a method for deep reinforcement learning classification for unlabeled, unbalanced and multi-noise data is proposed. The method feeds back reward information based on the relationship between the sample and the class cluster center, and improves the performance of unsupervised model training.

[0157] Optionally, based on the above embodiment, when the number of classifications corresponding to the noise classification model and / or the object classification model is less than or equal to a preset threshold, deep reinforcement learning is performed based on the unlabeled sample data. The preset threshold can be 2, 3, etc. The specific value of the preset threshold can be determined according to different classification accuracy requirements, and the embodiment does not limit the value of the preset threshold.

[0158] Taking the case where the noise classification includes three types: Gaussian noise, gamma noise and Rayleigh noise, and the object classification includes two types: normal and fault, and the preset threshold is 2, at this time, if the number of classifications corresponding to the noise classification model is greater than the preset threshold, deep reinforcement learning is performed based on the labeled sample data, and if the number of classifications corresponding to the object classification model is equal to the preset threshold, deep reinforcement learning is performed based on the unlabeled sample data.

[0159] In this embodiment, by selecting the deep reinforcement learning mode of the noise classification model and / or the object classification model based on the classification accuracy requirement, the training flexibility of the classification model can be improved.

[0160] Next, taking the case where the noise classification model is obtained by deep reinforcement learning based on labeled sample data, and the object classification model is obtained by deep reinforcement learning based on unlabeled sample data as an example. In the above embodiment, the sample data used to train the noise classification model is sample data with noise; and the sample data used to train the object classification model is denoised sample data.

[0161] Correspondingly, the noise classification model obtained by deep reinforcement learning based on labeled sample data includes the following steps:

[0162] Step 1, obtaining a training set composed of sample data with noise, the training set including sample data and noise classification labels corresponding to the sample data.

[0163] In one example, the training set is obtained by adding noise data to original data without noise, different original data corresponding to different classification of added noise data, to obtain sample data; setting noise classification labels corresponding to the sample data based on the noise classification of the added noise data in the sample data.

[0164] In other embodiments, the sample data in the training set can also be directly collected data with noise, and correspondingly, the noise classification labels corresponding to the sample data can be obtained by manual annotation. The present embodiment does not limit the way of obtaining the training set.

[0165] Step 2, constructing a data environment based on the training set.

[0166] Step 3, sequentially returning the i-th sample data to the agent through the data environment; the agent includes a reinforcement learning model, the reinforcement learning model includes a deep Q network and a target network, the network parameters of the deep Q network and the network parameters of the target network are initialized to be the same; i is a positive integer sequentially taking values from 1 to n, n is the total number of sample data.

[0167] Specifically, the state S fed back by the data environment to the agent is: each sample data with noise corresponds to a state, and the states at different time steps correspond to different samples.

[0168] Step 4, determining the i-th classification action corresponding to the i-th sample data through the deep Q network.

[0169] The i-th classification action indicates one of the noise classifications. For example: classification action A=(0, 1, 2), where 0, 1, and 2 respectively indicate three noise categories.

[0170] Step 5, returning the i-th classification action to the data environment through the agent, so that the data environment determines the i-th reward of the i-th classification action based on the comparison result between the i-th classification action and the class cluster label corresponding to the i-th sample data, and returns the i-th reward and the i+1-th sample data to the agent.

[0171] In some scenarios, most of the sample data contains Gaussian noise, and only a small part contains other noise. At this time, it is difficult to correctly identify other noise in the unbalanced data set. In order to better identify the data containing minority class noise, solve the problem of unbalanced noise data, the noise classification model should pay more attention to the minority class noise data. Therefore, in this embodiment, the sample data with labels includes sample data of a first noise classification (such as other noise) and sample data of a second noise classification (such as Gaussian noise), and the number of sample data of the first noise classification is less than the number of sample data of the second noise classification.

[0172] Correspondingly, in the deep reinforcement learning process based on the labeled sample data, the reward when the data environment interacts with the agent is determined by the following formula:

[0173]

[0174] Wherein, s t represents the current sample data (i.e. the ith sample) returned by the data environment to the agent, a t represents the classification action performed based on the current sample data s t (i.e. the ith classification action), R(s t ,a t ) represents the reward of performing the classification action based on the current sample data s t (i.e. the ith reward), l t represents the noise classification label corresponding to the current state s t , D minority represents a data set composed of sample data of the first noise classification, D majority represents a data set composed of sample data of the second noise classification, and p represents the ratio of the number of sample data of the first noise classification to the number of sample data of the second noise classification.

[0175] At this time, when the agent correctly or incorrectly classifies the sample data containing the second noise classification, the reward is p or -p; when the agent correctly or incorrectly classifies the sample data containing the first noise classification, the reward is 1 or -1. When the reward function faces sample data containing majority class noise, the reward or punishment is small, which can improve the classification performance of the noise classification model in identifying data containing minority class noise.

[0176] Step 6, store the ith experience data in the experience pool in the agent, and update i=i+1 to trigger step 3; wherein the ith experience data includes the ith sample data, the ith classification action, the ith reward and the ith+1 sample data.

[0177] Step 7, after the n sample data is traversed, randomly extract K groups of experience data from the experience pool; K is a positive integer.

[0178] Step 8, input each set of experience data into the deep Q network and the target network respectively, to update the deep Q network based on the difference between the first network result of the deep Q network and the second network result of the target network.

[0179] Step 9, synchronize the network parameters of the deep Q network to the target network every time W sets of experience data are traversed, until the traversal of each set of experience data is completed, and the network parameters of the deep Q network obtained by the current round of training are obtained, W is a positive integer.

[0180] Step 10, after the current round of training is completed, the order between each sample data in the data environment is disturbed, and step 3 is triggered until the training round reaches the preset requirement, and the training is stopped. The deep Q network obtained by the last training is determined as the noise classification model.

[0181] The related description of the embodiment is described in detail in the above deep reinforcement learning based on unlabeled sample data. The difference is that the class cluster label is replaced by the noise classification label, and the reward function is replaced by the reward function in step 5. This embodiment will not be repeated here.

[0182] Optionally, the test set can also be input into the trained noise classification model, and the noise classification result of each sample data in the test set is output, so as to determine the performance of the noise classification model based on the noise classification result. The sample data in the test set is different from the sample data in the training set.

[0183] As described above, in the embodiment, when facing sample data containing multiple class noise, the reward or punishment is small by setting the reward function, which can improve the classification performance of the noise classification model in identifying sample data containing few class noise.

[0184] Optionally, in other embodiments, the object classification model can also be obtained by deep reinforcement learning based on the above labeled sample data. At this time, the noise classification label in this embodiment is adaptively replaced by the object classification label, and the sample data with noise is adaptively replaced by the sample data without noise. The overall training idea is the same, and the training method of the object classification model is not limited.

[0185] Optionally, the training order of the noise classification model is before the training order of the object classification model. At this time, for a training set composed of sample data with noise, first, the noise classification model is trained by using the training set, then the noise classification model is used to classify the sample data with noise in the training set, and the noise classification corresponding to the noise reduction algorithm is called to reduce the noise of the sample data with noise. Get the sample data after noise reduction; then, use each sample data after noise reduction to train the object classification model.

[0186] In one example, the training process of the object classification model is illustrated taking the object classification model used in the fault detection scenario as an example, for reference Figure 3 The process at least includes the following steps:

[0187] Step 301, obtaining a training set, the training set including sample data with noise.

[0188] The related description of this step is detailed in the above-mentioned embodiments, which will not be repeated here in this embodiment. Figure 3 In one example, the noise classification includes three kinds, which are noise 1, noise 2 and noise 3, for example.

[0189] Step 302, classifying the sample data with noise based on the noise classification model to obtain the noise classification of each sample data with noise.

[0190] The training method of the noise classification model can be the labeled training method or the unlabeled training method in the above-mentioned embodiments, and the training method of the noise classification model is not limited in this embodiment.

[0191] Step 303, calling the noise classification corresponding denoising algorithm to process the sample data with noise to obtain the denoised sample data.

[0192] The related description of this step is detailed in the related description of step 203, which will not be repeated here in this embodiment.

[0193] According to Figure 3 It can be seen that the optimal wavelet threshold denoising algorithm corresponding to each noise classification is called to process the sample data of the noise classification respectively to obtain the denoised sample data corresponding to noise 1, the denoised sample data corresponding to noise 2 and the denoised sample data corresponding to noise 3.

[0194] Step 304, performing deep reinforcement learning based on the denoised sample data to obtain the object classification model.

[0195] In one example, the object classification model can be the labeled training method or the unlabeled training method in the above-mentioned embodiments, and the training method of the object classification model is not limited in this embodiment.

[0196] Step 305, based on the object classification model obtained by deep reinforcement learning, classifying the denoised data to obtain the object classification result.

[0197] The denoised data can be the denoised sample data corresponding to noise 1, the denoised sample data corresponding to noise 2 or the denoised sample data corresponding to noise 3.

[0198] In this embodiment, first, the data imbalance ratio is used to feedback the reward information, the noise data is taken as the model state, and the noise classification is taken as the optional action of the agent. An interactive environment of agent observation, action and reward is constructed to realize the deep reinforcement learning decision of the noise classification contained in the data. Then, the wavelet threshold denoising effects of different wavelet bases under single noise are compared to obtain the optimal denoising method for each type of noise, and the denoised data is obtained by denoising each type of noise data based on the optimal denoising method. Finally, the reward information is fed back by the relationship between the sample and the cluster center. By taking the denoised data as the state and the fault type as the optional action of the agent, a new interactive environment is constructed to realize the deep reinforcement learning fault detection, and the rotating machinery fault detection oriented to unlabeled, unbalanced and multi-noise data is realized.

[0199] In other embodiments, the denoised sample data and the sample data with noise can also come from different training sets. Accordingly, the noise classification model and the object classification model can also be trained synchronously. The present embodiment does not limit the training order of the noise classification model and the object classification model, and the acquisition method of the denoised sample data.

[0200] Next, the object classification model is applied to the fault detection scene, and the model performance of the object classification model trained by unlabeled data and the noise classification model trained by labeled data is evaluated.

[0201] Suppose that the above sample data comes from the CWRU (Case Western Reserve University) dataset provided by Case Western Reserve University. The dataset contains acceleration sensor data from different states, reflecting vibration signals of different types of bearing failures. If the normal data and the 0.007mm inner ring fault data are selected as the original experimental data according to the ratio of 10:1, and on this basis, Gaussian noise, gamma noise and Rayleigh noise are added to each type of data according to the ratio of 8:1:1, a bearing vibration signal dataset containing three types of noise is obtained. In this experiment, a long string of vibration signal data in the original data set is segmented according to the sample segmentation strategy with overlap (such as Figure 4 ), and the sample is expanded to 3000 samples, each sample length is 1200, and the length of the overlap part is 100.

[0202] Considering the unbalanced three types of noise data, in order to more comprehensively evaluate the performance of the object classification model trained by unlabeled data in different classifications, the weighted accuracy and weighted G value are used as the fault detection evaluation indicators, and the related formulas are as follows:

[0203]

[0204] where N represents the total number of samples, Ω = {w1, w2, …, wK} represents cluster division, C = {c1, c2, …, c J} represents the real fault classification, in the above process, the classification with the most number of samples in each cluster is considered as the corresponding real fault classification of the cluster.

[0205]

[0206] wherein, represents the weighted accuracy rate, Acc1, Acc2, and Acc3 represent the fault detection accuracy rates of the sample data corresponding to the Gaussian noise, gamma noise, and Rayleigh noise, respectively.

[0207] The G value is the geometric mean of the true positive rate and the true negative rate, and the higher the G value, the better the performance of the algorithm.

[0208]

[0209] wherein, TP is the true positive, FN is the false negative, TN is the true negative, and FP is the false positive.

[0210]

[0211] wherein, represents the weighted G value, G1, G2, and G3 represent the fault detection G values of the data corresponding to the Gaussian noise, gamma noise, and Rayleigh noise, respectively.

[0212] Taking the wavelet threshold denoising algorithm as an example, the results of wavelet threshold denoising are different under different noise by selecting different wavelet bases. In this application, wavelet threshold denoising is performed on data containing Gaussian noise, gamma noise, and Rayleigh noise, respectively, to obtain the optimal denoising method under the corresponding noise.

[0213] From the aspects of support length, vanishing moment, symmetry, regularity, and similarity, this embodiment selects db3 wavelet base and sym4 wavelet base for wavelet threshold denoising, and the specific results are shown in the following Table 1:

[0214] Table 1:

[0215]

[0216] According to Table 1, the detection accuracy rate and G value of data containing Gaussian noise after wavelet threshold denoising using db3 wavelet base are higher than those of sym4 wavelet base; the detection accuracy rate and G value of data containing gamma noise after wavelet threshold denoising using sym4 wavelet base are higher than those of db3 wavelet base; the detection accuracy rate of data containing Rayleigh noise after wavelet threshold denoising using db3 wavelet base is 1% lower than that of sym4 wavelet threshold denoising, and the G value is 0.03 higher, and comprehensive consideration shows that the effect of using db3 wavelet base is better.

[0217] Based on this, the optimal wavelet threshold denoising algorithm corresponding to the Gaussian noise classification can be set as using db3 wavelet basis for wavelet threshold denoising; the optimal wavelet threshold denoising algorithm corresponding to the gamma noise classification can be set as using sym4 wavelet basis for wavelet threshold denoising; and the optimal wavelet threshold denoising algorithm corresponding to the Rayleigh noise classification can be set as using db3 wavelet basis for wavelet threshold denoising.

[0218] In addition, in order to verify the superiority of the deep reinforcement learning method in noise classification, in this embodiment, a convolutional neural network (CNN) is used for noise classification as a comparative experiment. Figure 5 The situation of the agent obtaining the reward value in the training process is shown. According to the formula (1), the reward value is calculated as follows: Figure 5 It can be seen that, with the continuous learning of the agent (i.e., the increase of the episode), the decision-making ability gradually improves, and the cumulative return (i.e., the total reward) continuously increases, and finally reaches a stable state. Figure 6 The change of the accuracy rate in the training process is shown. Due to the improvement of the decision-making ability, the accuracy rate gradually increases.

[0219] The experiment is cross-validated for ten times, and the noise classification accuracy under different methods is shown in Table 2.

[0220] Table 2:

[0221] CNN Noise classification algorithm of the present application Accuracy of noise classification 76.44％ 83.33％

[0222] According to Table 2, the noise classification algorithm in this application improves the CNN classification accuracy rate by nearly 7%, that is, the deep reinforcement learning denoising classification method has more classification advantages than the convolutional neural network.

[0223] In addition, according to the setting mode of the denoising algorithm in the above, the data containing Gaussian noise and Rayleigh noise is denoised by using the wavelet threshold denoising method with db3 as the wavelet basis, and the data containing gamma noise is denoised by using the wavelet threshold denoising method with sym4 as the wavelet basis, to obtain the denoised data. The three types of data are respectively subjected to deep reinforcement learning fault detection, and the obtained fault detection performance is shown in Table 3:

[0224] Table 3:

[0225]

[0226] According to Table 3, after using different denoising algorithms to denoise the data classified by different noises and then performing fault detection, the accuracy rate is improved by 7% and the G value is improved by 0.11 when the data is unbalanced, which verifies that the model has better detection effect when the data is unbalanced.

[0227] In summary, the deep reinforcement learning fault detection method under complex noise conditions proposed in the application uses deep reinforcement learning to determine the noise type contained in each data. At the same time, the wavelet threshold denoising effects of different wavelet bases under single noise are compared to obtain the optimal denoising method for each noise. Based on this, the data of each type of noise is denoised respectively, and after obtaining the denoised data, deep reinforcement learning fault detection is performed. The accuracy and G value of fault detection are better when the data is unbalanced, and the advantages are more prominent.

[0228] Figure 7 Fig. 1 is a block diagram of an object classification device under complex noise conditions based on deep reinforcement learning according to an example embodiment. The device includes at least the following modules: a data acquisition module 710, a noise classification module 720, a data denoising module 730, and an object classification module 740.

[0229] The data acquisition module 710 is configured to acquire data to be classified.

[0230] The noise classification module 720 is configured to perform noise classification on the data to be classified based on a noise classification model obtained by deep reinforcement learning, to obtain a noise classification corresponding to the data to be classified.

[0231] The data denoising module 730 is configured to call a denoising algorithm corresponding to the noise classification to perform denoising processing on the data to be classified, to obtain denoised data. Different noise classifications correspond to different denoising algorithms.

[0232] The object classification module 740 is configured to perform object classification on the denoised data based on an object classification model obtained by deep reinforcement learning, to obtain an object classification result.

[0233] For details, please refer to the above method embodiments.

[0234] In some embodiments, the device provided by the embodiments of the present disclosure has functions or includes modules that can be used to perform the methods described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for brevity, will not be repeated here.

[0235] The embodiments of the present disclosure also propose a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions are executed by a processor to implement the above method. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0236] The embodiments of the present disclosure also propose an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0237] The embodiments of the present disclosure further provide a computer program product, comprising computer readable code, or a non-volatile computer readable storage medium carrying computer readable code, when the computer readable code is run in the processor of the electronic device, the processor in the electronic device performs the above method.

[0238] Figure 8 is a block diagram of an object classification apparatus 1900 based on deep reinforcement learning under complex noise conditions according to an exemplary embodiment. For example, the apparatus 1900 can be provided as a server or a terminal device. Referring to Figure 8 , the apparatus 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932, for storing instructions executable by the processing component 1922, such as an application program. The application program stored in the memory 1932 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above method.

[0239] The apparatus 1900 can also include a power supply component 1926 configured to perform power management of the apparatus 1900, a wired or wireless network interface 1950 configured to connect the apparatus 1900 to a network, and an input output interface 1958 (I / O interface). The apparatus 1900 can operate based on an operating system stored in the memory 1932, such as Windows Server TM , MacOS X TM , Unix TM , Linux TM , FreeBSD TM or the like.

[0240] In exemplary embodiments, a non-volatile computer readable storage medium, such as the memory 1932 including computer program instructions executable by the processing component 1922 of the apparatus 1900 to complete the above method is also provided.

[0241] The above has described the embodiments of the present disclosure, the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes are obvious to those skilled in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles, practical applications or technical improvements in the art of the embodiments, or to enable other ordinary skilled persons in the art to understand the embodiments disclosed herein.

Claims

1. A method for object classification based on deep reinforcement learning under complex noise conditions, characterized in that, The method includes: Acquire data to be classified, the data to be classified including noisy device signals; Based on the noise classification model obtained by deep reinforcement learning, the data to be classified is classified into noise to obtain the noise classification corresponding to the data to be classified. The noise reduction algorithm corresponding to the noise classification is called to perform noise reduction processing on the data to be classified, and the noise-reduced data is obtained; wherein, different noise classifications correspond to different noise reduction algorithms; Based on an object classification model obtained through deep reinforcement learning, the denoised data is classified into objects to obtain object classification results; the object classification model is used for fault diagnosis of equipment; the object classification includes fault signal classification and normal signal classification. The noise classification model and / or the object classification model are obtained by deep reinforcement learning based on unlabeled sample data. During the deep reinforcement learning process based on unlabeled sample data, reward information is fed back based on the relationship between the sample data and the cluster center. The deep reinforcement learning based on unlabeled sample data includes: The number of cluster centers A is determined based on the number of classifications. The cluster centers are initially set to the A sample data points that are farthest from the sample; where A is an integer greater than 1. For each sample data, determine the sample distance between the sample data and each cluster center, and determine the index of the cluster center with the smallest sample distance to the sample data as the cluster label of the sample data; A data environment is constructed based on each sample data and the corresponding cluster label of the sample data; Deep reinforcement learning is performed through the interaction between the intelligent agent and the data environment to obtain a trained reinforcement learning model; After this round of training is completed, the cluster centers are updated based on the mean of the sample data corresponding to each cluster center. This triggers the execution of the following steps: for each sample data, determining the sample distance between the sample data and each cluster center, and determining the index of the cluster center with the smallest sample distance to the sample data as the cluster label of the sample data, and so on, until the training rounds reach the preset requirements and training stops.

2. The method according to claim 1, characterized in that, The sample data used to train the noise classification model is noisy sample data; the sample data used to train the object classification model is denoised sample data.

3. The method according to claim 1, characterized in that, The process of performing deep reinforcement learning through the interaction between the agent and the data environment to obtain a trained reinforcement learning model includes: The data environment sequentially returns the i-th sample data to the agent; the agent includes a reinforcement learning model, which includes a deep Q-network and a target network, and the network parameters of the deep Q-network and the target network are initialized to be the same; i is a positive integer from 1 to n, and n is the total number of sample data; The i-th classification action corresponding to the i-th sample data is determined by the deep Q-network; wherein the i-th classification action indicates one of the cluster labels. The agent returns the i-th classification action to the data environment, which then determines the i-th reward for the i-th classification action based on the comparison result between the i-th classification action and the cluster label corresponding to the i-th sample data, and returns the i-th reward and the (i+1)-th sample data to the agent; if the comparison result is the same, the reward is... If the comparison results are different, the reward is -1; where, This represents the sample distance between the i-th sample data and the corresponding cluster center; The i-th experience data is stored in the experience pool of the agent, and i=i+1 is updated to trigger the execution of the step of sequentially returning the i-th sample data to the agent through the data environment and the subsequent steps; the i-th experience data includes the i-th sample data, the i-th classification action, the i-th reward and the i+1-th sample data; After traversing n sample data, K sets of experience data are randomly selected from the experience pool; where K is a positive integer. Each set of empirical data is input into the deep Q-network and the target network respectively, so as to update the deep Q-network based on the difference between the first network result of the deep Q-network and the second network result of the target network. For each W sets of experience data traversed, the network parameters of the deep Q network are synchronized to the target network until all sets of experience data have been traversed, thus obtaining the network parameters of the deep Q network obtained in this round of training, where W is a positive integer.

4. The method according to claim 1, characterized in that, When the number of classifications corresponding to the noise classification model and / or the object classification model is less than or equal to a preset threshold, it is obtained by deep reinforcement learning based on unlabeled sample data.

5. The method according to claim 1, characterized in that, The noise classification model is obtained through deep reinforcement learning based on labeled sample data, and the labeled sample data includes sample data for a first noise category and sample data for a second noise category, wherein the number of sample data for the first noise category is less than the number of sample data for the second noise category; accordingly, In deep reinforcement learning based on labeled sample data, the reward for interaction between the data environment and the agent is determined by the following formula: ； Among them, s t This represents the current sample data returned by the data environment to the agent, a. t Indicates based on the current sample data s t The classification action performed Indicates based on the current sample data s t Rewards for performing categorization actions, l t Indicates the current state s t The corresponding noise classification label, The dataset represents the sample data for the first noise classification. The dataset representing the sample data for the second noise classification. This represents the ratio of the number of sample data in the first noise category to the number of sample data in the second noise category.

6. The method according to claim 1, characterized in that, The deep reinforcement learning process of the object classification model includes: Obtain a training set, which includes noisy sample data; Based on the noise classification model, the noisy sample data is classified to obtain the noise classification of each noisy sample data. The noise reduction algorithm corresponding to the noise classification is called to perform noise reduction processing on the noisy sample data to obtain the noise-reduced sample data; The object classification model is obtained by performing deep reinforcement learning on the denoised sample data.

7. The method according to claim 1, characterized in that, The step of calling the noise reduction algorithm corresponding to the noise classification to perform noise reduction processing on the data to be classified, and obtaining the noise-reduced data, includes: Obtain the correspondence between noise classification and the optimal wavelet threshold denoising algorithm; The data to be classified is subjected to wavelet threshold denoising according to the optimal wavelet threshold denoising algorithm corresponding to the noise classification, and the denoised data is obtained.

8. The method according to claim 7, characterized in that, The method for obtaining the correspondence between noise classification and the optimal wavelet threshold denoising algorithm includes: For each noise category, wavelet threshold denoising is performed on the same noise data belonging to the noise category using different wavelet bases. The wavelet base with the best denoising effect is determined from the denoising results corresponding to different wavelet bases, and the optimal wavelet threshold denoising algorithm corresponding to the noise category is obtained.

9. An object classification device based on deep reinforcement learning under complex noise conditions, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of any one of claims 1 to 8 when executing instructions stored in the memory.

Images