Object classification method and device based on deep reinforcement learning under complex noise condition

Abstract

The invention relates to the technical field of computers, and discloses an object classification method and device based on deep reinforcement learning under a complex noise condition. The method comprises: acquiring to-be-classified data; performing noise classification on the to-be-classified data based on a noise classification model obtained by deep reinforcement learning to obtain noise classification corresponding to the to-be-classified data; a noise reduction algorithm corresponding to the noise classification is called to carry out noise reduction processing on the to-be-classified data, and the denoised data is obtained; performing object classification on the denoised data based on an object classification model obtained by deep reinforcement learning to obtain an object classification result; the problem that a traditional denoising algorithm is poor in noise removal effect can be solved. According to the method, the noise reduction algorithm can be adaptively switched to perform noise reduction processing under the condition of more noise types, so that the noise reduction effect of the to-be-classified data in a complex noise scene is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computers, and in particular to a method and device for object classification based on deep reinforcement learning under complex noise conditions. Background Art

[0002] Deep reinforcement learning has a wide range of applications in games, robot control and other fields, and is also widely used in object classification scenarios, such as equipment fault detection and diagnosis scenarios. In object classification based on deep reinforcement learning, in order to improve the performance of object classification, it is necessary to denoise the signal to be classified.

[0003] Traditional denoising methods include: using denoising algorithms to denoise the collected signals, effectively removing noise and interference signals, and then classifying objects based on deep reinforcement learning. Common denoising algorithms include but are not limited to: wavelet denoising and deep learning denoising, such as: analyzing the signal through three steps of parameter offline setting based on wavelet threshold denoising, Hilbert transform method and detection method based on principal component analysis, thus realizing turbine fault detection; or, wavelet threshold denoising based on genetic algorithm optimization, realizing rotating machinery fault diagnosis by combining variational mode decomposition; or, by using multi-layer convolutional neural networks to process different types of noise data, the purity of vibration signals is improved. Or, by using denoising fault-aware wavelet networks, wavelet convolution and dynamic hard thresholds are integrated to eliminate noise interference, thus realizing fault diagnosis under high noise conditions. Or, based on discrete graph wavelet frames, graph wavelet denoising convolutional neural networks realize signal denoising and fault diagnosis by extracting multi-scale features of graph structure data.

[0004] However, most of the above denoising algorithms only filter out single noise, while in the complex environment of actual production, the noise in the signal to be classified often exists in the form of mixed noise, and the noise removal effect of using traditional denoising algorithms is not good. Summary of the invention

[0005] In view of this, the present disclosure proposes an object classification method and device based on deep reinforcement learning under complex noise conditions. When there are many types of noise, the noise reduction algorithm can be adaptively switched to perform noise reduction processing, thereby improving the noise reduction effect of the data to be classified in complex noise scenes.

[0006] According to one aspect of the present disclosure, a method for object classification based on deep reinforcement learning under complex noise conditions is provided, the method comprising:

[0007] Obtain data to be classified;

[0008] Based on the noise classification model obtained by deep reinforcement learning, the data to be classified is subjected to noise classification to obtain the noise classification corresponding to the data to be classified;

[0009] Calling the noise reduction algorithm corresponding to the noise classification to perform noise reduction processing on the data to be classified to obtain noise-reduced data; wherein different noise classifications correspond to different noise reduction algorithms;

[0010] Based on the object classification model obtained by deep reinforcement learning, object classification is performed on the denoised data to obtain an object classification result.

[0011] In a possible implementation, the noise classification model is obtained by deep reinforcement learning based on unlabeled sample data, and the sample data used to train the noise classification model is sample data with noise; and / or, the object classification model is obtained by deep reinforcement learning based on unlabeled sample data, and the sample data used to train the object classification model is sample data after noise reduction;

[0012] The deep reinforcement learning based on unlabeled sample data includes:

[0013] Determine the number of cluster centers A based on the number of classifications, and initialize and set the cluster centers to be A sample data with the farthest sample distance; A is an integer greater than 1;

[0014] For each sample data, determine the sample distance between the sample data and each cluster center, and determine the index of the cluster center with the smallest sample distance to the sample data as the cluster label of the sample data;

[0015] Building a data environment based on each sample data and a cluster label corresponding to the sample data;

[0016] Performing deep reinforcement learning through interaction between the intelligent agent and the data environment to obtain a trained reinforcement learning model;

[0017] After this round of training is completed, based on the mean of the sample data corresponding to each cluster center, the cluster center is updated, triggering the execution of the steps of determining the sample distance between the sample data and each cluster center for each sample data, and determining the index of the cluster center with the smallest sample distance to the sample data as the cluster label of the sample data, and subsequent steps, until the training round reaches the preset requirements and stops the training.

[0018] In a possible implementation, performing deep reinforcement learning through interaction between the intelligent agent and the data environment to obtain a trained reinforcement learning model includes:

[0019] The i-th sample data is returned to the agent in sequence through the data environment; the agent includes a reinforcement learning model, the reinforcement learning model includes a deep Q network and a target network, and the network parameters of the initialized deep Q network are the same as the network parameters of the target network; i is a positive integer ranging from 1 to n in sequence, and n is the total number of the sample data;

[0020] Determine the i-th classification action corresponding to the i-th sample data through the deep Q network; wherein the i-th classification action indicates one of the cluster labels;

[0021] The agent returns the i-th classification action to the data environment, so that the data environment determines the i-th reward of the i-th classification action based on the comparison result between the i-th classification action and the cluster label corresponding to the i-th sample data, and returns the i-th reward and the i+1-th sample data to the agent; if the comparison result is the same, the reward is If the comparison result is different, the reward is -1; represents the sample distance between the i-th sample data and the corresponding cluster center;

[0022] The i-th experience data is stored in the experience pool of the agent, and i=i+1 is updated to trigger the execution of the step of returning the i-th sample data to the agent in sequence through the data environment and subsequent steps; the i-th experience data includes the i-th sample data, the i-th classification action, the i-th reward and the i+1-th sample data;

[0023] After the traversal of n sample data is completed, K groups of experience data are randomly selected from the experience pool; K is a positive integer;

[0024] Inputting each set of experience data into the deep Q network and the target network respectively, so as to update the deep Q network based on the difference between the first network result of the deep Q network and the second network result of the target network;

[0025] Every time W groups of experience data are traversed, the network parameters of the deep Q network are synchronized to the target network until the traversal of each group of experience data is completed, and the network parameters of the deep Q network obtained in this round of training are obtained, where W is a positive integer.

[0026] In a possible implementation, when the number of classifications corresponding to the noise classification model and / or the object classification model is less than or equal to a preset threshold, deep reinforcement learning is performed based on unlabeled sample data to obtain the model.

[0027] In a possible implementation, the noise classification model is obtained by deep reinforcement learning based on labeled sample data, and the labeled sample data includes sample data of a first noise classification and sample data of a second noise classification, and the number of sample data of the first noise classification is less than the number of sample data of the second noise classification; accordingly,

[0028] In the process of deep reinforcement learning based on labeled sample data, the reward when the data environment interacts with the agent is determined by the following formula:

[0029]

[0030] Among them, s t represents the current sample data returned by the data environment to the agent, a t Indicates that based on the current sample data s t The classification action performed, R(s t ,a t ) indicates that based on the current sample data s t The reward for performing the classification action, l t Indicates the current state s t The corresponding noise classification label, D minority Denotes the dataset consisting of sample data of the first noise classification, D majority represents a data set consisting of sample data of the second noise classification, and ρ represents the ratio of the number of sample data of the first noise classification to the number of sample data of the second noise classification.

[0031] In a possible implementation, the deep reinforcement learning process of the object classification model includes:

[0032] Obtaining a training set, wherein the training set includes sample data with noise;

[0033] Performing noise classification on the sample data with noise based on the noise classification model to obtain a noise classification of each sample data with noise;

[0034] Calling a noise reduction algorithm corresponding to the noise classification to perform noise reduction processing on the sample data with noise to obtain noise-reduced sample data;

[0035] Deep reinforcement learning is performed based on the denoised sample data to obtain the object classification model.

[0036] In a possible implementation, the calling of a noise reduction algorithm corresponding to the noise classification to perform noise reduction processing on the data to be classified to obtain noise-reduced data includes:

[0037] Obtain the correspondence between noise classification and optimal wavelet threshold denoising algorithm;

[0038] The data to be classified is subjected to wavelet threshold denoising according to the optimal wavelet threshold denoising algorithm corresponding to the noise classification to obtain the denoised data.

[0039] In a possible implementation, the obtaining of the correspondence between the noise classification and the optimal wavelet threshold denoising algorithm includes:

[0040] For each noise classification, different wavelet bases are used to perform wavelet threshold denoising on the same noise data belonging to the noise classification, and the wavelet base with the best denoising effect is determined from the denoising results corresponding to different wavelet bases to obtain the optimal wavelet threshold denoising algorithm corresponding to the noise classification.

[0041] In a possible implementation, the object classification model is used to perform fault diagnosis on a device; accordingly, the signal to be classified includes a device signal with noise; and the object classification includes fault signal classification and normal signal classification.

[0042] According to another aspect of the present disclosure, there is provided an object classification device based on deep reinforcement learning under complex noise conditions, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0043] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0044] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0045] By obtaining the data to be classified; performing noise classification on the data to be classified based on the noise classification model obtained by deep reinforcement learning to obtain the noise classification corresponding to the data to be classified; calling the denoising algorithm corresponding to the noise classification to perform denoising on the data to be classified to obtain the denoised data; performing object classification on the denoised data based on the object classification model obtained by deep reinforcement learning to obtain the object classification result; the problem of poor noise removal effect of traditional denoising algorithms can be solved; because before the object classification of the data to be classified is performed, the noise classification of the data to be classified is first determined, and then the denoising algorithm corresponding to the noise classification is called to perform denoising, when there are many noise types, the denoising algorithm can be adaptively switched for denoising, thereby improving the denoising effect of the data to be classified in complex noise scenes.

[0046] At the same time, using the noise classification model obtained based on deep reinforcement learning to perform noise classification on the unclassified data can avoid the problem of poor performance of the noise classification model caused by imbalanced noise samples. Using the object classification model obtained based on deep reinforcement learning to perform object classification on the denoised data can avoid the problem of poor performance of the object classification model caused by imbalanced object classification samples, thereby improving the accuracy of noise classification and object classification in scenarios with imbalanced samples.

[0047] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0049] Figure 1 A schematic diagram showing the principle of a Markov decision framework according to an embodiment of the present disclosure;

[0050] Figure 2 A flowchart showing an object classification method based on deep reinforcement learning under complex noise conditions according to an embodiment of the present disclosure is shown;

[0051] Figure 3 A schematic diagram showing a training process of an object classification model according to an embodiment of the present disclosure;

[0052] Figure 4 A schematic diagram showing expanded sample data according to an embodiment of the present disclosure;

[0053] Figure 5 A schematic diagram showing a curve diagram of changes in the total reward obtained by an agent during training according to an embodiment of the present disclosure;

[0054] Figure 6A schematic diagram showing a curve of changes in accuracy during training according to an embodiment of the present disclosure;

[0055] Figure 7 A block diagram of an object classification device based on deep reinforcement learning under complex noise conditions according to an embodiment of the present disclosure is shown;

[0056] Figure 8 A block diagram of an object classification device based on deep reinforcement learning under complex noise conditions according to another embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0057] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0058] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0059] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0060] First, several terms involved in this application are introduced.

[0061] Deep Reinforcement Learning (DRL) is a combination of reinforcement learning (RL) and deep learning (DL). Specifically, it can include reinforcement learning and neural networks. Reinforcement learning is a machine learning method in which an intelligent agent learns the optimal behavior strategy through interaction with the environment. Neural networks are used to model the mapping between states and actions and realize the processing of high-dimensional data. In reinforcement learning, the intelligent agent explores the environment by trying different actions and adjusts the strategy based on the feedback (reward) from the environment to maximize the long-term accumulated reward value. The intelligent agent can be a robot, a software program, or any entity that can interact with the environment and take actions.

[0062] Combined with the powerful advantages of deep reinforcement learning algorithms in board games, the object classification problem can be regarded as a guessing game: at each time step, the agent receives an environment state s represented by sample data, and then performs action a for classification under the guidance of the strategy. The environment returns a reward r and the next sample s' to the agent. The experience data consisting of the environment state s, action a, reward r and the next sample s' is stored in the agent's experience pool. After traversing all the sample data, multiple groups of experience data are randomly extracted from the experience pool to train the deep Q network in the agent. The interaction process between the agent and the environment is shown in the figure. Figure 1 As shown in Figure 2, the agent receives a reward r when performing an action. When the agent learns the optimal strategy from the interaction with the environment to obtain the maximum cumulative reward, the deep Q network in the agent can correctly classify the samples.

[0063] Will Figure 1 The Markov decision framework shown in the figure is transformed into a sequential decision problem, and a classification model based on a deep Q network (including the noise classification model and object classification model described below) can be obtained. Assume that the training data set is D = {x 1 ,x 2 ,…,x n}, where x i Represents the i-th sample in the training data set, then in the classification model based on deep reinforcement learning:

[0064] State S: Determined by the sample. At the beginning of training, the agent will receive the first sample x 1 As its initial state s 1 , the state s of the environment at each time step t will correspond to sample x t When a new episode begins, the environment randomly shuffles the order of samples in the training dataset, so that the order of samples fed back to the agent is different during different rounds of training. An episode is a transition trajectory from the initial state to the terminal state.

[0065] Action A: related to the category of the training data set. In the classification problem, the action taken by the agent is to judge the classification. For example: for the noise classification problem, A = (0, 1, 2), where 0, 1, and 2 represent three noise categories respectively; for another example: for the fault detection problem, A = (0, 1), where 0 represents normal and 1 represents fault.

[0066] Reward R: Feedback from the environment. The environment measures the success or failure of the agent’s behavior by giving back rewards.

[0067] Strategy π θ: Mapping function π: S→A, which means that the mapping function π is a function that maps the state S to the action A; where π θ (a t |s t ) indicates that the agent is in state s t Next, perform action a t probability.

[0068] In the classification model, the mapping function π is a function that returns the probability of all categories A based on the sample S. The mapping function π can be expressed as follows:

[0069] π(a|s)=P(a t =a|s t =s)

[0070] Among them, s t represents the sample returned by the environment to the agent at the tth time step, a t Indicates that the agent is based on sample s t The action to be performed, π(a|s) represents the probability of performing action a based on the current sample s, and the value of t is a positive integer from 1 to n, where n is the total number of samples.

[0071] When the agent correctly classifies the sample, the environment gives the agent a reward, which can be obtained by maximizing the cumulative reward g t Function, to achieve the goal of correctly classifying samples, g t The function can be expressed as follows:

[0072]

[0073] Among them, the discount factor γ∈[0, 1] is used to balance the current reward and future reward, r t+k represents the reward corresponding to the t+kth time step, t represents the tth time step, the value of t is a positive integer from 1 to n, and n is the total number of samples.

[0074] According to the Bellman equation, the action-value function (i.e., Q function) Q π (s,a)=E π [g t |s t =s,a t =a] can be expressed as:

[0075] Q π (s,a)=E π [r t +γQ π (s t+1 ,a t+1 )|s t =s,a t=a];

[0076] Among them, Q π (s,a) represents the action value (i.e., Q value) of the mapping function π based on sample s to perform action a, s t represents the sample returned by the environment to the agent at the tth time step, a t Indicates that the agent is based on sample s t Action performed, g t is the cumulative return function, r t represents the reward corresponding to the t-th time step fed back by the environment to the agent, γ represents the discount factor, and Q π (s t+1 ,a t+1 ) represents the mapping function π based on sample s t+1 Execute action a t+1 Q value, E π [] indicates calculating the expected value of the data in brackets.

[0077] According to the above definition and notation, the classification problem can be defined as finding an optimal strategy π * :S→A maximizes the cumulative return. The agent maximizes the cumulative return by solving the optimal strategy. The strategy π under the optimal Q function * That is the optimal strategy for the classification model, strategy π * It can be expressed by the following formula:

[0078]

[0079] Q * (s,a)=E π [r t +γmax a Q * (s t+1 ,a t+1 )|s t =s,a t =a];

[0080] Among them, π * (a|s) represents the probability of the classification model performing different actions a based on the current sample s, argmax a Q * (s,a) means finding the action a that can maximize the Q value under sample s. That is, if action a can make Q * (s,a) is the largest, then the strategy π * Determine the probability value corresponding to action a is 1; if action a cannot make Q * (s,a) is the largest, then the strategy π * Determine the probability value corresponding to action a is 0; Q *(s,a) is the action value function for calculating Q value, E π [] indicates the expected value of the data in brackets; Q * (s t+1 ,a t+1 ) represents the policy function π * Based on samples t+1 Execute action a t+1 Q value, s t represents the sample returned by the environment to the agent at the tth time step, a t Indicates that the agent is based on sample s t The action performed, r t represents the reward corresponding to the tth time step fed back by the environment to the agent, and γ represents the discount factor.

[0081] In the deep reinforcement learning classification algorithm of the present application, the Q function is fitted with a deep neural network to obtain a deep Q network. According to the experience replay mechanism, multiple sets of experience data are stored in the experience pool B. Each set of experience data can be expressed as: (s, a, r, s′), where s represents the sample corresponding to a time step, a represents the action performed based on the sample in the time step, r represents the reward obtained by performing action a in the time step, and s′ represents the sample corresponding to the next time step. Afterwards, the following steps of gradient descent are performed on the deep Q network according to the loss function:

[0082] L(θ k )=∑ (s,a,r,s′)∈B (yQ(s,a;θ k )) 2 ;

[0083] Among them, L(θ k ) indicates that the model parameter is θ k The loss function corresponding to the deep Q network; Q(s,a;θ k ) indicates that the deep neural network has model parameters θ k In the case of , the Q value is determined based on sample s and action a. y represents the true value of the Q function, and the value of y can be determined based on the value of the target network, which can be specifically expressed by the following formula:

[0084] y=r+γmax a′ Q(s′,a′;θ k-1 );

[0085] Among them, r represents the reward for performing action a based on sample s, γ represents the discount factor, s' represents the next sample, a' represents the action performed based on the next sample s', and max a′ Q(s′,a′;θ k-1 ) indicates that the target network has model parameters θ k-1In the case of , the maximum Q value is determined based on the next sample s' and action a'.

[0086] The derivative of the loss function with respect to θ can determine the adjustment direction of the model parameter θ in the deep Q network to minimize the loss function, thereby optimizing the prediction performance of the deep Q network. The derivative of the loss function with respect to θ can be expressed as follows:

[0087]

[0088] in, Represents the gradient of the loss function with respect to θ. The gradient is used to indicate the direction in which the loss function value increases fastest. When updating the model parameter θ in the deep Q network, the model parameter θ is updated along the opposite direction of the gradient (i.e., the negative gradient direction).

[0089] In this application, the loss function L(θ k ) to obtain the optimal Q function Q * , thereby obtaining the optimal strategy, achieving the maximum cumulative reward, and realizing object classification based on deep reinforcement learning.

[0090] Wavelet threshold denoising: It is a signal denoising method that decomposes noise data into sub-signals of different frequencies through wavelet transform, then thresholds the wavelet coefficients, and finally performs inverse wavelet transform to obtain the denoised signal. Different wavelet bases will affect the denoising effect. Several common wavelet bases include but are not limited to: Haar wavelet, Daubechies (dbN) wavelet, Symmetric (Symlets, symN) wavelet, Cosine integral (Coiflets, coifN) wavelet, Biorthogonal (BiorNr.Nd) wavelet, etc. This application will not list them one by one here.

[0091] Below, the object classification method based on deep reinforcement learning under complex noise conditions provided by the present application is described in detail. The application scenarios of the object classification method provided by the present application include but are not limited to: fault diagnosis scenarios (i.e., classifying whether the classified signal is a fault signal and the fault type), text classification scenarios (i.e., determining which text classification the classified signal belongs to), etc. This embodiment does not limit the application scenarios of the object classification method.

[0092] Taking fault diagnosis as an example, rotating machinery, as the most common mechanical component in the industrial field, is widely used in many key fields such as aviation, aerospace, transportation, and intelligent manufacturing. Once a serious fault occurs in a rotating machinery, it will lead to a series of chain reactions, causing huge economic losses and even casualties. Therefore, it is of great significance to establish a fast and effective fault detection mechanism, discover mechanical faults as early as possible, and handle faults in time to ensure the safe operation of rotating machinery and enhance the reliability and stability of rotating machinery.

[0093] In actual work, fault detection technology faces huge challenges. On the one hand, rotating machinery is in normal operation most of the time, and fault data is far less than normal data, so fault detection faces serious imbalance problems. On the other hand, in the complex environment of actual production, the operating state of mechanical equipment will inevitably be disturbed by the external environment, generating vibration signals containing various noises, making fault characteristics easily disturbed or even covered up.

[0094] Object classification methods in other scenarios also have the above-mentioned problems of complex noise signals and unbalanced sample data, which will not be elaborated in detail in this embodiment.

[0095] In recent years, deep reinforcement learning has demonstrated strong decision-making capabilities and intelligent application prospects in the fields of games and robot control. For deep reinforcement learning fault detection and diagnosis problems, the intelligent agent can use the deep network to extract data features and interact with the environment to perform self-motivated reinforcement learning to optimize the fault detection and diagnosis strategy. In this process, the learning performance is evaluated by the overall reward of the data. In other words, during the learning process, the intelligent agent cannot distinguish which samples are not well learned from the input signal. This method forces the intelligent agent to discover the inherent differences between fault modes, which makes deep reinforcement learning have great potential in fault detection and diagnosis. For example, the existing deep feature-enhanced reinforcement learning method for rolling bearing fault diagnosis breaks the reliance of previous fault diagnosis methods on model structure and successfully obtains higher recognition accuracy; or, the deep reinforcement learning fault diagnosis algorithm based on the gated recurrent unit (GRU) is successfully applied to nuclear power plant fault diagnosis; or, the hybrid combination of reinforcement learning and multi-agent system successfully detects local and global abnormal behaviors in microservice architecture; or, the classification problem is transformed into a sequential decision problem, and the reward function is constructed according to the imbalance ratio, and the deep reinforcement learning imbalance classification algorithm model is proposed; or, the reward function is improved according to the principle of K-Means algorithm, and it is experimentally verified that it works well in rolling bearing fault diagnosis; or, a dual experience pool deep reinforcement learning model solves the imbalance problem in fault diagnosis through balanced cross sampling. The above existing fault diagnosis methods reflect the advantages of deep reinforcement learning in fault detection and diagnosis in terms of solving complex working conditions and data imbalance problems.

[0096] Based on this, in this application, deep reinforcement learning is applied to noise classification and object classification. The noise classification model obtained based on deep reinforcement learning is used to perform noise classification on the data to be classified, which can avoid the problem of poor performance of the noise classification model caused by imbalanced noise samples. The object classification model obtained based on deep reinforcement learning is used to classify the denoised data, which can avoid the problem of poor performance of the object classification model caused by imbalanced object classification samples, and improve the accuracy of noise classification and object classification in sample imbalance scenarios. At the same time, by first determining the noise classification of the data to be classified before performing object classification on the data to be classified, and then calling the noise reduction algorithm corresponding to the noise classification to perform noise reduction processing, it is possible to adaptively switch the noise reduction algorithm to perform noise reduction processing when there are many types of noise, thereby improving the noise reduction effect of the data to be classified in complex noise scenarios.

[0097] Figure 2The flowchart of the object classification method based on deep reinforcement learning under complex noise conditions according to an embodiment of the present disclosure is shown. In this embodiment, the method is described by using an electronic device with processing capabilities such as a user terminal or a server as an example. The user terminal includes but is not limited to: a computer, a tablet computer, a mobile phone, etc. This embodiment does not limit the implementation method of the electronic device. Figure 2 As shown, the method includes:

[0098] Step 201, obtaining data to be classified.

[0099] The data to be classified refers to the data to be classified as an object. For different object classification scenarios, the data content of the data to be classified may be different. For example, in a fault diagnosis scenario, the data to be classified may be a device signal (such as a vibration signal of a motor, etc.); in a text classification scenario, the data to be classified may be a text to be classified. Generally, the data to be classified includes noise data. Among them, noise data refers to data that will be interfered by noise when the data to be classified is classified as an object. Optionally, when collecting the data to be classified, the noise data may include multiple types, such as Gaussian noise, gamma noise, etc. This embodiment does not limit the classification of noise data.

[0100] The data to be classified may be collected by an electronic device or sent by other electronic devices. This embodiment does not limit the method for obtaining the data to be classified.

[0101] Step 202: Based on the noise classification model obtained by deep reinforcement learning, noise classification is performed on the data to be classified to obtain the noise classification corresponding to the data to be classified.

[0102] The noise classification model is used to determine the noise classification of the data to be classified. In this embodiment, the noise classification model is obtained based on deep reinforcement learning. In this way, the noise classification performance of the noise classification model can be guaranteed when the number of noise samples corresponding to each noise classification is uneven. The specific model training process is referred to the embodiment below, and this embodiment will not be repeated here.

[0103] Step 203: Call the noise reduction algorithm corresponding to the noise classification to perform noise reduction processing on the data to be classified to obtain noise-reduced data.

[0104] Among them, different noise classifications correspond to different noise reduction algorithms.

[0105] In one example, a denoising algorithm corresponding to a noise classification is called to perform denoising on the classified data to obtain denoised data, including: obtaining a correspondence between the noise classification and an optimal wavelet threshold denoising algorithm; performing wavelet threshold denoising on the classified data according to the optimal wavelet threshold denoising algorithm corresponding to the noise classification to obtain denoised data.

[0106] The correspondence between the noise classification and the optimal wavelet threshold denoising algorithm is pre-stored in the electronic device. The method for obtaining the correspondence includes: for each noise classification, using different wavelet bases to perform wavelet threshold denoising on the same noise data belonging to the noise classification, determining the wavelet base with the best denoising effect from the denoising results corresponding to different wavelet bases, and obtaining the optimal wavelet threshold denoising algorithm corresponding to the noise classification. Afterwards, the correspondence stored in the electronic device is read to obtain the optimal wavelet threshold denoising algorithm corresponding to the noise classification determined by the current noise classification model, and the data denoising effect can be improved by performing denoising on the data to be classified through the optimal wavelet threshold denoising algorithm.

[0107] Optionally, different wavelet bases include but are not limited to: Haar wavelet, dbN wavelet, symN wavelet, coifN wavelet, biorNr.Nd wavelet, etc. This embodiment does not limit the implementation methods of different wavelet bases.

[0108] In other embodiments, different noise reduction algorithms may also include singular value decomposition (SVD) noise reduction, deep learning noise reduction, etc. This embodiment does not limit the implementation method of the noise reduction algorithm.

[0109] Step 204: Based on the object classification model obtained by deep reinforcement learning, the denoised data is subjected to object classification to obtain an object classification result.

[0110] The object classification model is used to determine the noise classification of the denoised data. In this embodiment, the object classification model is obtained based on deep reinforcement learning. In this way, the object classification performance of the object classification model can be guaranteed when the number of samples corresponding to each object classification is unbalanced. The specific model training process is referred to the embodiment below, and this embodiment will not be repeated here.

[0111] In summary, the object classification method based on deep reinforcement learning under complex noise conditions provided in this embodiment obtains the data to be classified; based on the noise classification model obtained by deep reinforcement learning, the data to be classified is noise classified to obtain the noise classification corresponding to the data to be classified; the denoising algorithm corresponding to the noise classification is called to perform denoising on the data to be classified to obtain the denoised data; based on the object classification model obtained by deep reinforcement learning, the denoised data is object classified to obtain the object classification result; the problem of poor noise removal effect of traditional denoising algorithms can be solved; because before the data to be classified is object classified, the noise classification of the data to be classified is first determined, and then the denoising algorithm corresponding to the noise classification is called to perform denoising, it is possible to adaptively switch the denoising algorithm for denoising when there are many noise types, thereby improving the denoising effect of the data to be classified in complex noise scenes.

[0112] At the same time, using the noise classification model obtained based on deep reinforcement learning to perform noise classification on the unclassified data can avoid the problem of poor performance of the noise classification model caused by imbalanced noise samples. Using the object classification model obtained based on deep reinforcement learning to perform object classification on the denoised data can avoid the problem of poor performance of the object classification model caused by imbalanced object classification samples, thereby improving the accuracy of noise classification and object classification in scenarios with imbalanced samples.

[0113] In some scenarios, the label information of the sample data used to train the noise classification model and / or the object classification model may be unknown. For example, the label information of the machine operation status is mostly unknown, and obtaining the fault type label depends on expert experience, which is cumbersome to collect. In this case, the cost of supervised model training is often very high. Therefore, the object classification method directly facing unlabeled, unbalanced, and multi-noise data has greater application value and broader application prospects.

[0114] However, in traditional deep reinforcement learning methods, intelligent agents mostly obtain rewards based on whether their actions are consistent with sample labels, and cannot get rid of their dependence on labels.

[0115] Based on this, in some traditional methods, fault detection and diagnosis can be performed based on unsupervised training methods. Traditional unsupervised fault detection and diagnosis methods mainly perform fault identification through cluster analysis. For example, the k-means clustering algorithm is used to realize the fault identification of bearings and centrifugal compressors respectively. Alternatively, the sequential two-stage clustering method is used to perform fault diagnosis on fan coil units, and good results have been achieved. Alternatively, the principal component analysis is combined with the K-Means clustering algorithm to automatically detect and diagnose faults of unlabeled data. Alternatively, the fuzzy logic controller and fuzzy mean clustering method are used in combination with the expert system to detect and diagnose early transformer faults. Although the above traditional unsupervised training methods have achieved good results, most of them do not consider the impact of unbalanced clusters on clustering results, and cannot be used to directly analyze unbalanced fault detection and diagnosis data. Therefore, improving the clustering performance of the algorithm when processing unbalanced data sets is an indispensable step to enhance the efficiency of unlabeled fault detection.

[0116] Based on this, in the present application, the noise classification model and / or object classification model is obtained by deep reinforcement learning based on unlabeled sample data, and reward information is fed back based on the relationship between the sample and the cluster center to improve the performance of unsupervised model training.

[0117] Specifically, deep reinforcement learning based on unlabeled sample data includes the following steps:

[0118] Step 1: Determine the number of cluster centers A based on the number of categories, and initialize the cluster centers to be the A sample data with the farthest sample distance; A is an integer greater than 1.

[0119] The number of classifications refers to the total number of classifications corresponding to the model (noise classification model and / or object classification model), and the value of A is equal to the number of classifications. For example, the noise classification model corresponds to 3 noise classifications, namely Gaussian noise, gamma noise and Rayleigh noise, then the number of classifications is 3, and accordingly, A = 3. Another example: the object classification model corresponds to 2 object classifications, namely normal and fault, then the number of classifications is 2, and accordingly, A = 2.

[0120] The sample distance is used to indicate the similarity between different sample data. Since the similarity between sample data of different categories is small, by initializing the A sample data with the farthest sample distance as the cluster center, the accuracy of the initialization to determine the cluster center can be improved as much as possible and the number of updates of the cluster center can be reduced.

[0121] Optionally, the sample distance may be Manhattan distance, Euclidean distance, etc. This embodiment does not limit the method for obtaining the sample distance.

[0122] Initializing and setting the cluster center to the A sample data with the farthest sample distances includes: for each sample data, determining the sample distance between the sample data and other sample data; and determining the initial cluster center based on the sample data with the farthest sample distances.

[0123] Since the calculation speed of Manhattan distance is faster, in this embodiment, the sample distance is taken as Manhattan distance as an example, assuming that each sample data is an n-dimensional vector, and determining the sample data x(x 1 ,x 2 ,…x n ) and other sample data y(y 1 ,y 2 ,…y n ) can be expressed as follows:

[0124]

[0125] Among them, d xy Represents the Manhattan distance between sample data x and sample data y, x k represents the k-th dimension vector in the sample data x, y k Represents the k-th dimension vector in the sample data y.

[0126] Schematically, when the number of categories is 2 aIn the case of , determining the initial cluster center based on the sample data with the farthest sample distance includes: determining the sample data corresponding to the largest first a sample distances as the initial cluster center. According to the above, each sample distance is determined based on two sample data, that is, each sample distance corresponds to two sample data. In this case, determining the sample data corresponding to the sample distance as the initial cluster center means: determining both sample data corresponding to the sample distance as the initial cluster center.

[0127] When the number of categories is 2 a +1, the initial cluster center is determined based on the sample data with the farthest sample distance, including: determining the sample data corresponding to the largest first a sample distances as the initial cluster center, and determining a sample data with the largest sample distance to the cluster center from the sample data that are not the cluster center, and determining the sample data as the initial cluster center.

[0128] Among them, determining a sample data with the largest sample distance from the cluster center from the sample data of the non-cluster center includes: for each sample data of the non-cluster center, determining the maximum distance value of the sample distance between the sample data and each cluster center; from the maximum distance values ​​corresponding to the sample data of each non-cluster center, determining the maximum value of the maximum distance value, and the sample data of the non-cluster center corresponding to the maximum value is: the sample data with the largest sample distance from the cluster center. For example: a is 1, then the cluster centers determined based on the sample data corresponding to the largest first a sample distances include two, 1 and 2, respectively, for each sample data x of the non-cluster center, determine the sample distances between x and cluster center 1 and cluster center 2 respectively, and select the maximum distance value; then, select the maximum value from the maximum distance values ​​corresponding to each sample data x, and the sample data x corresponding to the maximum value is also determined as the initial cluster center.

[0129] Step 2: For each sample data, determine the sample distance between the sample data and each cluster center, and determine the index of the cluster center with the smallest sample distance to the sample data as the cluster label of the sample data.

[0130] The method for determining the sample distance between the sample data and the cluster center is the same as the method for determining the sample distance between different numbers of samples. For related descriptions, refer to step 1, which will not be repeated in this embodiment.

[0131] The index of the cluster center is used to identify different cluster centers. For example, if A=2, the index of the cluster center includes {0, 1}, indicating different cluster centers. In other embodiments, if A=2, the index of the cluster center may also include {1, 2}. This embodiment does not limit the setting method of the index of the cluster center.

[0132] Assume that all cluster centers are represented by C i , i=0,…|A|-1, the index i of the cluster center to which the sample data belongs can be determined by the following formula:

[0133]

[0134] Among them, i represents the index of the cluster center, Represents sample data s and cluster center C i The sample distance between Indicates obtaining sample data s and each cluster center C i The minimum distance between samples.

[0135] Step 3: Build a data environment based on each sample data and the cluster label corresponding to the sample data.

[0136] Step 4: Perform deep reinforcement learning through the interaction between the intelligent agent and the data environment to obtain a trained reinforcement learning model.

[0137] In this application, an agent refers to a physical software program that interacts with a data environment, and the physical software program may be a process that runs the agent.

[0138] In one example, deep reinforcement learning is performed through the interaction between the agent and the data environment to obtain a trained reinforcement learning model, which includes the following steps:

[0139] Step 41, return the i-th sample data to the agent in sequence through the data environment; the agent includes a reinforcement learning model, the reinforcement learning model includes a deep Q network and a target network, and the network parameters of the initialized deep Q network are the same as the network parameters of the target network; i is a positive integer ranging from 1 to n, and n is the total number of sample data.

[0140] Assuming that the training set includes n unlabeled sample data, the cluster label corresponding to each sample data can be determined through step 2, and the corresponding relationship between each sample data and the cluster label can be obtained. At this time, the data environment randomly selects a sample data as the first sample data and returns it to the intelligent agent to interact with the intelligent agent. After that, it selects the sample data that has not been traversed as the second sample data and returns it to the intelligent agent to interact with the intelligent agent. This cycle is repeated until all the sample data in the training set are traversed and the interaction with the intelligent agent is completed.

[0141] Step 42: Determine the i-th classification action corresponding to the i-th sample data through the deep Q network.

[0142] The deep Q network is based on a neural network, which includes an input layer, a hidden layer, and an output layer. The dimension of the input layer is the same as the dimension of the sample data, and the dimension of the output layer is the same as the number of classifications of the model.

[0143] The i-th classification action indicates one of the cluster labels. For example, if the cluster labels include {0, 1}, then the i-th classification action is one of {0, 1}.

[0144] Step 43: The agent returns the i-th classification action to the data environment, so that the data environment can determine the i-th reward for the i-th classification action based on the comparison result between the i-th classification action and the cluster label corresponding to the i-th sample data, and return the i-th reward and the i+1th sample data to the agent; if the comparison result is the same, the reward is If the comparison result is different, the reward is -1. Specifically, the reward function in the data environment can be expressed as follows:

[0145]

[0146] in, represents the sample distance between the i-th sample data and the corresponding cluster center; s t represents the i-th sample data, a t represents the i-th classification action, R(s t ,a t ) represents the i-th reward, c t Represents the cluster label corresponding to the i-th sample data.

[0147] Step 44, store the i-th experience data through the experience pool in the agent, and update i=i+1 to trigger the execution of step 41; wherein the i-th experience data includes the i-th sample data, the i-th classification action, the i-th reward and the i+1-th sample data.

[0148] Step 45: After the traversal of n sample data is completed, K groups of experience data are randomly selected from the experience pool; K is a positive integer.

[0149] Step 46: Input each set of experience data into the deep Q network and the target network respectively, so as to update the deep Q network based on the difference between the first network result of the deep Q network and the second network result of the target network.

[0150] Specifically, the weight and bias value parameters of the deep Q network are trained by the stochastic gradient descent method to minimize the difference between the first network result and the second network result of the target network, thereby maximizing the Q function value and obtaining the optimal strategy of the deep Q network. The difference between the first network result and the second network result of the target network is calculated by a preset loss function, and the loss function can be established in the same way as L(θk ), in other embodiments, the loss function may also be implemented in other ways, and this embodiment does not limit the implementation of the loss function. The update direction of the deep Q network is updated by the loss function L(θ k ) The gradient determined by the derivative of θ is determined. When updating the model parameter θ in the deep Q network, the model parameter θ is updated along the opposite direction of the gradient (i.e., the negative gradient direction).

[0151] Step 47: After each traversal of W sets of empirical data, synchronize the network parameters of the deep Q network to the target network until all sets of empirical data are traversed, and obtain the network parameters of the deep Q network obtained in this round of training, where W is a positive integer.

[0152] Step 5: After this round of training is completed, the cluster center is updated based on the mean of the sample data corresponding to each cluster center, triggering the execution of step 2 until the training rounds meet the preset requirements and stop the training.

[0153] The deep Q network obtained after the last round of training is determined as the classification model obtained by unlabeled training.

[0154] An episode of training refers to the process in which the agent and the data environment interact to traverse each sample data in the training set and use K sets of experience data in the experience pool to complete the training of the deep Q network. In different rounds of training, the order in which the data environment returns each sample data to the agent is different.

[0155] Among them, the training rounds meeting the preset requirements include but are not limited to: the training rounds are equal to the preset training round threshold T; and / or the change in the loss function value after x consecutive rounds of training is less than the preset change threshold. This embodiment does not limit the implementation method of the preset conditions.

[0156] In this embodiment, a method for deep reinforcement learning classification for unlabeled, unbalanced, and multi-noise data is proposed. The method feeds back reward information based on the relationship between samples and cluster centers to improve the performance of unsupervised model training.

[0157] Optionally, based on the above embodiment, when the number of classifications corresponding to the noise classification model and / or the object classification model is less than or equal to a preset threshold, deep reinforcement learning is performed based on unlabeled sample data. The preset threshold may be 2, 3, etc. The specific value of the preset threshold may be determined according to different classification accuracy requirements. This embodiment does not limit the value of the preset threshold.

[0158] Taking the case that noise classification includes three types, namely Gaussian noise, gamma noise, and Rayleigh noise, and object classification includes two types, namely normal and fault, and the preset threshold is 2, at this time, if the number of classifications corresponding to the noise classification model is greater than the preset threshold, it is obtained by deep reinforcement learning based on labeled sample data, and the number of classifications corresponding to the object classification model is equal to the preset threshold, it is obtained by deep reinforcement learning based on unlabeled sample data.

[0159] In this embodiment, by selecting a deep reinforcement learning method for a noise classification model and / or an object classification model based on classification accuracy requirements, the training flexibility of the classification model can be improved.

[0160] Below, we take the example of performing deep reinforcement learning based on labeled sample data to obtain a noise classification model and performing deep reinforcement learning based on unlabeled sample data to obtain an object classification model. In the above embodiment, the sample data used to train the noise classification model is sample data with noise; the sample data used to train the object classification model is sample data after noise reduction.

[0161] Accordingly, deep reinforcement learning is performed based on labeled sample data to obtain a noise classification model, which includes the following steps:

[0162] Step 1: Obtain a training set consisting of sample data with noise, where the training set includes the sample data and a noise classification label corresponding to the sample data.

[0163] In one example, a method for obtaining a training set includes: adding noise data to original data that does not contain noise, and adding noise data of different classifications to different original data to obtain sample data; and setting a noise classification label corresponding to the sample data based on the noise classification of the noise data added to the sample data.

[0164] In other embodiments, the sample data in the training set may also be directly collected data with noise. Accordingly, the noise classification labels corresponding to the sample data may be obtained by manual annotation. This embodiment does not limit the method for obtaining the training set.

[0165] Step 2: Build a data environment based on the training set.

[0166] Step 3, return the i-th sample data to the agent in sequence through the data environment; the agent includes a reinforcement learning model, the reinforcement learning model includes a deep Q network and a target network, and the network parameters of the initialized deep Q network are the same as the network parameters of the target network; i is a positive integer ranging from 1 to n, and n is the total number of sample data.

[0167] Specifically, the state S that the data environment feeds back to the agent is: each sample data with noise data corresponds to a state, and the states at different time steps correspond to different samples.

[0168] Step 4: Determine the i-th classification action corresponding to the i-th sample data through the deep Q network.

[0169] The i-th classification action indicates one of the noise classifications. For example, classification action A = (0, 1, 2), where 0, 1, and 2 indicate three noise categories, respectively.

[0170] Step 5: Return the i-th classification action to the data environment through the intelligent agent, so that the data environment can determine the i-th reward of the i-th classification action based on the comparison result between the i-th classification action and the cluster label corresponding to the i-th sample data, and return the i-th reward and the i+1th sample data to the intelligent agent.

[0171] In some scenarios, most of the sample data contains Gaussian noise, and only a small part contains other noise. At this time, other noise is difficult to correctly identify in an unbalanced data set. In order to better identify data containing minority class noise and solve the problem of unbalanced noise data, the noise classification model should pay more attention to minority class noise data. Therefore, in this embodiment, the labeled sample data includes sample data of a first noise classification (such as other noise) and sample data of a second noise classification (such as Gaussian noise), and the number of sample data of the first noise classification is less than the number of sample data of the second noise classification.

[0172] Accordingly, in the process of deep reinforcement learning based on labeled sample data, the reward when the data environment interacts with the agent is determined by the following formula:

[0173]

[0174] Among them, s t represents the current sample data (i.e., the i-th sample) returned by the data environment to the agent, a t Indicates that based on the current sample data s t The classification action performed (i.e., the i-th classification action), R(s t ,a t ) indicates that based on the current sample data s t The reward for performing the classification action (i.e., the i-th reward), l t Indicates the current state s t The corresponding noise classification label, D minority Denotes the dataset consisting of sample data of the first noise classification, D majority represents a data set consisting of sample data of the second noise classification, and ρ represents the ratio of the number of sample data of the first noise classification to the number of sample data of the second noise classification.

[0175] At this time, when the agent correctly or incorrectly classifies the sample data containing the second noise classification, the reward is ρ or -ρ; when the agent correctly or incorrectly classifies the sample data containing the first noise classification, the reward is 1 or -1. When the reward function faces sample data containing majority class noise, the reward or penalty is small, which can improve the classification performance of the noise classification model in identifying minority class noise data.

[0176] Step 6: Store the i-th experience data through the experience pool in the agent, and update i=i+1 to trigger the execution of step 3; wherein the i-th experience data includes the i-th sample data, the i-th classification action, the i-th reward and the i+1-th sample data.

[0177] Step 7: After traversing n sample data, randomly select K groups of experience data from the experience pool; K is a positive integer.

[0178] Step 8: Input each set of empirical data into the deep Q network and the target network respectively to update the deep Q network based on the difference between the first network result of the deep Q network and the second network result of the target network.

[0179] Step 9: After each traversal of W sets of empirical data, synchronize the network parameters of the deep Q network to the target network until the traversal of each set of empirical data is completed, and the network parameters of the deep Q network obtained in this round of training are obtained, where W is a positive integer.

[0180] Step 10: After this round of training is completed, the order of each sample data in the data environment is disrupted, triggering the execution of step 3, and the training is stopped until the number of training rounds reaches the preset requirements, and the deep Q network obtained from the last training is determined as the noise classification model.

[0181] For detailed description of this embodiment, please refer to the above-mentioned deep reinforcement learning based on unlabeled sample data. The difference is that the cluster label is replaced by the noise classification label, and the reward function is replaced by the reward function in step 5. This embodiment will not be repeated here.

[0182] Optionally, the test set may be input into the trained noise classification model, and the noise classification result of each sample data in the test set may be output, so as to determine the performance of the noise classification model based on the noise classification result. The sample data in the test set is different from the sample data in the training set.

[0183] In summary, in this embodiment, by setting the reward function so that the reward or penalty is smaller when facing sample data containing majority class noise, the classification performance of the noise classification model in identifying data containing minority class noise can be improved.

[0184] Optionally, in other embodiments, the object classification model can also be obtained through the above-mentioned deep reinforcement learning based on labeled sample data. At this time, the noise classification labels in the embodiment are adaptively replaced with object classification labels, and the noisy sample data are adaptively replaced with noise-free sample data. The overall training idea is the same, and this application does not limit the training method of the object classification model.

[0185] Optionally, the training order of the noise classification model is before the training order of the object classification model. In this case, for a training set composed of sample data with noise, the noise classification model is first trained through the training set, and then the noise classification model is used to perform noise classification on the sample data with noise in the training set, and the denoising algorithm corresponding to the noise classification is called to perform denoising on the sample data with noise to obtain the denoised sample data; thereafter, the object classification model is trained using each denoised sample data.

[0186] In an example, the object classification model is used in a fault detection scenario to illustrate the training process of the object classification model. Figure 3 , the process includes at least the following steps:

[0187] Step 301: Obtain a training set, where the training set includes sample data with noise.

[0188] The relevant description of this step is detailed in the above embodiment, and will not be repeated here in this embodiment. Figure 3 In the figure, the noise classification includes three types, namely noise 1, noise 2 and noise 3, for illustration.

[0189] Step 302: Perform noise classification on the sample data with noise based on the noise classification model to obtain a noise classification of each sample data with noise.

[0190] The training method of the noise classification model may be the labeled training method or the unlabeled training method in the above embodiment. This embodiment does not limit the training method of the noise classification model.

[0191] Step 303: Call the noise reduction algorithm corresponding to the noise classification to perform noise reduction processing on the sample data with noise to obtain the sample data after noise reduction.

[0192] For detailed description of this step, please refer to the relevant description of step 203, which will not be repeated here in this embodiment.

[0193] according to Figure 3 It can be seen that the optimal wavelet threshold denoising algorithm corresponding to each noise classification is called to perform denoising on the sample data of the noise classification respectively, and the denoised sample data corresponding to noise 1, the denoised sample data corresponding to noise 2, and the denoised sample data corresponding to noise 3 are obtained.

[0194] Step 304: perform deep reinforcement learning based on the denoised sample data to obtain an object classification model.

[0195] In one example, the object classification model may be trained in a labeled manner or an unlabeled manner in the above-mentioned embodiment. This embodiment does not limit the training method of the object classification model.

[0196] Step 305: Based on the object classification model obtained by deep reinforcement learning, the denoised data is subjected to object classification to obtain an object classification result.

[0197] The denoised data may be the denoised sample data corresponding to noise 1, the denoised sample data corresponding to noise 2 or the denoised sample data corresponding to noise 3.

[0198] In this embodiment, firstly, the data imbalance ratio is used to feedback reward information, the noise data is used as the model state, and the noise classification is used as the optional action of the intelligent agent, so as to construct an interactive environment for the intelligent agent to observe, act and obtain rewards, and realize the classification of noise contained in deep reinforcement learning decision data; then, the wavelet threshold denoising effects of different wavelet bases under single noise are compared to obtain the optimal denoising method for each noise, and based on this, each type of noise data is denoised separately to obtain the denoised data; finally, the reward information is fed back based on the relationship between the sample and the cluster center, and a new interactive environment is constructed by using the denoised data as the state and the fault type as the optional action of the intelligent agent to realize deep reinforcement learning fault detection, and realize the rotating machinery fault detection for unlabeled, unbalanced and multiple noise data.

[0199] In other embodiments, the denoised sample data and the noisy sample data may also come from different training sets. Accordingly, the noise classification model and the object classification model may also be trained synchronously. This embodiment does not limit the training order of the noise classification model and the object classification model, and the method of obtaining the denoised sample data.

[0200] Next, the object classification model is applied to the fault detection scenario to evaluate the model performance of the object classification model obtained by unlabeled training and the noise classification model obtained by labeled training.

[0201] Assume that the above sample data comes from the CWRU dataset provided by Case Western Reserve University (CWRU). This dataset contains acceleration sensor data from different states, reflecting the vibration signals of different types of bearing failures. If normal data and 0.007mm inner ring fault data are selected as the original experimental data in a ratio of 10:1, and Gaussian noise, gamma noise, and Rayleigh noise are added to each type of data in a ratio of 8:1:1, a bearing vibration signal dataset containing three types of noise is obtained. In this experiment, according to the overlapping sample segmentation strategy (such as Figure 4 ) splits a long string of vibration signal data in the original data set and expands the samples to 3000 samples, each with a length of 1200 and an overlapping length of 100.

[0202] Considering the imbalance of the three types of noise data, in order to more comprehensively evaluate the performance of the object classification model obtained based on unlabeled training in different classifications, weighted accuracy and weighted G value are used as fault detection evaluation indicators. The relevant formulas are as follows:

[0203]

[0204] Where N represents the total number of samples, Ω={w 1 ,w 2 ,…,w K} represents cluster division, C={c 1 ,c 2 ,…,c J} represents the real fault classification. In the above process, the classification with the largest number of samples of the same classification in each cluster is considered to be the corresponding real fault classification of the cluster.

[0205]

[0206] in, represents the weighted accuracy, Acc1, Acc2, and Acc3 represent the fault detection accuracy of sample data corresponding to Gaussian noise, gamma noise, and Rayleigh noise, respectively.

[0207] The G value is the geometric mean of the true positive rate and the true negative rate. The higher the G value, the better the performance of the algorithm.

[0208]

[0209] Among them, TP is a true positive example, FN is a false negative example, TN is a true negative example, and FP is a false positive example.

[0210]

[0211] in, represents the weighted G value, G1, G2, and G3 represent the fault detection G values ​​of Gaussian noise, gamma noise, and Rayleigh noise data, respectively.

[0212] Taking the wavelet threshold denoising algorithm as an example, the results of wavelet threshold denoising under different noises are different when different wavelet bases are selected. In this application, wavelet threshold denoising is performed on data containing Gaussian noise, gamma noise, and Rayleigh noise, respectively, to obtain the optimal denoising method under the corresponding noise.

[0213] Taking into account the support length, vanishing moment, symmetry, regularity and similarity, this embodiment selects the db3 wavelet basis and the sym4 wavelet basis for wavelet threshold denoising. The specific results are shown in Table 1 below:

[0214] Table 1:

[0215]

[0216] According to Table 1, the detection accuracy and G value of the data containing Gaussian noise after wavelet threshold denoising using db3 wavelet basis are higher than those of sym4 wavelet basis; the detection accuracy and G value of the data containing gamma noise after wavelet threshold denoising using sym4 wavelet basis are higher than those of db3 wavelet basis; the accuracy of wavelet threshold denoising using db3 wavelet basis for data containing Rayleigh noise is 1% lower than that of sym4 wavelet threshold denoising, and the G value is 0.03 higher. Comprehensively considering, the effect of using db3 wavelet basis is better.

[0217] Based on this, the optimal wavelet threshold denoising algorithm corresponding to the Gaussian noise classification can be set as wavelet threshold denoising using the db3 wavelet basis; the optimal wavelet threshold denoising algorithm corresponding to the gamma noise classification can be set as wavelet threshold denoising using the sym4 wavelet basis; and the optimal wavelet threshold denoising algorithm corresponding to the Rayleigh noise classification can be set as wavelet threshold denoising using the db3 wavelet basis.

[0218] In addition, in order to verify the superiority of the deep reinforcement learning method in noise classification, in this embodiment, convolutional neural networks (CNN) are used for noise classification as a comparative experiment. Figure 5 It shows the reward value obtained by the agent during the training process. Figure 5 It can be seen that as the agent continues to learn (that is, the number of episodes continues to increase), the decision-making ability gradually improves, the cumulative return (that is, the total reward) continues to increase, and finally reaches a stable state. Figure 6 It shows the change in accuracy during the training process, which gradually increases due to the improvement of decision-making ability.

[0219] The experiment was cross-validated ten times, and the noise classification accuracy under different methods is shown in Table 2 below.

[0220] Table 2:

[0221] CNN This application noise classification algorithm Noise classification accuracy 76.44％ 83.33％

[0222] According to Table 2, the noise classification algorithm in this application has improved the classification accuracy by nearly 7% compared with CNN, that is, the deep reinforcement learning denoising classification method has more classification advantages than the convolutional neural network.

[0223] In addition, according to the setting of the denoising algorithm in the above text, the data containing Gaussian noise and Rayleigh noise are denoised by the wavelet threshold denoising method with db3 as the wavelet basis, and the data containing gamma noise is denoised by the wavelet threshold denoising method with sym4 as the wavelet basis to obtain the denoised data. The three types of data are respectively subjected to deep reinforcement learning fault detection, and the obtained fault detection performance is shown in the following Table 3:

[0224] Table 3:

[0225]

[0226] According to Table 3, by using different denoising algorithms to denoise data of different noise categories before performing fault detection, the accuracy is improved by 7% when the data is unbalanced, and the G value is improved by 0.11, which verifies that the model has better detection effect when the data is unbalanced.

[0227] To sum up, the deep reinforcement learning fault detection method under complex noise conditions proposed in this application uses deep reinforcement learning to decide the type of noise contained in each data, and compares the wavelet threshold denoising effects of different wavelet bases under a single noise, and obtains the optimal denoising method for each noise. Based on this, each type of noise data is denoised separately, and deep reinforcement learning fault detection is performed on the denoised data after obtaining the denoised data. When the data is unbalanced, the accuracy and G value of fault detection are better, and the advantages are more prominent.

[0228] Figure 7 7 is a block diagram of an object classification device based on deep reinforcement learning under complex noise conditions according to an exemplary embodiment. The device includes at least the following modules: a data acquisition module 710, a noise classification module 720, a data noise reduction module 730 and an object classification module 740.

[0229] A data acquisition module 710 is used to acquire data to be classified;

[0230] A noise classification module 720 is used to perform noise classification on the data to be classified based on the noise classification model obtained by deep reinforcement learning to obtain the noise classification corresponding to the data to be classified;

[0231] The data denoising module 730 is used to call the denoising algorithm corresponding to the noise classification to perform denoising on the data to be classified to obtain denoised data; wherein different noise classifications correspond to different denoising algorithms;

[0232] The object classification module 740 is used to perform object classification on the denoised data based on the object classification model obtained by deep reinforcement learning to obtain an object classification result.

[0233] For detailed description, please refer to the above method embodiment.

[0234] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0235] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0236] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0237] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0238] Figure 8 1 is a block diagram of an object classification device 1900 based on deep reinforcement learning under complex noise conditions according to an exemplary embodiment. For example, the device 1900 can be provided as a server or a terminal device. Figure 8 , the apparatus 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application, that can be executed by the processing component 1922. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0239] The device 1900 may also include a power supply component 1926 configured to perform power management of the device 1900, a wired or wireless network interface 1950 configured to connect the device 1900 to a network, and an input / output interface 1958 (I / O interface). The device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2000. TM , MacOS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0240] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the device 1900 to perform the above method.

[0241] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for object classification based on deep reinforcement learning under complex noise conditions, characterized in that: The method comprises: Obtain data to be classified; Based on the noise classification model obtained by deep reinforcement learning, the data to be classified is subjected to noise classification to obtain the noise classification corresponding to the data to be classified; Calling the noise reduction algorithm corresponding to the noise classification to perform noise reduction processing on the data to be classified to obtain noise-reduced data; wherein different noise classifications correspond to different noise reduction algorithms; Based on the object classification model obtained by deep reinforcement learning, object classification is performed on the denoised data to obtain an object classification result.

2. The method according to claim 1, characterized in that The noise classification model is obtained by deep reinforcement learning based on unlabeled sample data, and the sample data used to train the noise classification model is sample data with noise; and / or, the object classification model is obtained by deep reinforcement learning based on unlabeled sample data, and the sample data used to train the object classification model is sample data after noise reduction; The deep reinforcement learning based on unlabeled sample data includes: Determine the number of cluster centers A based on the number of classifications, and initialize and set the cluster centers to be A sample data with the farthest sample distance; A is an integer greater than 1; For each sample data, determine the sample distance between the sample data and each cluster center, and determine the index of the cluster center with the smallest sample distance to the sample data as the cluster label of the sample data; Building a data environment based on each sample data and a cluster label corresponding to the sample data; Performing deep reinforcement learning through interaction between the intelligent agent and the data environment to obtain a trained reinforcement learning model; After this round of training is completed, based on the mean of the sample data corresponding to each cluster center, the cluster center is updated, triggering the execution of the steps of determining the sample distance between the sample data and each cluster center for each sample data, and determining the index of the cluster center with the smallest sample distance to the sample data as the cluster label of the sample data, and subsequent steps, until the training round reaches the preset requirements and stops the training.

3. The method according to claim 2, characterized in that The deep reinforcement learning is performed through the interaction between the intelligent agent and the data environment to obtain a trained reinforcement learning model, including: The i-th sample data is returned to the agent in sequence through the data environment; the agent includes a reinforcement learning model, the reinforcement learning model includes a deep Q network and a target network, and the network parameters of the initialized deep Q network are the same as the network parameters of the target network; i is a positive integer ranging from 1 to n in sequence, and n is the total number of the sample data; Determine the i-th classification action corresponding to the i-th sample data through the deep Q network; wherein the i-th classification action indicates one of the cluster labels; The agent returns the i-th classification action to the data environment, so that the data environment determines the i-th reward of the i-th classification action based on the comparison result between the i-th classification action and the cluster label corresponding to the i-th sample data, and returns the i-th reward and the i+1-th sample data to the agent; if the comparison result is the same, the reward is If the comparison result is different, the reward is -1; represents the sample distance between the i-th sample data and the corresponding cluster center; The i-th experience data is stored in the experience pool of the agent, and i=i+1 is updated to trigger the execution of the step of returning the i-th sample data to the agent in sequence through the data environment and subsequent steps; the i-th experience data includes the i-th sample data, the i-th classification action, the i-th reward and the i+1-th sample data; After the traversal of n sample data is completed, K groups of experience data are randomly selected from the experience pool; K is a positive integer; Inputting each set of experience data into the deep Q network and the target network respectively, so as to update the deep Q network based on the difference between the first network result of the deep Q network and the second network result of the target network; Every time W groups of experience data are traversed, the network parameters of the deep Q network are synchronized to the target network until the traversal of each group of experience data is completed, and the network parameters of the deep Q network obtained in this round of training are obtained, where W is a positive integer.

4. The method according to claim 2, characterized in that: When the number of classifications corresponding to the noise classification model and / or the object classification model is less than or equal to a preset threshold, it is obtained by performing deep reinforcement learning based on unlabeled sample data.

5. The method according to claim 1, characterized in that The noise classification model is obtained by deep reinforcement learning based on labeled sample data, and the labeled sample data includes sample data of a first noise classification and sample data of a second noise classification, and the number of sample data of the first noise classification is less than the number of sample data of the second noise classification; accordingly, In the process of deep reinforcement learning based on labeled sample data, the reward when the data environment interacts with the agent is determined by the following formula: Among them, s t represents the current sample data returned by the data environment to the agent, a t Indicates that based on the current sample data s t The classification action performed, R(s t ,a t ) indicates that based on the current sample data s t The reward for performing the classification action, l t Indicates the current state s t The corresponding noise classification label, D minority Denotes the dataset consisting of sample data of the first noise classification, D majority represents a data set consisting of sample data of the second noise classification, and ρ represents the ratio of the number of sample data of the first noise classification to the number of sample data of the second noise classification.

6. The method according to claim 1, characterized in that The deep reinforcement learning process of the object classification model includes: Obtaining a training set, wherein the training set includes sample data with noise; Performing noise classification on the sample data with noise based on the noise classification model to obtain a noise classification of each sample data with noise; Calling a noise reduction algorithm corresponding to the noise classification to perform noise reduction processing on the sample data with noise to obtain noise-reduced sample data; Deep reinforcement learning is performed based on the denoised sample data to obtain the object classification model.

7. The method according to claim 1, characterized in that The calling of the noise reduction algorithm corresponding to the noise classification to perform noise reduction processing on the data to be classified to obtain the noise-reduced data includes: Obtain the correspondence between noise classification and optimal wavelet threshold denoising algorithm; The data to be classified is subjected to wavelet threshold denoising according to the optimal wavelet threshold denoising algorithm corresponding to the noise classification to obtain the denoised data.

8. The method according to claim 6, characterized in that The obtaining of the correspondence between the noise classification and the optimal wavelet threshold noise reduction algorithm includes: For each noise classification, different wavelet bases are used to perform wavelet threshold denoising on the same noise data belonging to the noise classification, and the wavelet base with the best denoising effect is determined from the denoising results corresponding to different wavelet bases to obtain the optimal wavelet threshold denoising algorithm corresponding to the noise classification.

9. The method according to any one of claims 1 to 8, characterized in that: The object classification model is used to perform fault diagnosis on equipment; accordingly, the signal to be classified includes equipment signals with noise; and the object classification includes fault signal classification and normal signal classification.

10. An object classification device based on deep reinforcement learning under complex noise conditions, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the method described in any one of claims 1 to 9 when executing the instructions stored in the memory.

Images