Fault diagnosis system and method based on deep reinforcement learning

Abstract

The invention discloses a fault diagnosis system and method based on deep reinforcement learning, and relates to the technical field of fault diagnosis, and the method comprises the steps: carrying out the reverse compensation of historical output data of a sensor, and obtaining first standard output data; performing kernel density estimation on the first standard output data of each sensor; analyzing the fault probability of the industrial robot under different environmental factors to obtain second standard output data; acquiring real-time sensor output data of the industrial robot, and diagnosing the fault type of the industrial robot through the deep reinforcement learning model of the fault type; by analyzing the influence of different environmental data on sensor output data, reverse compensation is carried out on the sensor output data, and the influence of environmental data difference on a diagnosis model is reduced; according to the fault probability density function of the industrial robot, the fault probabilities of the industrial robot under different environment data are consistent, continuous fault diagnosis under different environments is realized, and the fault diagnosis efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of fault diagnosis, in particular to a fault diagnosis system based on deep reinforcement learning and a method thereof. BACKGROUND

[0002] Industrial robot fault diagnosis is a complex and challenging field. The industrial environment is dynamic, and the working environment of the robot can be continuously adjusted. Factors such as vibration, humidity, and temperature changes in the dynamic environment can change the physical quantities detected by the sensor, causing the distribution of its output data to change. Therefore, fault diagnosis needs to be performed according to the characteristics of the environment, which brings difficulties to the continuous fault diagnosis of industrial robots. Therefore, how to continuously diagnose the faults of industrial robots in a dynamic environment has become a problem that needs to be solved. SUMMARY

[0003] The present application aims to provide a fault diagnosis system based on deep reinforcement learning and a method thereof to solve the problems in the prior art.

[0004] To achieve the above object, the application provides the following technical scheme: a fault diagnosis system based on deep reinforcement learning, comprising a data acquisition module, a data storage module, a data analysis module and a fault diagnosis module; the data acquisition module acquires operation data and environment data of an industrial robot through a sensor; the data storage module is used for storing historical operation data, environment data and fault data of the industrial robot; the data analysis module reversely compensates historical output data of the sensor to remove deviation caused by environmental factors and obtain first standard output data; analyzes fault probability of the industrial robot under different environmental factors, adjusts the reverse compensation mode to remove deviation caused by environmental factors and obtains second standard output data; the fault diagnosis module diagnoses the fault type of the industrial robot through a deep reinforcement learning model of the fault type; the fault diagnosis module further comprises a recovery unit and a deep reinforcement learning unit; the deep reinforcement learning unit trains the deep reinforcement learning model of the fault type according to the second standard output data of the sensor, and diagnoses the fault of the industrial robot according to the deep reinforcement learning model of the fault type; the recovery unit executes recovery operation of the industrial robot according to the fault type of the industrial robot; the data analysis module further comprises a fitting unit, an estimation unit and an optimization unit; the fitting unit trains a fitting model between a difference value caused by environment data to sensor output data, reversely compensates historical output data of the sensor according to the fitting model to remove deviation caused by environmental factors and obtains first standard output data; the estimation unit performs kernel density estimation on the first standard output data of each sensor to obtain a probability density function between fault probability of the industrial robot and the first standard output data of the sensor; the optimization unit analyzes fault probability of the industrial robot under different environmental factors, adjusts the reverse compensation mode to remove deviation caused by environmental factors and obtains second standard output data.

[0005] The optimization unit acquires historical environment data and sensor output data of the industrial robot under other environment data except reference environment data, obtains first standard output data S from the sensor output data; let Sbase represent sensor output data of the industrial robot under the reference environment data, generate an interval with a length of 2e centered on m, e represents noise, m∈[S-e, S+e]; obtain fault probability P1 under the reference environment data and fault probability P2 under other environment data except the reference environment data according to the probability density function under the reference environment data and other environment data except the reference environment data, determine m in the interval [m-e, m+e] that minimizes the deviation between P1 and P2 according to P1, obtain P2 according to m, P2= ; wherein f1(x) represents a probability density function under the reference environment data, and f2(x) represents a probability density function under other environment data than the reference environment data; and the second standard output data is obtained according to the determined m that minimizes the deviation between P1 and P2.

[0006] To achieve the above object, the present application provides the following technical solutions.

[0007] The fault diagnosis method based on deep reinforcement learning comprises the following steps:

[0008] Obtaining historical output data of sensors of the industrial robot;

[0009] Compensating the historical output data of the sensors in reverse to remove the deviation caused by the environmental factors, and obtaining first standard output data;

[0010] Performing kernel density estimation on the first standard output data of each sensor to obtain a probability density function between the fault probability of the industrial robot and the first standard output data of the sensor;

[0011] Analyzing the fault probability of the industrial robot under different environmental factors, adjusting the reverse compensation mode, removing the deviation caused by the environmental factors, and obtaining second standard output data;

[0012] Training a deep reinforcement learning model of the fault type according to the second standard output data of all sensors; obtaining real-time sensor output data of the industrial robot, and diagnosing the fault type of the industrial robot through the deep reinforcement learning model of the fault type.

[0013] Specifically, the reverse compensation of the historical output data of the sensors to remove the deviation caused by the environmental factors and to obtain the first standard output data further comprises the following steps:

[0014] Obtaining all environmental data of the industrial robot during operation, selecting one of the environmental data as reference environmental data and recording it as Eb; obtaining output data Outb and Out of the sensors of the industrial robot under the environmental data Eb and other environmental data, calculating the difference between Out and Outb, and training a fitting model between the difference and the environmental data;

[0015] The output data of the industrial robot sensor under the environmental data Eb and other environmental data is divided into a training set and a test set, the other environmental data is taken as input, the difference Out-Outb in the training set is taken as output, a neural network fitting model is trained, the other environmental data in the verification set is input into the neural network fitting model, a fitting value of the difference is obtained, the effect of the neural network fitting model is verified according to the fitting value and the actual value of the difference, the other environmental data is input into the verified neural network fitting model, a fitting value of the difference is obtained, and the first standard output data is obtained by operating the fitting value of the difference and the output data under the other environmental data.

[0016] Specifically, the first standard output data of each sensor is subjected to kernel density estimation, and a probability density function between the industrial robot failure probability and the sensor first standard output data is obtained, and the method further comprises the following steps:

[0017] The historical output data of the sensor of the industrial robot under all environmental data is obtained, the data with faults is labeled, and the historical output data of the sensor is removed from the deviation caused by environmental factors to obtain the first standard output data.

[0018] According to the historical smelting data, a data set is generated: the nodes x1, x2, …, xn of the sensor output data are set, wherein n is the number of nodes; let X represent the first standard output data random variable, and the probability P{X≥a1} of the labeled data appearing when the first standard output data random variable X is greater than or equal to a1 is obtained, P{X≥a1}=n a1 / N a1 , wherein n a1 represents the number of first standard output data greater than or equal to a1 and with faults, and N a1 represents the number of first standard output data greater than or equal to a1; in the same way as the sensor output data node x1, the probabilities P{X≥a2}, …, P{X≥an} of the labeled data appearing when the first standard output data X is greater than or equal to a2, …, an are obtained; n data points are formed by a1 and P{X≥a1}, a2 and P{X≥a2}, …, an and P{X≥an}, and the data points are respectively denoted as x1, x2, …, xn to obtain the data set.

[0019] The probability density function f(x) between the fault occurrence probability and the first standard output data is obtained by kernel density estimation; a kernel function with non-negativity and symmetry and an integral of 1 on the real number domain R is set; a constant greater than zero is selected as the bandwidth h of the kernel function K; the kernel function is scaled according to the bandwidth h to obtain Kh, where Kh(u)=1 / h×K(u / h), u is the input of the kernel function; the contribution rate Kh(x-xi) of the data point xi in the data set to the estimation point x is obtained; the contribution rates of all data points in the data set to the estimation point x are added to obtain the kernel density estimation value f(x) at the estimation point x; f(x)=1 / n∑Kh(x-xi); by changing the position of the estimation point x, the kernel density estimation of the fault occurrence probability on the entire data set is obtained; the data set is the interval between the minimum value and the maximum value of the sensor output data; the effect of the kernel function is verified, and the bandwidth h with the best verification effect is selected through cross verification.

[0020] Specifically, the fault probability of the industrial robot under different environmental factors is analyzed, the reverse compensation mode is adjusted, the deviation caused by the environmental factors is removed, and the second standard output data is obtained, which further includes the following steps:

[0021] The historical environment data and sensor output data of the industrial robot under other environment data except the reference environment data are obtained, and the first standard output data S is obtained according to the sensor output data; Sbase represents the sensor output data of the industrial robot under the reference environment data, and an interval with a length of 2e is generated with m as the center, e represents noise, and m∈[S-e,S+e]; the fault probability P1 under the reference environment data and the fault probability P2 under other environment data except the reference environment data are obtained according to the probability density functions under the reference environment data and other environment data except the reference environment data, P1 and P2 is determined according to P1, P2 is obtained according to m, P2= ; In the formula, f1(x) represents the probability density function under the reference environment data, and f2(x) represents the probability density function under other environment data except the reference environment data; the second standard output data is obtained according to the determined m which makes the deviation between P1 and P2 minimum; let y1 and y2 represent the first standard output data and the second standard output data, then y2=y1+(m-S).

[0022] Specifically, the deep reinforcement learning model of the fault type is trained according to the second standard output data of all sensors, which further includes the following steps:

[0023] The fault type of the industrial robot is binary coded, a DQN deep neural network is constructed, the second standard output data of the sensor is formed as a state space as input, the fault type of the industrial robot is formed as an action space, a reward function and hyperparameters of the DQN neural network are set, and the DQN neural network model is trained.

[0024] Compared with the prior art, the beneficial effects of the present application are: by analyzing the influence of different environmental data on the sensor output data, the sensor output data is compensated in reverse, and the influence of environmental data difference on the diagnosis model is reduced; according to the probability density function of the industrial robot fault, the fault probability of the industrial robot under different environmental data is consistent, continuous fault diagnosis under different environments is realized, and the efficiency of fault diagnosis is improved. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 Fig. 1 is a structural schematic diagram of the fault diagnosis system based on deep reinforcement learning of the present application;

[0026] Figure 2 Fig. 2 is a flowchart of the fault diagnosis method based on deep reinforcement learning of the present application;

[0027] Figure 3 Fig. 3 is a flowchart of the fault diagnosis network update of the present application. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0029] Embodiment: as Figure 1As shown, the application provides a fault diagnosis system based on deep reinforcement learning, comprising a data acquisition module, a data storage module, a data analysis module and a fault diagnosis module; the data acquisition module acquires the operation data and environmental data of the industrial robot through sensors; the data storage module is used for storing the historical operation data, environmental data and fault data of the industrial robot; the data analysis module reversely compensates the historical output data of the sensors to remove the deviation caused by environmental factors and obtains first standard output data; analyzes the fault probability of the industrial robot under different environmental factors, adjusts the reverse compensation mode, removes the deviation caused by environmental factors, and obtains second standard output data; the fault diagnosis module diagnoses the fault type of the industrial robot through the deep reinforcement learning model of the fault type. The fault diagnosis module further comprises a recovery unit and a deep reinforcement learning unit; the deep reinforcement learning unit trains the deep reinforcement learning model of the fault type according to the second standard output data of the sensors, and diagnoses the fault of the industrial robot according to the deep reinforcement learning model of the fault type; the recovery unit executes the recovery operation of the industrial robot according to the fault type of the industrial robot. The data analysis module further comprises a fitting unit, an estimation unit and an optimization unit; the fitting unit trains the fitting model between the difference value caused by the environmental data to the sensor output data according to the environmental data, reversely compensates the historical output data of the sensors according to the fitting model to remove the deviation caused by environmental factors, and obtains the first standard output data; the estimation unit performs kernel density estimation on the first standard output data of each sensor to obtain the probability density function between the fault probability of the industrial robot and the first standard output data of the sensor; the optimization unit analyzes the fault probability of the industrial robot under different environmental factors, adjusts the reverse compensation mode, removes the deviation caused by environmental factors, and obtains the second standard output data.

[0030] The optimization unit acquires the historical environmental data and sensor output data of the industrial robot under other environmental data except the reference environmental data, obtains the first standard output data S from the sensor output data; let Sbase represent the sensor output data of the industrial robot under the reference environmental data, generate an interval with a length of 2e centered on m, e represents noise, m∈[S-e,S+e]; obtain the fault probability P1 under the reference environmental data and the fault probability P2 under other environmental data except the reference environmental data according to the probability density function under the reference environmental data and other environmental data except the reference environmental data, determine m in the interval [m-e,m+e] that minimizes the deviation between P1 and P2 according to P1, obtain P2 according to m, P2= ; wherein f1(x) represents a probability density function under the reference environment data, and f2(x) represents a probability density function under other environment data than the reference environment data; and the second standard output data is obtained according to the determined m that minimizes the deviation between P1 and P2.

[0031] In another embodiment of the present application, the present application provides a fault diagnosis method based on deep reinforcement learning, comprising the following steps:

[0032] obtaining historical output data of sensors of the industrial robot;

[0033] compensating the historical output data of the sensors in reverse to remove the deviation caused by the environmental factors, to obtain first standard output data;

[0034] performing kernel density estimation on the first standard output data of each sensor to obtain a probability density function between the fault probability of the industrial robot and the first standard output data of the sensor;

[0035] analyzing the fault probability of the industrial robot under different environmental factors, adjusting the reverse compensation method to remove the deviation caused by the environmental factors, to obtain second standard output data;

[0036] training a deep reinforcement learning model of the fault type according to the second standard output data of all sensors; obtaining real-time sensor output data of the industrial robot, and diagnosing the fault type of the industrial robot through the deep reinforcement learning model of the fault type.

[0037] compensating the historical output data of the sensors in reverse to remove the deviation caused by the environmental factors, to obtain first standard output data further comprises the following steps:

[0038] obtaining all environmental data of the industrial robot during operation, selecting one of the environmental data as reference environmental data and recording it as Eb; obtaining output data Outb and Out of the sensors of the industrial robot under the environmental data Eb and other environmental data, calculating the difference between Out and Outb, and training a fitting model between the difference and the environmental data;

[0039] dividing the output data of the sensors of the industrial robot under the environmental data Eb and other environmental data into a training set and a test set, taking the other environmental data as input and the difference Out-Outb in the training set as output, training a neural network fitting model; inputting the other environmental data in the validation set into the neural network fitting model to obtain the fitting value of the difference, verifying the effect of the neural network fitting model according to the fitting value and the actual value of the difference; inputting the other environmental data into the verified neural network fitting model to obtain the fitting value of the difference, and performing operation on the fitting value of the difference and the output data under the other environmental data to obtain the first standard output data.

[0040] The sensor on the industrial robot works in a dynamic environment, the sensor is susceptible to various environmental factors, the output signal will fluctuate or error, in different environments, the sensor will get different output results, which brings difficulties to the continuous fault diagnosis of the industrial robot; therefore, the first standard output data of the sensor is obtained by reverse compensation. A fitting model is established by a neural network model, first, an environment data is selected as a reference, first, when the industrial robot is in the reference environment data, the output data of the sensor is the standard output data, when the industrial robot leaves the reference environment and is in other environment data, the output data of the sensor will be affected by the environmental factors and deviate from the standard output data, the error brought by the environmental factors to the output data of the sensor is captured, since the reference environment data is fixed, the input of the fitting model can be directly selected as other environment data; for each type of sensor, a neural network fitting model is trained respectively to obtain the first standard output data.

[0041] The first standard output data of each sensor is subjected to kernel density estimation to obtain the probability density function between the fault probability of the industrial robot and the first standard output data of the sensor, which further comprises the following steps:

[0042] The historical output data of the sensor of the industrial robot under all environment data is obtained, the data with faults is labeled, and the historical output data of the sensor is removed from the deviation caused by the environmental factors to obtain the first standard output data;

[0043] According to the historical smelting data, a data set is generated: set the nodes x1, x2, …, xn of the sensor output data, wherein n is the number of nodes; let X represent the first standard output data random variable, obtain the probability P{X≥a1} of the labeled data when the first standard output data random variable X is greater than or equal to a1, P{X≥a1}=n a1 / N a1 , wherein n a1 represents the number of first standard output data greater than or equal to a1 and with faults, N a1 represents the number of first standard output data greater than or equal to a1; in the same way as the sensor output data node x1, the probability P{X≥a2}, …, P{X≥an} of the labeled data when the first standard output data X is greater than or equal to a2, …, an is obtained; n data points are formed by a1 and P{X≥a1}, a2 and P{X≥a2}, …, an and P{X≥an}, and the data points are respectively denoted as x1, x2, …, xn to obtain the data set;

[0044] The probability density function f(x) between the fault occurrence probability and the first standard output data is obtained by kernel density estimation; a kernel function with non-negativity and symmetry and an integral of 1 on the real number domain R is set; a constant greater than zero is selected as the bandwidth h of the kernel function K; the kernel function is scaled according to the bandwidth h to obtain Kh, where Kh(u) = 1 / h x K(u / h), u is the input of the kernel function; the contribution rate Kh(x-xi) of the data point xi in the data set to the estimation point x is obtained; the contribution rates of all data points in the data set to the estimation point x are added to obtain the kernel density estimation value f(x) at the estimation point x; f(x) = 1 / n∑Kh(x-xi); by changing the position of the estimation point x, the kernel density estimation of the fault occurrence probability on the entire data set is obtained; the data set is the interval between the minimum value and the maximum value of the sensor output data; the effect of the kernel function is verified, and the bandwidth h with the best verification effect is selected through cross-validation.

[0045] The probability density function between the sensor output data and the industrial robot fault probability is obtained by kernel density estimation, and the probability density function is obtained under each environment data. Since the environment factors may not be completely the same, similar environment data can be optionally classified into a category by supervised or unsupervised classification, and the probability density function is obtained under the environment data in the same category; when supervised classification is used, labels are manually assigned to environment factors, environment factors that have similar effects on sensor output data are assigned the same category label, and then a classification model is trained; according to the probability density function under each environment data, the fault probability of the industrial robot under each environment data can be obtained.

[0046] The fault probability of the industrial robot under different environment factors is analyzed, the reverse compensation mode is adjusted, and the deviation caused by the environment factors is removed to obtain the second standard output data, which further includes the following steps:

[0047] The historical environment data and sensor output data of the industrial robot under other environment data except the reference environment data are obtained, and the first standard output data S is obtained according to the sensor output data; Sbase represents the sensor output data of the industrial robot under the reference environment data, and an interval with a length of 2e is generated with m as the center, e represents noise, and m∈[S-e,S+e]; the fault probability P1 under the reference environment data and the fault probability P2 under other environment data except the reference environment data are obtained according to the probability density functions under the reference environment data and other environment data except the reference environment data, P1 and P2 are determined according to P1, P2 is obtained according to m, P2= ; wherein f1(x) represents a probability density function under the reference environment data, and f2(x) represents a probability density function under other environment data other than the reference environment data; the second standard output data is obtained according to the determined m that makes the deviation between P1 and P2 minimum; y1 and y2 represent the first standard output data and the second standard output data, and y2=y1+(m-S).

[0048] For the same robot, if there is no sudden abnormal situation, the fault probability should be similar or the same under different environment data, because the change of environment leads to the change of sensor output data, and the fault probability of judgment changes greatly, which is unreasonable; the second standard output data is further adjusted on the basis of the first standard output data, and due to the influence of noise, there is an error of ±e when the output data of the sensor is converted into the first standard output data, so that the accurate standard output data is in the interval [S-e, S+e], in order to obtain the accurate standard output data, any point m in the interval [S-e, S+e] is taken as the center, and the fault probability P2 of other environment data other than the reference environment data is calculated when m is taken as the center. When this fault probability is closest to the fault probability under the reference environment data, that is, the requirement is met At this time, m is the second standard output data; the error m-S is obtained according to m and the first standard output data S, the error m-S can be updated periodically, and it is not necessary to analyze each first standard output data of the sensor; after the error m-S is obtained, the second standard output data is obtained by adding the first standard output data of the sensor.

[0049] According to the obtained second standard output data, when the environment of the industrial robot changes, the fault probability of the industrial robot will not change suddenly without the occurrence of abnormal events, and the fault diagnosis of the industrial robot can be continuously performed.

[0050] According to the second standard output data of all sensors, training the deep reinforcement learning model of the fault type further includes the following steps:

[0051] The fault type of the industrial robot is binary coded, a DQN deep neural network is constructed, the second standard output data of the sensor is formed as a state space as input, the fault type of the industrial robot is formed as an action space, a reward function and hyperparameters of the DQN neural network are set, and the DQN neural network model is trained.

[0052] The definition of fault type is an action, and a binary bit is used to represent whether it exists for each possible fault type. For example, there are three fault types, and the action space is represented by a three-bit binary number. 000 represents no fault, 001 represents only fault type 3, 010 represents only fault type 2, 011 represents the simultaneous existence of fault type 2 and fault type 3, and so on. Since the second standard output data of the sensor changes over time, the state space changes over time.

[0053] Specifically, the following steps are included:

[0054] DQN network construction: a DQN deep neural network is constructed, the input is the feature vector of the state space, and the output is the Q value corresponding to each action, each action represents a possible fault type; the weights and biases of the DQN deep neural network are initialized by random initialization; the values of the hyperparameters are set, including but not limited to learning rate, discount factor γ, size of experience replay buffer, size of mini-batch, etc.

[0055] An empty experience replay buffer is created to store the samples generated during the interaction between the agent and the environment, including state, action, reward, and next state information;

[0056] At each time step, an action is selected according to the current state through the ε-greedy strategy, and then the selected action is executed in the environment to obtain the reward signal and the next state returned by the environment, and the sample is stored in the experience replay buffer. When selecting an action, the Q value is calculated by the DQN deep neural network according to the current state, and the calculation result of the Q value is the output Q(s,a,θ) of the network, s represents the state, i.e. the second standard output data of the sensor, a is the action, i.e. the fault type of the industrial robot, and θ is the parameter of the DQN deep neural network. The reward function is positively correlated with the correctly diagnosed fault type and inversely correlated with the incorrectly diagnosed fault type, and is connected by weighted summation;

[0057] Every certain training interval, a mini-batch of samples is randomly selected from the experience replay buffer; for each sample, the maximum Q value Qmax corresponding to the next state is calculated using the target network, and the target Q value Qtarget is calculated according to the reward r, the maximum Q value Qmax and the discount factor γ, Qtarget=r+γ×Qmax; the Q value Qpre predicted by the online network is compared with the target Q value, and the loss L is calculated, , where N represents the size of the mini-batch, The two-norm of Qpre-Qtarget is represented; the gradient information of the loss function is propagated back to each layer of the online network through a back propagation algorithm, and the parameters of the online network are updated by using an Adam optimization algorithm to minimize the difference between the predicted Q value and the target Q value; after the online network is updated for a certain number of times, the parameters of the online network are copied to the target network.

[0058] Since the second standard output data of the sensor changes over time, the DQN deep neural network can output a maximum Q value and a corresponding action at each time point, obtaining a continuous action sequence in time, which is a fault type diagnosis sequence, and the fault type of the industrial robot is diagnosed according to the obtained fault type diagnosis sequence.

[0059] It will be obvious to a person skilled in the art that the application is not limited to the details of the above-described exemplary embodiments, but that the application can be implemented in other concrete forms without departing from the spirit or essential characteristics of the application. Therefore, the embodiments should be considered in all respects as illustrative and not restrictive, the scope of the application being defined by the appended claims rather than the above description, and it is intended to include all changes falling within the meaning and scope of the equivalent elements of the claims. Any reference signs in the claims should not be considered as limiting the claims involved.

Claims

1. A fault diagnosis method based on deep reinforcement learning, characterized in that: The following steps are involved: Obtain historical output data from sensors on industrial robots; Perform reverse compensation on the historical output data of the sensor to remove the deviation caused by environmental factors and obtain the first standard output data; Perform kernel density estimation on the first standard output data of each sensor to obtain a probability density function between the industrial robot failure probability and the first standard output data of the sensor; Analyze the failure probability of industrial robots under different environmental factors, adjust the reverse compensation method, remove the deviation caused by environmental factors, and obtain the second standard output data; Based on the second standard output data of all sensors, a deep reinforcement learning model of fault types is trained; the real-time sensor output data of the industrial robot is obtained, and the fault type of the industrial robot is diagnosed through the deep reinforcement learning model of the fault type.

2. The fault diagnosis method based on deep reinforcement learning according to claim 1, characterized in that: The method of performing reverse compensation on the historical output data of the sensor to remove the deviation caused by environmental factors to obtain the first standard output data further includes the following steps: Obtain all environmental data during the operation of the industrial robot, select one environmental data as the reference environmental data and record it as Eb; obtain the output data Outb and Out of the sensors on the industrial robot under the environmental data Eb and other environmental data, calculate the difference between Out and Outb, and train a fitting model between the difference and the environmental data; The output data of the sensors on the industrial robot under the environmental data Eb and other environmental data are divided into a training set and a test set. The other environmental data are used as input and the difference Out-Outb in the training set is used as output to train the neural network fitting model; the other environmental data in the verification set are input into the neural network fitting model to obtain the fitting value of the difference, and the effect of the neural network fitting model is verified according to the fitting value and the actual value of the difference; the other environmental data are input into the verified neural network fitting model to obtain the fitting value of the difference, and the fitting value of the difference is calculated with the output data under the other environmental data to obtain the first standard output data.

3. The fault diagnosis method based on deep reinforcement learning according to claim 2, characterized in that: The performing of kernel density estimation on the first standard output data of each sensor to obtain a probability density function between the industrial robot failure probability and the first standard output data of the sensor further comprises the following steps: Obtain historical output data of the industrial robot's sensors under all environmental data, mark the data with faults, and remove deviations caused by environmental factors from the historical output data of the sensors to obtain first standard output data; Generate a data set based on historical smelting data: Set the sensor output data nodes x1, x2, ..., xn, where n is the number of nodes; let X represent the first standard output data random variable, and obtain the probability P{X≥a1} of the labeled data appearing when the first standard output data random variable X is greater than or equal to a1, P{X≥a1}=n a1 / N a1 , where n a1 Indicates the number of first standard output data greater than or equal to a1 and with faults, N a1 represents the number of first standard output data greater than or equal to a1; in the same way as the sensor output data node x1, the probability of the occurrence of the labeled data when the first standard output data X is greater than or equal to a2, ..., an is obtained; a1 and P{X≥a1}, a2 and P{X≥a2}, ..., an and P{X≥an} form n data points, and the data points are recorded as x1, x2, ..., xn to obtain the data set; The probability density function f(x) between the fault occurrence probability and the first standard output data is obtained through kernel density estimation; a kernel function with non-negativity and symmetry and an integral of 1 in the real number domain R is set; a constant greater than zero is selected as the bandwidth h of the kernel function K; the kernel function is scaled according to the bandwidth h to obtain Kh, where Kh(u)=1 / h×K(u / h), and u is the input of the kernel function; the contribution rate Kh(x-xi) of the data point xi in the data set to the estimated point x is obtained; the contribution rates of all data points in the data set to the estimated point x are added to obtain the kernel density estimate f(x) at the estimated point x; f(x)=1 / n∑Kh(x-xi); by changing the position of the estimated point x, the kernel density estimate of the fault occurrence probability in the entire data set is obtained; the data set is the interval between the minimum and maximum values ​​of the sensor output data; the effect of the kernel function is verified, and the bandwidth h with the best verification effect is selected through cross-validation.

4. The fault diagnosis method based on deep reinforcement learning according to claim 3, characterized in that: The analyzing the failure probability of the industrial robot under different environmental factors, adjusting the reverse compensation method, removing the deviation caused by the environmental factors, and obtaining the second standard output data further includes the following steps: Obtain historical environmental data and sensor output data of the industrial robot under environmental data other than the reference environmental data, and obtain the first standard output data S according to the sensor output data; let Sbase represent the sensor output data of the industrial robot under the reference environmental data, generate an interval of length 2e with m as the center, e represents noise, m∈[Se,S+e]; according to the probability density function under the reference environmental data and the environmental data other than the reference environmental data, obtain the failure probability P1 under the reference environmental data and the failure probability P2 under the environmental data other than the reference environmental data, , according to P1, determine the m that minimizes the deviation between P1 and P2 in the interval [me,m+e], , according to m, we get P2, P2= ; Wherein f1(x) represents the probability density function under the reference environmental data, and f2(x) represents the probability density function under environmental data other than the reference environmental data; based on the m determined to minimize the deviation between P1 and P2, the second standard output data is obtained; let y1 and y2 represent the first standard output data and the second standard output data, then y2=y1+(mS).

5. The fault diagnosis method based on deep reinforcement learning according to claim 4 is characterized in that: The method of training the deep reinforcement learning model of the fault type according to the second standard output data of all sensors further includes the following steps: The fault types of industrial robots are binary-encoded, a DQN deep neural network is constructed, the second standard output data of the sensor is used as input to form a state space, the fault types of the industrial robot are used as action space, the reward function and hyperparameters of the DQN neural network are set, and the DQN neural network model is trained.

6. A fault diagnosis system based on deep reinforcement learning, characterized by: It includes a data acquisition module, a data storage module, a data analysis module and a fault diagnosis module; the data acquisition module obtains the operating data and environmental data of the industrial robot through sensors; the data storage module is used to store the historical operating data, environmental data and fault data of the industrial robot; the data analysis module performs reverse compensation on the historical output data of the sensor to remove the deviation caused by environmental factors and obtain first standard output data; analyzes the failure probability of the industrial robot under different environmental factors, adjusts the reverse compensation method, removes the deviation caused by environmental factors, and obtains second standard output data; the fault diagnosis module diagnoses the fault type of the industrial robot through a deep reinforcement learning model of the fault type.

7. The fault diagnosis system based on deep reinforcement learning according to claim 6, characterized in that: The fault diagnosis module further includes a recovery unit and a deep reinforcement learning unit; the deep reinforcement learning unit trains a deep reinforcement learning model of the fault type based on the second standard output data of the sensor, and performs fault diagnosis on the industrial robot based on the deep reinforcement learning model of the fault type; The recovery unit performs a recovery operation of the industrial robot according to the fault type of the industrial robot.

8. The fault diagnosis system based on deep reinforcement learning according to claim 6, characterized in that: The data analysis module also includes a fitting unit, an estimation unit and an optimization unit; the fitting unit trains a fitting model between the difference between the sensor output data and the environmental data based on the difference caused by the environmental data, and performs reverse compensation on the historical output data of the sensor according to the fitting model to remove the deviation caused by the environmental factors to obtain the first standard output data; the estimation unit performs kernel density estimation on the first standard output data of each sensor to obtain a probability density function between the failure probability of the industrial robot and the first standard output data of the sensor; the optimization unit analyzes the failure probability of the industrial robot under different environmental factors, adjusts the reverse compensation method, removes the deviation caused by the environmental factors, and obtains the second standard output data.

9. The fault diagnosis system based on deep reinforcement learning according to claim 8, characterized in that: The optimization unit obtains historical environmental data and sensor output data of the industrial robot under environmental data other than the reference environmental data, and obtains first standard output data S according to the sensor output data; Let Sbase represent the sensor output data of the industrial robot under the reference environment data, generate an interval of length 2e with m as the center, e represents noise, m∈[Se,S+e]; according to the probability density function under the reference environment data and other environment data outside the reference environment data, the failure probability P1 under the reference environment data and the failure probability P2 under other environment data outside the reference environment data are obtained. , according to P1, determine the m that minimizes the deviation between P1 and P2 in the interval [me,m+e], , according to m, we get P2, P2= ; Wherein f1(x) represents the probability density function under the reference environmental data, and f2(x) represents the probability density function under environmental data other than the reference environmental data; based on the m determined to minimize the deviation between P1 and P2, the second standard output data is obtained.

Images