Industrial robot dynamic feedback normalization reinforcement learning optimization method and system

Abstract

The invention discloses an industrial robot dynamic feedback normalization reinforcement learning optimization method and system. The method comprises the following steps: S1, constructing an experience playback buffer area D, Q network parameter theta and a target network parameter theta-in an industrial robot memory; s2, the industrial robot executes an action based on a control instruction output by the current Q network, and an original feedback signal rraw and a next state st + 1 are observed; s3, performing nonlinear normalization processing on the original feedback signal rraw; s4, if the industrial robot triggers a system failure protection mechanism, determining a negative system failure penalty term p; s5, the complete state transition tuple of the final feedback signal rfinal is stored in an experience playback buffer area D; s6, sampling training data from the experience playback buffer area D, calculating a target Q value, and updating a Q network parameter theta; and S7, when a preset network updating condition is satisfied, synchronizing the Q network parameter theta to the target network theta-, and updating the parameters alpha t and beta t in the nonlinear normalization based on the gradient statistics.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent control, and particularly relates to an industrial robot dynamic feedback normalization reinforcement learning optimization method and system. BACKGROUND

[0002] In the real-time path planning task of the industrial robot control system, the traditional reinforcement learning algorithm has three technical defects due to the limitation of thousands of training data:

[0003] 1) Hard clipping to [-1, 1] linear feedback normalization causes high-value feedback signals to be compressed, reducing the sensitivity of the agent to key events;

[0004] 2) Lack of survival constraint mechanism, frequent collisions of industrial robots increase hardware wear and tear, resulting in high maintenance costs;

[0005] 3) Fixed step n-step update causes reward estimation bias in the early training stage, and the bias rate is high, and the convergence speed decreases in the later stage due to the fixed n value.

[0006] In the disclosed CN114571459A industrial robot fault diagnosis method and system technical solution, the theoretical frequency of each component of the running joint is calculated according to the running speed of the running joint and the model of the industrial robot, and the theoretical frequency of each component is compared and analyzed with the frequency f_vel obtained by frequency spectrum acquisition, and the actual frequency f_acc is compared and analyzed in turn, thereby identifying the fault position and outputting an analysis report. However, this scheme does not have the ability to analyze and evaluate dynamic fault conditions, which requires technical personnel in the field to solve the corresponding technical problems. SUMMARY

[0007] The present application aims to solve the technical problems existing in the prior art, and particularly innovatively proposes an industrial robot dynamic feedback normalization reinforcement learning optimization method and system.

[0008] In order to achieve the above-mentioned purpose of the present application, the present application provides an industrial robot dynamic feedback normalization reinforcement learning optimization method, which comprises the following steps: S1, constructing an experience replay buffer D, Q network parameters θ and target network parameters θ- in the memory of the industrial robot, wherein the Q network is used to fit the state-action value function;

[0009] S2, the industrial robot performs an action based on the control instruction output by the current Q network, and observes the original feedback signal r raw and the next state s t+1 from the physical environment sensor;

[0010] S3, the original feedback signal r rawnonlinear normalization processing is performed to obtain a normalized feedback signal r norm =f(r raw ; a t , b t ), wherein a t is a dynamic compression coefficient, b t is a semi-saturation coefficient; f() represents nonlinear normalization processing;

[0011] S4, if the industrial robot triggers the system failure protection mechanism in step S2, a system failure penalty term p of a negative value is superimposed on the normalized feedback signal r norm , to obtain a final feedback signal;

[0012] S5, the complete state transition tuple (s t , a t , r final , s t+1 , done t , priority t ) of the final feedback signal r final is stored in the experience replay buffer D; wherein done t ∈{0,1} is a termination flag, and priority t is a priority value;

[0013] S6, training data is sampled from the experience replay buffer D, and a target Q value is calculated through a dynamically adjusted n-step return calculation mechanism and a dynamically adjusted discount factor g t , and the Q network parameter theta is updated based on the target Q value;

[0014] S7, when a preset network update condition is met, the Q network parameter theta is synchronized to the target network theta-, and the parameters a t and b t in the nonlinear normalization are updated based on gradient statistics.

[0015] In the above scheme, the S1 comprises:

[0016] In the industrial robot initialization stage, before the experience replay buffer D is constructed in the non-volatile memory of the controller, the multi-source heterogeneous data fusion module is used to perform unified dimension processing on the sensor data of the industrial robot.

[0017] In the above scheme, the unified dimension processing comprises:

[0018] Through a physical energy normalization framework, each physical quantity is uniformly mapped to a standardized mechanical efficiency space, and a unified efficiency function is defined:

[0019] E unified [t]=w1·E kin [t]+w2·Epot [t]+w3·E diss [t]+w4·E task [t];

[0020] E unified [t] is the unified performance index at time t; E kin [t] represents the kinetic energy term at time t; E pot [t] represents the potential energy term at time t; E diss [t] represents the dissipation energy term at time t; E task [t] represents the task energy term at time t; w1, w2, w3, and w4 are weighting coefficients.

[0021] S1-3. Calculate the dissipation efficiency component and perform temperature drift compensation;

[0022] Where Δt is the control period, T ref =25℃ is the reference temperature, and the exponential term compensates for temperature drift. This represents the highest dissipation efficiency in history. Let τ be the energy dissipation increment of the j-th part within a time step Δt; j [t] represents the j-th generalized force at time t. Let T be the j-th velocity at time t; ref For reference time, λ T [t] is the time correction factor at time t, where T j [t] represents the j-th time variable at time t;

[0023] S1-4. Calculate the task performance component, and dynamically weighted fusion of sensor reliability and communication quality.

[0024] Where, p target [t] represents the target pose data; d max For the maximum permissible tracking error, δ max To determine the maximum allowable communication delay, c s [t] represents the sensor reliability weighting, δ comm [t] represents the delay penalty; p[t] is the position vector at time t;

[0025] S1-5. Calculate the adaptive weight dynamic adjustment mechanism, and calculate the weight vector w = [w1, w2, w3, w4] adaptively using task relevance entropy:

[0026]

[0027] Among them, T w =0.5 is the temperature parameter, which controls the sparsity of the weight distribution, s 1,2,3,4[t] represents the task relevance score of all components involved in the weight calculation, s i [t] represents the task relevance score of the i-th component at time t:

[0028] Used for calculating the average angular velocity ratio, reflecting the intensity of motion. This is the maximum angular velocity;

[0029] Used for calculating the variance of height H variation, reflecting the frequency of vertical movement, H max Maximum height;

[0030] Used for calculating the percentage of negative work done, reflecting the degree of energy loss, P max This represents the maximum power loss.

[0031] s4[t]=1-η task [t], used for calculating the task success rate complement, η task [t] represents the task completion rate of the most recent 100 steps;

[0032] S1-6. Construct a unified state vector, fusing multi-source data into a state vector with unified dimensions.

[0033] Each component is normalized to the interval [0, 1] using its maximum value, {d s} = 7 represents the state space dimension; this vector serves as the input for subsequent reinforcement learning, ensuring the comparability of heterogeneous physical quantities in a unified performance space; s t Let E be the state vector at time t. unified [t] is the unified performance index at time t, τ max For the greatest generalized force, For the maximum generalized speed, E unified [t] is the unified performance index at time t, [ ] T c is the transpose function; s [t] represents the task state coefficient at time t; λ T [t] is the decay coefficient at time t; η task [t] represents the task efficiency index at time t.

[0034] In the above scheme, S1 further includes:

[0035] Establish a sparse tracker and maintain a sliding window buffer W. s Used to store the raw feedback signal {r} from the most recent L steps. raw Construct a first-order low-pass filter whose state variable is the filtered feedback trend value r. filtered The smoothing coefficient is λf Preset sparsity critical value η c Signal-to-noise ratio (SNR) benchmark b This forms a signal-to-noise ratio estimator;

[0036] The capacity of the experience playback buffer D is N. D A circular queue structure is used to store state transition tuples;

[0037] The Q-network is a fully connected feedforward neural network with an input layer dimension of d. s +d a , where d s d represents the dimension of the robot's state space. a The action space dimension is represented by d, the hidden layer uses the ReLU activation function, and the output layer dimension is d. a ,parameter M is the total number of network parameters, M = (d s +d a )·h+h2+h·d a +h+d a h is the number of neurons in the hidden layer; the structure of the target network θ- is completely consistent with the Q network and is initialized by hard copying: θ-←θ.

[0038] In the above scheme: S2 includes:

[0039] S2-1, Robot task execution cycle during the data acquisition phase Within the system, the industrial robot selects action a based on a greedy strategy derived from the current Q-network output. t =argmax {a∈A} Q(s t (a, θ), in the t-th control cycle, according to the current state s t The selected specific action; and sends control commands to the servo driver via the industrial bus. The control commands are converted from D / A to drive the joint motor; simultaneously, feedback signals from the physical environment are collected through multi-axis force / torque sensors and encoders, and converted from A / D to obtain: where a is a single action of the industrial robot, and A is the complete set of motion space;

[0040] S2-2, calculate a t Then, put a t The data is sent to the servo system to obtain new sensor readings, which are then used to calculate r. raw and the state vector s of the next control period t+1 t+1 Therefore a t It is a crucial bridge connecting decision-making with data for the next moment.

[0041] Original feedback signal Let r be the set of real numbers, and its physical meaning is the reciprocal of the trajectory tracking error of the end effector. raw =1 / (||p desired -p actual ||2+ε), p desired p is the desired pose vector. actual The actual pose vector, ε = 10 -6 To avoid the smallest constant of division by zero; the next state vector Includes the angles q of each joint i i = 1, ..., n joints angular velocity and end force / torque measurement value F ext .

[0042] In the above scheme: S3 includes:

[0043] S3-1. During the industrial robot signal preprocessing stage, the original feedback signal r... raw Adaptive

[0044] Nonlinear normalization processing Where sign(·) is the sign function, used to preserve the feedback polarity; α t ∈(0,1] represents the dynamic compression coefficient;

[0045] S3-2, based on the sparsity index η of the current training phase t. t Adaptive adjustment: α t =max(α) min ,1-k α ·tanh((η t -η c ) / δ η )), η t Let α be the percentage of zero feedback in the most recent k episodes; where α min =0.3, which is the minimum allowable compression factor to prevent excessive compression from causing signal distortion; k α =0.5 is the sparsity sensitivity coefficient, used to adjust the effect of sparsity changes on α. t The magnitude of the impact; based on signal sparsity α t The source of the dynamic calculation is the dynamic compression coefficient α. t The real-time sparsity index η is obtained through the following steps. t To confirm;

[0046] η t ∈[0,1] represents the dynamic sparsity index at the current time t, which is calculated based on the sliding window buffer W. s Internal feedback statistical characteristics; η c =0.7 is the sparsity critical value, when ηt When this value is exceeded, the feedback signal is considered highly sparse; δ η =0.1 is the sparsity smoothing parameter, which controls the transition bandwidth of the tanh function;

[0047] S3-3, Half-saturation coefficient β formed based on signal-to-noise ratio estimation t The dynamic calculation source is derived from the half-saturation coefficient β. t The signal-to-noise ratio (SNR) of the feedback signal is determined through the following steps. t Sure: Where β0 = 1.0, is the baseline half-saturation coefficient; k β =0.5, which is the signal-to-noise ratio gain coefficient used to adjust the SNR. t For β t Scope of influence; The feedback signal-to-noise ratio (SNR) at the current time t is physically defined as the ratio of the power of the effective trend component to the power of random noise in the feedback signal. b =15dB is the baseline value for signal-to-noise ratio, corresponding to a medium signal-to-noise ratio level in typical industrial robot tasks; σ SNR =5dB is the signal-to-noise ratio scaling parameter, which controls the transition slope of the arctan function.

[0048] In the above scheme: S3 includes:

[0049] S3-4, the dynamic sparsity index η t For: η t =w′1·η zero +w'2·η kurt +w'3·η TotalVariation ;

[0050] in,

[0051] For a zero feedback ratio, ε zero =10⁻³ is the threshold close to zero, and I(·) is the indicator function;

[0052] The normalized kurtosis sparsity index is given by μ₄, where μ₂ is the fourth central moment and K is the second central moment. max =10 is the upper limit of normalization;

[0053] r is a sparse index of total variation range This represents the difference between the maximum and minimum values ​​within the sliding window.

[0054] w1'=0.5, w'2=0.3, w'3=0.2 are weighting coefficients, satisfying w1'+w'2+w'3=1;

[0055] S3-5, the feedback signal-to-noise ratio (SNR) t The calculation uses an adaptive trend separation method:

[0056] in,

[0057] For the trend signal power, r filtered,i T represents the smoothed feedback trend value of the low-pass filter output. s =0.01s is the sampling period; L is the number of signal samples used in the power calculation;

[0058] For noise power, r residual,i =r raw,i -r filtered,i This is residual noise;

[0059] The low-pass filter is implemented using a first-order IIR filter: r filtered,t =λ f ·r filtered,t-1 +(1-λ f )·r raw,t , where λ f =0.95 is the smoothing coefficient, which is dynamically updated through the adaptive law of the forgetting factor: λ f ←clip(λ f +μ λ ·(P signal,t -P noise,t ),0.9,0.99),μ λ =10 -4 It is an adaptive learning rate.

[0060] In the above scheme: S6 includes:

[0061] S6-1, During the network update phase, data is retrieved from the experience replay buffer D according to priority and probability. Sample batch data B = {(s t ,a t ,r final ,s t+1 ,done t Batch size |B| = m, where m ∈ [32, 256], priority t Used only for calculating sampling probability P t This will not be included in the field of B; the target Q value is calculated through dynamically adjusted n-step returns and a dynamic discount factor:

[0062] Where k is the step number, k = 0 is the current step, k = 1 is the next step, and so on; n t For dynamic step size; Let r be the k-th decay coefficient at time t. i+k For the (i+k)th input signal, For time t, the nth t One attenuation coefficient, Action a at time t t The maximum value, Q() is the action value function. For the i+nth t A state vector;

[0063] The discount factor is dynamic, γ0 = 0.9 is the initial discount rate, and γ ∞ =0.99 is the asymptotic discount rate, τ γ =10 4 It is a time constant;

[0064] The loss function is calculated using Huber loss.

[0065] in,

[0066] δ i =y i -Q(s i a i ;θ) represents the time-series difference error, where s i Let a be the state vector of the i-th sample. i For the action of the i-th sample, y i Let be the target value for the i-th sample; κ = 1.0 is the threshold of the loss function;

[0067] The importance sampling weights are represented by I(*), which is the indicator function, set to 1 if the condition is met and 0 otherwise, used to switch between the quadratic and linear regions of the Huber loss; β = 0.4 is the annealing parameter; the learning rate of the Adam optimizer is η = 3 × 10⁻⁴. -4 The first-order moment decay coefficient β1 = 0.9, the second-order moment decay coefficient β2 = 0.999, and gradient descent is performed to update θ;

[0068] The overall error is obtained through Huber loss L(θ), where L(θ) represents δ. i Perform truncation smoothing to prevent outliers from driving up the gradient; simultaneously use importance sampling weights w i The distribution bias caused by priority sampling is offset; then L(θ) is handed over to the Adam optimizer, and Adam performs a gradient descent on the gradient of L(θ) with respect to θ to obtain new network parameters; at the update period specified by S7 (mod(t, C) = 0), these new parameters are copied to the target network θ- in a soft update manner.

[0069] S6-2, The dynamically adjusted dynamic step size n of the n-step return calculation mechanism is adopted. t Through the convergence index ζ t Adaptive determination:

[0070]

[0071] Where, n max =5 is the maximum number of steps, Δt n =5000 is the step size adjustment period; n min =1, representing the maximum and minimum step sizes respectively; Δn=4, representing the step size adjustment range; ζ is the infinite norm of the gradient of the current Q-network loss function, used as a convergence metric; c =0.1 is the convergence critical value; δ ζ =0.05 is the convergence scaling parameter; sigmoid(*) is the sigmoid activation function;

[0072] S6-3, through the calculation mechanism of the dynamic discount factor, based on the dynamic discount factor γ t Through task progress indicators τ task and exploration level index ε explore Jointly determined:

[0073] Where γ0 = 0.9 is the initial discount rate, γ ∞ =0.99 is the asymptotic discount rate, τ γ =10 4 It is a time constant;

[0074] τ task ∈[0,1] represents the task success rate, N success,t N represents the number of tasks successfully completed in the last 1000 steps. total,t This represents the total number of attempted tasks.

[0075] ∈ explore ∈[0,1] exploration rate, ∈ decay =10 5 To explore the decay step count; w ε =0.3 is the exploration weight coefficient.

[0076] The present invention also discloses a computer system, which includes a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the described industrial robot dynamic feedback normalization reinforcement learning optimization method.

[0077] In summary, the beneficial effects of this invention are:

[0078] The entire solution integrates industrial robots and reinforcement learning, from raw data to safety constraints, training efficiency to network convergence. Compared to traditional teaching or fixed PID parameter tuning, its advantages are mainly reflected in the following four aspects:

[0079] Unified units and dynamic n-step / discount reduce the number of interactions required for collision-free stable operation, η t SNR t Online adjustment of alpha t β t High-error samples are automatically weighted to avoid sample scarcity; temperature exponential decay, credibility-delay weighting, and task entropy weights are corrected in real time, reducing the impact of heat, wear, and communication packet loss on control accuracy; adaptive soft update τ and Hessian scaling suppress network divergence, reducing training variance by an order of magnitude and ensuring no performance drift in long-term operation; enabling robots to learn fast, stable, and safe motion strategies in complex, "invisible" workshops with minimal trial and error and in the shortest time. Attached Figure Description

[0080] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0081] Figure 1 This is a schematic diagram of the workflow of the present invention;

[0082] Figure 2 This is a comparison chart of linear and nonlinear feedback normalization in this invention. Detailed Implementation

[0083] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0084] like Figure 1 and 2 As shown: This invention discloses a dynamic feedback normalized reinforcement learning optimization method for industrial robots, comprising:

[0085] S1. Construct an experience playback buffer D, Q network parameters θ, and target network parameters θ- in the industrial robot's memory, wherein the Q network is used to fit the state-action value function;

[0086] During the initialization phase of an industrial robot, before constructing an experience playback buffer D in the controller's non-volatile memory, a multi-source heterogeneous data fusion module is used to perform unified dimensional processing on the data from various sensors of the industrial robot. The specific technical steps are as follows:

[0087]

[0088] Using a physical energy normalization framework, all physical quantities are uniformly mapped to a standardized mechanical efficiency space, and a unified efficiency function is defined:

[0089] E unified [t] = w1·E kin [t]+w2·E pot [t]+w3·E diss [t]+w4·E task [t];

[0090] E unified [t] is the unified performance index at time t; E kin [t] represents the kinetic energy term at time t; E pot [t] represents the potential energy term at time t; E diss [t] represents the dissipation energy term at time t; E task [t] represents the task energy term at time t; w1, w2, w3, and w4 are weighting coefficients.

[0091] S1-1, Calculate the kinetic energy efficiency component. Among them I j The equivalent moment of inertia of the j-th axis is kg·m 2 , This represents the historical maximum kinetic energy used for normalization; Let be the squared term of the j-th axis at time t, and n be the number of moving parts or degrees of freedom involved in the kinetic energy calculation;

[0092] S1-2, Calculate the potential energy efficiency component. Where m j For the mass of the connecting rod, g = 9.81 m / s² 2 h is the acceleration due to gravity. j [t] represents the height of the centroid. This represents the greatest potential energy in history.

[0093] S1-3. Calculate the dissipation efficiency component and perform temperature drift compensation;

[0094] Where Δt is the control period, T ref =25℃ is the reference temperature, and the exponential term compensates for temperature drift. This represents the highest dissipation efficiency in history. Let τ be the energy dissipation increment of the j-th part within a time step Δt; j [t] represents the j-th generalized force at time t. Let T be the j-th velocity at time t; refFor reference time, λ T [t] is the time correction factor at time t, where T j [t] represents the j-th time variable at time t;

[0095] S1-4. Calculate the task performance component, and dynamically weighted fusion of sensor reliability and communication quality.

[0096] Where, p target [t] represents the target pose data; d max For the maximum permissible tracking error, δ max To determine the maximum allowable communication delay, c s [t] represents the sensor reliability weighting, δ comm [t] represents the delay penalty; p[t] is the position vector at time t;

[0097] S1-5. Calculate the adaptive weight dynamic adjustment mechanism, and calculate the weight vector w = [w1, w2, w3, w4] adaptively using task relevance entropy:

[0098]

[0099] Among them, T w =0.5 is the temperature parameter, which controls the sparsity of the weight distribution, s 1,2,3,4 [t] represents the task relevance score of all components involved in the weight calculation, s i [t] represents the task relevance score of the i-th component at time t:

[0100] Used for calculating the average angular velocity ratio, reflecting the intensity of motion. This is the maximum angular velocity; Used for calculating the variance of height H variation, reflecting the frequency of vertical movement, H max Maximum height;

[0101] Used for calculating the percentage of negative work done, reflecting the degree of energy loss, P max This represents the maximum power loss.

[0102] s4[t]=1-η task [t], used for calculating the task success rate complement, η task [t] represents the task completion rate of the most recent 100 steps;

[0103] S1-6. Construct a unified state vector, fusing multi-source data into a state vector with unified dimensions.

[0104] Each component is normalized to the interval [0, 1] using its respective maximum value, {d s} = 7 represents the state space dimension; this vector serves as the input for subsequent reinforcement learning, ensuring the comparability of heterogeneous physical quantities in a unified performance space; s t Let E be the state vector at time t. unified [t] is the unified performance index at time t, τ max For the greatest generalized force, For the maximum generalized speed, E unified [t] is the unified performance index at time t, [ ] T c is the transpose function; s [t] represents the task state coefficient at time t; λ T [t] is the decay coefficient at time t; η task [t] represents the task efficiency index at time t;

[0105] S1-7. Construct an experience playback buffer D, Q network parameters θ, and target network parameters θ- in the controller's non-volatile memory, where:

[0106] Establish a sparsity tracker and maintain a sliding window buffer W of length L = 1000. s Used to store the raw feedback signal {r} from the most recent L steps. raw Construct a first-order low-pass filter whose state variable is the filtered feedback trend value r. filtered The smoothing coefficient is λ f =0.95; Preset sparsity critical value η c =0.7, Signal-to-Noise Ratio (SNR) reference value b =15dB, forming a signal-to-noise ratio estimator;

[0107] The capacity of the experience playback buffer D is N. D N D ∈[10 5 10 6 A circular queue structure is used to store state transition tuples;

[0108] The Q-network is a fully connected feedforward neural network with an input layer dimension of d. s +d a , where d s d represents the dimension of the robot's state space. a The action space dimension is represented by d, the hidden layer uses the ReLU activation function, and the output layer dimension is d. a ,parameter M is the total number of network parameters, M = (d s +d a )·h+h2+h·d a +h+da h is the number of neurons in the hidden layer; the structure of the target network θ- is completely consistent with the Q network and is initialized by hard copying: θ-←θ.

[0109] In the aforementioned unified dimensional calculation, a temperature exponential decay factor is introduced into traditional power calculations to address the sensor data distortion problem caused by thermal deformation of industrial robots; sensor reliability-communication delay joint weighting: through c s [t] and δ comm The product form of [t] enables the coupling of multi-dimensional quality assessment, rather than a simple linear superposition; the weights of each physical quantity are dynamically adjusted using task relevance scores to achieve a unified performance index E. unified [t] It has task awareness capabilities, which is different from traditional fixed-weight fusion methods.

[0110] S2. The industrial robot executes actions based on the control commands output by the current Q network, and observes the raw feedback signal r from the physical environment sensors. raw and the next state s t+1 ;

[0111] S2-1, Robot task execution cycle during the data acquisition phase Within the system, the industrial robot selects action a based on a greedy strategy derived from the current Q-network output. t =argmax {a∈A} Q(s t (a, θ), in the t-th control cycle, according to the current state s t The selected specific action; and sends control commands to the servo driver via the industrial bus. The control commands are converted from D / A to drive the joint motor; simultaneously, feedback signals from the physical environment are collected through multi-axis force / torque sensors and encoders, and converted from A / D to obtain: where a is a single action of the industrial robot, and A is the complete set of motion space;

[0112] S2-2, calculate a t Then, put a t The data is sent to the servo system to obtain new sensor readings, which are then used to calculate r. raw and the state vector s of the next control period t+1 t+1 Therefore a t It is a crucial bridge connecting decision-making with data for the next moment.

[0113] Original feedback signal Let r be the set of real numbers, and its physical meaning is the reciprocal of the trajectory tracking error of the end effector. raw =1 / (||p desired -p actual ||2+ε), p desired p is the desired pose vector.actual The actual pose vector, ε = 10 -6 To avoid the smallest constant of division by zero; the next state vector Includes the angles q of each joint i i = 1, ..., n joints angular velocity and end force / torque measurement value F ext .

[0114] S3, regarding the original feedback signal r raw Perform nonlinear normalization processing r norm =f(r raw ;α t ,β t ), to obtain the normalized feedback signal, where α t For dynamic compression coefficient, β t The half-saturation coefficient;

[0115] S3-1. During the industrial robot signal preprocessing stage, the original feedback signal r... raw Perform adaptive nonlinear normalization. Where sign(·) is the sign function, used to preserve the feedback polarity; α t ∈(0,1] represents the dynamic compression coefficient;

[0116] S3-2, based on the sparsity index η of the current training phase t. t Adaptive adjustment: α t =max(α) min ,1-k α ·tanh((η t -η c ) / δ η )), η t Let α be the percentage of zero feedback in the most recent k episodes; where α min =0.3, which is the minimum allowable compression factor to prevent excessive compression from causing signal distortion; k α =0.5 is the sparsity sensitivity coefficient, used to adjust the effect of sparsity changes on α. t The magnitude of the impact; based on signal sparsity α t The source of the dynamic calculation is the dynamic compression coefficient α. t The real-time sparsity index η is obtained through the following steps. t To confirm;

[0117] η t ∈[0,1] represents the dynamic sparsity index at the current time t, which is calculated based on the sliding window buffer W. s Internal feedback statistical characteristics; η c =0.7 is the sparsity critical value, when ηt When this value is exceeded, the feedback signal is considered highly sparse; δ η =0.1 is the sparsity smoothing parameter, which controls the transition bandwidth of the tanh function;

[0118] S3-3, Half-saturation coefficient β formed based on signal-to-noise ratio estimation t The dynamic calculation source is derived from the half-saturation coefficient β. t The signal-to-noise ratio (SNR) of the feedback signal is determined through the following steps. t Sure: Where β0 = 1.0, is the baseline half-saturation coefficient; k β =0.5, which is the signal-to-noise ratio gain coefficient used to adjust the SNR. t For β t Scope of influence; The feedback signal-to-noise ratio (SNR) at the current time t is physically defined as the ratio of the power of the effective trend component to the power of random noise in the feedback signal. b =15dB is the baseline value for signal-to-noise ratio, corresponding to a medium signal-to-noise ratio level in typical industrial robot tasks; σ SNR =5dB is the signal-to-noise ratio scaling parameter, which controls the transition slope of the arctan function;

[0119] S3-4, the dynamic sparsity index η t For: η t =w′1·η zero +w'2·η kurt +w'3·η TotalVariation ;

[0120] in,

[0121] For a zero feedback ratio, ε zero =10⁻³ is the threshold close to zero, and I(·) is the indicator function;

[0122] The normalized kurtosis sparsity index is given by μ₄, where μ₂ is the fourth central moment and K is the second central moment. max =10 is the upper limit of normalization;

[0123] r is a sparse index of total variation range This represents the difference between the maximum and minimum values ​​within the sliding window.

[0124] w1'=0.5, w'2=0.3, w'3=0.2 are weighting coefficients, satisfying w1'+w'2+w'3=1;

[0125] S3-5, the feedback signal-to-noise ratio (SNR) t The calculation uses an adaptive trend separation method:

[0126] in,

[0127] For the trend signal power, r filtered,i T represents the smoothed feedback trend value of the low-pass filter output. s =0.01s is the sampling period; L is the number of signal samples used in the power calculation;

[0128] For noise power, r residual,i =r raw,i -r filtered,i This is residual noise;

[0129] The low-pass filter is implemented using a first-order IIR filter: r filtered,t =λ f ·r filtered,t-1 +(1-λ f )·r raw,t , where λ f =0.95 is the smoothing coefficient, which is dynamically updated through the adaptive law of the forgetting factor: λ f ←clip(λ f +μ λ ·(P signal,t -P noise,t ),0.9,0.99),μ λ =10 -4 It is an adaptive learning rate.

[0130] Due to α t Compressing abnormal feedback will result in a more negative r for the same physical collision. final The policy gradient will rapidly lower the probability of that action; high η t High priority caused t Repeatedly reusing hazardous samples is equivalent to increasing the weight of safety incidents in the training set, thus improving the network's sensitivity to boundary moments and collision forces. In summary, η t It not only adjusts the shape of the reward, but also significantly shortens the time required for the robot to learn self-protection by amplifying the punishment and increasing the weight of dangerous sample replays, and provides a quantitative safety indicator that can be read in real time on site.

[0131] S4. If the industrial robot triggers the system failure protection mechanism in step S2, then in the normalized feedback signal r norm The final feedback signal is obtained by superimposing a negative system failure penalty term p.

[0132] During the safety constraint phase, if the industrial robot safety protection mechanism is triggered in S2, the specific determination condition is that the joint torque τ... iExceeding the rated value τ max 85% or end impact force F ext Exceeding threshold F threshold Then in the normalized feedback r norm System failure penalty items with superimposed negative values: Where λ∈[5,10] is the penalty intensity coefficient, set according to the task risk level; c=0.5 is the attenuation constant; Δt safety The time step is the time since the last safety trigger; the death penalty term is added together using a formula to form the final feedback signal.

[0133] Where δ is the collision penalty coefficient, for example, δ = -5, and the trigger condition is the distance d between the robot and the obstacle. safety <d threshold For example, d threshold =10cm; I crash This is a collision indicator function, which is 1 when a collision occurs and 0 otherwise. The safety distance weight, such as λ = 0.1, is used to penalize actions that approach dangerous areas; d safety The safe distance vector, for example [10cm, 20cm], is measured in real time by the lidar.

[0134] S5, the final feedback signal r final The complete state transition tuple, (s t ,a t ,r final ,s t+1 ,done t priority t Store in the experience replay buffer D;

[0135] The data storage phase will store the state transition tuple (s) t ,a t ,r final ,s t+1 ,done t priority t Store it in the experience replay buffer D, where done t ∈{0, 1} is the termination flag, set to 1 when the task is completed or a security mechanism is triggered; priority t =|δ t |+ζ represents the priority value, δ t =r final +γ·maxQ(s t+1 )-Q(s t a t ) represents the timing difference error, and ζ = 10⁻³ is a constant to avoid zero priority and is used for subsequent importance sampling.

[0136] S6. Sample training data from the experience replay buffer D, using a dynamically adjusted n-step reward calculation mechanism and a dynamically adjusted discount factor γ. t Calculate the target Q value and update the Q network parameters θ accordingly;

[0137] S6-1, During the network update phase, data is retrieved from the experience replay buffer D according to priority and probability. Sample batch data B = {(s t ,a t ,r final ,s t+1 ,done t Batch size |B| = m, where m ∈ [32, 256], priority t Used only for calculating sampling probability P t This will not be included in the field of B; the target Q value is calculated through dynamically adjusted n-step returns and a dynamic discount factor:

[0138] Where k is the step number, k = 0 is the current step, k = 1 is the next step, and so on; n t For dynamic step size; Let r be the k-th decay coefficient at time t. i+k For the (i+k)th input signal, For time t, the nth t One attenuation coefficient, Action a at time t t The maximum value, Q() is the action value function. For the i+nth t A state vector;

[0139] The discount factor is dynamic, γ0 = 0.9 is the initial discount rate, and γ ∞ =0.99 is the asymptotic discount rate, τ γ =10 4 It is a time constant;

[0140] The loss function is calculated using Huber loss.

[0141] in,

[0142] δ i =y i -Q(s i a i ;θ) represents the time-series difference error, where s i Let a be the state vector of the i-th sample. i For the action of the i-th sample, y iLet be the target value for the i-th sample; κ = 1.0 is the threshold of the loss function;

[0143] The importance sampling weights are represented by I(*), which is the indicator function, set to 1 if the condition is met and 0 otherwise, used to switch between the quadratic and linear regions of the Huber loss; β = 0.4 is the annealing parameter; the learning rate of the Adam optimizer is η = 3 × 10⁻⁴. -4 The first-order moment decay coefficient β1 = 0.9, the second-order moment decay coefficient β2 = 0.999, and gradient descent is performed to update θ;

[0144] The overall error is obtained through Huber loss L(θ), where L(θ) represents δ. i Perform truncation smoothing to prevent outliers from driving up the gradient; simultaneously use importance sampling weights w i The distribution bias caused by priority sampling is offset; then L(θ) is handed over to the Adam optimizer, and Adam performs a gradient descent on the gradient of L(θ) with respect to θ to obtain new network parameters; at the update period specified by S7 (mod(t, C) = 0), these new parameters are copied to the target network θ- in a soft update manner.

[0145] S6-2, The dynamically adjusted dynamic step size n of the n-step return calculation mechanism is adopted. t Through the convergence index ζ t Adaptive determination:

[0146]

[0147] Where, n max =5 is the maximum number of steps, Δt n =5000 is the step size adjustment period; n min =1, representing the maximum and minimum step sizes respectively; Δn=4, representing the step size adjustment range; ζ is the infinite norm of the gradient of the current Q-network loss function, used as a convergence metric; c =0.1 is the convergence critical value; δ ζ =0.05 is the convergence scaling parameter; sigmoid(*) is the sigmoid activation function;

[0148] S6-3, through the calculation mechanism of the dynamic discount factor, based on the dynamic discount factor γ t Through task progress indicators τ task and exploration level index ε explore Jointly determined:

[0149] Where γ0 = 0.9 is the initial discount rate, γ ∞ =0.99 is the asymptotic discount rate, τ γ =104 It is a time constant;

[0150] τ task ∈[0,1] represents the task success rate, N success,t N represents the number of tasks successfully completed in the last 1000 steps. total,t This represents the total number of attempted tasks.

[0151] ∈ explore ∈[0,1] exploration rate, ∈ decay =10 5 To explore the decay step count; w ε =0.3 is the exploration weight coefficient.

[0152] S7. When the preset network update conditions are met, synchronize the Q network parameters θ to the target network θ-, and update the parameters α in the nonlinear normalization based on the gradient statistics. t and β t .

[0153] S7-1, Parameter Synchronization Phase: When the update condition mod(t, C) = 0 is met, and C = 8000 (the update interval step number), execute:

[0154] The target network parameters are updated softly, θ-←(1-τ)·θ-+τ·θ, where τ=0.005 is the soft update coefficient;

[0155] Meta-learning of normalized parameters updates the compression coefficient α t β t The update not only relies on gradient information, but also introduces the diagonal element estimation of the Hessian matrix to achieve second-order adaptation:

[0156]

[0157] Where: μ α =10 -5 μ β =10 -5 The hyperparameter is the learning rate; clip(x, x) min x max () is a truncation function that ensures the parameter is within the valid range; Let L be the partial derivative of the loss function L with respect to α. The partial derivatives of the loss function L with respect to β are calculated using the chain rule, reflecting the impact of the normalization parameter on network convergence.

[0158] The Hessian diagonal element approximation of α is calculated using the finite difference method: Δα=10 -3 h is the difference step size;β Similarly, approximately; ρ = 10 -4 This is the damping coefficient, used to prevent division by zero.

[0159] S7-2. Adaptive Adjustment of the Target Network Soft Update Coefficient: The soft update coefficient τ is no longer fixed, but dynamically adjusted through the parameter difference Δθ.

[0160] Where, τ base =0.005 is the baseline update rate; θ scale =10.0 is the parameter difference scale; this mechanism slows down the update speed when the Q network and the target network are significantly different, and accelerates the fusion when the difference is small, thereby improving training stability.

[0161] The beneficial effect of the above technical solution is that the compression coefficient α is dynamically adjusted based on signal sparsity and signal-to-noise ratio. t β t This approach addresses the technical challenge of extremely uneven feedback distribution in industrial robot tasks, enhancing the controller's sensitivity to critical abnormal states. It quantifies robot safety triggering conditions into torque and collision force thresholds and designs a time-decreasing penalty term *p* to ensure sufficient negative feedback for high-risk actions, preventing physical damage. Furthermore, it adaptively adjusts *n* according to the training progress. t and γ t In the early stages of training, a short-range, low-discount strategy is employed to accelerate convergence, while a long-range strategy is switched to optimize steady-state performance in the later stages. Priority is calculated by combining temporal difference error and security event flags, ensuring that critical security-related samples are sampled more frequently, thus improving learning efficiency in high-risk scenarios. The normalization parameter α is directly optimized using the gradient of the loss function. t β t This enables joint training of feedback preprocessing and network learning.

Claims

1. A dynamic feedback normalized reinforcement learning optimization method for industrial robots, characterized in that, include: S1. Construct an experience playback buffer D, Q network parameters θ, and target network parameters θ- in the industrial robot's memory, wherein the Q network is used to fit the state-action value function; S2. The industrial robot executes actions based on the control commands output by the current Q network, and observes the raw feedback signal r from the physical environment sensors. raw and the next state s t+1 ; S3, regarding the original feedback signal r raw Nonlinear normalization is performed to obtain the normalized feedback signal r. norm =f(r raw ;α t ,β t ), where α t For dynamic compression coefficient, β t f is the half-saturation coefficient; f() represents the nonlinear normalization process; S4. If the industrial robot triggers the system failure protection mechanism in step S2, then in the normalized feedback signal r norm The final feedback signal is obtained by superimposing a negative system failure penalty term p. S5, the final feedback signal r final Complete state transition tuple (s t ,a t ,r final ,s t+1 ,done t priority t Store it in the experience replay buffer D; where done t ∈{0,1} is the termination marker, priority t This is the priority value; S6. Sample training data from the experience replay buffer D, using a dynamically adjusted n-step reward calculation mechanism and a dynamically adjusted discount factor γ. t Calculate the target Q value and update the Q network parameters θ accordingly; S7. When the preset network update conditions are met, synchronize the Q network parameters θ to the target network θ-, and update the parameters α in the nonlinear normalization based on the gradient statistics. t and β t .

2. The dynamic feedback normalized reinforcement learning optimization method for industrial robots according to claim 1, characterized in that, S1 includes: During the initialization phase of an industrial robot, before constructing an experience playback buffer D in the controller's non-volatile memory, the data from various sensors of the industrial robot need to be processed to a unified dimension through a multi-source heterogeneous data fusion module.

3. The dynamic feedback normalized reinforcement learning optimization method for industrial robots according to claim 2, characterized in that, The unified dimension processing includes: Using a physical energy normalization framework, all physical quantities are uniformly mapped to a standardized mechanical efficiency space, and a unified efficiency function is defined: E unified [t]=w1·E kin [t]+w2·E pot [t]+w3·E diss [t]+w4·E task [t]; E unified [t] is the unified performance index at time t; E kin [t] represents the kinetic energy term at time t; E pot [t] represents the potential energy term at time t; E diss [t] represents the dissipation energy term at time t; E task [t] represents the task energy term at time t; w1, w2, w3, and w4 are weighting coefficients. S1-3. Calculate the dissipation efficiency component and perform temperature drift compensation; Where Δt is the control period, T ref =25℃ is the reference temperature, and the exponential term compensates for temperature drift. This represents the highest dissipation efficiency in history. Let τ be the energy dissipation increment of the j-th part within a time step Δt; j [t] represents the j-th generalized force at time t. Let T be the j-th velocity at time t; ref For reference time, λ T [t] is the time correction factor at time t, where T j [t] represents the j-th time variable at time t; S1-4. Calculate the task performance component, and dynamically weighted fusion of sensor reliability and communication quality. Where, p target [t] represents the target pose data; d max For the maximum permissible tracking error, δ max To determine the maximum allowable communication delay, c s [t] represents the sensor reliability weighting, δ comm [t] represents the delay penalty; p[t] is the position vector at time t; S1-5. Calculate the adaptive weight dynamic adjustment mechanism, and calculate the weight vector w = [w1, w2, w3, w4] adaptively using task relevance entropy: Among them, T w =0.5 is the temperature parameter, which controls the sparsity of the weight distribution, s 1,2,3,4 [t] represents the task relevance score of all components involved in the weight calculation, s i [t] represents the task relevance score of the i-th component at time t: Used for calculating the average angular velocity ratio, reflecting the intensity of motion. This is the maximum angular velocity; Used for calculating the variance of height H variation, reflecting the frequency of vertical movement, H max Maximum height; Used for calculating the percentage of negative work done, reflecting the degree of energy loss, P max This represents the maximum power loss. s4[t]=1-η task [t], used for calculating the task success rate complement, η task [t] represents the task completion rate of the most recent 100 steps; S1-6. Construct a unified state vector, fusing multi-source data into a state vector with unified dimensions. Each component is normalized to the interval [0, 1] using its maximum value, {d s } = 7 represents the state space dimension; this vector serves as the input for subsequent reinforcement learning, ensuring the comparability of heterogeneous physical quantities in a unified performance space; s t Let E be the state vector at time t. unified [t] is the unified performance index at time t, τ max For the greatest generalized force, For the maximum generalized speed, E unified [t] represents the unified performance index at time t. T c is the transpose function; s [t] represents the task state coefficient at time t; λ T [t] is the decay coefficient at time t; η task [t] represents the task efficiency index at time t.

4. The dynamic feedback normalized reinforcement learning optimization method for industrial robots according to claim 1, characterized in that, S1 further includes: Establish a sparse tracker and maintain a sliding window buffer W. s Used to store the raw feedback signal {r} from the most recent L steps. raw Construct a first-order low-pass filter whose state variable is the filtered feedback trend value r. filtered The smoothing coefficient is λ f Preset sparsity critical value η c Signal-to-noise ratio (SNR) benchmark b This forms a signal-to-noise ratio estimator; The capacity of the experience playback buffer D is N. D A circular queue structure is used to store state transition tuples; The Q-network is a fully connected feedforward neural network with an input layer dimension of d. s +d a , where d s d represents the dimension of the robot's state space. a The action space dimension is represented by d, the hidden layer uses the ReLU activation function, and the output layer dimension is d. a ,parameter M is the total number of network parameters, M = (d s +d a )·h+h 2 +h·d a +h+d a h is the number of neurons in the hidden layer; the structure of the target network θ- is completely consistent with the Q network and is initialized by hard copying: θ-←θ.

5. The industrial robot dynamic feedback normalized reinforcement learning optimization method according to claim 1, characterized in that, S2 includes: S2-1, Robot task execution cycle during the data acquisition phase Within the system, the industrial robot selects action a based on a greedy strategy derived from the current Q-network output. t =argmax {a∈A} Q(s t (a, θ), in the t-th control cycle, according to the current state s t The selected specific action; and sends control commands to the servo driver via the industrial bus. The control commands are converted from D / A to drive the joint motor; simultaneously, feedback signals from the physical environment are collected through multi-axis force / torque sensors and encoders, and converted from A / D to obtain: where a is a single action of the industrial robot, and A is the complete set of motion space; S2-2, calculate a t Then, put a t The data is sent to the servo system to obtain new sensor readings, which are then used to calculate r. raw and the state vector s of the next control period t+1 t+1 Therefore a t It is a crucial bridge connecting decision-making with data for the next moment. Original feedback signal Let r be the set of real numbers, and its physical meaning is the reciprocal of the trajectory tracking error of the end effector. raw =1 / (||p desired -p actual ||2+ε), p desired p is the desired pose vector. actual The actual pose vector, ε = 10 -6 To avoid the smallest constant of division by zero; the next state vector Includes the angles q of each joint i i = 1, ..., n joints angular velocity and end force / torque measurement value F ext .

6. The dynamic feedback normalized reinforcement learning optimization method for industrial robots according to claim 1, characterized in that, S3 includes: S3-1. During the industrial robot signal preprocessing stage, the original feedback signal r... raw Perform adaptive nonlinear normalization. Where sign(·) is the sign function, used to preserve the feedback polarity; α t ∈(0,1] represents the dynamic compression coefficient; S3-2, based on the sparsity index η of the current training phase t. t Adaptive adjustment: α t =max(α) min ,1-k α ·tanh((η t -η c ) / δ η )), η t Let α be the percentage of zero feedback in the most recent k episodes; where α min =0.3, which is the minimum allowable compression factor to prevent excessive compression from causing signal distortion; k α =0.5 is the sparsity sensitivity coefficient, used to adjust the effect of sparsity changes on α. t The magnitude of the impact; based on signal sparsity α t The source of the dynamic calculation is the dynamic compression coefficient α. t The real-time sparsity index η is obtained through the following steps. t To confirm; η t ∈[0,1] represents the dynamic sparsity index at the current time t, which is calculated based on the sliding window buffer W. s Internal feedback statistical characteristics; η c =0.7 is the critical value for sparsity, when η t When this value is exceeded, the feedback signal is considered highly sparse; δ η =0.1 is the sparsity smoothing parameter, which controls the transition bandwidth of the tanh function; S3-3, Half-saturation coefficient β formed based on signal-to-noise ratio estimation t The dynamic calculation source is derived from the half-saturation coefficient β. t The signal-to-noise ratio (SNR) of the feedback signal is determined through the following steps. t Sure: Where β0 = 1.0, is the baseline half-saturation coefficient; k β =0.5, which is the signal-to-noise ratio gain coefficient used to adjust the SNR. t For β t Scope of influence; The feedback signal-to-noise ratio (SNR) at the current time t is physically defined as the ratio of the power of the effective trend component to the power of random noise in the feedback signal. b =15dB is the baseline value for signal-to-noise ratio, corresponding to a medium signal-to-noise ratio level in typical industrial robot tasks; σ SNR =5dB is the signal-to-noise ratio scaling parameter, which controls the transition slope of the arctan function.

7. The dynamic feedback normalized reinforcement learning optimization method for industrial robots according to claim 6, characterized in that, S3 includes: S3-4, the dynamic sparsity index η t For: η t =w1'·η zero +w'2·η kurt +w'3·η TotalVariation ; in, For a zero feedback ratio, ε zero =10⁻³ is the threshold close to zero, and I(·) is the indicator function; The normalized kurtosis sparsity index is given by μ₄, where μ₂ is the fourth central moment and K is the second central moment. max =10 is the upper limit of normalization; r is a sparse index of total variation range This represents the difference between the maximum and minimum values ​​within the sliding window. w1'=0.5, w'2=0.3, w'3=0.2 are weighting coefficients, satisfying w1'+w'2+w'3=1; S3-5, the feedback signal-to-noise ratio (SNR) t The calculation uses an adaptive trend separation method: in, For the trend signal power, r filtered,i T represents the smoothed feedback trend value of the low-pass filter output. s =0.01s is the sampling period; L is the number of signal samples used in the power calculation; For noise power, r residual,i =r raw,i -r filtered,i This is residual noise; The low-pass filter is implemented using a first-order IIR filter: r filtered,t =λ f ·r filtered,t-1 +(1-λ f )·r raw,t , where λ f =0.95 is the smoothing coefficient, which is dynamically updated through the adaptive law of the forgetting factor: λ f ←clip(λ f +μ λ ·(P signal,t -P noise,t ),0.9,0.99),μ λ =10 -4 It is an adaptive learning rate.

8. The dynamic feedback normalized reinforcement learning optimization method for industrial robots according to claim 1, characterized in that, S6 includes: S6-1, During the network update phase, data is retrieved from the experience replay buffer D according to priority and probability. Sample batch data B = {(s t ,a t ,r final ,s t+1 ,done t Batch size |B| = m, where m ∈ [32, 256], priority t Used only for calculating sampling probability P t This will not be included in the field of B; the target Q value is calculated through dynamically adjusted n-step returns and a dynamic discount factor: Where k is the step number, k = 0 is the current step, k = 1 is the next step, and so on; n t For dynamic step size; Let r be the k-th decay coefficient at time t. i+k For the (i+k)th input signal, For time t, the nth t One attenuation coefficient, Action a at time t t The maximum value, Q() is the action value function. For the i+nth t A state vector; The discount factor is dynamic, γ0 = 0.9 is the initial discount rate, and γ ∞ =0.99 is the asymptotic discount rate, τ γ =10 4 It is a time constant; The loss function is calculated using Huber loss. in, δ i =y i -Q(s i a i ;θ) represents the time-series difference error, where s i Let a be the state vector of the i-th sample. i For the action of the i-th sample, y i Let be the target value for the i-th sample; κ = 1.0 is the threshold of the loss function; The importance sampling weights are represented by I(*), which is the indicator function, set to 1 if the condition is met and 0 otherwise, used to switch between the quadratic and linear regions of the Huber loss; β = 0.4 is the annealing parameter; the learning rate of the Adam optimizer is η = 3 × 10⁻⁴. -4 The first-order moment decay coefficient β1 = 0.9, the second-order moment decay coefficient β2 = 0.999, and gradient descent is performed to update θ; The overall error is obtained through Huber loss L(θ), where L(θ) represents δ. i Perform truncation smoothing to prevent outliers from driving up the gradient; simultaneously use importance sampling weights w i The distribution bias caused by priority sampling is offset; then L(θ) is handed over to the Adam optimizer, and Adam performs a gradient descent on the gradient of L(θ) with respect to θ to obtain new network parameters; at the update period specified by S7 (mod(t, C) = 0), these new parameters are copied to the target network θ- in a soft update manner. S6-2, The dynamically adjusted dynamic step size n of the n-step return calculation mechanism is adopted. t Through the convergence index ζ t Adaptive determination: Where, n max =5 is the maximum number of steps, Δt n =5000 is the step size adjustment period; n min =1, representing the maximum and minimum step sizes respectively; Δn=4, representing the step size adjustment range; ζ is the infinite norm of the gradient of the current Q-network loss function, used as a convergence metric; c =0.1 is the convergence critical value; δ ζ =0.05 is the convergence scaling parameter; sigmoid(*) is the sigmoid activation function; S6-3, through the calculation mechanism of the dynamic discount factor, based on the dynamic discount factor γ t Through task progress indicators τ task and exploration level index ε explore Jointly determined: Where γ0 = 0.9 is the initial discount rate, γ ∞ =0.99 is the asymptotic discount rate, τ γ =10 4 It is a time constant; τ task ∈[0,1] represents the task success rate, N success,t N represents the number of tasks successfully completed in the last 1000 steps. total,t This represents the total number of attempted tasks. ∈ explore ∈[0,1] exploration rate, ∈ decay =10 5 To explore the decay step count; w ε =0.3 is the exploration weight coefficient.

9. A computer system comprising a memory and a processor, the memory storing a computer program, wherein when the computer program is executed by the processor, the processor performs the steps of the dynamic feedback normalization reinforcement learning optimization method for industrial robots as described in any one of claims 1 to 8.

Images