Single robot furniture assembly motion planning method based on noise pulse mixed reinforcement learning

Abstract

The application provides a single-robot furniture assembly motion planning method based on noise pulse mixed reinforcement learning, and specifically comprises the following steps: collecting observation information, building a strategy-value network to guide the robot to perform furniture assembly, and designing a noise pulse neural network as a strategy network; designing a coding layer based on group coding to convert the observation information into a pulse sequence; designing a pulse neural layer with noise, outputting the final pulse activity according to the pulse sequence; converting the pulse activity into an action expression through a non-pulse decoding layer with noise; designing a dynamic noise level adjustment optimization method, modifying a loss function to balance the noise-strategy; optimizing network parameters to obtain an optimal model to guide the robot to complete an assembly task. The application provides a guarantee for continuous operation of the robot in a resource-limited scene, and promotes the practical development of intelligent equipment in the field of intelligent manufacturing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of mechanical arm motion planning, and particularly relates to a single mechanical arm furniture assembly motion planning method based on noise pulse mixed reinforcement learning. BACKGROUND

[0002] Mechanical arms exhibit many significant advantages in furniture assembly tasks, which not only improve production efficiency, but also optimize product quality and production safety. First, mechanical arms can significantly improve the efficiency and quality of furniture assembly. Its high precision and repetitive operation capability enables it to quickly and accurately complete the tasks of part grabbing, positioning and assembly, avoiding fatigue and mistakes that may occur in manual operation. For example, in the assembly project of IKEA furniture, mechanical arms can efficiently complete the assembly task in a short time through 3D camera recognition and precise force control technology. Second, the flexibility and programmability of mechanical arms enable them to adapt to furniture parts of different sizes and shapes. By replacing end effectors or adjusting programs, mechanical arms can easily switch between different assembly tasks to meet diverse production needs. This flexibility not only improves the adaptability of the production line, but also reduces the adjustment cost caused by product changes.

[0003] In addition, the application of mechanical arms also significantly reduces labor costs and labor intensity. In traditional furniture manufacturing, assembly work is usually labor-intensive and repetitive, which can easily lead to worker fatigue and injuries. The introduction of mechanical arms can replace manual labor to complete these heavy tasks, reduce dependence on human labor, and reduce material waste and product defects caused by human errors. For example, the Optimal collaborative robot reduces assembly time and reduces the risk of employee injury through precise control and high-speed motion capabilities. In terms of quality control, the high-precision operation of mechanical arms can ensure the consistency and stability of furniture assembly. Its repeat positioning accuracy can reach millimeters or even microns, effectively reducing product quality problems caused by assembly errors. For example, some advanced mechanical arms use six-axis force sensors to achieve precise force feedback, ensuring precise operation at each assembly stage. In summary, the use of mechanical arms provides great flexibility and adaptability for industrial production.

[0004] As a neural network architecture that simulates the working mode of the biological nervous system, the spiking neural network has many unique advantages. It transmits information through discrete spike signals, which can more realistically mimic the behavior patterns of neurons in the brain. This architecture not only excels in processing spatiotemporal information, but also achieves efficient computation with lower energy consumption due to its characteristic of triggering spike firing only when a threshold is reached. When noise is injected into the spiking neural network, the performance and robustness of the network can be further improved. This noise injection mechanism is similar to the randomness in the biological nervous system, which can enhance the sensitivity of the network to input signals, enabling it to better adapt and learn in complex or uncertain environments. At the same time, the introduction of noise can also improve the training process of the network, by increasing the diversity of training, helping the network to avoid falling into local optima, and thus improving the generalization ability of the model. SUMMARY

[0005] The present application aims to solve the problems of low flexibility and high energy consumption in robot motion planning, and provides a single robot furniture assembly motion planning method based on noise pulse hybrid reinforcement learning, which aims to balance the high flexibility and low energy consumption of robot motion planning, and promote the practical development of intelligent equipment in the field of intelligent manufacturing.

[0006] To achieve the above technical purposes, the present application provides the following technical solutions:

[0007] A single robot furniture assembly motion planning method based on noise pulse hybrid reinforcement learning, comprising the following steps:

[0008] S1, collect the observation information of the robot in the furniture assembly task scene, build a policy-value network to guide the robot to perform furniture assembly, and design a noise pulse neural network as the policy network; the noise pulse neural network comprises an encoding layer, a pulse neural layer and a decoding layer;

[0009] S2, design the encoding layer based on group coding, convert the observation information into a deterministic pulse sequence, and the pulse sequence is a set of discrete time pulses generated by neurons in the encoding process, representing the time and intensity characteristics of the observation information;

[0010] S3, design a pulse neural layer with noise, output the final pulse activity according to the pulse sequence obtained in step S2, and the pulse activity is the overall dynamic behavior of neurons in the neural layer, reflecting the activation state of neurons;

[0011] S4, then convert the input pulse activity into action representation through the non-pulse decoding layer with noise;

[0012] S5, design a dynamic noise level optimization method, realize the dynamic adjustment of noise by modifying the loss function, and improve the training efficiency while ensuring sufficient exploration;

[0013] S6. Optimize the parameters of the noise pulse neural network based on the deep reinforcement learning framework SAC to obtain the optimal model; use the trained optimal model to guide the robotic arm to complete the furniture assembly task.

[0014] Furthermore, the observation information in step S1 is specifically:

[0015] The state r of the robot arm and the state f of the furniture component at each time step t; the observation information o is expressed as:

[0016] o={r(h1,h2;g1;e1,e2,e3,e4),f(p n ,q n )};

[0017] Among them, h1 and h2 represent the position and velocity of the robot arm joint respectively; g1 represents the position of the robot arm gripper; e1~e4 represent the position, quaternion, linear velocity and angular velocity of the robot arm end effector respectively; p n ,q n are the 3D coordinates and orientation of the nth furniture component respectively.

[0018] Furthermore, step S2 specifically includes:

[0019] S21, observation data o at time i for observation information o i Any data o of dimension j in ij , calculate the group activation value p of the observation data of the jth dimension through the Gaussian function j , the formula is:

[0020]

[0021] Among them, u j It represents the response center of the observation data of the jth dimension, σ j It represents the response width of the observation data in the jth dimension;

[0022] S22, at each time step t, the neuron's membrane potential cumulative population activation value p j , and then update the membrane potential state; the update rules are as follows:

[0023] v j (t) = v j (t-1)+p j ;

[0024]

[0025] v j (t) = v j (t)-Vth ·s j (t);

[0026] where v j (t) represents the membrane potential state of the neuron related to the jth dimension of observation data at the current t time, which is 0 at the initial time; is a function of judging whether the voltage accumulation exceeds the threshold value V th , if it exceeds the threshold value, it takes 1, otherwise it takes 0, s j (t) represents the generated pulse sequence.

[0027] Further, step S3 specifically includes:

[0028] S31, design a pulse neural layer with noise of circuit leakage-charge-discharge working principle, wherein the current of any one pulse neuron is updated as follows:

[0029]

[0030] where δ represents the current decay rate, X t represents external input; l represents the number of layers of the second shock neural network where the neuron is located, and m represents the mth neuron in the layer;

[0031] S32, introduce noise in the charging process of the neuron, then the charging voltage of any one pulse neuron with noise is updated as follows:

[0032]

[0033] where v represents the membrane voltage after discharge, η represents the voltage decay rate, and σ v represents the noise parameter, ε v represents noise extracted from any random process;

[0034] S33, according to the updated charging voltage, the pulse signal of any one pulse neuron is updated and fired as follows:

[0035]

[0036] where Θ(·) processes each element in the input charge point cloud, if it exceeds the threshold value V th , it takes 1, otherwise it takes 0; σ s represents the noise parameter, ε s represents noise extracted from any random process;

[0037] S34, after firing the pulse signal, reset the membrane potential, and the update rule of the membrane potential v(t) after discharge is as follows:

[0038]

[0039] wherein V rest represents the resting potential.

[0040] Further, the step S4 specifically comprises:

[0041] S41, design a neuron with non-pulse integral-discharge working principle of injecting noise as a decoding layer; at each time step t, the membrane potential v r (t) of the rth neuron in the decoding layer is updated according to the pulse signal emitted by the pulse neural layer, and the update rule is as follows:

[0042] v r (t) = v r (t-1) + x t ;

[0043] wherein x t represents the pulse signal input at the current time step; if the model is currently in the training mode, noise is also added to the input, and the update mode of the membrane potential after adding the noise is:

[0044] v r (t) = v r (t-1) + x t + σ s ⊙ ε s ;

[0045] wherein σ s represents a noise parameter, and ε s represents noise extracted from any random process;

[0046] S42, obtain an intuitive action expression through the membrane potential [v r1 , v r2 , …, v rT ] of the neuron in the decoding layer, and the specific decoding method is to calculate the average membrane potential of the entire time sequence, which is expressed by the formula:

[0047]

[0048] wherein out represents the obtained action expression.

[0049] More specifically, the action expression obtained after the step S42 decoding is specifically:

[0050] the mechanical arm action a = {a1, a2, …, a9} at each time step t;

[0051] Wherein, a1-a7 represent the speed of seven mechanical arm joints respectively; a8 represents selection, that is, selecting one from a plurality of furniture parts to be grabbed; and a9 represents connection, which is used to represent the interaction with the environment and objects.

[0052] Further, the step S5 specifically comprises:

[0053] S51, taking the constructed noise pulse neural network as a policy network, and calculating its original loss L old ; and taking the sum of the noise variance of the non-pulse neurons in the decoding layer as a new loss function L new of the policy network, which is expressed by the formula as:

[0054]

[0055] Wherein, N A represents the action dimension, σ i is the noise standard deviation of the i-th non-pulse neuron, and k is a coefficient for controlling the update ratio between the original gradual update direction and the noise reduction direction, the value of which is adjusted according to the current reward and the task dynamics;

[0056] S52, when the reward value is low, a smaller k value encourages the mechanical arm to explore new strategies using noise; when the reward value is high, a larger k value controls the mechanical arm to reduce the use of noise and focus on using existing strategies to obtain greater rewards; the formula of k is expressed as:

[0057]

[0058] Wherein, k0 is a factor for controlling the size of the loss term; R mean represents the average reward value after multiple training; R max represents the highest reward value; and R min represents the lowest reward value.

[0059] Further, the step S6 specifically comprises:

[0060] S61, using a double Q network to minimize the loss function to train the value network; the formula is expressed as:

[0061] L critic = (Q target -Q real1 ) 2 + (Q target -Q real2 ) 2 ;

[0062] Wherein, Q target represents the prediction of future rewards, which is calculated by the target network and the policy network; Q real1 and Q real2are two different value estimates of the value network for the same state-action pair; Q target The calculation formula is as follows:

[0063] Q target = gamma * (1-d t ) * (minC target (o t+1 ,a t+1 )- alpha * logpi (o t+1 ))+ r t ;

[0064] Wherein, gamma is a discount factor of future rewards, d t is a Boolean value; if d t = 1, it indicates that the current step is a terminal state, otherwise d t = 0; C target (o t+1 ,a t+1 ) is the estimated Q value of the target network for the next state o t+1 and action a t+1 ; alpha is a temperature parameter for adjusting the entropy of the policy; logpi (o t+1 ) is the entropy of the policy of the next state; r t is the reward at time t;

[0065] S62, the policy network in the deep reinforcement learning framework is composed of a noise pulse neural network, and the value network is composed of two MLP instances, each of which is composed of three fully connected layers;

[0066] S63, the parameters of the entire reinforcement learning framework are randomly initialized, the gradient of the loss with respect to the policy network parameters is calculated, and the Adam optimizer is used to optimize the weights of the noise pulse neural network;

[0067] S64, the obtained optimal configuration of the noise pulse neural network is deployed in a furniture assembly task scenario, steps S1-S4 are performed, and the mechanical arm is guided to complete the furniture assembly task in the task scenario.

[0068] Based on the above technical solution, the present application has at least the following beneficial effects:

[0069] 1、The present application designs a policy-value network for guiding furniture assembly motion planning of a mechanical arm, introduces a noise pulse neural network in the policy network, and still uses a deep neural network in the value network, realizes efficient calculation with lower energy consumption through the introduced pulse neural network only triggering pulse firing when a threshold is reached, and the value network still retains the deep neural network, so that the entire policy-value network is not weaker than the original performance.

[0070] 2. By designing a pulse neural layer with noise neurons, time-dependent noise is introduced during the charging and transmission of neurons, improving the exploration efficiency in diversified environments, and helping to generate diversified decisions in complex environments; by designing a non-pulse decoding module with noise, the output pulse signal is converted to a more intuitive action space expression; and by using a dynamic noise level adjustment method, the balance between strategy and noise is achieved;

[0071] 3. Finally, the parameters of the network are optimized based on the deep reinforcement learning algorithm framework, and finally an intelligent decision-making model that can be transferred to the furniture assembly task of the robot arm is constructed; this method finally realizes the construction of an energy-efficient motion generation system for the robot arm, enabling it to maintain continuous operation capability in resource-constrained scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0072] Figure 1 A strategy-value network diagram for guiding the furniture assembly motion planning of the robot arm constructed in the method proposed in the present application;

[0073] Figure 2 A pulse neuron model with noise in the method proposed in the present application;

[0074] Figure 3 A simulation diagram for the furniture assembly of the robot arm in the present application. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical scheme and advantages of the present application clearer and more understandable, the present application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0076] The steps in the present application are arranged with labels, but are not used to limit the order of the steps, unless the order of the steps is explicitly stated or the execution of a step requires other steps as a basis, otherwise the relative order of the steps can be adjusted. It can be understood that the term "and / or" used herein relates to and covers any and all possible combinations of one or more of the associated listed items.

[0077] With the development of robotic arm technology, more and more robotic arms are applied to practical tasks with complex high-dimensional observation and action spaces. These robotic arms usually rely on limited on-board energy, so it is necessary to develop energy-efficient continuous control solutions to extend their operating time. Although deep reinforcement learning methods based on policy gradient have successfully learned optimal control policies in complex tasks, these methods are often accompanied by the disadvantage of high energy consumption, limiting their application in many scenarios. In contrast, spiking neural networks provide a low-energy alternative that can effectively solve this problem. On this basis, the application also adds relevant noise to the spiking neural network, and appropriate noise levels can help the network avoid unstable phenomena during the learning process, produce more diverse decisions, and be closer to the working mechanism of biological neural networks. Therefore, the application designs a noise-added spiking neural network model framework, including an encoding layer, a noise-added spiking neural layer, and a noise-added non-spiking decoding module. In addition, in order to balance the relationship between the policy and the noise, the application also proposes a noise reduction method for non-spiking neurons. Based on the deep reinforcement learning algorithm framework, the parameters of the above modules are optimized, and finally an intelligent decision-making model that can be transferred to the robotic arm furniture assembly task is constructed.

[0078] The present research aims to construct an energy-efficient motion generation system for robotic arms, enabling them to maintain continuous operation in resource-limited scenarios and promoting the practical development of intelligent equipment in the field of intelligent manufacturing. As shown in Figure 1 The present research aims to construct an energy-efficient motion generation system for robotic arms, enabling them to maintain continuous operation in resource-limited scenarios and promoting the practical development of intelligent equipment in the field of intelligent manufacturing. As shown in

[0079] S1, collect the observation information of the robotic arm in the furniture assembly task scenario, build a policy-value network to guide the robotic arm to perform furniture assembly, and design a noise-added spiking neural network as the policy network; the noise-added spiking neural network includes an encoding layer, a spiking neural layer, and a decoding layer;

[0080] As a preferred embodiment, the observation information in step S1 is specifically:

[0081] The state of the robotic arm r and the state of the furniture component f at each time step t; the observation information o is expressed by the formula:

[0082] o={r(h1,h2;g1;e1,e2,e4),f(p n ,q n )};

[0083] Where h1 and h2 represent the position and velocity of the robotic arm joints, respectively; g1 represents the position of the robotic arm gripper; e1-e4 represent the position, quaternion, linear velocity, and angular velocity of the robotic arm end effector, respectively; p n ,qn are the 3D coordinates and direction of the nth furniture component, respectively.

[0084] S2, encoding layer design based on population coding, converting observation information into deterministic pulse sequence, the pulse sequence is a discrete time pulse set generated by neurons in the encoding process, representing the time and intensity characteristics of the observation information;

[0085] As a preferred embodiment, step S2 specifically comprises:

[0086] S21, for observation data o i of observation information o at time i in any dimension j of o ij , in this embodiment, j∈{1, 2, …, 64}, indicating that there are 64 dimensions of data in the mechanical arm state r and the furniture component state; the population activation value p j of the observation data of the jth dimension is calculated by the Gaussian function (the population activation value represents the response intensity of the observation information in the population coding), and the formula is:

[0087]

[0088] Wherein, u j represents the response center of the observation data of the jth dimension, and σ j represents the response width of the observation data of the jth dimension; in this embodiment, 10 neurons are set for each dimension for population coding;

[0089] S22, at each time step t, the membrane potential of the neuron accumulates the population activation value p j , and then the membrane potential state is updated; the update rule is as follows:

[0090] v j (t)=v j (t-1)+p j ;

[0091]

[0092] v j (t)=v j (t)—V th ·s j (t);

[0093] Wherein, v j (t) represents the membrane potential state of the neuron related to the jth dimension observation data at the current t time, which is 0 initially; is a function for judging whether the voltage accumulation exceeds the threshold value V th , if it exceeds the threshold value, it takes 1, otherwise it takes 0, s j(t) represents the generated spike train; the above three equations describe the accumulation of population activation value, the determination of spike firing and the adjustment of membrane potential according to whether a spike is generated, respectively.

[0094] S3, design a pulse neural layer with noise, output the final pulse activity according to the pulse sequence obtained in step S2, the pulse activity is the overall dynamic behavior of neurons in the neural layer, reflecting the activation state of neurons; appropriate noise level can help the network avoid unstable phenomena in the learning process, and also produce more diversified decisions, and closer to the working mechanism of biological neural networks;

[0095] As a preferred embodiment, as shown in Figure 2 , step S3 specifically includes:

[0096] S31, design a pulse neural layer with noise, the current of any one pulse neuron in the neural layer is updated according to the following rules:

[0097]

[0098] Where δ represents the current decay rate, X t represents the external input; l represents the layer number of the second pulse neural network where the neuron is located, and m represents the mth neuron in the layer;

[0099] S32, introduce noise in the charging process of the neuron, then the charging voltage of any one pulse neuron with noise is updated according to the following rules:

[0100]

[0101] Where v represents the membrane voltage after discharge, η represents the voltage decay rate, and σ v represents the noise parameter, ε v represents the noise extracted from any random process;

[0102] S33, update and fire the pulse signal of any one pulse neuron according to the updated charging voltage according to the following rules:

[0103]

[0104] Where Θ(·) processes each element in the input charge point cloud, and if it exceeds the threshold V th , it takes 1, otherwise it takes 0; σ s represents the noise parameter, ε s represents the noise extracted from any random process;

[0105] S34, after the pulse signal is sent, the membrane potential is reset, and the membrane potential v(t) after discharge is updated as follows:

[0106]

[0107] wherein V rest represents the resting potential.

[0108] In addition, in the embodiment, three noise-pulse neural layers connected in sequence are selected as the final neural layers of the noise-pulse neural network, as shown in Figure 1 : layer 1 and layer 2 are hidden layers, with 256 pulse neurons, and layer 3 is an output layer, with 90 pulse neurons (considering that the robot arm has a total of nine actions, in the embodiment, 10 neurons are used for each action); at the same time, the input data of each layer is converted into a format suitable for pulse neuron processing through a linear layer in advance.

[0109] The application introduces time-dependent noise in the charging and discharging process of neurons, thereby improving the exploration efficiency in a diversified environment; and after the above leakage-charging-discharging process, the pulse neural layer finally outputs pulse activity reflecting the overall dynamic behavior.

[0110] S4, the input pulse activity is converted into an action representation through a non-pulse decoding layer with noise; the design of the non-pulse decoding layer is based on continuous numerical calculation, which can more accurately decode the pulse activity and reduce the information loss caused by the sparseness of pulses or the time step limit;

[0111] As a preferred embodiment, step S4 specifically comprises:

[0112] S41, neurons with noise-injected non-pulse integral-discharge working principle are designed as the decoding layer; at each time step t, the membrane potential v r (t) of the rth neuron in the decoding layer is updated according to the pulse signal emitted by the pulse neural layer, and the update rule is as follows:

[0113] v r (t) = v r (t-1) + x t ;

[0114] wherein x t represents the pulse signal input at the current time step; if the model is currently in a training mode, noise is also added to the input, and the update mode of the membrane potential after adding the noise is:

[0115] v r (t) = v r (t-1) + x t + σs ⊙ε s ;

[0116] wherein σ s represents a noise parameter, ε s represents noise extracted from any random process;

[0117] S42, obtaining the intuitive action representation by decoding the membrane potential [v r1 ,v r2 ,…,v rT ] of the neurons in the decoding layer, and the specific decoding manner is to calculate the average membrane potential of the entire time sequence, which is expressed by the formula:

[0118]

[0119] wherein out represents the obtained action representation; this decoding manner assumes that the most useful information is contained in the average dynamics of the entire time sequence, so the output is the average value of all time step membrane potentials; the action representation obtained by the decoding layer is more intuitive, and when the furniture assembly task of the mechanical arm is performed, the obtained action representation is specifically:

[0120] the mechanical arm action a = {a1, a2, …, a9} at each time step t;

[0121] wherein a1-a7 represent the speeds of the seven joints of the mechanical arm respectively; a8 represents selection, that is, selecting one from a plurality of furniture parts to be grasped; and a9 represents connection, which is used to represent the interaction with the environment and the object.

[0122] So far, the pulse signal has been converted into the action to be performed by the mechanical arm, and the following two steps mainly discuss the balance between noise and strategy and the optimization of network parameters.

[0123] S5, designing a dynamic adjustment noise level optimization method, and realizing the dynamic adjustment of the noise by modifying the loss function, so as to improve the training efficiency while ensuring sufficient exploration;

[0124] As a preferred embodiment, step S5 specifically comprises:

[0125] S51, taking the constructed noise pulse neural network as a strategy network, and calculating the original loss L old ; and taking the sum of the noise variance of the non-pulse neurons in the decoding layer and the original loss as the new loss function L new of the strategy network, which is expressed by the formula:

[0126]

[0127] wherein N A represents the action dimension, σ iis the noise standard deviation of the ith non-spiking neuron, k is a coefficient that controls the update ratio between the original gradual update direction and the denoising direction, and its value is adjusted according to the current reward and task dynamics;

[0128] S52, when the reward value is low, a smaller k value indicates that the influence of the noise variance term on the loss function is small, and then the robot arm is encouraged to explore new strategies using noise; when the reward value is high, a larger k value indicates that the influence of the noise variance term on the loss function is strong, and then the robot arm is controlled to reduce the use of noise and focus on using existing strategies to obtain greater rewards; thereby achieving a balance between noise use and strategy rewards; the formula of k is:

[0129]

[0130] wherein k0 is a factor that controls the size of the loss term; R mean represents the average reward value after multiple training; R max represents the highest reward value; R min represents the lowest reward value.

[0131] S6, based on the deep reinforcement learning framework SAC, the parameters of the noise spiking neural network are optimized to obtain an optimal model; the trained optimal model is used to guide the robot arm to complete the furniture assembly task;

[0132] As a preferred embodiment, step S6 specifically comprises:

[0133] S61, the value network is trained using a double Q network to minimize the loss function; the formula is:

[0134] L critic =(Q target -Q real1 ) 2 +(Q target -Q real2 ) 2 ;

[0135] wherein Q target represents the prediction of future rewards, which is calculated by the target network and the policy network; Q real1 and Q real2 are two different value estimates of the value network for the same state-action pair; the calculation formula of Q target is as follows:

[0136] Q target =γ·(1-d t )·(minC target (o t+1 ,a t+1 )-α·logπ(o t+1 ))+r t ;

[0137] where γ is a discount factor for future rewards, d t is a Boolean value; if d t = 1, it means the current step is a terminal state, otherwise d t = 0; C target (o t+1 , a t+1 ) is the estimated Q-value of the target network for the next state o t+1 and action a t+1 ; α is a temperature parameter that adjusts the policy entropy; logπ(o t+1 ) is the policy entropy of the next state; r t is the reward at time t;

[0138] S62, the policy network in the deep reinforcement learning framework is composed of a noise pulse neural network, and the value network is composed of two MLP instances, each of which is composed of three fully connected layers;

[0139] S63, the parameters of the entire reinforcement learning framework are randomly initialized, the gradient of the loss with respect to the policy network parameters is calculated, and the Adam optimizer is used to optimize the weights of the noise pulse neural network;

[0140] S64, the obtained optimal configuration of the noise pulse neural network is deployed in a furniture assembly task scenario, and steps S1-S4 are executed, as shown in Figure 3 , guiding the robot arm to complete the furniture assembly task in the task scenario.

[0141] So far, the method proposed in the present application is a single-robot arm furniture assembly motion planning method based on noise pulse hybrid reinforcement learning, which constructs a low-energy policy value network for robot arm furniture assembly tasks. This method replaces the traditional deep network with a noise pulse neural network as a policy network to significantly reduce energy consumption when performing mechanical motion planning. At the same time, due to the addition of noise, this method can also make diversified decisions in complex environments. In this way, this research can build an energy-efficient motion generation system for robot arms, enabling them to maintain continuous operation capabilities in resource-constrained scenarios and promote the practical development of intelligent equipment in the field of intelligent manufacturing.

[0142] It will be obvious to a person skilled in the art that the application is not limited to the details of the foregoing exemplary embodiments and can be implemented in other concrete forms without departing from the spirit or essential characteristics of the application. The embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. No reference signs in the claims should be considered as limiting the scope of the claims to the identity of the reference signs therein.

[0143] Furthermore, it should be understood that although the description is made on the basis of the embodiments, not every embodiment contains only one independent technical solution, and the description of the specification is only for the sake of clarity, and those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that those skilled in the art can understand.

Claims

1. A single robot arm furniture assembly motion planning method based on noise pulse mixed reinforcement learning, characterized in that, Specifically comprising the following steps: S1, collecting observation information of a robot arm in a furniture assembly task scene, building a policy-value network to guide the robot arm to perform furniture assembly, and designing a noisy spiking neural network as a policy network; The noisy spiking neural network comprises an encoding layer, a spiking neural layer, and a decoding layer; S2, designing the encoding layer based on group coding, converting the observation information into a deterministic spike sequence, the spike sequence being a discrete-time spike set generated by neurons in the encoding process, representing the time and intensity characteristics of the observation information; S3, designing a spiking neural layer with noise, outputting the final spiking activity according to the spike sequence obtained in step S2, the spiking activity being the overall dynamic behavior of neurons in the neural layer, reflecting the activation state of the neurons; step S3 specifically comprises: S31, design a pulse neural layer with the working principle of circuit leakage-charge-discharge with noise, the current of any one pulse neuron in it update as follows: wherein δ represents a current decay rate, X t represents an external input; l represents a layer number of a pulse neural network in which the neuron is located, and m represents an mth neuron of the layer. S32, introduce noise during the charging of the neuron, then the charging voltage of any one of the noisy spiking neurons update as follows: where v denotes the film voltage after discharging, η denotes the voltage decay rate, where σ v denotes a noise parameter, ε v denotes noise extracted from any random process; S33、According to the updated charging voltage, update any one pulse neuron pulse signal Update and fire as follows: where Θ(·) processes each element in the input charge point cloud, taking 1 if it exceeds a threshold V th , and 0 otherwise; σ s denotes a noise parameter, and ε s denotes noise drawn from any random process. S34, after emitting the spike signal, resetting the membrane potential, and the update rule of the membrane potential v(t) after discharge is as follows: wherein V rest represents the resting potential; S4, further converting the input spike activity into action representation through the non-spiking decoding layer with noise, specifically comprising: S41, design a neuron of non-pulsed integral-discharge working principle of injecting noise as a decoding layer; at each time step t, the membrane potential v of the rth neuron in the decoding layer is r (t) update according to the pulse signal emitted by the pulse neural layer, and the update rule is as follows: v r (t) = v r (t-1) + x t ; where x t represents the input of the pulse signal at the current time step; if the model is currently in the training mode, noise is also added to the input, and the update mode of the membrane potential after adding noise is: v r (t) = v r (t-1) + x t + σ s ⊙ ε s ; where σ s denotes a noise parameter, ε s denotes noise drawn from any random process; S42, obtain intuitive action representation by decoding membrane potential [v r1 ,v r2 ,…,v rT ] of neurons in the layer, and the specific decoding manner is to calculate the average membrane potential of the entire time sequence, which is expressed by the formula: Wherein, out represents the obtained action representation; S5, designing a dynamic noise level optimization method, modifying the loss function to realize dynamic adjustment of the noise, ensuring sufficient exploration while improving training efficiency; step S5 specifically comprises: S51, the constructed noise pulse neural network is taken as a policy network, and its original loss L is calculated old ; and the sum of the noise variances of the non-pulse neurons in the decoding layer is taken as a new loss function L of the policy network new , which is expressed by a formula as follows: where N A represents the action dimension, σ i is the noise standard deviation of the i-th non-spiking neuron, k is a coefficient that controls the update ratio between the original gradual update direction and the noise reduction direction, and its value is adjusted according to the current reward and task dynamics; S52, when the reward value is low, a smaller k value encourages the robot arm to explore new strategies using noise; when the reward value is high, a larger k value controls the robot arm to reduce the use of noise and focus on using existing strategies to obtain greater rewards; the formula of k is: where k0 is a factor to control the size of the loss term; R mean represents the average reward value after multiple training; R max represents the highest reward value; R min represents the lowest reward value; S6, optimizing the parameters of the noisy spiking neural network based on the deep reinforcement learning framework SAC to obtain an optimal model; using the trained optimal model to guide the robot arm to complete the furniture assembly task.

2. The single robot arm furniture assembly motion planning method based on noise pulse hybrid reinforcement learning according to claim 1, wherein, The observation information in step S1 is specifically: The state of the robot arm r and the state of the furniture component f at each time step t; the observation information o is expressed by the formula: o = { r(hl, h2; gl; el, e2, e3, e4), f(p n , q n )}; wherein h1, h2 represent the position and velocity of the joints of the robot arm, respectively; g1 represents the position of the robot arm gripper; e1-e4 represent the position, quaternion, linear velocity and angular velocity of the robot arm end effector, respectively; p n n are the 3D coordinates and orientation of the nth furniture component, respectively.​ 3. The single robot arm furniture assembly motion planning method based on noise pulse hybrid reinforcement learning according to claim 1, wherein, The action representation obtained after step S42 decoding is specifically: The robot arm action a = {a1, a2, …, a9} at each time step t; Wherein, a1-a7 represent the speeds of the seven robot arm joints respectively; a8 represents selection, that is, selecting one from a large number of furniture components for grasping; a9 represents connection, which is used to represent interaction with the environment and objects.

4. The single robot arm furniture assembly motion planning method based on noise pulse hybrid reinforcement learning according to claim 1, wherein, Step S6 specifically comprises: S61, using a double Q network to minimize the loss function to train the value network; the formula is: L critic = (Q target - Q real1 ) 2 + (Q target - Q real2 ) 2 ; where Q target represents the prediction of future rewards, computed by the target and policy networks; Q real1 and Q real2 are two different value estimates for the same state-action pair by the value network; Q target is computed as follows: Q target = γ · (1 - d t ) · (minC target (o t+1 , a t+1 ) - α · logπ(o t+1 )) + r t ; where γ is a discount factor for future rewards, d t is a Boolean value; if d t = 1, it indicates that the current step is a terminal state, otherwise d t = 0; C target (o t+1 , a t+1 ) is the estimated Q-value of the target network for the next state o t+1 and action a t+1 ; a is a temperature parameter that adjusts the policy entropy; logπ(o t+1 ) is the policy entropy for the next state; r t is the reward at time t. S62, the policy network in the deep reinforcement learning framework is composed of a noisy spiking neural network, and the value network is composed of two MLP instances, each of which is composed of three fully connected layers; S63, randomly initializing the parameters of the entire reinforcement learning framework, calculating the gradient of the loss with respect to the policy network parameters, and using the Adam optimizer to optimize the weights of the noisy spiking neural network; S64, deploying the obtained optimal configuration of the noisy spiking neural network to the furniture assembly task scene, executing steps S1-S4, and guiding the robot arm in the task scene to complete the furniture assembly task.

Images