Industrial robot adaptive trajectory planning and control system based on reinforcement learning

Abstract

The invention relates to the technical field of industrial robot intelligent control, in particular to an industrial robot adaptive trajectory planning and control system based on reinforcement learning, which comprises a data acquisition module, a state reward construction module, a decision model training module, an online decision control module and an instruction execution module. The data acquisition module collects multi-modal sensing data of an industrial robot, a rotary positioner and a processing area sensing network, and the state reward construction module constructs a model input state and generates a reward signal according to the composite manufacturing process requirement. The decision model training module sets continuous actions including joint micro position increment, laser power adjustment amount and feeding speed, and completes model training in a digital twinning environment. The on-line decision control module outputs an action sequence according to the real-time state, and the instruction execution module converts the action sequence into a corresponding driving instruction. According to the system, multi-parameter cooperative adaptive control is realized, and the synchronism and real-time adaptation capability of trajectory planning and process adjustment are improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology for industrial robots, and in particular to an adaptive trajectory planning and control system for industrial robots based on reinforcement learning. Background Technology

[0002] When industrial robots work with rotary positioners to perform composite manufacturing processes, existing control methods mostly rely on manual teaching and offline programming to set fixed motion trajectories and process parameters. Simple feedback adjustment is achieved only through a single type of sensor data. When deep reinforcement learning is applied to robot control, it often adopts discrete motion decision-making or a single control output mode. Model training mostly relies on trial and error with physical prototypes, without integrating multimodal sensor network data of the processing area or combining it with a digital twin simulation environment to complete model iteration.

[0003] Fixed trajectory and parameter control methods cannot adapt to real-time changes in working conditions during the processing. Robot motion, laser power, and feed speed are mostly independently decoupled and controlled, which can easily lead to deviations in processing accuracy and instability in process quality. Discrete actions or single parameter outputs are difficult to meet the control requirements of continuous fine processing. Physical prototype trial and error training has the problems of high cost and long cycle. The quality requirements of composite manufacturing processes cannot be transformed into accurate model training reward signals. Multimodal sensing data cannot be effectively transformed into effective input states for decision-making models.

[0004] To achieve integrated continuous motion output of robot joint micro-position, laser power, and end effector feed speed, it is necessary to rely on a digital twin simulation environment and combine the quality requirements of composite manufacturing processes to construct reward signals to complete deep reinforcement learning model training, so as to adapt to the adaptive trajectory planning and control requirements of industrial robots driven by multimodal sensor data. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing an adaptive trajectory planning and control system for industrial robots based on reinforcement learning.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: an industrial robot adaptive trajectory planning and control system based on reinforcement learning, comprising:

[0007] The data acquisition module acquires multimodal sensor data streams from the industrial robot, rotary positioner, and sensor network of the processing area.

[0008] The state reward construction module processes the multimodal sensing data stream, constructs the input state of the deep reinforcement learning decision model, and constructs a reward signal according to the quality requirements of the composite manufacturing process.

[0009] The decision model training module defines the output action of the deep reinforcement learning decision model. The output action is a continuous action that includes the incremental position of the robot joints, the instantaneous adjustment of the laser power, and the feed speed of the end effector. In the digital twin simulation environment, the deep reinforcement learning decision model is trained by reward signals, so that the deep reinforcement learning decision model learns the mapping relationship between the output action and the input state.

[0010] The online decision control module deploys a trained deep reinforcement learning decision model in a physical environment. The deep reinforcement learning decision model generates a corresponding output action sequence based on the real-time acquired input state.

[0011] The instruction execution module converts the output motion sequence into drive instructions for each control joint of the robot, power control instructions for the laser, and speed control instructions for the end effector.

[0012] As a further aspect of the present invention, the specific process of constructing the input state of a deep reinforcement learning decision model includes:

[0013] The input states include the real-time pose of the robot end effector, robot joint angle feedback, real-time contact force during the machining process, real-time topographic deviation of the machining area, and real-time pose of the positioner.

[0014] The real-time coordinates and orientation angles of the robot end effector in the base coordinate system are read from the industrial robot controller to form the real-time pose of the robot end effector.

[0015] Read the real-time angle measurement values ​​of the encoders of each joint of the industrial robot to form the robot joint angle feedback;

[0016] The real-time contact force between the tool and the workpiece during the machining process is collected by a force sensor installed on the robot's end effector.

[0017] The surface image of the current processing area is captured by a vision sensor in the processing area. The surface image is registered and compared with a reference topography model to calculate the real-time topography deviation.

[0018] The real-time angle of the rotation axis is read from the rotary positioner controller to form the real-time pose of the positioner;

[0019] The robot end effector's real-time pose, robot joint angle feedback, real-time contact force, real-time shape deviation, and positioner's real-time pose are arranged and combined in a predetermined order to generate a vector with fixed dimensions. This vector is the input state of the deep reinforcement learning decision model.

[0020] As a further aspect of the present invention, the specific process of constructing the reward signal includes:

[0021] The reward signal is at least related to profile accuracy deviation, process constraint violation, and joint motion smoothness;

[0022] A method for calculating profile accuracy deviation is defined, wherein the profile accuracy deviation is the norm of the deviation between the real-time profile deviation of the current processing area and the target reference profile;

[0023] Define a method for judging process constraint violations. When the real-time contact force exceeds a preset force threshold or the heat input power of the processing area exceeds a preset power threshold, it is judged that a process constraint violation has occurred.

[0024] A quantitative method for defining joint motion smoothness is provided, wherein the joint motion smoothness is measured by calculating the norm of the robot joint feedback angular acceleration;

[0025] The negative values ​​of profile accuracy deviation, penalty values ​​for process constraint violation, and negative values ​​of joint motion smoothness are weighted and summed, with the weighting coefficients being profile accuracy weight, process constraint weight, and motion smoothness weight, respectively.

[0026] The weighted summation result is used as the immediate reward signal obtained by the deep reinforcement learning decision-making model after receiving the input state and executing the corresponding output action within the current decision-making cycle.

[0027] As a further aspect of the present invention, the specific process of guiding the training of a deep reinforcement learning decision model through reward signals in a digital twin simulation environment includes:

[0028] Virtual models of industrial robots, workpieces, rotary positioners, and the processing environment are created in a digital twin simulation environment.

[0029] In the virtual model, the same dynamic and kinematic parameters are configured for the industrial robot, workpiece, rotary positioner, and processing environment as for the physical environment.

[0030] A deep reinforcement learning decision model is initialized in a simulation environment, the deep reinforcement learning decision model comprising an actor network and a critic network;

[0031] At each simulation step, multimodal sensing data streams in the simulation environment are acquired and processed into input states to provide to the deep reinforcement learning decision model.

[0032] The actor network of the deep reinforcement learning decision model generates output actions based on the input state and executes the output actions in a simulation environment.

[0033] The reward signal is calculated based on the simulation environment data after execution. The input state, output action, reward signal and new input state are stored as an experience tuple in the simulation experience playback buffer.

[0034] A batch of experience tuples is periodically sampled from the simulation experience replay buffer to update the parameters of the actor network and critic network of the deep reinforcement learning decision model;

[0035] Through iterative training with a large number of simulated step sizes, the actor network of the deep reinforcement learning decision model learns to generate output action strategies that maximize the cumulative reward signal.

[0036] As a further aspect of the present invention, the process of deploying a trained deep reinforcement learning decision model in a physical environment, wherein the deep reinforcement learning decision model generates a corresponding output action sequence based on real-time acquired input states includes:

[0037] In each control cycle of the physical processing, the real-time pose of the robot end effector, the robot joint angle feedback, the real-time contact force, the real-time shape deviation, and the real-time pose of the positioner are acquired synchronously.

[0038] The real-time pose of the robot end effector, the robot joint angle feedback, the real-time contact force, the real-time shape deviation, and the real-time pose of the positioner are combined to form the input state of the current control cycle.

[0039] The input state of the current control cycle is fed into the actor network of the already trained deep reinforcement learning decision model;

[0040] The actor network processes the input state and outputs a multidimensional continuous vector. Each dimension of the multidimensional continuous vector corresponds to the minute position increment of each joint of the robot, the instantaneous adjustment of the laser power, and the feed speed of the end effector.

[0041] This multidimensional continuous vector is used as the output action generated by the deep reinforcement learning decision model in the current control cycle, and this process is repeated in the next control cycle to generate a time-varying sequence of output actions.

[0042] As a further aspect of the present invention, the process of converting the output motion sequence into drive commands for each control joint of the robot, power control commands for the laser, and speed control commands for the end effector includes:

[0043] The output action corresponding to each control cycle in the output action sequence is analyzed, and the minute position increments of each joint of the robot, the instantaneous adjustment of laser power, and the feed speed of the end effector contained in the output action are extracted.

[0044] For each joint of the robot, based on the angle feedback of the joint in the current control cycle, plus the small position increment of the joint specified in the output action, the expected angle of the joint in the next control cycle is calculated.

[0045] The difference between the desired angle of the joint and the current angle feedback is input into the servo controller of each joint of the robot and converted into corresponding drive current or torque commands.

[0046] For lasers, the current laser power setting value is used as the basis, plus the instantaneous laser power adjustment amount specified in the output action, as the power setting value for the next control cycle, and the corresponding power control command is generated.

[0047] For the end effector, the end effector feed speed specified in the output action is directly converted into a speed control command in the robot's tool coordinate system.

[0048] As a further aspect of the present invention, it also includes:

[0049] The execution and learning module issues corresponding drive commands, power control commands, and speed control commands to the industrial robot and laser, driving the robot to execute the processing trajectory and control the laser process parameters. During the execution of the industrial robot, the module simultaneously collects actual execution result data, associates the actual execution result data with the input state and output action sequence to form training experience data, and feeds the training experience data back to the deep reinforcement learning decision model for online fine-tuning.

[0050] The specific process of synchronously collecting actual execution result data during the execution of an industrial robot includes:

[0051] The actual execution result data includes the actual pose of the end effector, the actual contact force, the actual shape deviation, and the actual joint acceleration;

[0052] After each control cycle, the actual position feedback provided by the industrial robot controller is read to obtain the actual pose of the end effector;

[0053] Read the actual readings of the force sensor installed on the robot's end effector to obtain the actual contact force;

[0054] The latest image of the processed surface is acquired by a vision sensor in the processing area and compared with the target model to calculate the actual shape deviation.

[0055] Read the feedback data from the servo drives of each joint of the robot, calculate the actual angular velocity of the joint, and estimate the actual acceleration of the joint by calculating the difference in angular velocity.

[0056] The actual pose, actual contact force, actual shape deviation, and actual joint acceleration of the end effector collected in each control cycle are timestamped and associated with the input state and generated output action corresponding to the control cycle.

[0057] As a further aspect of the present invention, the actual execution result data is associated with the input state and output action sequence to form training experience data, and the training experience data is fed back to the deep reinforcement learning decision model for online fine-tuning. The specific process includes:

[0058] Establish a physical experience replay buffer to store training experience data collected from the physical environment. The training experience data is a tuple consisting of input state, output action, actual execution result data, and input state at the next moment.

[0059] Set a trigger threshold; when the amount of training experience data stored in the physical experience replay buffer reaches the trigger threshold, the online fine-tuning process will be started.

[0060] During online fine-tuning, a batch of training experience data is randomly sampled from the physical experience replay buffer, and the reward signal is calculated using the training experience data.

[0061] Using the calculated reward signal and sampled training experience data, the parameters of the critic network and actor network in the deep reinforcement learning decision model are updated in one round.

[0062] After completing a round of parameter updates, you can choose to clear or partially clear the physics experience replay buffer, continue to collect new training experience data, and wait for the next online fine-tuning to be triggered.

[0063] As a further aspect of the present invention, the adaptive adjustment process of the weight coefficients involved in the construction of the reward signal includes:

[0064] Record the actual values ​​of profile accuracy deviation, process constraint violation, and joint motion smoothness within each control cycle when performing machining tasks in a physical environment;

[0065] Calculate the average value of profile accuracy deviation, the frequency of process constraint violation, and the average value of joint motion smoothness within a preset statistical window;

[0066] The profile accuracy weight is dynamically adjusted based on the deviation of the average profile accuracy deviation from the preset target.

[0067] The weights of process constraints are dynamically adjusted based on the frequency of process constraint violations.

[0068] The motion smoothing weights are dynamically adjusted based on the deviation of the average joint motion smoothness from the desired level.

[0069] As a further aspect of the present invention, the system establishes a model parameter synchronization mechanism between the digital twin simulation environment and the physical environment. The specific working process of the model parameter synchronization mechanism includes:

[0070] In a digital twin simulation environment, reinforcement learning training based on simulation data is continuously performed, and updated parameters of the deep reinforcement learning decision model are generated periodically.

[0071] In physical environments, updated parameters for deep reinforcement learning decision models are also generated through online fine-tuning.

[0072] Set a model parameter fusion cycle, and in each fusion cycle, extract the critic network parameters and actor network parameters from the model parameters obtained from training in the simulation environment;

[0073] Extract critic network parameters and actor network parameters from model parameters obtained through online fine-tuning in the physical environment;

[0074] The network parameters are obtained by weighted averaging of the network parameters of the commenters from the simulation environment and the network parameters of the actors from the simulation environment and the physical environment.

[0075] The fused parameters of the commentator network and the actor network are synchronously updated to the deep reinforcement learning decision model deployed in the digital twin simulation environment and the physical environment.

[0076] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0077] By setting the output actions of the deep reinforcement learning decision model as continuous actions of the industrial robot's joint micro-position increments, laser power instantaneous adjustments, and end effector feed speeds, coupled output of robot motion parameters, laser processing parameters, and end effector feed parameters can be achieved. This replaces discrete action forms or single control quantity output modes, eliminates the control deviation caused by the independent regulation of motion control and process parameter control, and shortens the response delay of control commands by synchronously outputting multi-dimensional control quantities. This ensures that the control output keeps pace with the real-time state during the processing. The continuous action output form can match the dynamic regulation requirements of composite manufacturing processes, and the multi-parameter collaborative output can keep the trajectory planning and process adjustment consistent. The action output driven by multi-modal sensor data can fit the real-time working condition changes of the processing area, improving the matching degree between control commands and the actual processing scenario.

[0078] In a digital twin simulation environment, reward signals are constructed based on the quality requirements of composite manufacturing processes to guide the training of deep reinforcement learning decision-making models. This can completely transform the multimodal sensor data streams of industrial robots, rotary positioners, and processing area sensor networks into model input states, avoiding the additional resource consumption and extended training cycle problems caused by trial and error training with physical prototypes. The reward signals are directly related to the quality requirements of composite manufacturing processes, enabling the model to accurately learn the mapping logic between input states and output actions. The digital twin simulation environment can completely reproduce the working conditions of physical processing, enhancing the model's adaptability to complex processing environments. The fusion processing of multimodal sensor data can improve the model's perception accuracy of processing states. The trained model can be directly deployed in the physical environment to execute online decisions, and the generated action sequences can be directly converted into drive instructions for the corresponding equipment, realizing the adaptive and collaborative operation of industrial robot trajectory planning and processing technology control. Attached Figure Description

[0079] Figure 1 This is a timing diagram of the reinforcement learning-based adaptive trajectory planning and control system for industrial robots described in this invention.

[0080] Figure 2 A flowchart illustrating the input states for constructing a deep reinforcement learning decision-making model;

[0081] Figure 3 A flowchart for constructing the reward signal;

[0082] Figure 4 Output action sequence graphs for deep reinforcement learning models;

[0083] Figure 5 This is a diagram showing the data stream acquisition of multimodal sensing data from an industrial robot end effector. Detailed Implementation

[0084] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0085] In the description of this invention, it should be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, in the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0086] See Figure 1 This invention provides an adaptive trajectory planning and control system for industrial robots based on reinforcement learning. The system integrates industrial robots, rotary positioners, force sensors, and vision sensors in hardware, and constructs a closed loop for data stream processing, model training, and decision control in software. System operation begins with a data acquisition module, which continuously acquires multimodal sensor data streams from the industrial robot controller, rotary positioner controller, and a sensor network deployed in the processing area. A state reward construction module processes the raw sensor data, constructing it into a formatted input state required by the deep reinforcement learning decision model, and calculating reward signals in real time based on the quality indicators and constraints of the composite manufacturing process. The decision model training module runs in a digital twin simulation environment. It defines the output action of the decision model as a set of continuous control variables, including minute position increments of each robot joint, instantaneous adjustments to laser power, and the feed speed of the end effector. By providing the input state to the decision model, the model outputs actions and executes them in the simulation environment. Reward signals are calculated based on the consequences, and these reward signals are used to continuously adjust the model's internal parameters through a reinforcement learning algorithm, enabling it to learn to output action strategies that yield high cumulative rewards. The online decision control module deploys the trained deep reinforcement learning decision model into the real-time control loop of the physical system. This model generates corresponding output action sequences based on the input states obtained in real time from the physical world. The instruction execution module is responsible for translating and converting the abstract action sequences output by the model into low-level drive instructions, power control instructions, and speed control instructions that can be directly understood and executed by the servo drives of each joint of the industrial robot, the laser power controller, and the robot motion controller, thereby driving the physical equipment to complete adaptive processing tasks.

[0087] In one embodiment of the present invention, the input state is a fixed-dimensional vector composed of multiple key process parameters arranged in a predetermined order. This vector includes the real-time pose of the robot end effector, robot joint angle feedback, real-time contact force during processing, real-time topographic deviation of the processing area, and real-time pose of the positioner. See also... Figure 2The real-time pose of the robot's end effector is determined by reading its real-time coordinates and orientation angles in the base coordinate system from the industrial robot controller. Robot joint angle feedback is obtained by reading the real-time angle measurements from the encoders of each joint of the industrial robot. Real-time contact force is acquired by a force sensor mounted on the robot's end effector, which directly measures the interaction force between the tool and the workpiece during machining. The calculation of real-time topographic deviation relies on a vision sensor of the machining area, which captures a surface image of the current machining area. An image processing algorithm registers and compares this surface image with a preset target reference topographic model to calculate the deviation. The real-time pose of the positioner is determined by reading the real-time angle of its rotation axis from the rotary positioner controller. These five data points are concatenated in a fixed order to generate a one-dimensional vector, which serves as the input state for the deep reinforcement learning decision model at each decision moment.

[0088] In practical implementation, the specific process of constructing the input state of the deep reinforcement learning decision model takes a six-DOF industrial robot performing laser cladding on a complex curved workpiece as an example. The construction of the input state requires real-time fusion of data from multiple sensor sources. The input state of the deep reinforcement learning decision model is a fixed-dimensional vector, composed of the robot end effector's real-time pose, robot joint angle feedback, real-time contact force during processing, real-time topographic deviation of the processing area, and the rotary positioner's real-time pose. In practice, the robot end effector's real-time pose is acquired through the open interface of the industrial robot controller, reading the end effector's three-dimensional spatial coordinates in the robot's base coordinate system and the attitude angles represented by Euler angles or quaternions. The robot joint angle feedback obtains the instantaneous angles of the six joints by reading the real-time measurements from the absolute or incremental encoders of the industrial robot's six rotary joints.

[0089] In practical implementation, the real-time contact force during processing is acquired using a six-dimensional force sensor. This sensor is installed between the end flange of the industrial robot's wrist and the laser cladding nozzle, directly measuring the forces and moments in three directions generated by the interaction between the cladding head and the workpiece surface. In practical implementation, the calculation of real-time topographic deviation in the processing area relies on a fixed three-dimensional line laser scanner. This scanner continuously scans the clad weld area during robot processing, acquiring three-dimensional point cloud data of the workpiece surface. In some embodiments, the acquired real-time three-dimensional point cloud data is registered with the workpiece's computer-aided design model in the same coordinate system, and the topographic deviation is quantified by calculating the distance from the point cloud to the surface of the computer-aided design model. Optionally, the real-time topographic deviation can be represented using a statistic, such as the root mean square error of the distance from all points in the real-time point cloud to the surface of the computer-aided design model, calculated using the following formula:

[0090]

[0091] Where: symbol This represents the real-time topographic deviation scalar value calculated during the current control cycle, with the symbol... Indicates the total number of valid points in the currently scanned point cloud, symbol Indicating the first point cloud The shortest symbolic distance from a point to the surface of the computer-aided design model. In some embodiments, the real-time pose of the rotary positioner is obtained by reading the real-time angle value of its spindle rotation axis from the servo driver of the rotary positioner via an industrial fieldbus, thus constituting the positioner pose.

[0092] In practical implementation, the data from the above five parts are combined into a one-dimensional vector in a preset, fixed order to form the input state of the deep reinforcement learning decision model. It can be understood that the dimension of the input state vector is predetermined. The robot end effector's real-time pose contains six scalars: position and orientation. The robot joint angle feedback contains six scalars: joint angles. The real-time contact force contains six scalars: three-dimensional force and three-dimensional torque. The real-time topography deviation is a scalar, and the rotational positioner's real-time pose is a scalar. Optionally, these scalars are concatenated in the order of "pose-joint angle-contact force-topography deviation-positioner angle" to generate a 20-dimensional input state vector. In each control cycle, the state reward construction module executes a complete construction process from the multimodal sensor data stream to this 20-dimensional vector. This uniformly formatted vector is the sole state input for the deep reinforcement learning decision model to make decisions.

[0093] In one embodiment of the invention, the reward signal is constructed to guide a deep reinforcement learning decision model to learn strategies that satisfy process quality, comply with process constraints, and maintain smooth motion. The reward signal is correlated with three factors: profile accuracy deviation, process constraint violation, and joint motion smoothness. See also... Figure 3The method for calculating the profile accuracy deviation is to use the norm of the deviation between the real-time profile deviation of the current processing area and the target reference profile as the quantized value. The judgment of process constraint violation is based on real-time monitoring data. When the real-time contact force exceeds a preset force threshold, or the thermal input power of the processing area calculated by laser power and feed speed exceeds a preset power threshold, a process constraint violation event is determined to have occurred. The quantification of joint motion smoothness is achieved by calculating the norm of the robot joint feedback angular acceleration, which reflects the degree of motion jitter. When constructing the instant reward signal, the negative values ​​of profile accuracy deviation, process constraint violation penalties, and joint motion smoothness are weighted and summed, where the profile accuracy weight, process constraint weight, and motion smoothness weight are preset weighting coefficients. The result of the weighted sum is the instant reward obtained by the model after receiving a specific input state and executing the corresponding output action. During system operation, the weighting coefficients can be adaptively adjusted according to the actual processing performance. The system records the actual values ​​of profile accuracy deviation, whether a process constraint violation has occurred, and the actual values ​​of joint motion smoothness in each control cycle when performing processing tasks in the physical environment. The system calculates the average value of profile accuracy deviation, the frequency of process constraint violation events, and the average value of joint motion smoothness within a preset time window. Based on the deviation of the average profile accuracy deviation from the preset quality target, the profile accuracy weight is dynamically adjusted upwards or downwards. Based on the frequency of process constraint violations, the process constraint weight is dynamically adjusted; a higher frequency violation results in a higher weight to strengthen the constraint. Based on the deviation of the average joint motion smoothness from the desired motion smoothness level, the motion smoothness weight is dynamically adjusted. Through this mechanism, the reward function can better adapt to the dynamic needs of the actual machining process.

[0094] In practical implementation, the specific process of constructing the reward signal and adaptively adjusting its weight coefficients is detailed in the example scenario of laser cladding repair using an industrial robot. The construction of the reward signal aims to guide the deep reinforcement learning decision model towards high-precision and high-stability processing behavior. In practice, the immediate reward signal obtained by the deep reinforcement learning decision model is associated with three core evaluation indicators: profile accuracy deviation, process constraint violation, and joint motion smoothness. The profile accuracy deviation value originates from the output of the real-time profile deviation calculation module. In each control cycle, this module calculates the deviation between the current cladding layer surface scanned point cloud and the target computer-aided design model and outputs a scalar value. Process constraint violation is determined through two real-time monitoring criteria. The first criterion is monitoring the real-time contact force reading from a six-dimensional force sensor; when the absolute value of any axial component exceeds a preset force threshold, a force constraint violation is determined. The second criterion is monitoring the real-time thermal input power calculated from the laser power and the cladding head feed speed; when the thermal input power exceeds a preset power threshold, a thermal constraint violation is determined. Joint motion smoothness is measured by calculating the norm of the robot joint feedback angular acceleration. In each control cycle, the real-time angular velocity of each joint is read from the robot servo driver, and the instantaneous angular acceleration of each joint is estimated by subtracting the angular velocity from the previous cycle and dividing by the control cycle duration, and then its norm is calculated.

[0095] In practical implementation, the process of constructing the instant reward signal requires dimensionless processing and weighted summation of the three quantitative indicators with different physical dimensions. It can be understood that the profile accuracy deviation is a value with a length dimension, and the joint motion smoothness is a value with an angular acceleration dimension. In some embodiments, the profile accuracy deviation is divided by a preset reference topography tolerance value, and the joint motion smoothness is divided by a preset reference angular acceleration value, thereby normalizing the two to a dimensionless ratio. Process constraint violations are represented as a dimensionless penalty term in the reward function. Within a control cycle, if any process constraint violation occurs, the penalty term is 1; otherwise, it is 0. The final instant reward signal is obtained by weighted summation of these three normalized values. Optionally, the formula for calculating the instant reward within a control cycle can be designed as follows:

[0096]

[0097] Where: symbol This represents the instantaneous reward signal calculated during the current control cycle, with the symbol... The dimensionless shape accuracy weighting coefficient is represented by the symbol. This represents the original value of the profile accuracy deviation with the dimension of length, denoted by the symbol. This represents a preset reference topographic tolerance value with the same length dimension, denoted by the symbol. Represents the dimensionless process constraint weighting coefficient, symbol This represents a dimensionless penalty term for a violation of process constraints, with a value of 0 or 1. (symbol) Represents the dimensionless motion smoothing weighting coefficient, symbol Represents the raw value of joint motion smoothness with angular acceleration dimension, symbol This represents a preset reference angular acceleration value with the same angular acceleration dimension. Through this design, each term in the formula is dimensionless, ensuring dimensional consistency on both sides of the formula.

[0098] In practical implementation, weighting coefficient , , The system adaptively adjusts based on actual processing performance. During the physical system's processing tasks, the execution and learning module continuously records the actual values ​​of profile accuracy deviation, Boolean flags indicating whether process constraint violations occurred, and actual values ​​of joint motion smoothness within each control cycle. The system maintains a preset statistical window of the most recent 1000 control cycles and calculates in real-time the average value of the actual profile accuracy deviation, the frequency of process constraint violation events, and the average value of the actual joint motion smoothness within this window. The profile accuracy weighting coefficient can be understood as... The adjustment is based on the degree of deviation of the average profile accuracy deviation from the preset quality target. If the average profile accuracy deviation continues to be higher than the quality target, the profile accuracy weighting coefficient is adjusted upward by a fixed step. This is to strengthen the guidance for accuracy optimization in the reward function. Process constraint weighting coefficients. The adjustment is based on the frequency of process constraint violations. If the violation frequency exceeds the allowable upper limit, the process constraint weight coefficient is increased. To strengthen the penalty for constraint violations. Motion smoothing weight coefficients. The adjustment is based on the degree of deviation of the average joint motion smoothness from the desired smoothness level. In some embodiments, the dynamic adjustment of weights is performed after each model parameter update cycle, and optionally, the adjustment magnitude of the weights is proportional to the degree of deviation from the target. Through this adaptive mechanism, the reward function can dynamically respond to changes in the state of the processing, keeping the optimization objective of the deep reinforcement learning decision model synchronized with real-time process requirements.

[0099] In one embodiment of the invention, a virtual model comprising an industrial robot, a workpiece, a rotary positioner, and a processing environment is established in a digital twin simulation environment, and these virtual entities are configured with the same dynamic and kinematic parameters as their physical counterparts. A deep reinforcement learning decision model is initialized in the simulation environment; this model typically includes an actor network for decision-making and a critic network for evaluation. At each simulation step, the system simulates and acquires multimodal sensor data streams, processing them into input states according to a predetermined method. This input state is provided to the deep reinforcement learning decision model, whose actor network generates an output action based on this state. This output action is executed in the simulation environment, changing the state of the virtual model. The system calculates a reward signal based on the new state data after execution and stores the input state before execution, the executed output action, the obtained reward signal, and the new input state after execution together as an experience tuple in a dedicated experience replay buffer. During training, the system periodically samples a batch of experience tuples randomly from this buffer and updates the parameters of the actor network and critic network using a reinforcement learning algorithm, with the goal of enabling the actor network to learn to generate action policies that maximize future cumulative rewards. Through iterative simulations with numerous step sizes, the model learns preliminary adaptive strategies in the virtual environment. The system establishes a model parameter synchronization mechanism between the digital twin simulation environment and the physical environment. In the simulation environment, continuous reinforcement learning training periodically generates updated parameters for the deep reinforcement learning decision model. In the physical environment, online execution and data collection also generate updated model parameters. The system sets a fixed model parameter fusion cycle. At the end of each fusion cycle, the parameters of the critic network and the actor network are extracted from the model parameters obtained from training in the simulation environment, and the corresponding network parameters are also extracted from the model parameters obtained from online fine-tuning in the physical environment. A weighted average is performed on the critic network parameters from both environments, and a weighted average is also performed on the actor network parameters from both environments to obtain the fused new network parameters. Finally, the fused network parameters are synchronously updated in the deep reinforcement learning decision models deployed in both the digital twin simulation environment and the physical environment, achieving bidirectional transfer and fusion of knowledge between the virtual and real worlds.

[0100] In practical implementation, the specific process of training a deep reinforcement learning decision model in a digital twin simulation environment and synchronizing model parameters between the simulation and the physical environment is illustrated by an example of offline training and online synchronization for laser repair of an aero-engine blade. In the implementation, a virtual model is established in the digital twin simulation environment, including a six-axis industrial robot, the blade to be repaired, a dual-axis rotary positioner, and a work cabin. The virtual model is constructed using 3D computer-aided design software and imported into the physics engine. In the physics engine, the industrial robot virtual model is configured with dynamic and kinematic parameters such as link length, mass distribution, and joint moment of inertia that are completely consistent with the actual robot. The blade virtual model is given thermodynamic properties such as melting point and thermal conductivity that match the actual material. And the rotary positioner virtual model is configured with precise reduction ratios and rotation ranges.

[0101] In the implementation, a deep reinforcement learning decision model is initialized in the simulation environment. This model employs an actor-critic framework, comprising an actor network for outputting action policies and a critic network for evaluating state-action values. At each simulation step, the simulation engine generates a multimodal sensor data stream based on the current state of the virtual model, including virtual end-effector pose, virtual joint encoder angles, virtual force sensor readings, and virtual 3D scan point clouds. In this implementation, this virtual sensor data is input to the state reward construction module, which processes it using the same logic as the physical environment to generate an input state vector. This input state vector is then provided to the deep reinforcement learning decision model, whose actor network generates an output action based on the input state vector, including joint increments, power adjustments, and feed rates. In this implementation, the output action is executed in the simulation environment, and the physics engine calculates the new states of the virtual robot, workpiece, and environment after execution. Based on the new virtual states, the simulation reward calculation module calculates the reward signal using the same reward function as the physical environment. It is understandable that the input state before execution, the output action after execution, the obtained reward signal, and the new input state after execution are collectively stored as an experience tuple in a dedicated simulation experience replay buffer. In some embodiments, the system periodically samples a batch of experience tuples randomly from the simulation experience replay buffer and uses a reinforcement learning algorithm, such as the flexible actor-critic algorithm, to update the parameters of the actor network and the critic network. Optionally, the goal of parameter updating is to minimize the temporal difference error of the critic network and maximize the expected reward of the actor network's output action. Through iterative training with a large number of simulation steps, the deep reinforcement learning decision model learns a preliminary strategy for co-optimizing laser cladding trajectory and process parameters in the virtual environment.

[0102] In practical implementation, the system establishes a model parameter synchronization mechanism between the digital twin simulation environment and the physical environment to achieve knowledge fusion. Training in the digital twin simulation environment and online operation in the physical environment each generate parameter updates for the deep reinforcement learning decision model. It can be understood that critic network parameters and actor network parameters are extracted from the model parameters obtained from training in the digital twin simulation environment, and corresponding critic network parameters and actor network parameters are also extracted from the model parameters obtained from online fine-tuning in the physical environment. In some embodiments, the system sets a fixed model parameter fusion cycle, for example, triggering a fusion after every 8 hours of physical processing tasks. In each fusion cycle, a parameter fusion operation is performed. The core of the model parameter fusion operation is to perform a weighted average of the network parameters from the two environments, the calculation formula of which can be expressed as:

[0103]

[0104] Where: symbol Represents the fused network parameter vector, with the symbol... This represents a network parameter vector from a digital twin simulation environment, with the symbol... This represents a vector of network parameters from the physical environment, with the symbol... This is a fusion weight coefficient between 0 and 1, used to balance the influence of virtual training and real-world experience. Optionally, the above formula can be applied independently to the commentator network parameters and the actor network parameters for fusion, but different fusion weight coefficients can be used. See Table 1, which shows a parameter fusion weight configuration.

[0105] Table 1: Model Parameter Fusion Weight Configuration Table

[0106]

[0107] In practice, based on predefined weight configurations, the parameters of the fused critic network and actor network are calculated separately. These fused network parameters are then synchronously updated in the corresponding networks of the deep reinforcement learning decision-making models deployed in both the digital twin simulation environment and the physical environment. This periodic bidirectional parameter synchronization effectively combines the general strategies learned through extensive low-cost trial and error in the digital twin simulation environment with the precise adaptive strategies learned through actual interaction in the physical environment, thereby improving the overall performance and adaptability of the deep reinforcement learning decision-making model.

[0108] In one embodiment of the present invention, during each control cycle of the physical processing, the system synchronously acquires the real-time pose of the robot's end effector, robot joint angle feedback, real-time contact force, real-time topographic deviation, and the real-time pose of the positioner. These real-time data are immediately combined to form the input state for the current control cycle. This input state is then fed into the actor network of a pre-trained deep reinforcement learning decision model. The actor network processes the input state and outputs a multi-dimensional continuous vector after forward propagation. The dimensions of this vector are predefined, with each dimension precisely corresponding to the minute position increments of each robot joint, the instantaneous adjustment of laser power, and the feed speed of the end effector. This multi-dimensional continuous vector is considered the output action generated by the deep reinforcement learning decision model in the current control cycle. The system repeats the above process in each control cycle, thereby generating a continuously changing sequence of output actions over time. The instruction execution module is responsible for converting the output action sequence into drive instructions for the underlying actuators. This module parses the output action sequence corresponding to each control cycle, extracting the three components: the minute position increments of each robot joint, the instantaneous adjustment of laser power, and the feed speed of the end effector. For robot joint control, the expected angle of each joint in the next control cycle is calculated based on the actual angle feedback value of each joint in the current control cycle, plus the minute position increment specified in the output action. The difference between this expected angle and the current actual angle is used as the input of the position loop and sent to the servo controller corresponding to each robot joint, which converts it into corresponding drive current or torque commands. For the laser, the laser power setpoint in the current control cycle is used as the basis, plus the instantaneous laser power adjustment specified in the output action; the sum is used as the laser power setpoint for the next control cycle, and a corresponding power control command is generated and sent to the laser. For the feed speed of the end effector, the system directly converts the speed value specified in the output action into a speed control command in the robot tool coordinate system and sends it to the robot motion controller.

[0109] In practical implementation, the process of generating output action sequences through online decision-making in the physical environment and converting these sequences into specific control commands is illustrated by a real-time control scenario of a six-DOF industrial robot performing laser cladding repair on a turbine disk. During each control cycle of the physical processing, the online decision-making control module synchronously acquires real-time sensor data from multiple hardware interfaces. This includes the real-time pose of the robot's end effector read from the robot controller, the robot joint angle feedback read from the joint encoders, the real-time contact force read from the six-dimensional force sensor, the real-time topographic deviation calculated from the three-dimensional vision sensor using a point cloud registration algorithm, and the real-time pose of the positioner read from the rotary positioner controller. In a system with a control cycle of 10 milliseconds, this data is synchronously sampled and locked at the beginning of the cycle. In practical implementation, the state reward construction module combines the acquired real-time pose of the robot's end effector, robot joint angle feedback, real-time contact force, real-time topographic deviation, and positioner real-time pose in the same predetermined order as during model training, forming a fixed-dimensional vector. This vector represents the input state of the deep reinforcement learning decision model for the current control cycle. The input state of the deep reinforcement learning decision model is immediately fed into the actor network of the already trained deep reinforcement learning decision model. The actor network can be understood as a deep neural network that processes the input state of the deep reinforcement learning decision model, outputting a multi-dimensional continuous vector after forward propagation through its internal fully connected layers and nonlinear activation functions. In some embodiments, for a six-DOF robot with a configuration of one laser power control variable and one feed rate control variable, the actor network outputs an 8-dimensional continuous vector. Refer to Table 2, which illustrates the mapping relationship between the dimensions of the output action vector and the control variables.

[0110] Table 2: Output Action Vector Dimension Mapping Table

[0111]

[0112] In practice, the 8-dimensional continuous vector output by the actor network is defined as the output action generated by the deep reinforcement learning decision model in the current control cycle. The online decision control module repeats the "sensing-building-decision" process in each control cycle, generating a continuously changing sequence of output actions over time. The instruction execution module receives this sequence and converts it into instructions that the underlying actuators can directly execute. The instruction execution module analyzes the output actions corresponding to each control cycle, extracting the minute position increments of each robot joint, the instantaneous adjustment of laser power, and the end effector feed rate. For robot joint control, the instruction execution module provides the angle feedback value of each joint in the current control cycle. Based on this, plus the minute position increment of the joint specified in the output motion. Calculate the expected angle of the joint in the next control cycle. The calculation formula is:

[0113]

[0114] Where: symbol This represents the calculated expected angle of the joint in the next control cycle, with the sign... This represents the angle feedback value of the joint read from the joint encoder during the current control cycle, with the symbol... This represents the minute position increment of the joint, resolved from the output motion vector. It can be understood as the desired angle of the joint. Feedback value of current angle The difference is used as the position loop error and input to the servo controller of the corresponding joint. The servo controller converts this error into a corresponding drive current or torque command according to its internal proportional-integral-derivative algorithm. For laser control, the command execution module uses the laser's power setpoint for the current control cycle. Based on this, plus the instantaneous adjustment of the laser power specified in the output action. The power setpoint for the next control cycle is obtained. This setting is then sent to the laser power supply via an analog output board or industrial Ethernet command to generate a power control command. Optionally, for end effector feed speed control, the command execution module directly sets the end effector feed speed value specified in the output action as a linear speed command in the tool coordinate system through the robot controller's application programming interface, driving the robot end effector to move at this speed. In some embodiments, the calculation and issuance of all the above commands must be completed within the current control cycle to ensure real-time control. Through the above process, the abstract action sequence output by the deep reinforcement learning decision model is accurately and in real-time converted into low-level control signals that drive the physical device to perform specific processing actions.

[0115] See Figure 4This is a sequence diagram of the action output by a deep reinforcement learning model. The laser power adjustment exhibits high-frequency, large-amplitude, and bidirectional fluctuations. Positive adjustments increase power to compensate for insufficient heat input during processing, ensuring the bonding strength of the cladding layer. Negative adjustments decrease power to prevent excessive heat input from causing workpiece deformation or ablation. The high-frequency fluctuations demonstrate the model's real-time decision-making capability at the 10ms level, adapting to complex processing conditions. The end effector feed speed is low-frequency, small-amplitude, and generally stable within the 10-15mm / s range. Speed ​​increases improve efficiency when processing is stable, while speed decreases ensure processing accuracy when morphological deviations exceed tolerances or contact forces are excessive. The overall stability reflects the model's balanced optimization of processing efficiency and quality. With a control cycle of 10ms / step, the model continuously outputs continuous actions, proving that the system meets the real-time requirements of industrial processing. The bidirectional dynamic adjustment of laser power intuitively demonstrates the model's ability to autonomously optimize process parameters based on processing conditions, distinguishing it from traditional fixed-parameter control.

[0116] In one embodiment of the present invention, the workflow of the system execution and learning module includes the collection of actual execution result data and online fine-tuning. After the instruction execution module issues drive commands, power control commands, and speed control commands to the industrial robot and laser, it drives the robot to execute the processing trajectory and controls the laser process parameters. During this physical execution process, the execution and learning module synchronously collects actual execution result data, which includes the actual pose of the end effector, the actual contact force, the actual shape deviation, and the actual joint acceleration. After each control cycle, the module reads the actual position feedback from the industrial robot controller to obtain the actual pose of the end effector. It reads the actual readings of the force sensors installed on the robot's end effector to obtain the actual contact force. It obtains the latest image of the processed surface through the vision sensor of the processing area and compares it with the target shape model to calculate the actual shape deviation. By reading the feedback data of the servo drives of each joint of the robot, the actual angular velocity of the joint can be calculated, and the actual acceleration of the joint can be estimated by the difference between the angular velocities of adjacent cycles. The collected actual execution result data will be precisely timestamped and associated with the input state corresponding to the same control cycle that produced these results, as well as the output action generated by the model in that cycle, forming a complete training experience data set. The system establishes a physical experience replay buffer to store the training experience data collected from the physical environment. Each data set is a tuple consisting of the input state, output action, actual execution result data, and the input state for the next time step calculated based on the actual result. The system sets a quantity trigger threshold; when the number of training experience data entries stored in the physical experience replay buffer reaches this threshold, an online fine-tuning process is automatically initiated. During the online fine-tuning process, a batch of training experience data is randomly sampled from the physical experience replay buffer, and the information in this batch of data is used to calculate the reward signal. Subsequently, the calculated reward signal and the sampled training experience data are used to update the parameters of the critic network and actor network in the deep reinforcement learning decision model. After completing this round of parameter updates, you can choose to clear or partially clear the physics experience replay buffer so that the system can continue to collect new physics execution experience data. Once the accumulated data reaches a threshold, the next online fine-tuning will be triggered, thereby enabling the model to continuously learn and adapt in the physical environment.

[0117] In practical implementation, the workflow of the system execution and learning module, including the collection of actual execution result data and online fine-tuning, is illustrated by a scenario where an industrial robot performs constant-force grinding on a complex curved workpiece. After the instruction execution module issues drive and speed control commands to the industrial robot and force control device, it drives the robot to execute an adaptive grinding trajectory. During execution, the execution and learning module simultaneously collects actual execution result data. This data includes the actual pose of the end effector, actual contact force, actual shape deviation, and actual joint acceleration. After each control cycle, the execution and learning module reads the filtered actual position feedback from the industrial robot controller to obtain the actual coordinates and orientation of the end effector in the base coordinate system, i.e., the actual pose of the end effector. The execution and learning module also reads the actual readings from the six-dimensional force sensor installed between the robot's end effector and the grinding head to obtain the force and torque in three directions, i.e., the actual contact force. The actual shape deviation is acquired using a 3D vision sensor integrated outside the processing area. After the robot completes a grinding path segment, the 3D vision sensor scans the processed surface, acquires point cloud data, and compares it with the computer-aided design model of the workpiece to calculate the actual shape deviation value corresponding to that path segment. The actual joint acceleration is acquired by reading the real-time feedback data from the servo drives of each joint of the robot. The servo drives provide high-frequency actual angular velocity information for each joint. By differentiating the actual angular velocities of two consecutive control cycles and dividing by the control cycle duration, the actual joint acceleration is estimated.

[0118] In practical implementation, the execution and learning module strictly timestamps and associates the actual pose, actual contact force, actual shape deviation, and actual joint acceleration of the end effector collected in each control cycle with the input state corresponding to the start of that control cycle and the output action generated by the deep reinforcement learning decision model in that cycle. It can be understood that since the measurement frequency of the actual shape deviation may be lower than the control frequency, in control cycles where the actual shape deviation measurement value is updated, it is associated with the current state and action; in control cycles where it is not updated, the previous measurement value is used or marked as invalid. After association and association, a complete training experience data set is formed, which is a structured data tuple. The execution and learning module establishes a physical experience replay buffer to sequentially store this training experience data collected from the physical environment. Each training experience data set contains four core elements: the input state before the action is executed, the action output by the deep reinforcement learning decision model, the reward signal calculated from the actual execution result data, and the input state for the next moment derived from the actual execution result data. In some embodiments, the actual topographic deviations, process constraint violations, and other data required to calculate the reward signal are all derived from actual execution result data, and the calculation method of the reward function is consistent with the simulation environment.

[0119] In practice, the physical experience replay buffer has a fixed storage capacity. When the number of training experience data stored in the buffer reaches a preset trigger threshold, the system automatically initiates an online fine-tuning process. The trigger threshold can be set to 2000 experience data points. During the online fine-tuning process, a batch of training experience data (256 data points) is randomly sampled from the physical experience replay buffer. This batch of training experience data is used to calculate the loss function and update the network parameters. Input state, action, reward signal, and next-time input state are extracted from the sampled training experience data. These data are used to calculate the parameter gradients of the critic network and the actor network in the deep reinforcement learning decision model. It can be understood that the goal of updating the critic network is to make its estimation of state-action value more accurate, while the goal of updating the actor network is to make its output action obtain a higher expected reward. The loss function formula used in the online fine-tuning is:

[0120]

[0121] Where: symbol The value of the loss function of the commentator network is represented by the symbol. This represents the batch size of training experience data sampled from the physical experience replay buffer, with the symbol […]. Indicates based on target network and reward signal Calculated target value, symbol This indicates that the critic network has parameters Next state and actions Value estimation. Using the calculated loss. The parameters of the critic network are updated using the backpropagation algorithm. The parameters of the actor network are updated according to the policy gradient method. After one round of parameter updates, the physical experience replay buffer can be cleared to collect new training experience data. Optionally, a first-in-first-out (FIFO) policy can be used to partially clear the buffer, retaining the most recent set of experience data. In some embodiments, the online fine-tuning process is performed asynchronously in a background thread of the real-time control system to avoid interfering with high-priority real-time control tasks. In this way, the deep reinforcement learning decision model can continuously optimize its policy using real data generated during the operation of the physical system, adapting to uncertainties such as tool wear and changes in material properties during the machining process.

[0122] See Figure 5This is a multimodal sensor data stream acquisition diagram of an industrial robot end effector, representing core visualization data for robot motion control and path planning. The X-axis displacement (approximately 100mm) is significantly greater than the Y-axis displacement (approximately 50mm), indicating that the main path of this machining / motion extends along the X-axis. Neither curve exhibits drastic fluctuations, demonstrating smooth robot motion control without sudden stops, jitter, or abrupt trajectory changes. From 0 to 40 seconds, the Z-axis drops to a negative trough (approximately -5mm), potentially representing workpiece sinking, machining head sinking, or machining into workpiece grooves / depth directions. From 40 to 80 seconds, the Z-axis rapidly rises to a positive peak (approximately 10mm), reflecting dynamic compensation of the Z-axis, possibly to adapt to curved surface machining, height changes, or clearance adjustments. From 80 to 100 seconds, the Z-axis descends again, completing the finishing motion. This visually demonstrates the system's Z-axis height adaptive capability, dynamically adjusting the height according to real-time operating conditions, corresponding to the "pose feedback" and "motion adjustment" stages in a reinforcement learning control system.

[0123] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. An industrial robot adaptive trajectory planning and control system based on reinforcement learning, characterized in that, include: The data acquisition module acquires multimodal sensor data streams from the industrial robot, rotary positioner, and sensor network of the processing area. The state reward construction module processes the multimodal sensing data stream, constructs the input state of the deep reinforcement learning decision model, and constructs a reward signal according to the quality requirements of the composite manufacturing process. The decision model training module defines the output action of the deep reinforcement learning decision model. The output action is a continuous action that includes the incremental position of the robot joints, the instantaneous adjustment of the laser power, and the feed speed of the end effector. In the digital twin simulation environment, the deep reinforcement learning decision model is trained by reward signals, so that the deep reinforcement learning decision model learns the mapping relationship between the output action and the input state. The online decision control module deploys a trained deep reinforcement learning decision model in a physical environment. The deep reinforcement learning decision model generates a corresponding output action sequence based on the real-time acquired input state. The instruction execution module converts the output motion sequence into drive instructions for each control joint of the robot, power control instructions for the laser, and speed control instructions for the end effector.

2. The industrial robot adaptive trajectory planning and control system based on reinforcement learning according to claim 1, characterized in that, The specific process of constructing the input state of a deep reinforcement learning decision model includes: The input states include the real-time pose of the robot end effector, robot joint angle feedback, real-time contact force during the machining process, real-time topographic deviation of the machining area, and real-time pose of the positioner. The real-time coordinates and orientation angles of the robot end effector in the base coordinate system are read from the industrial robot controller to form the real-time pose of the robot end effector. Read the real-time angle measurement values ​​of the encoders of each joint of the industrial robot to form the robot joint angle feedback; The real-time contact force between the tool and the workpiece during the machining process is collected by a force sensor installed on the robot's end effector. The surface image of the current processing area is captured by a vision sensor in the processing area. The surface image is registered and compared with a reference topography model to calculate the real-time topography deviation. The real-time angle of the rotation axis is read from the rotary positioner controller to form the real-time pose of the positioner; The robot end effector's real-time pose, robot joint angle feedback, real-time contact force, real-time shape deviation, and positioner's real-time pose are arranged and combined in a predetermined order to generate a vector with fixed dimensions. This vector is the input state of the deep reinforcement learning decision model.

3. The industrial robot adaptive trajectory planning and control system based on reinforcement learning according to claim 2, characterized in that, The specific process of constructing the reward signal includes: The reward signal is at least related to profile accuracy deviation, process constraint violation, and joint motion smoothness; A method for calculating profile accuracy deviation is defined, wherein the profile accuracy deviation is the norm of the deviation between the real-time profile deviation of the current processing area and the target reference profile; Define a method for judging process constraint violations. When the real-time contact force exceeds a preset force threshold or the heat input power of the processing area exceeds a preset power threshold, it is judged that a process constraint violation has occurred. A quantitative method for defining joint motion smoothness is provided, wherein the joint motion smoothness is measured by calculating the norm of the robot joint feedback angular acceleration; The negative values ​​of profile accuracy deviation, penalty values ​​for process constraint violation, and negative values ​​of joint motion smoothness are weighted and summed, with the weighting coefficients being profile accuracy weight, process constraint weight, and motion smoothness weight, respectively. The weighted summation result is used as the immediate reward signal obtained by the deep reinforcement learning decision-making model after receiving the input state and executing the corresponding output action within the current decision-making cycle.

4. The industrial robot adaptive trajectory planning and control system based on reinforcement learning according to claim 3, characterized in that, In a digital twin simulation environment, the specific process of guiding the training of a deep reinforcement learning decision model through reward signals includes: Virtual models of industrial robots, workpieces, rotary positioners, and the processing environment are created in a digital twin simulation environment. In the virtual model, the same dynamic and kinematic parameters are configured for the industrial robot, workpiece, rotary positioner, and processing environment as for the physical environment. A deep reinforcement learning decision model is initialized in a simulation environment, the deep reinforcement learning decision model comprising an actor network and a critic network; At each simulation step, multimodal sensing data streams in the simulation environment are acquired and processed into input states to provide to the deep reinforcement learning decision model. The actor network of the deep reinforcement learning decision model generates output actions based on the input state and executes the output actions in a simulation environment. The reward signal is calculated based on the simulation environment data after execution. The input state, output action, reward signal and new input state are stored as an experience tuple in the simulation experience playback buffer. A batch of experience tuples is periodically sampled from the simulation experience replay buffer to update the parameters of the actor network and critic network of the deep reinforcement learning decision model; Through iterative training with a large number of simulated step sizes, the actor network of the deep reinforcement learning decision model learns to generate output action strategies that maximize the cumulative reward signal.

5. The industrial robot adaptive trajectory planning and control system based on reinforcement learning according to claim 4, characterized in that, Deploying a trained deep reinforcement learning decision model in a physical environment, wherein the deep reinforcement learning decision model generates a corresponding output action sequence based on real-time acquired input states, includes the following steps: In each control cycle of the physical processing, the real-time pose of the robot end effector, the robot joint angle feedback, the real-time contact force, the real-time shape deviation, and the real-time pose of the positioner are acquired synchronously. The real-time pose of the robot end effector, the robot joint angle feedback, the real-time contact force, the real-time shape deviation, and the real-time pose of the positioner are combined to form the input state of the current control cycle. The input state of the current control cycle is fed into the actor network of the already trained deep reinforcement learning decision model; The actor network processes the input state and outputs a multidimensional continuous vector. Each dimension of the multidimensional continuous vector corresponds to the minute position increment of each joint of the robot, the instantaneous adjustment of the laser power, and the feed speed of the end effector. This multidimensional continuous vector is used as the output action generated by the deep reinforcement learning decision model in the current control cycle, and this process is repeated in the next control cycle to generate a time-varying sequence of output actions.

6. The industrial robot adaptive trajectory planning and control system based on reinforcement learning according to claim 5, characterized in that, The process of converting the output motion sequence into drive commands for each control joint of the robot, power control commands for the laser, and speed control commands for the end effector includes: The output action corresponding to each control cycle in the output action sequence is analyzed, and the minute position increments of each joint of the robot, the instantaneous adjustment of laser power, and the feed speed of the end effector contained in the output action are extracted. For each joint of the robot, based on the angle feedback of the joint in the current control cycle, plus the small position increment of the joint specified in the output action, the expected angle of the joint in the next control cycle is calculated. The difference between the desired angle of the joint and the current angle feedback is input into the servo controller of each joint of the robot and converted into corresponding drive current or torque commands. For lasers, the current laser power setting value is used as the basis, plus the instantaneous laser power adjustment amount specified in the output action, as the power setting value for the next control cycle, and the corresponding power control command is generated. For the end effector, the end effector feed speed specified in the output action is directly converted into a speed control command in the robot's tool coordinate system.

7. The industrial robot adaptive trajectory planning and control system based on reinforcement learning according to claim 6, characterized in that, Also includes: The execution and learning module issues corresponding drive commands, power control commands, and speed control commands to the industrial robot and laser, driving the robot to execute the processing trajectory and control the laser process parameters. During the execution of the industrial robot, the module simultaneously collects actual execution result data, associates the actual execution result data with the input state and output action sequence to form training experience data, and feeds the training experience data back to the deep reinforcement learning decision model for online fine-tuning. The specific process of synchronously collecting actual execution result data during the execution of an industrial robot includes: The actual execution result data includes the actual pose of the end effector, the actual contact force, the actual shape deviation, and the actual joint acceleration; After each control cycle, the actual position feedback provided by the industrial robot controller is read to obtain the actual pose of the end effector; Read the actual readings of the force sensor installed on the robot's end effector to obtain the actual contact force; The latest image of the processed surface is acquired by a vision sensor in the processing area and compared with the target model to calculate the actual shape deviation. Read the feedback data from the servo drives of each joint of the robot, calculate the actual angular velocity of the joint, and estimate the actual acceleration of the joint by calculating the difference in angular velocity. The actual pose, actual contact force, actual shape deviation, and actual joint acceleration of the end effector collected in each control cycle are timestamped and associated with the input state and generated output action corresponding to the control cycle.

8. The industrial robot adaptive trajectory planning and control system based on reinforcement learning according to claim 7, characterized in that, The actual execution result data is associated with the input state and output action sequence to form training experience data. This training experience data is then fed back to the deep reinforcement learning decision model for online fine-tuning. The specific process includes: Establish a physical experience replay buffer to store training experience data collected from the physical environment. The training experience data is a tuple consisting of input state, output action, actual execution result data, and input state at the next moment. Set a trigger threshold; when the amount of training experience data stored in the physical experience replay buffer reaches the trigger threshold, the online fine-tuning process will be started. During online fine-tuning, a batch of training experience data is randomly sampled from the physical experience replay buffer, and the reward signal is calculated using the training experience data. Using the calculated reward signal and sampled training experience data, the parameters of the critic network and actor network in the deep reinforcement learning decision model are updated in one round. After completing a round of parameter updates, you can choose to clear or partially clear the physics experience replay buffer, continue to collect new training experience data, and wait for the next online fine-tuning to be triggered.

9. The industrial robot adaptive trajectory planning and control system based on reinforcement learning according to claim 8, characterized in that, The adaptive adjustment process of the weight coefficients involved in the construction of the reward signal includes: Record the actual values ​​of profile accuracy deviation, process constraint violation, and joint motion smoothness within each control cycle when performing machining tasks in a physical environment; Calculate the average value of profile accuracy deviation, the frequency of process constraint violation, and the average value of joint motion smoothness within a preset statistical window; The profile accuracy weight is dynamically adjusted based on the deviation of the average profile accuracy deviation from the preset target. The weights of process constraints are dynamically adjusted based on the frequency of process constraint violations. The motion smoothing weights are dynamically adjusted based on the deviation of the average joint motion smoothness from the desired level.

10. The industrial robot adaptive trajectory planning and control system based on reinforcement learning according to claim 9, characterized in that, The system establishes a model parameter synchronization mechanism between the digital twin simulation environment and the physical environment. The specific working process of the model parameter synchronization mechanism includes: In a digital twin simulation environment, reinforcement learning training based on simulation data is continuously performed, and updated parameters of the deep reinforcement learning decision model are generated periodically. In physical environments, updated parameters for deep reinforcement learning decision models are also generated through online fine-tuning. Set a model parameter fusion cycle, and in each fusion cycle, extract the critic network parameters and actor network parameters from the model parameters obtained from training in the simulation environment; Extract critic network parameters and actor network parameters from model parameters obtained through online fine-tuning in the physical environment; The network parameters are obtained by weighted averaging of the network parameters of the commenters from the simulation environment and the network parameters of the actors from the simulation environment and the physical environment. The fused parameters of the commentator network and the actor network are synchronously updated to the deep reinforcement learning decision model deployed in the digital twin simulation environment and the physical environment.

Images