Adaptive control method and system for hyper-redundant robot based on double-loop evolutionary reinforcement learning

Abstract

The application relates to a hyper-redundant robot adaptive control method and system based on double-loop evolutionary reinforcement learning. The method comprises the following steps: acquiring robot data; constructing an outer loop evolutionary evaluation layer and an inner loop behavior execution layer; the outer loop evolutionary layer generates a plurality of candidate weight vectors of a reward function; the inner loop behavior execution layer trains according to the plurality of candidate weight vectors and the robot data and generates a plurality of fitness indexes; the candidate weight vector corresponding to the highest fitness index is retained, and the plurality of candidate weight vectors are dynamically optimized again to generate the plurality of candidate weight vectors for execution training until the training is completed, and the optimal weight ratio is obtained; and the adaptive control of the robot is realized. By introducing the double-loop evolutionary reinforcement learning mechanism, the dependence on the setting of the reward function weight of the artificial experience in the reinforcement learning control process is reduced, the automatic adjustment of the reward parameter and the adaptive optimization of the strategy performance are realized, and therefore the autonomous decision-making ability and the control robustness of the hyper-redundant robot in a limited complex environment are improved.

Description

Technical Field

[0001] This invention relates to the field of mechanical control, and more specifically, to an adaptive control method and system for a super-redundant robotic arm based on dual-loop evolutionary reinforcement learning. Background Technology

[0002] With the continuous development of industrial automation and intelligent control technologies, hyper-redundant manipulators, especially snake-shaped manipulators with multiple joints and high flexibility, have shown significant advantages in applications such as maintenance of confined compartments in aerospace, detection of confined spaces in nuclear power plants, and medical surgery because they can use redundant degrees of freedom to bypass obstacles and adapt to complex spatial structures.

[0003] However, when hyper-redundant robotic arms perform motion control and path planning tasks in confined environments with narrow spaces and complex obstacle distribution, existing control methods still face several technical bottlenecks, mainly in the following aspects: The computational redundancy and singularity problems introduced by hyper-redundant degrees of freedom: For snake-like robotic arms with a large number of joints, their inverse kinematics solution space typically exhibits an infinite number of solutions. In confined environments, the robotic arm not only needs to ensure that the end effector accurately reaches the target position, but also needs to simultaneously constrain all joints and links of the entire robotic arm to avoid collisions with surrounding obstacles. Existing analytical methods or inverse kinematics methods based on numerical iteration suffer from high computational complexity and poor real-time performance when dealing with such high-dimensional, multi-constraint problems. They are also prone to failure near local optima or singular poses, making it difficult to meet the engineering application requirements for continuous and stable control. Summary of the Invention

[0004] Therefore, it is necessary to provide an adaptive control method and system for super-redundant robotic arms based on dual-loop evolutionary reinforcement learning, which has better control performance, to address the aforementioned technical problems.

[0005] The adaptive control method for super-redundant robotic arms based on dual-loop evolutionary reinforcement learning includes:

[0006] S1: Acquire robotic arm data; S2: Construct the outer ring evolution evaluation layer and the inner ring behavior execution layer; S3: Multiple candidate weight vectors for the outer ring evolutionary layer reward function; S4: The inner loop behavior execution layer is trained based on multiple candidate weight vectors and robotic arm data and generates multiple fitness metrics; S5: Retain the candidate weight vector corresponding to the highest fitness index, and perform dynamic optimization to generate multiple candidate weight vectors again. Return to step S4 and continue until training is completed to obtain the optimal weight ratio. S6: Achieve adaptive control of the robotic arm based on the optimal weight ratio.

[0007] Furthermore, in step S1, specifically: multiple sensor groups are set on the robotic arm, and the sensor groups collect high-frequency data from the robotic arm. The real-time state of each degree of freedom and the environmental characteristics within the confined space are used as data for the robotic arm.

[0008] Furthermore, the sensor array high-frequency data acquisition robot arm The real-time state of each degree of freedom and the environmental characteristics within the confined space are specifically used as data for the robotic arm: S1.1: Joint pose feature extraction and encoding: By deploying in Sensor arrays on each joint acquire the current angle information of each joint. ;right Each joint angle is mapped using sine and cosine mappings to generate... A 3D attitude-aware vector; S1.2: Physical limit margin sensing: The rotational extreme values ​​of each joint are preset according to the hardware specifications of the robotic arm. Through calculation ,get Dimensional constraint margin characteristics; S1.3: Task Space Goal-Oriented Feature Extraction: The real-time world coordinates of the end effector and the target point coordinates are determined by the sensor array; the 3D relative displacement vector and the 1D scalar Euclidean distance between the two are calculated to synthesize 4D target guidance features; S1.4: Obstacle avoidance feature extraction based on discrete sampling of the entire torso: Divide the full arm into Independent links are used in each segment; environmental point clouds are acquired using depth vision sensors or LiDAR, and the shortest distance of each link relative to the nearest obstacle is calculated in real time. ; S1.5: Calculation of the direction vector of the most dangerous obstacle point: Extract the obstacle point with the minimum distance value within the entire arm's range, and calculate the 3D unit direction vector from that point to the corresponding link of the robotic arm. ; S1.6: Sequential Action Memory Integration: Record the control signal issued at the previous moment. Incremental joint movement commands .

[0009] Further, in step S3, the outer ring evolution layer generates the candidate weight vector for the reward function specifically as follows: the outer ring evolution layer initializes the... A population of particles, each particle randomly distributed in a preset area. Within the dimensional weighted search space, the outer ring evolution layer randomly selects particles and maps them to a set of standardized reward function weight vectors based on the spatial position information of the currently selected particles.

[0010] Further, in step S4, the training of the inner loop behavior execution layer based on the candidate weight vector of the reward function and the robotic arm data includes: setting the total reward function based on the candidate weight vector of the reward function; and training based on the total reward function. Total reward function Defined as the weighted vector sum of multiple hierarchical sub-items, its mathematical expression is as follows:

[0011] in, The candidate weight vector for the reward function. Rewards are designed to guide the recipient's goals. Punishment for joint limitation. For static collision penalties, For motion smoothing penalty, The core of this invention is the dynamic trend obstacle avoidance reward. A reward for successful task completion; Goal-Driven Rewards Based on the Euclidean distance between the end effector and the target point, an exponential decay function is used.

[0012] Joint restriction punishment Static collision penalty The logarithmic penalty term or inverse proportional penalty term with boundary mutation characteristics is adopted. When the joint angle or link distance is close to the physical limit, the penalty value increases nonlinearly as the margin decreases. Motion smoothing rewards Based on the energy distribution of motion increments, nonlinear suppression is applied to high-frequency jumps. Dynamic control mechanism for fusion vector projection and dual threshold decoupling Dynamic trend-aware reward items based on vector projection mechanism Real-time calculation of the motion trend vector generated by the current action command of the robotic arm Normal vector of the most dangerous obstacle dot product projection between ,according to The symbol determines the intention of movement: when When it is determined to be a risk of kinetic collision and a proportional penalty is applied; when If the direction is determined to be safe and the vehicle passes or moves away from the target, no penalty will be triggered. During training, a dual-threshold control strategy is employed, decoupling the task space and safety space, by setting non-overlapping safety intervention thresholds. With the task success threshold and satisfy The logical constraints are such that when the robotic arm's end effector enters the task success threshold range, the obstacle avoidance repulsion force weight of the end joint is dynamically reduced and a high task reward is activated. .

[0013] Further, in step S4, the fitness index is calculated specifically by: based on the average cumulative reward value during the training process. and task success rate Calculate fitness: .

[0014] Furthermore, in step S5, the dynamic optimization specifically involves: adjusting the total reward function... The weight parameters in the equation are encoded as position vectors of evolving particles, and the first... The position vector of each particle for:

[0015] In this context, each dimension of the position vector corresponds to a weight coefficient for the target guidance reward, physical constraint penalty, static collision penalty, motion smoothing penalty, and dynamic trend obstacle avoidance reward; the particle's search space is restricted to a preset physical reasonable range. ; During the outer ring evolution process, each particle represents a specific reward evaluation system, and the weight vector is iteratively optimized according to the following velocity and position update formula:

[0016]

[0017] in, This indicates the increment step size for weight updates. and These represent the individual optimal weight configuration and the global optimal weight configuration, respectively. The acceleration constant, For random operators; A cosine annealing strategy was introduced during the evolutionary iteration to dynamically adjust the inertia weight. Its mathematical expression is:

[0018] in, For the current evolutionary generation, This represents the total number of iterations.

[0019] Furthermore, it also includes: during the training process in step S4, maintaining a time-series success rate sequence using a sliding window technique. Calculate the weighted average success rate within the current window. ,in This is the time decay factor; When weighted average success rate Continuous The course advancement threshold is consistently exceeded within each cycle. When the super-redundant robotic arm has learned the obstacle avoidance and positioning paradigm under the current constraints, the scene evolution command is executed to increase the task difficulty to the next order of magnitude by dynamically adjusting the topology of the colliders or the physical boundary parameters in the simulation environment.

[0020] Furthermore, it also includes: upon activation of the freeze command, immediately initiating the evolution freeze logic: forcibly locking the position vectors of the entire current particle swarm within the outer ring evolution layer. This causes the reward weight configuration to enter a static hold state; During the freeze period, the behavior reshaping phase begins; at this time, the inner-loop behavior execution layer, guided by a fixed reward evaluation system, concentrates computing resources on the new scenario. Multimodal observation data is reconstructed to correct policy biases caused by changes in environmental width or geometric features; fitness volatility is defined. Only when the performance of the policy network in the new environment tends to stabilize, and satisfies continuously below the preset smoothing threshold Only then can the evolution freeze state be lifted.

[0021] After the freeze is lifted, the outer loop resumes the parameter optimization logic based on cosine annealing weights, and uses the stable feedback of the reshaped inner loop to perform fine-grained iteration of the weights.

[0022] Furthermore, this invention provides an adaptive control system for a super-redundant robotic arm based on dual-loop evolutionary reinforcement learning, comprising: Acquisition module: Acquires data from the robotic arm; Construction modules: Construct the outer ring evolution evaluation layer and the inner ring behavior execution layer; Candidate weight generation module: The outer ring evolution layer generates multiple candidate weight vectors for the reward function; Training module: The inner loop behavior execution layer is trained based on multiple candidate weight vectors and robotic arm data, and generates multiple fitness metrics; Iterative module: Retain the candidate weight vector corresponding to the highest fitness index, and dynamically optimize it to generate multiple candidate weight vectors again. Return to step S4 and continue until training is complete to obtain the optimal weight ratio. Control module: Implements adaptive control of the robotic arm based on the optimal weight allocation.

[0023] This invention introduces a dual-loop evolutionary reinforcement learning mechanism to reduce the reliance on manually set reward function weights during reinforcement learning control, thereby enabling automatic adjustment of reward parameters and adaptive optimization of strategy performance. This enhances the autonomous decision-making ability and control robustness of the hyper-redundant robotic arm in constrained and complex environments. Attached Figure Description

[0024] Figure 1 This is a flowchart of an adaptive control method for an over-redundant robotic arm based on dual-loop evolutionary reinforcement learning in one embodiment. Figure 2 This is a block diagram of an adaptive control system for a super-redundant robotic arm based on dual-loop evolutionary reinforcement learning in one embodiment. Figure 3 This describes the overall system architecture and workflow of the control method in one embodiment. Figure 4 This is a schematic diagram of multimodal perception and feature sampling in one embodiment; Figure 5 This is an example of the dynamic trend perception and dual threshold decoupling principle in one embodiment; Figure 6 A schematic diagram of the reward weight optimization and cosine annealing strategy in one embodiment; Figure 7 Here is a flowchart of the course-based transfer and progressive freeze control in one embodiment; Detailed Implementation To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0025] Example 1: This embodiment provides, as follows: Figure 1 The adaptive control method for a super-redundant robotic arm based on dual-loop evolutionary reinforcement learning, as shown, includes: S1: Acquire robotic arm data; S2: Construct the outer ring evolution evaluation layer and the inner ring behavior execution layer; S3: Multiple candidate weight vectors for the outer ring evolutionary layer reward function; S4: The inner loop behavior execution layer is trained based on multiple candidate weight vectors and robotic arm data and generates multiple fitness metrics; S5: Retain the candidate weight vector corresponding to the highest fitness index, and perform dynamic optimization to generate multiple candidate weight vectors again. Return to step S4 and continue until training is completed to obtain the optimal weight ratio. S6: Achieve adaptive control of the robotic arm based on the optimal weight ratio.

[0026] This embodiment introduces a dual-loop evolutionary reinforcement learning mechanism to reduce the reliance on manually set reward function weights during reinforcement learning control, thereby enabling automatic adjustment of reward parameters and adaptive optimization of strategy performance. This enhances the autonomous decision-making ability and control robustness of the hyper-redundant robotic arm in constrained and complex environments.

[0027] Example 2: Furthermore, in step S1, specifically: multiple sensor groups are set on the robotic arm, and the sensor groups collect high-frequency data from the robotic arm. The real-time state of each degree of freedom and the environmental characteristics within the confined space are used as data for the robotic arm.

[0028] Furthermore, the sensor array high-frequency data acquisition robot arm The real-time state of each degree of freedom and the environmental characteristics within the confined space are specifically used as data for the robotic arm: S1.1: Joint pose feature extraction and encoding: By deploying in Sensor arrays on each joint acquire the current angle information of each joint. ;right Each joint angle is mapped using sine and cosine mappings to generate... A 3D attitude-aware vector; S1.2: Physical limit margin sensing: The rotational extreme values ​​of each joint are preset according to the hardware specifications of the robotic arm. Through calculation ,get Dimensional constraint margin characteristics; S1.3: Task Space Goal-Oriented Feature Extraction: The real-time world coordinates of the end effector and the target point coordinates are determined by the sensor array; the 3D relative displacement vector and the 1D scalar Euclidean distance between the two are calculated to synthesize 4D target guidance features; S1.4: Obstacle avoidance feature extraction based on discrete sampling of the entire torso: Divide the full arm into Independent links are used in each segment; environmental point clouds are acquired using depth vision sensors or LiDAR, and the shortest distance of each link relative to the nearest obstacle is calculated in real time. ; S1.5: Calculation of the direction vector of the most dangerous obstacle point: Extract the obstacle point with the minimum distance value within the entire arm's range, and calculate the 3D unit direction vector from that point to the corresponding link of the robotic arm. ; S1.6: Sequential Action Memory Integration: Record the control signal issued at the previous moment. Incremental joint movement commands .

[0029] Further, in step S3, the outer ring evolution layer generates the candidate weight vector for the reward function specifically as follows: the outer ring evolution layer initializes the... A population of particles, each particle randomly distributed in a preset area. Within the dimensional weighted search space, the outer ring evolution layer randomly selects particles and maps them to a set of standardized reward function weight vectors based on the spatial position information of the currently selected particles.

[0030] Further, in step S4, the training of the inner loop behavior execution layer based on the candidate weight vector of the reward function and the robotic arm data includes: setting the total reward function based on the candidate weight vector of the reward function; and training based on the total reward function. Total reward function Defined as the weighted vector sum of multiple hierarchical sub-items, its mathematical expression is as follows:

[0031] in, The candidate weight vector for the reward function. Rewards are designed to guide the recipient's goals. Punishment for joint limitation. For static collision penalties, For motion smoothing penalty, The core of this invention is the dynamic trend obstacle avoidance reward. A reward for successful task completion; Goal-Driven Rewards Based on the Euclidean distance between the end effector and the target point, an exponential decay function is used.

[0032] Joint restriction punishment Static collision penalty The logarithmic penalty term or inverse proportional penalty term with boundary mutation characteristics is adopted. When the joint angle or link distance is close to the physical limit, the penalty value increases nonlinearly as the margin decreases. Motion smoothing rewards Based on the energy distribution of motion increments, nonlinear suppression is applied to high-frequency jumps. Dynamic control mechanism for fusion vector projection and dual threshold decoupling Dynamic trend-aware reward items based on vector projection mechanism Real-time calculation of the motion trend vector generated by the current action command of the robotic arm Normal vector of the most dangerous obstacle dot product projection between ,according to The symbol determines the intention of movement: when When it is determined to be a risk of kinetic collision and a proportional penalty is applied; when If the direction is determined to be safe and the vehicle passes or moves away from the target, no penalty will be triggered. During training, a dual-threshold control strategy is employed, decoupling the task space and safety space, by setting non-overlapping safety intervention thresholds. With the task success threshold and satisfy The logical constraints are such that when the robotic arm's end effector enters the task success threshold range, the obstacle avoidance repulsion force weight of the end joint is dynamically reduced and a high task reward is activated. .

[0033] Further, in step S4, the fitness index is calculated specifically by: based on the average cumulative reward value during the training process. and task success rate Calculate fitness:

[0034] Furthermore, in step S4, the dynamic optimization specifically involves: adjusting the total reward function... The weight parameters in the equation are encoded as position vectors of evolving particles, and the first... The position vector of each particle for:

[0035] In this context, each dimension of the position vector corresponds to a weight coefficient for the target guidance reward, physical constraint penalty, static collision penalty, motion smoothing penalty, and dynamic trend obstacle avoidance reward; the particle's search space is restricted to a preset physical reasonable range. ; During the outer ring evolution process, each particle represents a specific reward evaluation system, and the weight vector is iteratively optimized according to the following velocity and position update formula:

[0036]

[0037] in, This indicates the increment step size for weight updates. and These represent the individual optimal weight configuration and the global optimal weight configuration, respectively. The acceleration constant, For random operators; A cosine annealing strategy was introduced during the evolutionary iteration to dynamically adjust the inertia weight. Its mathematical expression is:

[0038] in, For the current evolutionary generation, This represents the total number of iterations.

[0039] Furthermore, it also includes: during the training process in step S4, maintaining a time-series success rate sequence using a sliding window technique. Calculate the weighted average success rate within the current window. ,in This is the time decay factor; When weighted average success rate Continuous The course advancement threshold is consistently exceeded within each cycle. When the super-redundant robotic arm has learned the obstacle avoidance and positioning paradigm under the current constraints, the scene evolution command is executed to increase the task difficulty to the next order of magnitude by dynamically adjusting the topology of the colliders or the physical boundary parameters in the simulation environment.

[0040] Furthermore, it also includes: upon activation of the freeze command, immediately initiating the evolution freeze logic: forcibly locking the position vectors of the entire current particle swarm within the outer ring evolution layer. This causes the reward weight configuration to enter a static hold state; During the freeze period, the behavior reshaping phase begins; at this time, the inner-loop behavior execution layer, guided by a fixed reward evaluation system, concentrates computing resources on the new scenario. Multimodal observation data is reconstructed to correct policy biases caused by changes in environmental width or geometric features; fitness volatility is defined. Only when the performance of the policy network in the new environment tends to stabilize, and satisfies continuously below the preset smoothing threshold Only then can the evolution freeze state be lifted.

[0041] After the freeze is lifted, the outer loop resumes the parameter optimization logic based on cosine annealing weights, and uses the stable feedback of the reshaped inner loop to perform fine-grained iteration of the weights.

[0042] This embodiment introduces a dual-loop evolutionary reinforcement learning mechanism to reduce the reliance on manually set reward function weights during reinforcement learning control, thereby enabling automatic adjustment of reward parameters and adaptive optimization of strategy performance. This enhances the autonomous decision-making ability and control robustness of the hyper-redundant robotic arm in constrained and complex environments.

[0043] Example 3: Furthermore, this embodiment provides, as follows: Figure 2 The diagram shows an adaptive control system for a super-redundant robotic arm based on dual-loop evolutionary reinforcement learning, including: Acquisition module: Acquires data from the robotic arm; Construction modules: Construct the outer ring evolution evaluation layer and the inner ring behavior execution layer; Candidate weight generation module: The outer ring evolution layer generates multiple candidate weight vectors for the reward function; Training module: The inner loop behavior execution layer is trained based on multiple candidate weight vectors and robotic arm data, and generates multiple fitness metrics; Iterative module: Retain the candidate weight vector corresponding to the highest fitness index, and dynamically optimize it to generate multiple candidate weight vectors again. Return to step S4 and continue until training is complete to obtain the optimal weight ratio. Control module: Implements adaptive control of the robotic arm based on the optimal weight allocation.

[0044] This embodiment introduces a dual-loop evolutionary reinforcement learning mechanism to reduce the reliance on manually set reward function weights during reinforcement learning control, thereby enabling automatic adjustment of reward parameters and adaptive optimization of strategy performance. This enhances the autonomous decision-making ability and control robustness of the hyper-redundant robotic arm in constrained and complex environments.

[0045] Example 4: This embodiment proposes a super-redundant robotic arm control scheme based on dual-loop evolutionary reinforcement learning. Its core architecture consists of an outer-loop evolutionary evaluation layer, an inner-loop behavior execution layer, and the controlled object and perception system. This architecture combines high-dimensional policy learning with global weight evolution to achieve autonomous obstacle avoidance and precise movement of the physical robotic arm in complex and confined spaces.

[0046] System Overall Architecture The system's overall control architecture consists of two core closed loops: ① Outer ring evolution evaluation layer (evolutionary ring): A heuristic optimization algorithm is used as the global search engine, preferably Particle Swarm Optimization (PSO), but Genetic Algorithm (GA) or Differential Evolution (DE) can also be used. Its main responsibility is to encode the various weight parameters in the reward function (such as target attraction, collision penalty, etc.) into the position information of the evolving particles, and to continuously iterate and optimize the weight vector W based on the fitness index fed back from the inner ring during the macro-training cycle.

[0047] ② Inner Loop Behavior Execution Layer (Learning Loop): This layer employs a deep reinforcement learning algorithm as the underlying controller, preferably a soft actor-commentator algorithm based on maximum entropy (SAC), but can also use a dual-delay deep deterministic policy gradient algorithm (TD3) or a proximal policy optimization algorithm (PPO). This layer receives the current optimal weight vector injected from the outer loop and interacts frequently with the confined space in the physical simulation environment. It iteratively learns the optimal joint action output through the policy gradient of the neural network. .

[0048] Dual-loop collaborative workflow The system achieves collaborative optimization of the two-tier architecture through the following steps: ① System Multidimensional Initialization and Sensing Preparation: After system startup, the initialization of the underlying hardware and high-level algorithms is performed synchronously. The sensor group begins high-frequency data acquisition from the robotic arm. The real-time state of each degree of freedom and the environmental characteristics within the confined space. Simultaneously, the outer ring evolution layer initialization includes... A population of particles, each particle randomly distributed in a preset area. Within the dimensional weight search space.

[0049] ② Outer ring weight mapping and parameter injection: The outer ring evolution layer maps the spatial position information of the currently selected particle into a set of standardized reward function weight vectors. This weight vector is injected in real time into the reward evaluation module of the inner loop behavior execution layer, serving as the sole guiding criterion for the inner loop to determine the "quality" of actions within the current training cycle, thereby achieving adaptive dynamic configuration of the evaluation criteria.

[0050] ③ Multimodal state extraction and observation space construction: The inner loop sensing module transforms the raw signals collected by the sensors into... through feature engineering. Dimensional normalization of observation vectors. The specific process includes: performing sine and cosine encoding on joint angles to eliminate physical discontinuities; calculating the shortest distance of each link relative to obstacles; and extracting the unit direction vector pointing to the most dangerous obstacle point on the entire arm. It also integrates the action memory from the previous moment. This step ensures that the control strategy has full-body obstacle avoidance perception and temporal correlation.

[0051] ④ Inner-loop policy generation and incremental action execution: Inner-loop deep reinforcement learning network reception The dimensional observation vector, through forward computation via the Actor Network, outputs... Continuous incremental action signal The signal is obtained through the incremental mapping formula. The values ​​are converted into target positions for each joint, and then physically driven by the underlying controller. This incremental control mode effectively avoids abrupt motion changes in high-dimensional spaces in overly redundant systems. After completing a preset number of training cycles in the inner loop, the system calculates the average cumulative reward value and task success rate within that cycle, and feeds them back to the outer loop evolution layer as fitness indicators.

[0052] ⑤ Real-time dynamic reward calculation and experience storage: The system calculates the instant reward value based on the physical feedback after the action is executed. The core of the system is the vector projection mechanism: calculating the motion trend vector. With the obstacle normal vector The dot product is used to apply differentiated penalties to actions with a collision tendency. Simultaneously, decoupled dual-threshold logic is used to dynamically adjust the penalty boundaries after the endpoint enters the task region. All interaction trajectories are stored in an asynchronous experience replay pool.

[0053] ⑥ Policy Gradient Iteration and Fitness Evaluation: The inner loop randomly samples from the experience pool and updates the neural network parameters according to the maximum entropy optimization criterion to improve the expected cumulative reward of the policy. After completing a preset number of training cycles, the system calculates the average reward value and end-point localization accuracy within that cycle and uses them as fitness indicators to feed back to the outer loop. The outer loop updates the individual best (pbest) and global best (gbest) states of the particles based on their fitness.

[0054] ⑦ Course Scene Migration and Weight Freeze Protection: The system monitors the task success rate in real time. When the success rate remains stable above a preset threshold, it automatically triggers a course difficulty upgrade command, loading more complex and restricted conditions. In the initial stage after scene switching, the system executes "evolution freeze protection": suspending outer loop weight updates and only allowing the inner loop network to adapt to the physical characteristics of the new environment until the training curve returns to a stable state.

[0055] Methods for constructing multimodal high-dimensional sensing state spaces This embodiment uses a multimodal sensing system to collect real-time information on the internal state and external environment of an ultra-redundant robotic arm, and transforms this information into a data structure with high feature representation capabilities. 3D standardized observation vector This vector provides a complete decision-making basis for the inner loop behavior execution layer. The specific construction steps are as follows: Step 1: Joint pose feature extraction and encoding The system is deployed in High-precision encoders on each joint acquire the current angle information of each joint. To eliminate the impact of numerical discontinuities in angle at physical limits on neural network convergence, the system... Each joint angle is mapped using sine and cosine mappings to generate... A 3D attitude-aware vector.

[0056] Step 2: Physical limit margin sensing The system presets the rotational extreme values ​​of each joint according to the hardware specifications of the robotic arm. Through calculation ,get The dimension of the limit margin feature. This feature enables the control strategy to sense whether the joint is approaching its physical limits, thereby planning avoidance actions in advance.

[0057] Step 3: Task Space Goal-Oriented Feature Extraction The system uses an external positioning device to determine the real-time world coordinates of the end effector and the target point coordinates. It calculates the 3D relative displacement vector and the 1D scalar Euclidean distance between the two, and synthesizes a 4D target guidance feature to indicate the movement direction and remaining stroke of the robotic arm end effector.

[0058] Step 4: Obstacle avoidance feature extraction based on discrete sampling of the entire torso To address the obstacle avoidance blind spots caused by the excessively long links in the super-redundant robotic arm, the system divides the entire arm into sections. Independent link segments. Environmental point clouds are acquired using depth vision sensors or LiDAR, and the shortest distance of each link segment relative to the nearest obstacle is calculated in real time. The obstacle avoidance perception vector generated by this process can characterize the collision risk distribution of the entire torso of the robotic arm, and is not limited to end-effector perception.

[0059] Step 5: Calculate the direction vector of the most dangerous obstacle point Based on step 4, the system extracts the obstacle point with the smallest distance value within the entire arm range and calculates the 3D unit direction vector from that point to the corresponding link of the robotic arm. This vector provides a crucial geometric reference for subsequent "vector projection reward," enabling it to identify whether the robotic arm is approaching or deviating from a high-risk obstacle zone.

[0060] Step 6: Integrating Sequential Action Memory The system records the control commands issued at the previous control moment. Incremental joint movement commands By incorporating historical actions into the current observation, the temporal relevance of the Markov decision-making process is maintained, effectively improving the smoothness of the motion trajectory and avoiding drastic jumps in action commands between adjacent moments.

[0061] Constructed through the above steps This method utilizes a multidimensional state space to comprehensively cover the five dimensions of "attitude, constraint, task, obstacle avoidance, and memory" for an ultra-redundant robotic arm. It effectively addresses the fine-grained problem of obstacle avoidance with long links by employing piecewise sampling, and simultaneously transforms the complex physical environment into standardized input through feature engineering, significantly reducing the training difficulty of reinforcement learning algorithms in high-dimensional nonlinear spaces.

[0062] A reward function modeling method that integrates motion trend perception with task-safety decoupling To address the challenge of coordinating obstacle avoidance safety and end-effector positioning accuracy in complex, narrow environments with multiple physical constraints, the core of the reward function modeling method described in this embodiment lies in constructing a composite evaluation system that includes multi-dimensional physical constraints and dynamic risk perception. This system enables precise responses to the weights injected into the outer loop and real-time guidance of the motion state of the super-redundant robotic arm.

[0063] (1) Design idea and mathematical model of hierarchical reward function The system's total reward function Defined as the weighted vector sum of multiple hierarchical sub-items, its mathematical expression is as follows:

[0064] in, The dynamic weight vector injected into the outer ring evolution layer. Rewards are designed to guide the recipient's goals. Punishment for joint limitation. For static collision penalties, For motion smoothing penalty, The core of this embodiment is the dynamic trend obstacle avoidance reward. A reward for successful completion of the task.

[0065] The design concept of the hierarchical reward function is to resolve the conflict of evaluation criteria in nonlinear composite tasks through the parallel decoupling and dynamic aggregation of multi-dimensional physical indicators. ① Decoupling and convergence design for multi-objective decision-making The system extracts features from fundamentally different physical quantities such as positioning accuracy, full-arm obstacle avoidance, and physical limits of the mechanism through independent operators, and performs weighted value aggregation at the final output, allowing the system to dynamically switch behavioral preferences under different working conditions by adjusting the weight ratio.

[0066] ② Dynamic game of weights between task space and constraint space The weight parameters are optimized online by the outer-loop evolution algorithm, which enables the reward function to adaptively adjust the penalty intensity according to the severity of environmental constraints, ensuring that the training of the inner-loop policy is always in the optimal gradient descent range.

[0067] (2) Nonlinear value mapping of prior physical knowledge This embodiment introduces prior physical knowledge to perform differentiated nonlinear value mapping on different categories of physical feedback signals, thereby achieving efficient search in a high-dimensional configuration space. Its specific design logic is as follows: ①Exponential gain mapping at the target guidance level The target-guided reward Based on the Euclidean distance between the end effector and the target point, an exponential decay function is preferred. This nonlinear mapping ensures that the robotic arm has a stable motion gradient when it is far from the target, and significantly improves accuracy sensitivity when it is close to the target point.

[0068] ② Barrier-type penalty mapping at the physical constraint level Joint restriction punishment Static collision penalty The preferred approach is to use a logarithmic penalty term or an inverse proportional penalty term with boundary mutation characteristics. When the joint angle or link distance approaches the physical limit, the penalty value increases nonlinearly as the margin decreases, thus constructing a virtual "repulsive barrier" in mathematical logic, forcing the strategy network to avoid high-risk action spaces.

[0069] ③ Second-order smooth mapping of action space hierarchy The smooth reward of the action Based on the energy distribution of motion increments, nonlinear suppression of high-frequency jumps ensures the synergy of joint movements.

[0070] Dynamic control mechanism for fusion vector projection and dual threshold decoupling The key innovation of the reward function modeling method described in this embodiment lies in the introduction of a dynamic trend-aware reward term based on a vector projection mechanism. This mechanism calculates the motion trend vector generated by the current action command of the robotic arm in real time. Normal vector of the most dangerous obstacle dot product projection between The system is based on The symbol determines the intention of movement: when When it is determined to be a risk of kinetic collision and a proportional penalty is applied; when If the movement is deemed safe to tangentially pass over or move away from the obstacle, no penalty is triggered. This logic identifies the geometric features of the movement intention, allowing the super-redundant robotic arm to achieve tangential movement that conforms to the obstacle surface in extremely narrow working conditions, eliminating motion deadlock in traditional solutions.

[0071] Furthermore, to resolve the antagonistic conflict between attraction and repulsion forces during end-position insertion, this embodiment employs a dual-threshold control strategy that decouples the task space and safety space. This strategy involves setting non-overlapping safety intervention thresholds. With the task success threshold and satisfy Logical constraints. When the robotic arm's end effector enters the task success threshold range, the system dynamically reduces the obstacle avoidance repulsion force weight of the end joint and activates a high task reward. This eliminates the interference of the obstacle repulsion field on the end-positioning guidance force from a mathematical logic perspective, ensuring that the super-redundant robotic arm can achieve millimeter-level precision positioning accuracy while ensuring the safety of obstacle avoidance throughout the entire torso.

[0072] Weight adaptive tuning mechanism based on particle swarm evolution and cosine annealing The complex physical coupling and nonlinear antagonism among the components of the reward function present a significant technical challenge in manually setting weights to adapt to all task conditions. This embodiment employs a nested optimization framework based on an "evolutionary outer loop - learning inner loop." It uses the Particle Swarm Optimization (PSO) algorithm to heuristically search the weight vector of the reward function within the global parameter space, and combines this with a cosine annealing strategy to dynamically adjust the evolutionary step size. This allows for the automatic acquisition of the optimal evaluation parameter configuration for a specific constrained space without the need for expert intervention.

[0073] (1) Particle encoding and spatial mapping of weight vectors The weighted reward function The weight parameters in the equation are encoded as position vectors of evolving particles, and the first... The position vector of each particle for:

[0074] In this context, each dimension of the position vector corresponds to a weight coefficient for the target guidance reward, physical constraint penalty, static collision penalty, motion smoothness penalty, and dynamic trend obstacle avoidance reward. The particle's search space is restricted to a pre-defined, physically reasonable range. This is to ensure the convergence stability of the inner loop reinforcement learning training.

[0075] (2) Adaptive Iterative Logic Based on Particle Swarm Operator During the outer ring evolution, each particle represents a specific reward evaluation system. The system iteratively optimizes the weight vector according to the following velocity and position update formulas:

[0076]

[0077] in, This indicates the increment step size for weight updates. and These represent the individual optimal weight configuration and the global optimal weight configuration, respectively. The acceleration constant, It is a random operator.

[0078] Dynamic control of inertia weights in fusion cosine annealing To balance global exploration capability with local convergence accuracy, this embodiment introduces a cosine annealing strategy during evolutionary iteration to dynamically adjust the inertia weight. Its mathematical expression is:

[0079] in, For the current evolutionary generation, This represents the total number of iterations.

[0080] In the early stages of evolution, Maintaining relatively large values ​​endows the weight vector with strong global search capabilities to cover diverse reward allocation schemes; as iterations progress, Based on the smooth decay of the cosine curve, particles are guided to converge precisely towards the high-fitness region. Compared to the traditional linear decay strategy, the cosine annealing logic can more effectively escape the local optimum trap of the reward function, ensuring that the final weight vector injected into the inner loop has stronger environmental robustness.

[0081] Inner and outer loop closed-loop interaction and fitness evaluation The optimization mechanism described in this embodiment uses a fitness function. Achieve data closure between inner and outer loops: ① Parameter Injection: The outer ring will inject the current particle's position vector. Injected into the inner loop behavior execution layer as the reward weight parameter of the deep learning algorithm.

[0082] ② Inner loop training: The inner loop performs ultra-redundant robotic arm trajectory planning training based on this set of weights and records the task execution performance.

[0083] ③ Feedback Evaluation: The evolutionary layer evaluates the average cumulative reward value based on the inner loop feedback. and task success rate Calculate fitness:

[0084] in, To evaluate the weights, the system ultimately selects the optimal weight combination that enables the super-redundant robotic arm to learn a high-precision, high-safety motion strategy in the shortest possible time through the above closed-loop iteration.

[0085] A Method for Scenario Transfer and Stability Control Based on Course Stage Perception Training control strategies for hyperredundant robotic arms in complex dynamic environments faces severe "curse of dimensionality" and "exploration barriers." If training is initially conducted directly in extremely confined scenarios, the policy network is highly susceptible to deadlock with zero-incremental feedback under sparse rewards due to the difficulty of reaching the target point through random exploration. When the system migrates from a simple to a complex environment, drastic changes in physical constraints often cause significant performance fluctuations in the inner-loop behavior execution layer. If the outer-loop evolution layer maintains high-frequency updates at this time, it is prone to misinterpreting short-term performance degradation and incorrectly adjusting reward weights, leading to convergence failure of the entire control evaluation system.

[0086] Progressive Course Scenario Design Logic This embodiment pre-sets a multi-level course scenario sequence from easy to difficult: ① Basic navigation stage: The environment is free of obstacles or has only a few loose obstacles, which aims to guide the robotic arm to learn basic kinematic mapping and target approach capabilities.

[0087] ② Intermediate obstacle avoidance stage: Introduce static random obstacle blocks, activate the whole body segment distance perception and static collision penalty, so that the robotic arm can learn the initial spatial avoidance behavior.

[0088] ③ Advanced Constraint Stage: Construct an extremely constrained, narrow working condition. At this point, the vector projection reward logic is fully activated, forcing the robotic arm to learn a precise operation strategy of sliding tangentially along the surface of the obstacle.

[0089] Stage-based perception triggering and scenario evolution logic based on success rate threshold The stage-aware triggering mechanism described in this embodiment constructs a highly sensitive state monitoring operator to determine in real time the coverage of the current control strategy to a specific complexity environment. During the inner-loop training process, the system uses a sliding window technique to maintain a time-series success rate sequence. To eliminate random noise interference during the reinforcement learning exploration process, the system does not directly use the results of a single training round, but instead calculates the weighted average success rate within the current window. ,in This is a time decay factor designed to increase the weight of recent performance.

[0090] When the weighted average success rate Continuous The course advancement threshold is consistently exceeded within each cycle. At this point, the environment manager determines that the hyper-redundant robotic arm has learned the obstacle avoidance and positioning paradigm under the current constraints. The system then executes a scenario evolution command, dynamically adjusting the topology of colliders or physical boundary parameters in the simulation environment to increase the task difficulty to the next order of magnitude. This triggering mechanism ensures that the system can switch between "exploration" and "exploitation" with optimal step sizes, avoiding overfitting in simple scenarios and providing a logical prerequisite for ultimately learning control strategies under extremely constrained conditions.

[0091] Evolutionary Freeze and Behavioral Remodeling Regulation Mechanisms During Environmental Switching To address the risk of strategy failure due to sudden changes in physical constraints during course transitions, this embodiment proposes a temporally decoupled stability control scheme. Its core lies in implementing a two-stage transition of "evolutionary freeze" and "behavior reshaping." Upon activation of the perception trigger command, the system immediately initiates the evolutionary freeze logic: in the outer ring evolution layer, it forcibly locks the position vectors of the entire current particle swarm. This allows the reward weight configuration to enter a static hold state, thereby cutting off the sensitive response path of the outer loop parameters to the instantaneous performance degradation of the inner loop.

[0092] During the freeze period, the system enters the behavior reshaping phase. At this time, the inner-loop behavior execution layer (SAC), guided by a fixed reward evaluation system, concentrates computing resources on the new scenario. Multimodal observation data is reconstructed to correct policy biases caused by changes in environment width or geometric features. Fitness volatility is defined. Only when the performance of the policy network in the new environment tends to stabilize, and satisfies continuously below the preset smoothing threshold Only then can the system unfreeze its evolution.

[0093] After unfreezing, the outer loop resumes its parameter optimization logic based on cosine annealing weights, and uses the reshaped stable feedback of the inner loop for fine-tuning the weight iterations. This stability control method, through asynchronous updates of the physical scene and evaluation parameters, effectively isolates the interference of environmental non-stationarity on the evolutionary algorithm, preventing the weights from converging to extremely conservative regions due to the high collision rate in the initial migration stage. This ensures the monotonic convergence and long-term robustness of the ultra-redundant robotic arm control system during the migration process across challenging tasks.

[0094] The overall system logic block diagram is as follows: Figure 3 As shown, the system comprises three core logical modules: the outer-loop evolutionary evaluation layer, the inner-loop behavior execution layer, and the physical system and perception layer. The diagram clearly illustrates the interactions and data flows between these modules, including: the "weight vector injection" implemented by the outer-loop evolutionary algorithm based on global optimization results to the inner loop; the "incremental action signals" generated by the inner-loop policy network to drive physical entities; the "multimodal observation" data collected and fed back to the inner loop by physical system sensors (such as encoders and LiDAR); and the "fitness feedback" fed back to the outer loop based on task performance for parameter iteration. This diagram macroscopically reveals the complete closed-loop workflow from evolutionary weight optimization and policy gradient learning to precise control of physical entities.

[0095] A schematic diagram of multimodal perception and feature sampling is shown below. Figure 4As shown, this diagram illustrates the feature extraction and mapping logic of the system from physical environment characteristics to high-dimensional mathematical observation vectors. The left side of the image shows a redundant robotic arm entity in a constrained obstacle environment, with detailed markings of encoder signal acquisition points distributed across each joint, the shortest distance sampling lines from each link segment to the obstacle, the task displacement vector pointing from the robotic arm's end effector to the target, and the normal vector extracted at high-risk collision points to characterize risk trends. The right side shows the structured, standardized observation vector components, encompassing the integrated process from attitude encoding, physical limit margins, target guidance features to full-arm segmented obstacle avoidance features, and historical action memory. This diagram intuitively reveals how this invention transforms complex physical constraints into high-dimensional digital representations by sampling full-body features of an ultra-redundant robotic arm, providing complete and accurate decision inputs for the inner-loop policy network.

[0096] Dynamic trend perception and dual threshold decoupling principle as follows Figure 5 The diagram illustrates the core geometric discrimination logic in reward function modeling. The left side explains the motion trend perception mechanism, which determines the sign of collision risk by calculating the dot product of the motion trend vector and the obstacle normal vector in real time, thus allowing the robotic arm to perform tangential sweeping movements without triggering high penalties. The right side demonstrates the decoupling mechanism between the task space and the safety space. By setting non-overlapping double-layer distance thresholds, it ensures that after the end effector enters the precision work area, the positioning guidance force can dominate motion generation, eliminating the attraction-repulsion antagonism phenomenon in the neighborhood of the target point from a physical perspective.

[0097] The principle diagram of reward weight optimization and cosine annealing strategy is as follows: Figure 6 The diagram illustrates the dynamics of the outer ring evolution layer adaptively adjusting the reward weights of the inner ring. The upper left region describes the particle swarm dynamics. The search behavior within the weighted space is illustrated, with each particle's position vector corresponding to a specific set of reward weights; the right-hand area displays the inertia weights. The cosine annealing curve changes with the number of evolutionary generations. This mechanism ensures the global optimization capability of weight search by maintaining high exploratory activity in the early stage of evolution and implementing fine convergence in the later stage. The closed-loop path on the lower left shows the process of calculating the fitness index based on the task success rate and average cumulative reward based on the inner loop feedback. This feedback guides the particle swarm to migrate to the region with the optimal weight ratio, thereby realizing the automated construction of the evaluation system for the control strategy of the super-redundant robotic arm.

[0098] The flowchart for curriculum-based transfer and progressive freeze control is as follows: Figure 7The diagram macroscopically illustrates the environmental evolution logic and state control strategies executed by the control system as the training progresses. The top layer shows the increasing sequence of task scenario complexity from easy to difficult, covering the entire process of evolution from basic open environment navigation to extremely restricted narrow-channel operations; the middle layer reveals the average success rate based on the sliding window (…). ) and preset promotion threshold ( The phase perception trigger logic for real-time comparison establishes the criteria for automatic scene migration. The timeline view at the bottom of the figure intuitively shows the temporal decoupling control mechanism at the moment of switching between environments of different difficulty. That is, when the system enters a new scene, it is forced to enter the "evolution freeze stage" and temporarily locks the outer ring reward weight parameters until the inner ring policy network completes the behavior reshaping and no longer produces drastic fitness fluctuations. Only then is the "parameter optimization state" of the outer ring particles restored, thereby ensuring the robustness and convergence stability of the entire migration process.

[0099] (1) Effectively eliminate motion deadlock and obstacle avoidance hypersensitivity in confined spaces: By introducing a trend perception mechanism based on geometric vector projection, the system can accurately identify the tangential sweeping intention of the robotic arm, rather than relying solely on scalar distance to perform repulsive actions. This enables the redundant robotic arm to smoothly pass through the surface of obstacles in extremely narrow working conditions, fundamentally solving the motion deadlock problem caused by repulsive force balance in confined spaces using the traditional artificial potential field method.

[0100] (2) Significantly improves end-effector pose convergence accuracy under complex working conditions: This invention eliminates the physical antagonism between attraction and repulsion in the target neighborhood through a dual-threshold decoupling strategy between the task space and the safety boundary. When the end of the robotic arm approaches the target point, the system can dynamically weaken the interference of the obstacle avoidance force field, ensuring that the positioning guidance force dominates, so that the redundant robotic arm can still achieve millimeter-level precision positioning even when it is close to an obstacle.

[0101] (3) Significantly reduces the cost of algorithm parameter tuning and deployment threshold for complex and redundant systems: This invention utilizes a dual-loop collaborative architecture of "evolutionary outer loop - behavioral inner loop" to achieve automatic optimization of reward weights through the PSO algorithm combined with the cosine annealing strategy. This mechanism replaces the cumbersome manual parameter tuning process, enabling the control system to have adaptive deployment capabilities for different unstructured environments, and significantly improving the engineering practicality of the algorithm.

[0102] (4) Ensures the stability of strategy convergence during complex task migration: By introducing scenario migration logic based on course stage awareness and evolution freeze mechanism, the system can guide the robotic arm to smoothly transition from simple working conditions to extremely restricted working conditions. The evolution freeze mechanism effectively isolates the drastic performance fluctuations caused by environmental changes, prevents the training collapse of the reward evaluation system, and ensures the monotonic convergence of the control strategy in long-term training.

[0103] (5) Balancing the smoothness of mechanism motion with physical safety: This invention integrates full-torso segmented distance sampling into the perception features and couples motion smoothness penalty and physical limit constraint into the reward function. This not only ensures the coordination of joint motions in 12-DOF or higher super-redundant mechanisms, but also significantly reduces the risk of physical collisions between the mechanism and the environment, and extends the service life of the robotic arm hardware.

[0104] Example 5: This embodiment selects a super-redundant serpentine robotic arm with 12 degrees of freedom (DOF) as the actuator. The robotic arm consists of six sets of two-DOF gimbal structural units connected in series, each gimbal capable of coordinated movement in both yaw and pitch dimensions. This mechanism performs autonomous navigation and precise positioning tasks in S-shaped narrow passage conditions with multiple physical constraints.

[0105] Step 1: Sensing System Configuration and Environment Modeling In the experimental environment, a continuous S-shaped curve simulating a narrow industrial environment was first constructed, with the width of the passage set to 1.5 times the maximum envelope diameter of the robotic arm. A sensing layer was deployed throughout the robotic arm's torso, using 12 high-precision absolute encoders to acquire the angular pose signals of each joint in real time. Meanwhile, multiple sampling points are pre-set on the surface of the 12 connecting rods, and the shortest segment distance between each part of the robotic arm and the inner wall of the channel is obtained in real time using a range sensor array. In addition, the robotic arm's end effector is equipped with a position feedback unit to calculate the task vector between itself and a preset target point in real time. .

[0106] Step 2: Construction of multimodal high-dimensional observation vectors After the system starts, the main control module will collect the physical signals according to... Figure 2 The mapping logic shown is standardized to construct a 67-dimensional standardized observation vector. This vector encompasses 24-dimensional joint angle sine and cosine features, 30-dimensional segmented obstacle distance features, 3-dimensional end-effector pose deviation features, 3-dimensional motion trend projection features, and 1-dimensional physical limit margin. All raw feature data have undergone zero-mean normalization to eliminate dimensional differences and ensure that the inner-loop policy network receives a stable and complete state representation.

[0107] Step 3: Optimize and tune the dual-loop collaborative logic The system enters the execution phase, where the outer-loop evolutionary layer initializes a particle swarm containing 20 sets of candidate weights. Each set of weights is injected into the inner-loop reinforcement learning module through the outer-loop controller, serving as guiding parameters for the current evaluation function. The inner loop employs an Actor-Critic architecture, utilizing a SAC policy network for action generation. During trial iterations, the system records the task success rate and path smoothness for each set of weights. Based on the feedback fitness index, the evolutionary layer uses a PSO algorithm with cosine annealing to evaluate the performance. Global optimization is performed with equal weights until the optimal weight ratio that enables the 12-DOF robotic arm to efficiently navigate S-shaped curves is selected.

[0108] Step 4: Integrating trend perception with action generation decoupled from dual thresholds When the robotic arm approaches a narrow bend, the system performs dynamic trend perception calculations in real time. When the predicted motion vector from the inner loop tends towards the obstacle, the system calculates the dot product of the motion trend and the obstacle's normal vector. The system identifies collision risks and applies non-linear penalties. When the robotic arm performs a tangential sweep along the inner wall of the passage, the system automatically assesses its safety and maintains a low-penalty state, thus ensuring the smooth movement of the entire arm in extremely narrow spaces. When the robotic arm's end effector enters the preset 15-centimeter task success threshold from the target point, the system automatically activates decoupling logic, weakening the intervention of obstacle avoidance repulsion and strengthening the target attraction, guiding the robotic arm's end effector to achieve final positioning with a accuracy better than 1 centimeter.

[0109] Step 5: Environmental Transfer Guided by Course Learning To verify the system's stability, the experiment adopted a scenario migration mode from easy to difficult. The robotic arm first completed policy initialization in a basic open environment, then gradually introduced random obstacles, and finally migrated to an S-shaped extremely narrow passage. After each upgrade in scenario difficulty, the system performed an evolution freeze protection for 1000 iteration cycles to maintain stable reward weights and train only the behavior reconstruction ability of the inner loop network. Once the system's fitness volatility in the new environment decreased to below 5%, outer loop optimization was resumed, thus ensuring the monotonic convergence of the control system as the task complexity increased.

[0110] Experimental results Through the above implementation methods, the results of 100 independent tests on the 12-DOF super-redundant robotic arm in an S-shaped channel show that the average success rate of the present invention reaches more than 95%, the end-effector positioning accuracy is stable at less than 1 cm, and the average collision rate of the entire arm is reduced by 40% when passing through the curve, which effectively proves the practicality and advanced nature of the present invention in complex and confined space operations.

Claims

1. An adaptive control method for a super-redundant robotic arm based on dual-loop evolutionary reinforcement learning, characterized in that, include: S1: Acquire robotic arm data; S2: Construct the outer ring evolution evaluation layer and the inner ring behavior execution layer; S3: Multiple candidate weight vectors for the outer ring evolutionary layer reward function; S4: The inner loop behavior execution layer is trained based on multiple candidate weight vectors and robotic arm data and generates multiple fitness metrics; S5: Retain the candidate weight vector corresponding to the highest fitness index, and perform dynamic optimization to generate multiple candidate weight vectors again. Return to step S4 and continue until training is completed to obtain the optimal weight ratio. S6: Achieve adaptive control of the robotic arm based on the optimal weight ratio.

2. The adaptive control method for a super-redundant robotic arm based on dual-loop evolutionary reinforcement learning according to claim 1, characterized in that, In step S1, specifically: multiple sensor groups are set on the robotic arm, and the sensor groups collect high-frequency data from the robotic arm. The real-time state of each degree of freedom and the environmental characteristics within the confined space are used as data for the robotic arm.

3. The adaptive control method for a super-redundant robotic arm based on dual-loop evolutionary reinforcement learning according to claim 2, characterized in that, Sensor group high-frequency acquisition robotic arm The real-time state of each degree of freedom and the environmental characteristics within the confined space are specifically used as data for the robotic arm: S1.1: Joint pose feature extraction and encoding: By deploying in Sensor arrays on each joint acquire the current angle information of each joint. ;right Each joint angle is mapped using sine and cosine mappings to generate... A 3D attitude-aware vector; S1.2: Physical limit margin sensing: The rotational extreme values ​​of each joint are preset according to the hardware specifications of the robotic arm. ; through calculation ,get Dimensional constraint margin characteristics; S1.3: Task Space Goal-Oriented Feature Extraction: The real-time world coordinates of the end effector and the target point coordinates are determined by the sensor array; the 3D relative displacement vector and the 1D scalar Euclidean distance between the two are calculated to synthesize 4D target guidance features; S1.4: Obstacle avoidance feature extraction based on discrete sampling of the entire torso: Divide the full arm into Independent links are used in each segment; environmental point clouds are acquired using depth vision sensors or LiDAR, and the shortest distance of each link relative to the nearest obstacle is calculated in real time. ; S1.5: Calculation of the direction vector of the most dangerous obstacle point: Extract the obstacle point with the minimum distance value within the entire arm range, and calculate the 3D unit direction vector from the obstacle point to the corresponding link of the robotic arm. ; S1.6: Sequential Action Memory Integration: Record the control signal issued at the previous moment. Incremental joint movement commands .

4. The adaptive control method for a super-redundant robotic arm based on dual-loop evolutionary reinforcement learning according to claim 1, characterized in that, In step S3, the outer ring evolution layer generates candidate weight vectors for the reward function, specifically as follows: the outer ring evolution layer initialization includes... A population of particles, each particle randomly distributed in a preset area. Within the dimensional weighted search space, the outer ring evolution layer randomly selects particles and maps them to a set of standardized reward function weight vectors based on the spatial position information of the currently selected particles.

5. The adaptive control method for a super-redundant robotic arm based on dual-loop evolutionary reinforcement learning according to claim 1, characterized in that, In step S4, the inner loop behavior execution layer is trained based on the candidate weight vector of the reward function and the robotic arm data, including: setting the total reward function based on the candidate weight vector of the reward function; and training based on the total reward function. Total reward function Defined as the weighted vector sum of multiple hierarchical sub-items, its mathematical expression is as follows: in, The candidate weight vector for the reward function. Rewards are designed to guide the recipient's goals. Punishment for joint limitation. For static collision penalties, For motion smoothing penalty, The core of this invention is the dynamic trend obstacle avoidance reward. A reward for successful task completion; Goal-Driven Rewards Based on the Euclidean distance between the end effector and the target point, an exponential decay function is used. Joint restriction punishment Static collision penalty The logarithmic penalty term or inverse proportional penalty term with boundary mutation characteristics is adopted. When the joint angle or link distance is close to the physical limit, the penalty value increases nonlinearly as the margin decreases. Motion smoothing rewards Based on the energy distribution of motion increments, nonlinear suppression is applied to high-frequency jumps. Dynamic control mechanism for fusion vector projection and dual threshold decoupling Dynamic trend-aware reward items based on vector projection mechanism Real-time calculation of the motion trend vector generated by the current action command of the robotic arm Normal vector of the most dangerous obstacle dot product projection between ,according to The symbol determines the intention of movement: when When it is determined to be a risk of kinetic collision and a proportional penalty is applied; when If the direction is determined to be safe and the vehicle passes or moves away from the target, no penalty will be triggered. During training, a dual-threshold control strategy is employed, decoupling the task space and safety space, by setting non-overlapping safety intervention thresholds. With the task success threshold and satisfy The logical constraints are such that when the robotic arm's end effector enters the task success threshold range, the obstacle avoidance repulsion force weight of the end joint is dynamically reduced and a high task reward is activated. .

6. The adaptive control method for a super-redundant robotic arm based on dual-loop evolutionary reinforcement learning according to claim 1, characterized in that, In step S4, the fitness index is calculated specifically by: based on the average cumulative reward value during the training process. and task success rate Calculate fitness: 。 7. The adaptive control method for a super-redundant robotic arm based on dual-loop evolutionary reinforcement learning according to claim 1, characterized in that, In step S5, dynamic optimization specifically involves training the feature matching model to obtain the total reward function. The weight parameters in the equation are encoded as position vectors of evolving particles, and the first... The position vector of each particle for: In this context, each dimension of the position vector corresponds to a weight coefficient for the target guidance reward, physical constraint penalty, static collision penalty, motion smoothing penalty, and dynamic trend obstacle avoidance reward; the particle's search space is restricted to a preset physical reasonable range. ; During the outer ring evolution process, each particle represents a specific reward evaluation system, and the weight vector is iteratively optimized according to the following velocity and position update formula: in, This indicates the increment step size for weight updates. and These represent the individual optimal weight configuration and the global optimal weight configuration, respectively. The acceleration constant, For random operators; A cosine annealing strategy was introduced during the evolutionary iteration to dynamically adjust the inertia weight. Its mathematical expression is: in, For the current evolutionary generation, This represents the total number of iterations.

8. The adaptive control method for a super-redundant robotic arm based on dual-loop evolutionary reinforcement learning according to claim 1, characterized in that, Also includes: During the training process in step S4, a time-series success rate sequence is maintained using the sliding window technique. Calculate the weighted average success rate within the current window. ,in This is the time decay factor; When weighted average success rate Continuous The course advancement threshold is consistently exceeded within each cycle. At that time, it is determined that the super-redundant robotic arm has learned the obstacle avoidance and positioning paradigm under the current constraints; At this point, the scenario evolution command is executed, which dynamically adjusts the topology or physical boundary parameters of the colliders in the simulation environment, raising the task difficulty to the next level.

9. The adaptive control method for a super-redundant robotic arm based on dual-loop evolutionary reinforcement learning according to claim 1, characterized in that, Also includes: Once the freeze command is activated, the evolution freeze logic is immediately initiated: within the outer ring evolution layer, the position vectors of the entire current particle swarm are forcibly locked. This causes the reward weight configuration to enter a static hold state; During the freeze period, the behavior reshaping phase begins; at this time, the inner-loop behavior execution layer, guided by a fixed reward evaluation system, concentrates computing resources on the new scenario. Multimodal observation data is reconstructed to correct policy biases caused by changes in environmental width or geometric features; fitness volatility is defined. Only when the performance of the policy network in the new environment tends to stabilize, and satisfies continuously below the preset smoothing threshold Only then can the evolution freeze state be lifted; After the freeze is lifted, the outer loop resumes the parameter optimization logic based on cosine annealing weights, and uses the stable feedback of the reshaped inner loop to perform fine-grained iteration of the weights.

10. An adaptive control system for a super-redundant robotic arm based on dual-loop evolutionary reinforcement learning, characterized in that, include: Acquisition module: Acquires data from the robotic arm; Construction modules: Construct the outer ring evolution evaluation layer and the inner ring behavior execution layer; Candidate weight generation module: The outer ring evolution layer generates multiple candidate weight vectors for the reward function; Training module: The inner loop behavior execution layer is trained based on multiple candidate weight vectors and robotic arm data, and generates multiple fitness metrics; Iterative module: Retain the candidate weight vector corresponding to the highest fitness index, and dynamically optimize it to generate multiple candidate weight vectors again. Return to step S4 and continue until training is complete to obtain the optimal weight ratio. Control module: Implements adaptive control of the robotic arm based on the optimal weight allocation.

Images