Mechanical arm safety control method and system

Abstract

The application relates to a mechanical arm safety control method and system, comprising: multi-modal environment perception and data fusion, human-computer interaction behavior prediction and potential collision prediction; energy boundary control mechanism; dynamic safety barrier based on risk entropy and self-adaptive adjustment; hierarchical safety response mechanism. The application can significantly improve the operation safety of the mechanical arm and essentially avoid collision damage risks.

Description

Technical Field

[0001] This invention relates to the field of robotic arm technology, and specifically to a robotic arm safety control method and system. Background Technology

[0002] With the rapid development of embodied intelligence technology, robots are increasingly widely used in human-robot collaboration scenarios in industrial manufacturing, logistics and warehousing, and other fields. Their operational safety is directly related to the personal safety of operators and production efficiency, and has become a core bottleneck restricting the large-scale implementation of embodied intelligence technology. Currently, existing robot safety control technologies still have many shortcomings, making it difficult to meet the safety protection and production continuity requirements in complex human-robot collaboration scenarios. The core problems are mainly reflected in two aspects. First, the perception dimension is too singular, resulting in insufficient reliability of collision detection. Most robot safety control solutions rely on only a single perception method for collision detection, either relying on current loop detection, which has low sensitivity and is easily affected by factors such as friction, making it prone to missing minor collisions and failing to detect potential safety hazards in a timely manner; or relying on single visual monitoring, but visual systems have obvious detection blind spots and are greatly affected by environmental factors such as occlusion and changes in lighting, resulting in poor monitoring stability and difficulty in comprehensively and accurately capturing the surrounding environment and personnel status. Secondly, the control strategy is rigid and cannot balance safety and production continuity. Traditional robot safety control generally adopts a single strategy of "collision-based emergency stop". Although this strategy can quickly terminate the robot's operation to ensure personnel safety when a collision occurs, it will directly interrupt the production process, seriously disrupt production continuity, and increase production costs. Especially in human-robot collaboration scenarios, this strategy cannot effectively distinguish between intentional contact and unintentional collision. Normal manual pushing, pulling, teaching and other collaborative operations will also trigger an emergency stop, which will greatly affect the flexibility and efficiency of human-robot collaboration.

[0003] Therefore, it is necessary to develop a new method and system for the safety control of robotic arms. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for controlling the safety of robotic arms, which can significantly improve the operational safety of robotic arms and fundamentally avoid the risk of collision damage.

[0005] In a first aspect, the present invention provides a robotic arm safety control method comprising the following steps:

[0006] Step 1: Multimodal environment perception and data fusion: Collect relevant data on the human body, robotic arm and environment in the work area through the visual perception layer and the tactile / force perception layer, and construct a dynamic semantic map through fusion processing;

[0007] Step 2: Human-computer interaction behavior prediction and potential collision prediction: Based on the time series model, combined with the human body and robotic arm related data in the dynamic semantic map output in Step 1, predict the movement intention of the human body and robotic arm, and estimate the time of potential collision and the part of the human body that comes into contact with the collision or the human body skeletal points.

[0008] Step 3: Energy Boundary Control Mechanism: Define the system's real-time total energy, combine the prediction results of potential collision contact points output in Step 2, set and dynamically adjust the safe energy limit according to the sensitivity of the contact points and the operating mode;

[0009] Step 4: Dynamic safety barrier and adaptive adjustment based on risk entropy: Combining the dynamic semantic map output from Step 1, the collision prediction results output from Step 2, and the safety energy limit output from Step 3, a dynamic virtual safety barrier is generated in real time, and the impedance control parameters of the robotic arm are adjusted in real time through a fuzzy controller to achieve flexible adaptive adjustment.

[0010] Step 5: Layered safety response mechanism: Establish a multi-level redundant safety response library. Based on the dynamic virtual safety barrier location output in Step 4, the system energy state output in Step 3, and the collision prediction results output in Step 2, execute the corresponding level of safety response and output safety response instructions.

[0011] In the above technical solution, through the coordinated operation of steps 1 to 5, the technical goals of active protection, accurate prediction, flexible control, and hierarchical response in human-machine collaboration scenarios are achieved. By constructing a dynamic semantic map through multimodal environmental perception and data fusion, it solves the problems of blind spots and insufficient sensitivity of single perception methods, providing a reliable data foundation for subsequent control. Combined with a time-series model, it achieves accurate prediction of motion intention and collision information, making up for the defects of traditional passive protection. Through the energy boundary control mechanism, it limits collision energy from a physical perspective. By using dynamic safety barriers and adaptive adjustment, it achieves a balance between safety and efficiency. Through a hierarchical safety response mechanism, it avoids the disruption of production continuity caused by over-protection. The whole system forms a closed-loop control, solving the core technical pain points of existing robotic arm safety control and achieving a balance between safety and collaborative efficiency.

[0012] Optionally, it also includes:

[0013] Control of heterogeneous computing devices based on safety redundancy: A dual-track safety processor is used to perform redundant calculations on the relevant data from steps 1 to 5 and the safety response instructions output from step 5. Through synchronous locking, independent decision and arbitration logic, the safety response instructions are executed to achieve safe and reliable control of the robotic arm.

[0014] In the above technical solution, a control step for heterogeneous computing devices based on security redundancy is added. Through the redundancy design of dual-track security processors, synchronous lockstep, independent decision-making, and arbitration logic, the reliability of system operation and the compliance of engineering implementation are improved, avoiding security risks caused by the failure of a single computing device, and forming a dual security guarantee.

[0015] Optionally, in step 1, the visual perception layer uses a depth camera and visual recognition algorithm to collect and identify three-dimensional map data of human skeleton points, robotic arm joints, end tools, and workpieces in the work area in real time. The human skeleton points include at least the torso, head, and hands. The tactile / force perception layer uses a robotic arm joint torque sensor and a flexible capacitive or piezoresistive electronic skin covering the robotic arm housing to collect the spatial distribution, magnitude, and direction of external contact forces. The fusion processing involves aligning the human position information in the visual space with the tactile information of the electronic skin in time and space to construct a dynamic semantic map containing the human body, robotic arm, and environment.

[0016] In the above technical solution, the multimodal perception in step 1 is refined. By clarifying the specific composition and fusion processing method of the visual perception layer and the tactile / force perception layer, the accuracy, comprehensiveness and reliability of perception are further improved, providing better data support for subsequent steps, ensuring the accuracy of human-machine relative position capture and contact force detection, and improving the system's safety control precision.

[0017] Optionally, in step 2, the temporal model is a Long Short-Term Memory Network or a Transformer temporal model. Based on the human body and robotic arm related data in the dynamic semantic map output in step 1, combined with the human body's motion trajectory parameters and the current motion state of the robotic arm, the movement intention of the human body and robotic arm within the next 200ms-500ms is predicted. The movement intention includes intentional collaborative interaction behavior, accidental fall of collaborators, unexpected collision behavior, and the motion state and path under each behavior state. The human body's motion trajectory parameters include speed, acceleration, and direction. Based on the current motion state of the robotic arm, path planning, and the prediction results of the human body's movement intention, the potential collision time and the part of the human body that comes into contact with the collision or the human body's skeletal points are estimated.

[0018] In the above technical solution, the temporal model in step 2 is refined to clarify the temporal model type, prediction time window, specific type of motion intent, and collision prediction content. By leveraging the temporal feature capture capabilities of long short-term memory networks or Transformer temporal models, the accuracy and timeliness of motion intent prediction and collision prediction are improved, and intentional interaction and unintentional collision are accurately distinguished, providing a more reliable basis for subsequent safety control and reducing the risk of collision damage.

[0019] Optionally, in step 3, the system's real-time total energy... Defined as:

[0020]

[0021] in, For the mass of each link, For the velocity of the center of mass, For rotational inertia, Angular velocity, The height difference between the center of mass and the point of contact with the human body is given by g, where g is the acceleration due to gravity. Based on the predicted potential collision contact points output in step 2, a safe energy upper limit is set. It is dynamically adjusted based on the sensitivity of the contact area and the operating mode. If the potential contact area is predicted to be the head or torso, Even at the lowest setting, each of the remaining parts of the human body has its own safe energy limit.

[0022] The operating modes include a precision assembly mode and a handling mode; the precision assembly mode corresponds to a lower... The transport mode corresponds to a higher And the corresponding transport mode It does not exceed the energy limit that each part of the human body can withstand.

[0023] In the above technical solution, the energy boundary control mechanism in step 3 is refined. By clarifying the real-time total energy calculation formula of the system, and combining the sensitivity of the collision contact part and the operating mode to dynamically adjust the upper limit of the safe energy, the energy boundary is accurately and dynamically controlled. This provides differentiated protection for vulnerable parts of the human body and takes into account the adaptation needs of different working conditions, thereby improving safety from a physical perspective and ensuring the efficiency of collaboration under different working conditions.

[0024] Optionally, in step 4, a dynamic virtual safety barrier is generated in real time by combining the dynamic semantic map output in step 1, the collision prediction results output in step 2, and the safety energy limit output in step 3.

[0025] The virtual safety barrier dynamically contracts and expands based on the sensitivity of the human joints, the current load on the robotic arm, and the relative motion state between the robotic arm and the human body. The human joint sensitivity ensures that the barrier radius at the head is greater than that at the hands, and the robotic arm load ensures that the barrier radius is greater when handling heavy objects than when not handling them. Simultaneously, the current motion state of the robotic arm and the system's real-time total energy are considered. The fuzzy controller is constructed using the human-machine distance d, the predicted collision time, and the parameters of the safety virtual barrier as inputs. The damping D and stiffness K of the robotic arm are adjusted in real time by the fuzzy controller. When the robotic arm is outside the safety barrier, the stiffness K is in a large state and the damping D is in a moderate state. When the robotic arm approaches the safety barrier or the predicted intention of the human to actively approach causes the safety barrier to approach the robotic arm, the stiffness K is adjusted to approach 0 and the damping D is increased based on the predicted collision time, so that the robotic arm is in a flexible state.

[0026] In the above technical solution, the dynamic safety barrier and adaptive adjustment in step 4 are refined, the adaptation factors of the dynamic virtual safety barrier and the input parameters of the fuzzy controller are clarified, the dynamic adaptation of the safety barrier and the precise online tuning of the robot arm impedance parameters are realized, so that the robot arm can maintain flexibility within the safe range, avoid collision risks in advance, reduce the downtime rate of intentional interaction scenarios, and further improve the system's safety, collaborative flexibility and efficiency.

[0027] Optionally, the multi-level redundant safety response library in step 5 includes four levels of response, which, combined with the dynamic virtual safety barrier location output in step 4, the real-time total system energy output in step 3, and the collision prediction results output in step 2, executes the corresponding response:

[0028] When the robotic arm approaches the safety barrier area but does not enter, the operating speed is reduced linearly or exponentially based on the distance between the robot and the human, thus reducing the system's total energy consumption in real time. Reduced to the safe energy limit Within the range;

[0029] When the robotic arm is near the safety barrier and ≤ When a collision or slight contact is predicted, the system will actively avoid the collision by moving along the tangential direction of the human body's movement, while controlling the contact force at the point of contact to be less than 10N.

[0030] When the robotic arm enters the safety barrier and > When this happens, a rapid shutdown action is performed and an alarm is triggered.

[0031] When it is detected that contact or collision has occurred and > When the machine stops, the brake is triggered for emergency stop, and the residual stress is released after the machine stops by using the reverse electromotive force of the servo motor or the built-in spring buffer device.

[0032] In the above technical solution, the hierarchical safety response mechanism in step 5 is refined, and the triggering conditions and specific actions of the four-level response are clarified to achieve the precision and hierarchical nature of the safety response. Through the hierarchical actions of early warning speed reduction, compliant avoidance, safe shutdown, emergency stop with brake, and residual stress release, the interference with the production process is minimized while ensuring safety, and the continuity and safety of human-machine collaboration are improved.

[0033] Optionally, in step 6, the dual-track safety processor includes track A and track B, where track A is a high-performance main controller that runs perception data fusion, intent prediction network, dynamic safety barrier and impedance control, and trajectory planning.

[0034] Track B is a highly reliable safety processor that runs a hardware-based safety response mechanism. It independently reads data from the joint encoder and torque sensor and directly determines whether a collision has occurred. If a collision occurs, it directly triggers a level four response.

[0035] The dual-track redundancy calculation adopts synchronous lock-step, independent decision and arbitration logic to perform redundancy calculation on the relevant data of steps 1 to 5 and the safety response command output in step 5. If neither triggers the level 4 response, the track A command is executed. If they are inconsistent, the track B command is executed first and an alarm is triggered to achieve safe and reliable control of the robotic arm.

[0036] In the above technical solution, the dual-track safety processor of claim 2 is further refined, clarifying the functional division of track A and track B, the independent safety response of track B, and the dual-track redundant calculation logic, thereby further improving the reliability of redundant calculation and the timeliness of safety response, strengthening the system's compliance and engineering adaptability, covering a variety of abnormal scenarios, and providing a more solid guarantee for the engineering implementation of the technical solution.

[0037] Secondly, the robotic arm safety control system of the present invention includes a multimodal perception module, a behavior and collision prediction module, an energy boundary control module, a dynamic safety barrier and adaptive adjustment module, and a hierarchical safety response module.

[0038] The multimodal perception module is used to collect and fuse data related to the human body, robotic arm and environment in the work area, and output a dynamic semantic map.

[0039] The behavior and collision prediction module is used to receive the dynamic semantic map, predict the movement intentions of the human body and the robotic arm and the time of potential collisions, the contact points of the human body or the human skeleton points, and output the collision prediction results.

[0040] The energy boundary control module is used to receive the collision prediction results, define the real-time total energy of the system, and dynamically adjust the safe energy limit based on the sensitivity of the contact part and the operating mode, and output the safe energy limit.

[0041] The dynamic safety barrier and adaptive adjustment module is used to receive the dynamic semantic map, collision prediction results and safety energy limit, generate a dynamic virtual safety barrier in real time and adjust the impedance control parameters of the robotic arm in real time through a fuzzy controller to achieve flexible adaptive adjustment.

[0042] The layered safety response module is used to receive the location of the dynamic virtual safety barrier, the system energy state, and the collision prediction results, execute the corresponding level of safety response, and output the safety response command.

[0043] Optionally, it also includes a heterogeneous redundant computing module, which employs a dual-track safety processor to perform redundant calculations on the relevant data of the multimodal perception module, behavior and collision prediction module, energy boundary control module, dynamic safety barrier and adaptive adjustment module, and hierarchical safety response module, as well as the safety response commands output by the hierarchical safety response module. The safety response commands are executed through synchronous lockstep, independent decision and arbitration logic to achieve safe and reliable control of the robotic arm.

[0044] Compared with the prior art, the robotic arm safety control method and system provided by the present invention have the following beneficial effects:

[0045] 1. Significantly improves the operational safety of robotic arms, fundamentally avoiding the risk of collision injuries. This invention constructs a dynamic semantic map through multimodal environmental perception and data fusion, combined with a human-computer interaction behavior prediction network to accurately predict motion intentions and potential collisions. Coupled with a dynamic safety virtual barrier and energy boundary control mechanism, it transforms the passive protection mode of post-event collision force limitation in existing technologies into an active protection mode of pre-event collision energy control. Even in extreme cases where the motion control algorithm fails, it can still ensure that the collision energy of the robotic arm remains below the safe threshold that the human body can withstand, fundamentally guaranteeing personnel safety during human-machine collaboration and solving the safety hazards caused by missed collision detection and delayed protection in existing technologies.

[0046] 2. Enhance the continuity of human-computer interaction and significantly improve collaboration efficiency. This invention, through multimodal perception data fusion technology and intention prediction network, can accurately distinguish between intentional human-computer interaction and unintentional collisions. Under the premise of strictly adhering to the safety energy boundary, it effectively reduces the probability of robot arm shutdown in intentional interaction scenarios, avoids the disruption of production process continuity caused by traditional collision-based emergency stop strategies, significantly improves the flexibility and efficiency of human-computer collaboration, and solves the technical pain points of rigid control strategies and low human-computer collaboration efficiency in existing technologies.

[0047] 3. Highly adaptable and possessing a high degree of platform versatility, this invention can be widely adapted to various robot models. Based on a configurable energy boundary management mechanism and personalized parameter profiles, this invention can seamlessly adapt to various robot models, from light collaborative robots to heavy-duty industrial robots (such as 20kg payloads), without requiring significant modifications to the core architecture. It adapts to the usage requirements of different loads and working conditions, reduces the adaptation cost of technical solutions, expands its application scope, and solves the problems of poor adaptability and difficulty in accommodating multiple robot models in existing technologies.

[0048] 4. Enhance system reliability and lay a compliant foundation for engineering implementation. This invention adopts a safe and redundant heterogeneous computing architecture. Through the synchronous locking, independent decision-making, and arbitration logic of dual-track safety processors, it can effectively cover abnormal scenarios such as sensor data fusion failure, identification deviation, and inaccurate prediction model calculation results. This ensures that the safety of human-machine collaboration can still be maintained under various abnormal conditions, meeting the compliance requirements of industrial fields for the safety control of robotic arms. It provides a reliable guarantee for the engineering implementation of the technical solution and solves the defects of insufficient reliability and inability to meet the compliance requirements of engineering applications in existing technologies. Attached Figure Description

[0049] Figure 1 This is a flowchart of the robotic arm safety control method in the embodiments of this application;

[0050] Figure 2 This is a schematic diagram of the robotic arm safety control system in the embodiments of this application;

[0051] Figure 3 This is a schematic diagram of the hardware components involved in the robotic arm safety control system in the embodiments of this application. Detailed Implementation

[0052] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0053] like Figure 1 As shown, a safety control method for a robotic arm includes the following steps:

[0054] Step 1: Multimodal environment perception and data fusion: Collect relevant data on the human body, robotic arm and environment in the work area through the visual perception layer and the tactile / force perception layer, and construct a dynamic semantic map through fusion processing.

[0055] Step 2: Human-computer interaction behavior prediction and potential collision prediction: Based on the time series model, combined with the human body and robotic arm related data in the dynamic semantic map output in Step 1, predict the movement intention of the human body and robotic arm, and estimate the time of potential collision and the part of the human body that comes into contact with the collision or the human body skeletal points.

[0056] Step 3: Energy boundary control mechanism: Define the system's real-time total energy, combine the prediction results of potential collision contact points output in Step 2, set and dynamically adjust the safe energy limit according to the sensitivity of the contact points and the operating mode.

[0057] Step 4: Dynamic safety barrier and adaptive adjustment based on risk entropy: Combining the dynamic semantic map output from Step 1, the collision prediction results output from Step 2, and the safety energy limit output from Step 3, a dynamic virtual safety barrier is generated in real time, and the impedance control parameters of the robotic arm are adjusted in real time through a fuzzy controller to achieve flexible adaptive adjustment.

[0058] Step 5: Layered safety response mechanism: Establish a multi-level redundant safety response library. Based on the dynamic virtual safety barrier location output in Step 4, the system energy state output in Step 3, and the collision prediction results output in Step 2, execute the corresponding level of safety response and output safety response instructions.

[0059] In one possible embodiment, a robotic arm safety control method further includes:

[0060] Heterogeneous computing device control based on safety redundancy: A dual-track safety processor is used to perform redundant calculations on the relevant data from steps 1 to 5 and the safety response command output from step 5. Through synchronous locking, independent decision and arbitration logic (outputting the arbitrated control command), it is sent to each joint driver and joint encoder of the robotic arm to execute the safety response command and realize the safe and reliable control of the robotic arm.

[0061] In one possible embodiment, in step 1, multimodal environment perception and data fusion are performed:

[0062] Visual perception layer: Employs depth cameras and visual recognition algorithms to collect and identify 3D map data of human skeleton points, robotic arm joints, end tools, and workpieces in the work area in real time. Human skeleton points include at least the torso, head, and hands.

[0063] Tactile / force sensing layer: Employs a robotic arm joint torque sensor and a flexible capacitive or piezoresistive electronic skin covering the robotic arm housing to collect the spatial distribution, magnitude, and direction of external contact forces.

[0064] Fusion processing: The human body position information in the visual space is spatiotemporally aligned with the tactile information of the electronic skin to construct a dynamic semantic map that includes the human body, the robotic arm, and the environment.

[0065] In one possible embodiment, step 2 involves human-computer interaction behavior prediction and potential collision prediction:

[0066] The temporal model employs a Long Short-Term Memory (LSTM) network or a Transformer temporal model. Based on the human body and robotic arm-related data in the dynamic semantic map output from step 1, and combining the human body's motion trajectory parameters and the robotic arm's current motion state, it predicts the motion intentions of the human body and robotic arm within the next 200ms-500ms. These motion intentions include intentional collaborative interaction, accidental falls by collaborators, unexpected collisions, and the motion state and path for each behavior. Human body motion trajectory parameters include velocity, acceleration, and direction. Based on the robotic arm's current motion state, path planning, and the predicted human body motion intentions, the model estimates the potential collision time and the point of contact or skeletal location for the human body.

[0067] In this embodiment, the temporal model is used to predict the movement intention within the next 200ms-500ms based on the relevant data (mainly motion data) of the human body and the robotic arm extracted from the dynamic semantic map, and directly output the collision prediction result. The temporal model uses a sequence-to-sequence spatiotemporal Transformer network as its core framework, but it can also be replaced by variants such as a two-stream long short-term memory network or a graph spatiotemporal network.

[0068] 1. Input data and preprocessing of time series models

[0069] The input data for the time series model comes directly from the dynamic semantic map output in step 1. From this dynamic semantic map, data is extracted within a fixed time window T. win Sliding sampling is performed at a time interval of 1.0s and a sampling frequency of f=50Hz to form a historical frame sequence S. in =[F t-Twin ,…,F t ], F t Let t represent the t-th historical frame, where each frame contains the following structured feature vector:

[0070] Human motion trajectory parameter sequence H i It contains the 3D coordinates (x, y, z) of 15 predefined skeletal keypoints (such as head, neck, shoulders, elbows, wrists, hips, knees, and ankles), as well as the calculated resultant velocity v, resultant acceleration a, and motion direction unit vector d for each point. That is, each human body frame is a 15 × 7 = 105-dimensional vector.

[0071] robotic arm state parameter sequence R i : The current angle θ of the six joints of the robotic arm 1-6 Joint angular velocity ω 1-6The three-dimensional pose (x', y', z', roll, pitch, yaw) and end-effector velocity v of the end effector ee This forms a 43-dimensional vector. Here, x' is the X-axis position coordinate of the end effector in the world coordinate system, y' is the Y-axis position coordinate of the end effector in the world coordinate system, z' is the Z-axis position coordinate of the end effector in the world coordinate system, roll is the roll angle of the end effector around its own X-axis, pitch is the pitch angle of the end effector around its own Y-axis, and yaw is the yaw angle of the end effector around its own Z-axis.

[0072] Interaction context set C: Static semantics resolved by the dynamic semantic map, including the minimum Euclidean distance d between the human body and the robotic arm's end effector. min The confidence score of marking a human body as "intentional contact" (such as actively reaching out). int Represented by a 2D vector. S within the time window in Constructed as a dimension (T) win The input tensor X, which is 50×150, is arranged by time step (105+43+2)×f)×(105+43+2).

[0073] 2. Model structure, hierarchy, and connections

[0074] The temporal model employs a multi-task Transformer architecture specifically designed for human-machine collaborative prediction. It consists of an input embedding layer, a spatiotemporal Transformer encoder, an intent-aware decoder, and a multi-task prediction head, all cascaded together. The input embedding layer maps the input tensor X to an embedding space of model dimension dmodel=256 via a fully connected linear projection layer, outputting a feature matrix of dimension 50×256. ,in, Represents a 50-row, 256-column two-dimensional matrix space with all elements being real numbers, superimposed with learnable sinusoidal position codes. This achieves the encoding and preservation of temporal positional information in the sequence. The spatiotemporal Transformer encoder consists of four stacked encoder layers with identical structures. Each encoder layer includes a multi-head self-attention module and a position feedforward network. The multi-head self-attention module uses eight attention heads to capture long-range dependencies between 50 time steps, modeling the temporal evolution relationship between human and robotic arm motions. The position feedforward network employs a two-layer linear transformation structure with the GELU activation function. The feature dimension is first increased to 1024 and then reduced back to 256. Each sub-module is configured with residual connections and layer normalization operations. The encoder finally outputs temporal context features with a dimension of 50×256. .

[0075] Intent-aware decoder: To simultaneously predict motion intent and collision information, the intent-aware decoder employs a combination of autoregression and cross-attention. The planned future path points of the robotic arm are used as the main query input to the decoder. Specifically, a preset sequence of future 200ms-500ms path points for the robotic arm (interpolated to 10 steps) is taken and mapped to a 10×256 query sequence Q through an independent Multi-Layer Perceptron (MLP). plan The intent-aware decoder also consists of four layers. Each layer performs masked multi-head self-attention (only looking at the previous position) + residual & layer normalization, with Q... plan For Query, Z enc The network employs cross-attention for Key and Value, along with residuals and layer normalization, and a location-fedforward network with residuals and layer normalization. The final output, denoted as Z, fuses historical motion features with the planned path, providing the context for each future prediction step. dec . This represents a two-dimensional matrix space with 10 rows and 256 columns, where all elements are real numbers.

[0076] Multi-task prediction head: in Z dec At each decoding time step, three sets of parallel prediction heads are attached, and the final output is generated by taking the last decoding step or by weighted summation, forming an end-to-end multi-task learning structure.

[0077] ① Motion Intent Classification Head: Consists of a fully connected layer and Softmax, outputting four types of probabilities: normal collaboration, intentional interaction, accidental fall, and unexpected collision.

[0078] ② Human Motion State and Path Regression Head: Composed of two fully connected layers (256→128→90), it outputs a 3D coordinate prediction sequence of 15 skeletal points corresponding to the next 10 time steps (covering 200-500ms), i.e., a 10×15×3=450-dimensional vector. This vector is then reshaped (i.e., only the data format is reshaped, without modifying the data content) to obtain the future trajectory. This human motion state and path regression head also outputs the confidence score of the regressed behavioral state.

[0079] ③ Collision Prediction Head: Also a fully connected layer, it outputs two parallel sub-items. Sub-item one is Time-to-Collision (TCC) regression: outputting a scalar Δt∈[0,500]ms, representing the predicted time offset of a potential collision. Sub-item two is collision site classification: outputting a 15-dimensional vector, activated by Softmax, indicating the index of the human skeleton point most likely to make contact; simultaneously outputting a 6-dimensional vector indicating the robotic arm joints that may be impacted.

[0080] 3. Model training and inference

[0081] During model training, forward propagation calculates Z. dec The model outputs all prediction heads, and backpropagation updates all parameters. During inference, the input is a real-time sliding window sequence, which is directly computed in parallel through a single forward propagation. The outputs are the classification intent, the future human skeleton motion path, and the collision time and collision skeleton point, thereby predicting the potential collision time and the location of human contact. To ensure model convergence and obtain usable prediction accuracy, training is performed according to the following steps and parameters:

[0082] Constructing a training dataset: Multimodal data was collected on the robotic arm's operation under various motion states, including intentional collaboration (passing tools, pressing buttons), accidental falls (tripping, slipping), and unexpected collisions (sudden intrusion, stopping), under high / low speed, different loads, and different path conditions. The OptiTrack motion capture system and the robotic arm's joint encoder were used for synchronous recording, resulting in over 100,000 valid samples, which were divided into training, validation, and test sets in an 8:1:1 ratio. Each sequence sample must include: Input sequence S in (As mentioned earlier, a 1-second historical frame), tag data (the trajectory sequence of 15 real skeletal points within the next 200-500ms; collision event annotation, if there is a collision, the actual collision time Δt is recorded). gt (and collision bone points; motion intention category label).

[0083] Data augmentation and normalization: Random rotation, scaling (±10%), and translation are applied to the coordinates of skeletal points; Gaussian noise is added to the velocity and acceleration. All input features are Z-score normalized.

[0084] Loss function and optimizer configuration: A multi-task joint loss function is adopted, with total loss = 0.3 × classification loss + 0.4 × trajectory regression loss + 0.3 × collision prediction loss, where:

[0085] ① Classification loss: Labeled smoothing (coefficient 0.1) cross-entropy loss, which solves the class imbalance problem;

[0086] ②Trajectory regression loss: Smooth L1 loss to reduce the impact of outliers on training;

[0087] ③ Collision prediction loss: 0.6×TTC regression MSE loss + 0.4×bone point collision probability binary cross-entropy loss; the optimizer used is AdamW, with an initial learning rate of 1×10⁻⁶. −4 Weight decay 1×10 −4 The batch size is 64. Training is performed for 200 epochs, using cosine annealing scheduling to reduce the learning rate.

[0088] In one possible embodiment, in step 3, the system's real-time total energy... Defined as:

[0089]

[0090] in, For the mass of each link, For the velocity of the center of mass, For rotational inertia, Angular velocity, Let g be the height difference between the center of mass and the point of contact between the human body and g is the acceleration due to gravity.

[0091] Based on the predicted potential collision contact points output from step 2, a safety energy upper limit is set. It is dynamically adjusted based on the sensitivity of the contact area and the operating mode. If the potential contact area is predicted to be the head or torso, When the energy level is lowered to the lowest setting, each of the other parts of the human body has its own safe energy limit.

[0092] The operating modes include precision assembly mode and material handling mode; precision assembly mode corresponds to lower costs. The transport mode corresponds to a higher And the corresponding transport mode It does not exceed the energy limit that each part of the human body can withstand.

[0093] In one possible embodiment, step 4 involves dynamic security barriers and adaptive adjustment based on risk entropy:

[0094] By combining the dynamic semantic map output in step 1, the collision prediction results output in step 2, and the safety energy limit output in step 3, a dynamic virtual safety barrier is generated in real time.

[0095] The virtual safety barrier dynamically contracts and expands based on the sensitivity of the human joints, the current load on the robotic arm, and the relative motion between the robotic arm and the human body. The human joint sensitivity ensures that the barrier radius at the head is greater than that at the hands, and the robotic arm load ensures that the barrier radius is greater when handling heavy objects than when not handling them. Simultaneously, the barrier's motion state and the system's real-time total energy are considered. The fuzzy controller is constructed using the human-machine distance d, the predicted collision time, and the safety virtual barrier parameters as inputs. The damping D and stiffness K of the robotic arm are adjusted in real time by the fuzzy controller.

[0096] When the robotic arm is outside the safety barrier, the stiffness K is at a high level (high stiffness, trajectory accuracy and execution efficiency are the top priorities), and the damping D is at a moderate level.

[0097] When the robotic arm approaches the safety barrier, or when a human's intention to approach is predicted, causing the safety barrier to approach the robotic arm, the stiffness K is adjusted to approach 0 (flexible state) based on the predicted collision time, and the damping D is increased to prevent impact, thus placing the robotic arm in a flexible state. Even if a collision occurs, no harm will be caused, or a human-robot collaborative state will be entered. This ensures that the robotic arm enters a flexible state when entering the safety barrier area or when a human actively approaches the robotic arm.

[0098] In one possible embodiment, the multi-level redundant safety response library in step 5 includes four levels of response, which, combined with the dynamic virtual safety barrier location output in step 4, the real-time total system energy output in step 3, and the collision prediction results output in step 2, executes the corresponding response:

[0099] Level 1 Response (Early Warning and Speed ​​Reduction): When the robotic arm approaches the safety barrier area but does not enter, the operating speed is reduced linearly or exponentially based on the distance between the robot and the human (e.g., from 100% to 30%), ensuring the system's total energy is reduced in real time. Reduced to the safe energy limit Within safe limits. If the energy itself is already within safe limits, no additional safety control actions are required.

[0100] Level 2 Response (Compliant Avoidance): When the robotic arm is near a safety barrier and ≤ When a collision or slight contact is predicted, the system actively avoids the collision by moving along the tangential direction of the human body (based on the artificial potential field method), while maintaining an extremely low contact force at the point of contact (e.g., < 10N).

[0101] Level 3 Response (Safe Shutdown): When the robotic arm enters the safety barrier and... > When this happens, a rapid shutdown action is executed and an alarm is triggered.

[0102] Level 4 Response (Emergency Stop and Reverse Drive): When contact or collision is detected and > When the machine stops, the brake is triggered for emergency stop, and the residual stress is released after the machine stops by using the reverse electromotive force of the servo motor or the built-in spring buffer device to prevent secondary pulling damage caused by inertia at the moment of stopping.

[0103] In one possible embodiment, in step 6, the heterogeneous computing device is based on security redundancy:

[0104] The dual-track safety processor consists of Track A and Track B. Track A is a high-performance main controller, and Track B is a high-reliability safety processor.

[0105] Track A: This track integrates sensing data fusion, intending to predict networks, dynamic safety barriers and impedance control, trajectory planning, etc., and is deployed on high-performance computing platforms such as NVIDIA Jetson Orin or Intel Xeon D. It outputs the desired S5 safety response mechanism and control commands to the joints (such as desired position, torque, speed, etc.).

[0106] Track B: Based on FPGA or safety-grade MCU (such as Infineon AURIX), it runs a hardware-based "safety response mechanism". The processor independently reads the joint encoder and torque sensor data to directly determine whether a collision has occurred. If a collision occurs, it directly triggers the S5's four-level response mechanism.

[0107] The dual-track redundancy calculation adopts synchronous lock-step, independent decision and arbitration logic to perform redundancy calculation on the relevant data of steps 1 to 5 and the safety response command output in step 5. If neither triggers the level 4 response, the track A command is executed. If they are inconsistent, the track B command is executed first and an alarm is triggered to achieve safe and reliable control of the robotic arm.

[0108] like Figure 2 As shown in the embodiments of this application, a robotic arm safety control system includes a multimodal perception module, a behavior and collision prediction module, an energy boundary control module, a dynamic safety barrier and adaptive adjustment module, and a hierarchical safety response module. The multimodal perception module collects and fuses data related to the human body, robotic arm, and environment within the work area, outputting a dynamic semantic map. The behavior and collision prediction module receives the dynamic semantic map, predicts the movement intentions of the human body and robotic arm, the timing of potential collisions, the point of contact between the human body and the collision site, or the location of the human skeleton, and outputs the collision prediction result. The energy boundary control module receives the collision prediction result, defines the real-time total energy of the system, and dynamically adjusts the upper limit of safe energy based on the sensitivity of the contact point and the operating mode, outputting the upper limit of safe energy. The dynamic safety barrier and adaptive adjustment module receives the dynamic semantic map, the collision prediction result, and the upper limit of safe energy, generates a dynamic virtual safety barrier in real time, and adjusts the impedance control parameters of the robotic arm in real time through a fuzzy controller to achieve flexible adaptive adjustment. The hierarchical safety response module receives the position of the dynamic virtual safety barrier, the system energy state, and the collision prediction result, executes the corresponding level of safety response, and outputs safety response commands.

[0109] In one possible embodiment, a robotic arm safety control system further includes a heterogeneous redundant computing module, employing a dual-track safety processor, for redundantly computing relevant data from the multimodal perception module, behavior and collision prediction module, energy boundary control module, dynamic safety barrier and adaptive adjustment module, and hierarchical safety response module, as well as safety response commands output by the hierarchical safety response module. The system executes safety response commands through synchronous lockstep, independent decision-making, and arbitration logic to achieve safe and reliable control of the robotic arm.

[0110] like Figure 3 The diagram shows the hardware components involved in the robotic arm safety control system, including a depth camera, electronic skin, torque sensor, high-performance main controller, safety processor, joint actuator, and joint encoder. The depth camera, electronic skin, and torque sensor are connected to the high-performance main controller, and are also connected to the safety processor. The high-performance main controller is connected to the safety processor, and both the high-performance main controller and the safety processor are connected to the joint actuator and the joint encoder, respectively.

[0111] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for safety control of a robotic arm, characterized in that, Includes the following steps: Step 1: Multimodal environment perception and data fusion: Collect relevant data on the human body, robotic arm and environment in the work area through the visual perception layer and the tactile / force perception layer, and construct a dynamic semantic map through fusion processing; Step 2: Human-computer interaction behavior prediction and potential collision prediction: Based on the time series model, combined with the human body and robotic arm related data in the dynamic semantic map output in Step 1, predict the movement intention of the human body and robotic arm, and estimate the time of potential collision and the part of the human body that comes into contact with the collision or the human body skeletal points. Step 3: Energy Boundary Control Mechanism: Define the system's real-time total energy, combine the prediction results of potential collision contact points output in Step 2, set and dynamically adjust the safe energy limit according to the sensitivity of the contact points and the operating mode; Step 4: Dynamic safety barrier and adaptive adjustment based on risk entropy: Combining the dynamic semantic map output from Step 1, the collision prediction results output from Step 2, and the safety energy limit output from Step 3, a dynamic virtual safety barrier is generated in real time, and the impedance control parameters of the robotic arm are adjusted in real time through a fuzzy controller to achieve flexible adaptive adjustment. Step 5: Layered safety response mechanism: Establish a multi-level redundant safety response library. Based on the dynamic virtual safety barrier location output in Step 4, the system energy state output in Step 3, and the collision prediction results output in Step 2, execute the corresponding level of safety response and output safety response instructions.

2. The robotic arm safety control method according to claim 1, characterized in that, Also includes: Control of heterogeneous computing devices based on safety redundancy: A dual-track safety processor is used to perform redundant calculations on the relevant data from steps 1 to 5 and the safety response instructions output from step 5. Through synchronous locking, independent decision and arbitration logic, the safety response instructions are executed to achieve safe and reliable control of the robotic arm.

3. The robotic arm safety control method according to claim 1, characterized in that, In step 1, the visual perception layer uses a depth camera and visual recognition algorithm to collect and identify 3D map data of human skeleton points, robotic arm joints, end tools, and workpieces in the work area in real time. The human skeleton points include at least the torso, head, and hands. The tactile / force perception layer uses a robotic arm joint torque sensor and a flexible capacitive or piezoresistive electronic skin covering the robotic arm shell to collect the spatial distribution, magnitude, and direction of external contact forces. The fusion processing involves aligning the human position information in the visual space with the tactile information of the electronic skin in time and space to construct a dynamic semantic map that includes the human body, robotic arm, and environment.

4. The robotic arm safety control method according to claim 1, characterized in that, In step 2, the temporal model is a Long Short-Term Memory Network or a Transformer temporal model. Based on the human body and robotic arm related data in the dynamic semantic map output in step 1, combined with the human body's motion trajectory parameters and the current motion state of the robotic arm, the movement intention of the human body and robotic arm within the next 200ms-500ms is predicted. The movement intention includes intentional collaborative interaction behavior, accidental fall of collaborators, unexpected collision behavior, and the motion state and path under each behavior state. The human body's motion trajectory parameters include speed, acceleration, and direction. Based on the current motion state of the robotic arm, path planning, and the prediction results of the human body's movement intention, the potential collision time and the part of the human body that comes into contact with the collision or the human body's skeletal points are estimated.

5. The robotic arm safety control method according to claim 1, characterized in that, In step 3, the system's real-time total energy Defined as: in, For the mass of each link, For the velocity of the center of mass, For rotational inertia, Angular velocity, The height difference between the center of mass and the point of contact with the human body is given by g, where g is the acceleration due to gravity. Based on the predicted potential collision contact points output in step 2, a safe energy upper limit is set. It is dynamically adjusted based on the sensitivity of the contact area and the operating mode. If the potential contact area is predicted to be the head or torso, Even at the lowest setting, each of the remaining parts of the human body has its own safe energy limit. The operating modes include a precision assembly mode and a handling mode; the precision assembly mode corresponds to a lower... The transport mode corresponds to a higher And the corresponding transport mode It does not exceed the energy limit that each part of the human body can withstand.

6. The robotic arm safety control method according to claim 1, characterized in that, In step 4, a dynamic virtual safety barrier is generated in real time by combining the dynamic semantic map output in step 1, the collision prediction results output in step 2, and the safety energy limit output in step 3. The virtual safety barrier dynamically contracts and expands based on the sensitivity of the human joints, the current load on the robotic arm, and the relative motion state between the robotic arm and the human body. The human joint sensitivity ensures that the barrier radius at the head is greater than that at the hands, and the robotic arm load ensures that the barrier radius is greater when handling heavy objects than when not handling them. Simultaneously, the current motion state of the robotic arm and the system's real-time total energy are considered. The fuzzy controller is constructed using the human-machine distance d, the predicted collision time, and the parameters of the safety virtual barrier as inputs. The damping D and stiffness K of the robotic arm are adjusted in real time by the fuzzy controller. When the robotic arm is outside the safety barrier, the stiffness K is in a large state and the damping D is in a moderate state. When the robotic arm approaches the safety barrier or the predicted intention of the human to actively approach causes the safety barrier to approach the robotic arm, the stiffness K is adjusted to approach 0 and the damping D is increased based on the predicted collision time, so that the robotic arm is in a flexible state.

7. The robotic arm safety control method according to claim 1, characterized in that, The multi-level redundant safety response library in step 5 includes four levels of response. Combining the dynamic virtual safety barrier location output in step 4, the real-time total system energy output in step 3, and the collision prediction results output in step 2, the corresponding response is executed: When the robotic arm approaches the safety barrier area but does not enter, the operating speed is reduced linearly or exponentially based on the distance between the robot and the human, thus reducing the system's total energy consumption in real time. Reduced to the safe energy limit Within the range; When the robotic arm is near the safety barrier and ≤ When a collision or slight contact is predicted, the system will actively avoid the collision by moving along the tangential direction of the human body's movement, while controlling the contact force at the point of contact to be less than 10N. When the robotic arm enters the safety barrier and > When this happens, a rapid shutdown action is performed and an alarm is triggered. When it is detected that contact or collision has occurred and > When the machine stops, the brake is triggered for emergency stop, and the residual stress is released after the machine stops by using the reverse electromotive force of the servo motor or the built-in spring buffer device.

8. The robotic arm safety control method according to claim 2, characterized in that, In step 6, the dual-track safety processor includes track A and track B. Track A is a high-performance main controller that runs perception data fusion, intent prediction network, dynamic safety barrier and impedance control, and trajectory planning. Track B is a highly reliable safety processor that runs a hardware-based safety response mechanism. It independently reads data from the joint encoder and torque sensor and directly determines whether a collision has occurred. If a collision occurs, it directly triggers a level four response. The dual-track redundancy calculation adopts synchronous lock-step, independent decision and arbitration logic to perform redundancy calculation on the relevant data of steps 1 to 5 and the safety response command output in step 5. If neither triggers the level 4 response, the track A command is executed. If they are inconsistent, the track B command is executed first and an alarm is triggered to achieve safe and reliable control of the robotic arm.

9. A safety control system for a robotic arm, characterized in that, It includes a multimodal perception module, a behavior and collision prediction module, an energy boundary control module, a dynamic safety barrier and adaptive adjustment module, and a hierarchical safety response module; The multimodal perception module is used to collect and fuse data related to the human body, robotic arm and environment in the work area, and output a dynamic semantic map. The behavior and collision prediction module is used to receive the dynamic semantic map, predict the movement intentions of the human body and the robotic arm and the time of potential collisions, the contact points of the human body or the human skeleton points, and output the collision prediction results. The energy boundary control module is used to receive the collision prediction results, define the real-time total energy of the system, and dynamically adjust the safe energy limit based on the sensitivity of the contact part and the operating mode, and output the safe energy limit. The dynamic safety barrier and adaptive adjustment module is used to receive the dynamic semantic map, collision prediction results and safety energy limit, generate a dynamic virtual safety barrier in real time and adjust the impedance control parameters of the robotic arm in real time through a fuzzy controller to achieve flexible adaptive adjustment. The layered safety response module is used to receive the location of the dynamic virtual safety barrier, the system energy state, and the collision prediction results, execute the corresponding level of safety response, and output the safety response command.

10. The robotic arm safety control system according to claim 9, characterized in that, It also includes a heterogeneous redundancy computing module, which uses a dual-track safety processor to perform redundant calculations on the relevant data of the multimodal perception module, behavior and collision prediction module, energy boundary control module, dynamic safety barrier and adaptive adjustment module, and hierarchical safety response module, as well as the safety response commands output by the hierarchical safety response module. The safety response commands are executed through synchronous lockstep, independent decision and arbitration logic to achieve safe and reliable control of the robotic arm.

Images