Mechanical arm working area safety protection method and system based on multi-mode sensing

Abstract

The invention discloses a mechanical arm working area safety protection method and system based on multi-modal sensing, relates to the technical field of mechanical arm control protection, and solves the problem that multi-modal feature fusion lacks dynamic adaptation and space-time correlation modeling capability. In order to solve the technical problems that a risk assessment system is incomplete and a scenarized dynamic adjustment mechanism is lacked, the key security feature expression is accurately enhanced through a dynamic weight distribution mechanism based on a real-time security risk level; a two-dimensional spatial-temporal characteristic incidence matrix is constructed by combining a dynamic time warping algorithm and unified coordinate system mapping, time sequence and spatial correlation information of multi-modal data is fully mined, the defect that characteristic fusion lacks dynamic adaptation and spatial-temporal modeling is overcome, the generated fusion characteristic vector can comprehensively and accurately reflect multi-dimensional safety information, and the safety performance of the system is improved. The index attribute difference and dimension influence are eliminated through attribute-based normalization calibration, and the accuracy of core index quantification is ensured.

Description

Technical Field

[0001] This invention relates to the field of robotic arm control and protection technology, specifically to a method and system for safety protection of the working area of ​​a robotic arm based on multimodal sensing. Background Technology

[0002] With the rapid development of industrial automation and intelligent manufacturing, robotic arms have been widely used in automobile manufacturing, electronic assembly, precision machining, and other fields. The human-machine collaboration scenarios in their work areas are increasing, making safety protection a core requirement for ensuring personnel safety and reliable equipment operation. Traditional robotic arm safety protection often adopts passive protection methods such as physical fencing isolation. While this can avoid direct collisions, it limits the flexibility of human-machine collaboration and cannot cope with sudden risks in dynamic working environments.

[0003] To overcome the limitations of passive protection, multimodal sensing technology is increasingly being applied to the safety protection of robotic arms. By collecting data from multiple sensor sources, it enables comprehensive perception of the working environment and equipment status. However, existing multimodal sensing safety protection technologies still have significant shortcomings: The weight allocation is fixed during feature fusion, which cannot be dynamically adjusted according to real-time safety risks and operational scenarios. Key safety features are not sufficiently strengthened, and the spatiotemporal correlation information of multimodal data is not fully explored. The safety orientation of the fused features is weak, and a collaborative evaluation mechanism for multi-dimensional indicators has not been established. The adaptability to different operational scenarios is poor, resulting in a high rate of misjudgment and missed judgment in risk level determination. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method and system for safety protection of the working area of ​​a robotic arm based on multimodal sensing. It solves the problems of lack of dynamic adaptation and spatiotemporal correlation modeling capabilities in multimodal feature fusion, imperfect risk assessment system, and lack of scenario-based dynamic adjustment mechanism.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a method for safety protection of the working area of ​​a robotic arm based on multimodal sensing, the method specifically including the following steps: Step 1: Collect multimodal data through multi-source sensors, and sequentially perform data cleaning, spatiotemporal alignment, format standardization and data completion processing to obtain preprocessed multimodal data; Step 2: Extract single-modal features from the preprocessed multimodal data, adjust the weight coefficients of each modal feature, construct a spatiotemporal feature correlation matrix, and perform element-wise fusion of each modal feature with the spatiotemporal feature correlation matrix to generate a fused feature vector. Step 3: Input the fused feature vector into the multi-dimensional risk assessment model, and output three core safety indicators: collision risk value, contact safety, and equipment stability. Step 4: Based on the real-time operation scenario of the robotic arm, dynamically allocate the weights of the three core indicators, perform attribute normalization calibration on the core indicators, calculate the comprehensive risk index and map it to different safety risk levels, and execute corresponding safety protection actions according to different levels.

[0006] As a further aspect of the present invention, the multi-source sensor includes a vision system, a force / torque sensor, a lidar, a tactile array / electronic skin, and an IMU inertial measurement unit, and the multi-source data includes visual data, force / torque data, lidar point cloud, and tactile / IMU data.

[0007] As a further aspect of the present invention, data cleaning includes: visual data is denoised by Gaussian filtering and median filtering, and invalid points in the depth map are removed by thresholding; force / torque data is smoothed by moving average filtering and abnormal peaks are removed by the 3σ criterion; LiDAR point cloud is removed by statistical filtering to remove isolated points and effective areas are extracted by pass-through filtering; tactile / IMU data is filtered by Kalman filtering to suppress drift and remove invalid data that exceeds the range. Spatiotemporal alignment includes: using high-frequency sensors as a reference, interpolating and completing low-frequency sensors, and calibrating timestamps for hysteresis sensors; establishing a unified coordinate system through hand-eye calibration, and mapping spatial data from different sensors to the unified coordinate system.

[0008] As a further aspect of the present invention, single-modal feature extraction includes: extracting texture / semantic features and spatial geometric features from visual data using CNN and PointNet respectively; extracting temporal and frequency domain features from force / torque data; extracting target contour features from lidar point clouds through clustering; extracting contact area and pressure distribution features from tactile data; and extracting attitude angle and vibration features from IMU data.

[0009] As a further aspect of the present invention, in the spatiotemporal feature correlation matrix: the temporal correlation submatrix is ​​M×N dimensional, which uses a dynamic time warping algorithm to align the visual target motion trajectory with the force / torque temporal data, where M is the number of temporal frames and N is the sum of the temporal feature dimensions of each modality; the spatial correlation submatrix is ​​P×Q dimensional, which maps the three-dimensional coordinates of the lidar obstacle and the two-dimensional tactile contact coordinates based on a unified coordinate system, where P is the number of spatial grids and Q is the sum of the spatial feature dimensions of each modality.

[0010] As a further aspect of the present invention, the method for outputting the three core safety indicators—collision risk value, contact safety, and equipment stability—is as follows: The collision risk value is calculated based on the dynamic collision time and the force / torque correction factor. The specific collision risk value = dynamic TTC normalized score × 0.6 + force / torque correction factor × 0.4. The contact safety is assessed based on the weighted evaluation of tactile and force / torque characteristics. The equipment stability is determined based on the fusion of visual, IMU and force / torque vibration characteristics.

[0011] As a further aspect of the present invention, the attribute-based normalization calibration includes: The collision risk value adopts a forward normalization formula, while the contact safety and equipment stability adopt a reverse normalization formula. The normalization interval is [0, 1]. The forward normalization formula is: Normalized value = (Original index value - Minimum value of the effective range of the scenario) / (Maximum value of the effective range of the scenario - Minimum value of the effective range of the scenario); The reverse normalization formula is: Normalized value = (maximum value of the effective range of the scenario - original index value) / (maximum value of the effective range of the scenario - minimum value of the effective range of the scenario).

[0012] As a further aspect of the present invention, the effective range of the scenario includes: In human-machine collaboration scenarios, the collision risk score is [0, 80], the contact safety score is [50, 100], and the equipment stability score is [50, 100]. In unmanned operation scenarios, the collision risk score is [0, 70], the contact safety score is [40, 100], and the equipment stability score is [60, 100]. In precision assembly scenarios, the collision risk score is [0, 60], the contact safety score is [60, 100], and the equipment stability score is [70, 100].

[0013] As a further embodiment of the present invention, the comprehensive risk index = normalized collision risk value × corresponding weight + normalized contact safety value × corresponding weight + normalized equipment stability value × corresponding weight. If the comprehensive risk index is within the range of [0, 0.25), it indicates no risk, the robotic arm operates normally, and the indicator changes are continuously monitored. If the comprehensive risk index is within the range of [0.25, 0.45), it indicates a risk warning, an audible and visual alarm is triggered, and data is recorded. The robotic arm maintains its current parameters. If the comprehensive risk index is within the range of [0.45, 0.7), it indicates a medium risk, the robotic arm slows down to 30%-50% of its rated speed, and the path is pre-adjusted. If the comprehensive risk index is within the range of [0.7, 1.0], it indicates a high risk, an immediate dual-circuit emergency stop is initiated, and alarm information and indicator data are uploaded.

[0014] A safety protection system for the working area of ​​a robotic arm based on multimodal sensing, comprising: The multimodal data acquisition and processing module uses a multi-source sensor system consisting of a vision system, force / torque sensors, LiDAR, tactile array / electronic skin, and IMU inertial measurement unit to acquire visual data, force / torque data, LiDAR point cloud, and tactile / IMU data. The module then performs data cleaning, spatiotemporal alignment, format standardization, and data completion processing on the multimodal data to obtain preprocessed multimodal data, which is then transmitted to the multimodal feature extraction and fusion module. The multimodal feature extraction and fusion module extracts single-modal features from preprocessed multimodal data, dynamically adjusts the weight coefficients of each modal feature based on the real-time safety risk level of the robotic arm's working area, and constructs a spatiotemporal feature correlation matrix containing temporal correlation submatrices and spatial correlation submatrices. The module then performs element-level fusion of the dimension-aligned modal features with the spatiotemporal feature correlation matrix to generate a fused feature vector containing the target motion state, contact force change trend, obstacle spatial location, and tactile contact distribution. The fused feature vector is then transmitted to the risk indicator assessment and analysis module. The risk indicator assessment and analysis module integrates feature vectors into a multi-dimensional risk assessment model and outputs three core safety indicators: collision risk value, contact safety, and equipment stability. The collision risk value is calculated based on dynamic collision time and force / torque correction factor; the contact safety is assessed based on tactile and force / torque feature weighting; and the equipment stability is determined by fusion of visual, IMU, and force / torque vibration features. The core safety indicators are then transmitted to the risk level determination and protection module. The risk level assessment and protection module dynamically allocates the weights of three core indicators based on the real-time operation scenario of the robotic arm, performs attribute-based normalization calibration on the core indicators, calculates the comprehensive risk index and maps it to four levels of safety risk: no risk, warning risk, medium risk and high risk, and executes corresponding safety protection actions according to different levels.

[0015] This invention provides a method and system for safety protection of the working area of ​​a robotic arm based on multimodal sensing. Compared with the prior art, it has the following advantages: This invention precisely strengthens the expression of key security features through a dynamic weight allocation mechanism based on real-time security risk levels; it constructs a two-dimensional spatiotemporal feature correlation matrix by combining a dynamic time warping algorithm with a unified coordinate system mapping, fully mining the temporal and spatial correlation information of multimodal data, and solving the defects of feature fusion in existing technologies that lack dynamic adaptation and spatiotemporal modeling. The generated fused feature vector can comprehensively and accurately reflect multidimensional security information.

[0016] This invention constructs a multi-dimensional risk assessment model, designs differentiated calculation logic for collision risk, contact safety, and equipment stability, and establishes a multi-indicator collaborative assessment mechanism. By eliminating the differences in indicator attributes and the influence of dimensions through attribute normalization calibration, it ensures the accuracy of core indicator quantification, solves the problems of single and ambiguous risk assessment in existing technologies, and reduces the misjudgment rate and missed judgment rate of risk quantification.

[0017] This invention dynamically allocates the weights of core indicators and adjusts the normalized effective range based on the real-time operation scenario of the robotic arm. It establishes a closed-loop mechanism covering the entire chain of data preprocessing, feature fusion, risk assessment, level determination, and protection execution, achieving accurate mapping of multi-level safety risk levels and targeted protection actions. While ensuring personnel safety and equipment stability, it also takes into account operational efficiency and flexibility, making it particularly suitable for complex scenarios such as human-machine collaboration, precision assembly, and unmanned operation. It solves the pain points of poor scenario adaptability and difficulty in balancing safety and efficiency in existing technologies. Attached Figure Description

[0018] Figure 1 This is a flowchart of the safety protection method steps of the present invention; Figure 2 This is a system block diagram of the security protection method of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] First Embodiment Please see Figure 1 This application provides a method for safety protection of the working area of ​​a robotic arm based on multimodal sensing, including the following steps: Step 1: Acquire multimodal data through multi-source sensors, including vision systems, force / torque sensors, LiDAR, tactile arrays / electronic skin, and IMU inertial measurement units. The multimodal data includes visual data, force / torque data, LiDAR point clouds, and tactile / IMU data. The acquired multimodal data is then preprocessed to obtain preprocessed multimodal data, which includes data cleaning, spatiotemporal alignment, format standardization, and data completion. For data cleaning, visual data (RGB-D / 3D camera) was cleaned using Gaussian and median filtering to remove salt-and-pepper noise, and invalid points in the depth map were removed using a thresholding method. Force / torque data was cleaned using moving average filtering to smooth high-frequency vibration noise, and abnormal peaks triggered by collisions were removed using the 3σ criterion. LiDAR point clouds were cleaned using statistical filtering to remove isolated points, and pass-through filtering was used to extract the effective point cloud within the robotic arm's working area. Tactile / IMU data were cleaned using Kalman filtering to suppress sensor drift and invalid data exceeding the physical measurement range was removed. For spatiotemporal alignment, high-frequency sensors are used as a reference, low-frequency sensors are interpolated and completed, and lagging sensors are timestamped to ensure that all data are synchronized to the same time scale. At the same time, hand-eye calibration is used to establish the transformation relationship between vision, LiDAR and the robotic arm base coordinate system, and to map the spatial data of different sensors to a unified coordinate system (such as the robotic arm base coordinate system) to avoid positional deviation. To standardize the format, data of different dimensions are mapped to the range of [0, 1] or [-1, 1]. For example, force data and distance data are normalized by Min-Max to eliminate the influence of dimensions. Irregular data is structured, such as point cloud data being downsampled to a fixed number of points, tactile array data being flattened into a two-dimensional matrix, and visual images being scaled to a fixed resolution. For data completion, short-term missing data is replaced by forward interpolation or adjacent frame data, and long-term missing data is completed based on predictions from other modal data.

[0021] Step 2: Extract the multimodal features corresponding to the preprocessed multimodal data, and the specific extraction and fusion methods are as follows: For visual data, 2D images are extracted using CNN to extract texture / semantic features, and 3D point clouds are extracted using PointNet to extract spatial geometric features, outputting target category, bounding box coordinates, and distance information. For force / torque sensing data, temporal and frequency domain features are extracted, outputting contact force magnitude, collision intensity, and force distribution trend. For LiDAR point clouds, target contours are extracted through point cloud clustering, and target center distance, volume, and motion speed are calculated, while obstacle position, size, and movement trajectory are output. For tactile / IMU data, tactile data is extracted for contact area, pressure center coordinates, and pressure distribution entropy, outputting contact state and force uniformity; IMU data is extracted for attitude angle, angular acceleration, and vibration frequency, outputting robotic arm motion state and abnormal vibration characteristics. Feature fusion is performed based on the extracted multimodal features. The weight coefficients of each modal feature are dynamically adjusted based on the real-time safety risk level of the robotic arm's working area. The safety risk level is jointly quantified and generated by the real-time distance of personnel intrusion, the probability of obstacle collision, the robotic arm's operating speed, and load parameters. This strengthens the expression of key features that are strongly correlated with safety decisions. At the same time, an adaptive threshold filtering mechanism is used to suppress environmental interference noise in the original data of each modality. Further, a spatiotemporal feature correlation matrix is ​​constructed, which includes a temporal correlation submatrix and a spatial correlation submatrix. The temporal correlation submatrix is ​​constructed by using a dynamic time warping algorithm to perform frame-level alignment between the temporal data of the target motion trajectory collected by the visual sensor and the temporal data of the contact force-torque collected by the six-axis force / torque sensor, establishing a temporal correspondence between motion state and force change. The temporal correlation submatrix is ​​M×N dimensional, where M is the number of temporal frames and N is the sum of the dimensions of each modal temporal feature. The matrix elements represent the temporal correlation strength of the multimodal features of the corresponding frame. The spatial correlation submatrix is ​​constructed based on a unified coordinate system obtained from hand-eye calibration. The three-dimensional spatial coordinates of the obstacle output after the clustering of the lidar point cloud are spatially mapped and transformed with the two-dimensional planar coordinates of the tactile sensor array to establish a spatial correspondence between the spatial position of the obstacle and the distribution of the contact area. The spatial correlation submatrix is ​​P×Q dimensional, where P is the number of spatial grids in the working area and Q is the sum of the dimensions of each modal spatial feature. The matrix elements represent the spatial matching degree of the multimodal features of the corresponding grid. By using zero-filling or interpolation operations, the modal feature vectors and the spatiotemporal correlation matrix are adjusted to the same dimension. The modal features after dynamic weight adjustment are then fused with the two-dimensional spatiotemporal feature correlation matrix at the element level. The final result is a fused feature vector containing four core safety dimensions: target motion state, contact force change trend, obstacle spatial position, and tactile contact distribution. Step 3: Based on the obtained fusion feature vector, input it into the multi-dimensional risk assessment model and output core safety indicators including collision risk value, contact safety, and equipment stability. The specific collision risk value is obtained by correcting the static TTC value through target motion trajectory prediction to obtain the dynamic collision time. Then, the contact force peak value, force change rate, and torque fluctuation value in the force / torque features are introduced as correction factors. The collision risk value is calculated according to the weighted formula: Collision Risk Value = Dynamic TTC Normalized Score × 0.6 + Force / Torque Correction Factor × 0.4. The higher the score, the higher the collision risk. The tactile features for contact safety include contact area, pressure distribution entropy, and pressure center of gravity offset; the force / torque features include contact force uniformity, peak force duration, and torque balance. A weighted total score is calculated based on the correlation weight between each feature and contact safety to form the contact safety level. The judgment threshold is dynamically adjusted based on the type of robotic arm operation. If the value is lower than the corresponding threshold, it is judged as abnormal contact. Equipment stability: Visual features include robotic arm joint attitude angles and motion trajectory smoothness; IMU features include angular acceleration and vibration amplitude; force / torque data are transformed by FFT and the vibration dominant frequency and harmonic energy are used. The abnormal contribution of each feature is calculated by a Bayesian network model, and the equipment stability score is obtained by weighted summation. At the same time, a dynamic threshold is set. When the score is lower than the threshold, it is judged as a precursor to equipment failure.

[0022] Step 4: Conduct a comprehensive risk analysis based on the obtained core indicators. Dynamically allocate weight coefficients for the three main indicators based on the real-time operation scenario of the robotic arm and the priority of safety requirements. Specifically: Human-machine collaboration scenario: Collision risk value weight 0.5, contact safety weight 0.3, equipment stability weight 0.2, collision risk value range [0, 80] points, contact safety range [50, 100] points, equipment stability range [50, 100] points; Unmanned operation scenario: Equipment stability weight 0.4, collision risk value weight 0.35, contact safety weight 0.25, collision risk value range is [0, 70] points, contact safety range is [40, 100] points, equipment stability range is [60, 100] points; Precision assembly scenario: contact safety weight 0.4, collision risk value weight 0.3, equipment stability weight 0.3, collision risk value range [0, 60] points, contact safety range [60, 100] points, equipment stability range [70, 100] points; Collision risk value, contact safety, and equipment stability are normalized and calibrated. Adaptive normalization formulas are designed for positive and negative indicators respectively. Collision risk value normalization formula: Normalized value = (Original indicator value - Minimum value of scene effective range) / (Maximum value of scene effective range - Minimum value of scene effective range). When the original indicator value < the minimum value of scene effective range, the normalized value = 0; when the original indicator value > the maximum value of scene effective range, the normalized value = 1. (Contact safety and equipment stability) Normalization formula: Normalized value = (maximum value of effective range of scene - original index value) / (maximum value of effective range of scene - minimum value of effective range of scene). When the original index value > the maximum value of effective range of scene, the normalized value = 0. When the original index value < the minimum value of effective range of scene, the normalized value = 1. Next, based on the obtained normalized values, a weighted summation method is used to calculate the comprehensive risk index. The comprehensive risk index = (normalized collision risk value × weight 1) + (normalized contact safety value × weight 2) + (normalized equipment stability value × weight 3). If the comprehensive risk index is in the range of [0, 0.25), it indicates no risk, the robotic arm operates normally, and the indicator changes are continuously monitored. If the comprehensive risk index is in the range of [0.25, 0.45), it indicates a warning risk, an audible and visual alarm is triggered, and data is recorded. The robotic arm maintains its current parameters. If the comprehensive risk index is in the range of [0.45, 0.7), it indicates a medium risk. The robotic arm slows down to 30%-50% of its rated speed, and the path is pre-adjusted. If the comprehensive risk index is in the range of [0.7, 1.0], it indicates a high risk. An immediate dual-circuit emergency stop is initiated, and alarm information and indicator data are uploaded.

[0023] Second Embodiment Please see Figure 2 This application provides a safety protection system for the working area of ​​a robotic arm based on multimodal sensing, including: a multimodal data acquisition and processing module, a multimodal feature extraction and fusion module, a risk indicator assessment and analysis module, and a risk level determination and protection module, wherein the information between the above functional modules is transmitted in one direction.

[0024] The multimodal data acquisition and processing module uses a multi-source sensor system consisting of a vision system, force / torque sensors, LiDAR, tactile array / electronic skin, and IMU inertial measurement unit to acquire visual data, force / torque data, LiDAR point cloud, and tactile / IMU data. The module then performs data cleaning, spatiotemporal alignment, format standardization, and data completion processing on the multimodal data to obtain preprocessed multimodal data, which is then transmitted to the multimodal feature extraction and fusion module. The multimodal feature extraction and fusion module extracts single-modal features from preprocessed multimodal data, dynamically adjusts the weight coefficients of each modal feature based on the real-time safety risk level of the robotic arm's working area, and constructs a spatiotemporal feature correlation matrix containing temporal correlation submatrices and spatial correlation submatrices. The module then performs element-level fusion of the dimension-aligned modal features with the spatiotemporal feature correlation matrix to generate a fused feature vector containing the target motion state, contact force change trend, obstacle spatial location, and tactile contact distribution. The fused feature vector is then transmitted to the risk indicator assessment and analysis module. The risk indicator assessment and analysis module integrates feature vectors into a multi-dimensional risk assessment model and outputs three core safety indicators: collision risk value, contact safety, and equipment stability. The collision risk value is calculated based on dynamic collision time and force / torque correction factor; the contact safety is assessed based on tactile and force / torque feature weighting; and the equipment stability is determined by fusion of visual, IMU, and force / torque vibration features. The core safety indicators are then transmitted to the risk level determination and protection module. The risk level assessment and protection module dynamically allocates the weights of three core indicators based on the real-time operation scenario of the robotic arm, performs attribute-based normalization calibration on the core indicators, calculates the comprehensive risk index and maps it to four levels of safety risk: no risk, warning risk, medium risk and high risk, and executes corresponding safety protection actions according to different levels.

[0025] Some of the data in the above formulas are numerical calculations with dimensions removed, and the contents not described in detail in this specification are all prior art known to those skilled in the art.

[0026] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A method for safety protection of the working area of ​​a robotic arm based on multimodal sensing, characterized in that, The method specifically comprises the following steps: Step one, collect multi-modal data through multi-source sensors, and sequentially perform data cleaning, space-time alignment, format standardization and data completion processing to obtain pre-processed multi-modal data; Step two, extract single-modal features of the pre-processed multi-modal data, adjust the weight coefficients of the modal features, and simultaneously construct a space-time feature correlation matrix, perform element-level fusion of the modal features and the space-time feature correlation matrix, and generate a fusion feature vector; Step three, input the fusion feature vector into a multi-dimensional risk assessment model, and output three core safety indexes of collision risk value, contact safety degree and equipment stability; Step four, dynamically allocate weights of the three core indexes based on real-time operation scenes of the mechanical arm, perform attribute normalization calibration on the core indexes, calculate a comprehensive risk index and map it to different safety risk levels, and perform corresponding safety protection actions according to different levels.

2. The method of claim 1, wherein, The multi-source sensors include a vision system, a force / torque sensor, a laser radar, a touch array / electronic skin and an IMU inertial measurement unit, and the multi-source data includes vision data, force / torque data, laser radar point cloud and touch / IMU data. 3.The method of claim 1, wherein, The data cleaning includes: the vision data is denoised by Gaussian filtering and median filtering, and invalid points in the depth map are removed by threshold method; the force / torque data is smoothed by moving average filtering, and abnormal peaks are removed by 3 sigma criterion; the laser radar point cloud is filtered by statistical filtering to remove isolated points, and the effective region is intercepted by straight-through filtering; the touch / IMU data is suppressed by Kalman filtering to inhibit drift, and out-of-range invalid data is removed; The space-time alignment includes: taking a high-frequency sensor as a reference, interpolating and completing a low-frequency sensor, and calibrating a lagging sensor timestamp; a unified coordinate system is established through hand-eye calibration, and space data of different sensors are mapped to the unified coordinate system.

4. The method of claim 1, wherein, The single-modal feature extraction includes: the vision data extracts texture / semantic features and spatial geometric features through CNN and PointNet respectively; the force / torque data extracts time domain and frequency domain features; the laser radar point cloud extracts target contour features through clustering; the touch data extracts contact area and pressure distribution features; and the IMU data extracts attitude angle and vibration features.

5. The method of claim 1, wherein, In the space-time feature correlation matrix: a time sequence correlation sub-matrix aligns the motion trajectory of a vision target and the time sequence data of a force / torque by using a dynamic time warping algorithm, and has an MxN dimension, where M is the time sequence frame number and N is the sum of the dimensions of the modal time sequence features; A spatial correlation sub-matrix maps three-dimensional coordinates of laser radar obstacles and two-dimensional contact coordinates of touch based on the unified coordinate system, and has a PxQ dimension, where P is the number of spatial grids and Q is the sum of the dimensions of the modal spatial features.

6. The method of claim 1, wherein, The manner of outputting the three core safety indexes of collision risk value, contact safety degree and equipment stability is: The collision risk value is calculated based on a dynamic collision time and a force / torque correction factor, and specifically, the collision risk value = dynamic TTC normalized score x 0.6 + force / torque correction factor x 0.4, the contact safety degree is evaluated based on touch and force / torque features, and the equipment stability is determined based on fusion of vision, IMU and force / torque vibration features.

7. The method of claim 1, wherein, The attribute normalization calibration includes: The collision risk value adopts a positive normalization formula, the contact safety degree and the equipment stability adopt a reverse normalization formula, and the normalization interval is [0, 1]; The positive normalization formula is: normalized value = (original index value - minimum value of scene effective interval) / (maximum value of scene effective interval - minimum value of scene effective interval); The reverse normalization formula is: normalized value = (maximum value of scene effective interval - original index value) / (maximum value of scene effective interval - minimum value of scene effective interval). 8.The method of claim 1, wherein, The scene effective interval includes: The collision risk value in the human-machine cooperation scene is [0, 80] points, the contact safety degree is [50, 100] points, and the equipment stability is [50, 100] points; the collision risk value in the unmanned operation scene is [0, 70] points, the contact safety degree is [40, 100] points, and the equipment stability is [60, 100] points; the collision risk value in the precision assembly scene is [0, 60] points, the contact safety degree is [60, 100] points, and the equipment stability is [70, 100] points.

9. The method of claim 1, wherein, The comprehensive risk index = collision risk value normalized value x corresponding weight + contact safety degree normalized value x corresponding weight + equipment stability normalized value x corresponding weight; If the comprehensive risk index is in the interval [0, 0.25), it indicates no risk, the robot arm operates normally, and the index changes are continuously monitored; if the comprehensive risk index is in the interval [0.25, 0.45), it indicates a warning risk, and the sound and light alarm and data recording are performed, and the robot arm maintains the current parameters; if the comprehensive risk index is in the interval [0.45, 0.7), it indicates a medium risk, and the robot arm is reduced to 30%-50% of the rated speed, the path is pre-adjusted; if the comprehensive risk index is in the interval [0.7, 1.0], it indicates a high risk, and the robot arm is immediately stopped by double circuits, and the alarm information and index data are uploaded.

10. A multi-modal sensor based safety guarding system for a robot working area, configured to perform the safety guarding method according to any one of claims 1-9, characterized in that, It includes: A multi-modal data acquisition and processing module, which acquires visual data, force / torque data, laser radar point cloud and tactile / IMU data through a multi-source sensor composed of a vision system, a force / torque sensor, a laser radar, a tactile array / electronic skin and an IMU inertial measurement unit; the multi-modal data is sequentially subjected to data cleaning, space-time alignment, format standardization and data completion processing to obtain preprocessed multi-modal data, which is transmitted to a multi-modal feature extraction and fusion module; A multi-modal feature extraction and fusion module, which extracts single-modal features of the preprocessed multi-modal data, dynamically adjusts the weight coefficients of each modal feature based on the real-time safety risk level of the robot arm working area, simultaneously constructs a space-time feature correlation matrix containing a time sequence correlation sub-matrix and a spatial correlation sub-matrix, and performs element-level fusion of the dimension-aligned modal features and the space-time feature correlation matrix to generate a fusion feature vector containing target motion state, contact force change trend, obstacle spatial position and tactile contact distribution, and the fusion feature vector is transmitted to a risk index evaluation and analysis module; The risk index evaluation analysis module inputs the fusion feature vector into a multi-dimensional risk evaluation model, and outputs three core safety indexes of collision risk value, contact safety degree and equipment stability; wherein, the collision risk value is calculated based on dynamic collision time and force / torque correction factor weighting, the contact safety degree is evaluated based on touch and force / torque characteristics weighting, the equipment stability is determined based on visual, IMU and force / torque vibration characteristics fusion, and the core safety indexes are transmitted to the risk level determination and protection module; The risk level determination and protection module dynamically allocates weights of the three core indexes based on the real-time operation scene of the robot arm, performs attribute normalization calibration on the core indexes, calculates a comprehensive risk index and maps it into four safety risk levels of no risk, early warning risk, medium risk and high risk, and executes corresponding safety protection actions according to different levels.

Images