Human-robot interaction safety detection method and system based on convolutional neural network

Abstract

The present invention discloses a human-robot interaction safety detection method and system based on a convolutional neural network, the method comprising the following steps: collecting RGB-D images of a human-robot interaction environment and robot joint positions; extracting a human joint point sequence from the RGB-D image using a joint point detection algorithm, and extracting user posture parameters using a human body model reconstruction network; constructing a safety detection network based on a convolutional neural network, the safety detection network receives input human joint point sequences, posture parameters and robot joint positions, outputs the distance and contact state between a human and a robot in the interaction process in real time, detects whether a collision occurs or there is a potential collision risk; responds to a collision or a potential collision risk, sends a corresponding warning message or sends a corresponding operation instruction to the robot. The present invention can improve the accuracy of safety detection in a human-machine interaction environment and ensure the real-time performance of safety detection.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent manufacturing technology, and in particular relates to a human-robot interaction safety detection method and system based on a convolutional neural network. Background Art

[0002] With the advancement of robotics technology and the expansion of application scenarios, collaboration between humans and robots has become more and more common, especially in the fields of industrial automation, service robots, and smart homes. In these interactions, collision accidents may cause equipment damage, personal injury, or reduced work efficiency, so detecting the relative position relationship between the user and the robot is crucial to the human-machine collaborative system. Currently, commonly used methods include sensor-based detection and vision-based detection. Sensor technology usually involves the use of lidar, ultrasonic sensors, infrared sensors, etc., which can provide high-precision distance measurement and object recognition. Vision-based detection uses cameras and image processing algorithms to analyze visual information to determine potential collision risks.

[0003] The Chinese invention patent application with publication number CN118009874A discloses a human operator behavior safety detection system and method for industrial robots. The system includes: a depth vision sensor module, a data processing module, a collision warning module, a command conversion module, and a robot communication and execution module; the depth vision sensor module collects the depth image information of the human operator in the robot working area; the data processing module establishes the three-dimensional coordinates of the key points of the human operator's behavior and the robot's axis joints based on the world coordinate system; the collision warning module monitors the relative distance between the key points of the human operator and the robot's axis joints, and performs safety assessment and judgment; the command conversion module calculates the robot's motion parameters for obstacle avoidance, converts the obstacle avoidance command into an industrial robot motion command and outputs it; the robot communication and execution module executes the robot command to achieve obstacle avoidance movement. In this way, the safety of production personnel and the normal operation of equipment are guaranteed in real time.

[0004] The traditional collision interference detection method based on 3D geometric models has problems such as low real-time performance and assembly occlusion, and cannot meet the detection accuracy and real-time requirements in the safe production process. At the same time, the use of contact-based collision detection and braking response methods in the industrial field will greatly reduce the efficiency of processing and production and the safety of interaction. Summary of the invention

[0005] The present invention provides a human-robot interaction safety detection method and system based on convolutional neural network, aiming to solve the problems of low detection accuracy and real-time performance in the prior art.

[0006] In order to solve the above technical problems, the safety detection method proposed by the present invention comprises the following steps:

[0007] Collect RGB-D images of the human-robot interaction environment and robot joint positions;

[0008] Extracting a human joint point sequence from the RGB-D image using a joint point detection algorithm, and extracting a user's body posture parameters using a human body model reconstruction network;

[0009] Construct a safety detection network based on a convolutional neural network, which receives input of a human joint point sequence, user posture parameters, and robot joint positions, outputs the distance and contact status between the human and the robot in real time during the interaction, and detects whether a collision occurs or there is a potential collision risk;

[0010] In response to a collision or a potential collision, a corresponding warning message or a corresponding operation instruction is sent to the robot.

[0011] Preferably, the safety detection network includes a feature extraction module, a feature fusion module, an output optimization module and a task output module which are arranged in sequence;

[0012] The feature extraction module outputs the human joint point sequence features, the user posture parameter features and the robot joint position sequence features;

[0013] The feature fusion module outputs human-robot interaction features;

[0014] The output optimization module outputs user virtual nodes, robot virtual nodes and position compensation vectors.

[0015] Preferably, the feature extraction module is one of a convolutional neural network, a graph convolutional neural network, an LSTM network and a Mamba model, and receives input human joint point sequences, user posture parameters and robot joint point positions.

[0016] Preferably, the feature fusion module includes a maximum pooling submodule, an average pooling submodule, a convolution submodule and a fully connected layer, and the processing flow of the feature fusion module is specifically as follows:

[0017] The maximum pooling submodule and the average pooling submodule take the user's body parameter features as input data and output the maximum feature and the average feature of the user's body parameter;

[0018] The maximum feature and the average feature are concatenated and passed through a convolution submodule to obtain a posture weight coefficient;

[0019] The robot joint position sequence features are serially input into several fully connected layers, and the robot interaction features that are consistent with the human joint point sequence dimension are output;

[0020] The human joint sequence features are multiplied by the posture weight coefficient to generate the user mixed features, and then the robot interaction features are subtracted from the user mixed features to generate the human-robot interaction features.

[0021] Preferably, the output optimization module includes a voting module and a fully connected layer, and the processing flow is specifically as follows:

[0022] The human-robot interaction features output by the feature fusion module are received, and the human-robot interaction features are respectively input into the voting module and two fully connected layers, one fully connected layer outputs the user virtual node, the other fully connected layer outputs the robot virtual node, and the voting module outputs the position compensation vector.

[0023] Preferably, the task output module performs a voting operation on the human-robot interaction features, and votes out optimized user virtual nodes and robot virtual nodes along the direction of the feature compensation vector; the optimized user virtual nodes and robot virtual nodes are input into the distance calculation module, and the predicted minimum interaction distance is output.

[0024] Preferably, the calculation method of the minimum interaction distance is:

[0025] According to the relative position characteristics of humans and robots in three-dimensional coordinates represented by user virtual nodes and robot virtual nodes, the distance formula between two points in space is used to calculate the value of the minimum distance to be estimated in the regression task.

[0026] Preferably, the data set of the safety detection network is prepared as follows:

[0027] The dataset preparation includes four steps: obtaining the coordinates of human joint points and user posture parameters, modeling the robot capsule, coordinate system I, minimum distance and contact state label calculation;

[0028] Among them, the robot capsule modeling simulates the different postures of the robot in the interaction process by setting the angle sequence of each joint of the six-axis robot during the interaction process, generates an action sequence model based on the angle sequence of the robot urdf file model, completes the reconstruction of the robot interaction model, and obtains the input data of the robot.

[0029] Preferably, the minimum distance and contact status label calculation method is specifically as follows:

[0030] Traverse each vertex in the user's SMPL model and each part in the reconstructed robot interaction model, calculate the minimum distance, and use the VTK library to find the point in the user's SMPL model vertex that is closest to the robot model, and calculate the minimum distance and the point on the user's SMPL model that is closest to the robot; set the label of the data set according to the calculation result; finally, obtain the label of the data set, which includes the minimum interaction distance, interaction contact state and interaction contact position.

[0031] In another aspect of the present invention, a human-robot interaction safety detection system based on a convolutional neural network is also proposed. The system is used to perform the above-mentioned detection method, including:

[0032] The imaging module is used to collect RGB-D images of the human-robot interaction environment and send them to the human joint point detection module and the human body model reconstruction network.

[0033] The human joint point detection module detects the human joint point sequence based on the received RGB-D image and outputs it to the safety detection network.

[0034] The human body model reconstruction network detects the user's body parameters based on the received RGB-D image and outputs them to the safety detection network.

[0035] A robot is deployed with a robot operating system, which obtains the robot joint position sequence in real time and sends it to a safety detection network.

[0036] The output module generates operation instructions to the robot and / or generates warning information for display to the user according to the minimum distance and contact status between the human and the robot during the interaction process output by the safety detection network.

[0037] The safety detection network further includes a feature extraction module, a feature fusion module, an output optimization module and a task output module.

[0038] The feature extraction module is one of a convolutional neural network, a graph convolutional neural network, a LSTM network and a Mamba model, receives input human joint point sequence and user posture parameters, and outputs human joint point sequence features, user posture parameter features and robot joint position sequence features to a feature fusion module.

[0039] The feature fusion module includes a maximum pooling submodule, an average pooling submodule, a convolution submodule and a fully connected layer, and outputs human-robot interaction features to an output optimization module after processing the input features.

[0040] The output optimization module inputs the human-robot interaction features into the voting module and two fully connected layers respectively, and outputs the user virtual node, the robot virtual node and the position compensation vector.

[0041] The task output module performs a voting operation on the human-robot interaction features, and votes out optimized user virtual nodes and robot virtual nodes along the direction of the feature compensation vector; the optimized user virtual nodes and robot virtual nodes are input into the distance calculation module, and the predicted minimum interaction distance is output.

[0042] Compared with the prior art, the present invention has the following technical effects:

[0043] 1. The safety detection method proposed in the present invention collects RGB-D images of the human-robot interaction environment and the robot joint positions, extracts the human joint point sequence and user posture parameters from them, and constructs a safety detection network based on a convolutional neural network to detect the distance and contact status between the human and the robot during the interaction process. It can predict the minimum contact distance of the user and the robot interaction state in real time, and on the premise of ensuring the accuracy of safety detection, it also meets the real-time nature of human-robot interaction safety detection.

[0044] 2. When constructing a safety detection network, the safety detection method proposed in the present invention simulates the postures of the user and the robot during the human-machine interaction process, and prepares a virtual data set for collision detection through three steps: coordinate system conversion, capsule bounding box construction, and minimum distance calculation. This data set includes as many simulated interaction scenes between users of different postures and body shapes and robots of different postures as possible. The human posture morphological parameters and robot posture parameters are used as input data of the data set, and labels such as the closest distance, contact state, and contact position are produced according to different tasks. This can improve the accuracy and applicability of the safety detection model prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a schematic diagram of the process of the safety detection method of the present invention;

[0046] Figure 2 is a schematic diagram of the structure of the security detection network according to an embodiment of the present invention;

[0047] Figure 3 It is a schematic diagram of the safety detection system described in an embodiment of the present invention.

[0048] Figure numerals: 10, computer; 20, industrial robot; 30, depth camera. DETAILED DESCRIPTION

[0049] In order to make the objectives, technical solutions and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in combination with specific embodiments of the present application and with reference to the accompanying drawings.

[0050] Embodiment 1

[0051] This embodiment is a human-robot interaction safety detection method based on convolutional neural network. Figure 1 As shown, the steps include steps 1 to 4:

[0052] Step 1: Collect RGB-D images of the human-robot interaction environment and the joint positions of the industrial robot; in the specific implementation process, an RGB-D camera can be used to collect RGB-D images of the interaction environment. In addition to the R (Red), G (Green), and B (Blue) of the traditional RGB image, the RGB-D image also includes a D (Depth) channel, where the D channel represents the distance from each pixel to the sensor. By setting a reasonable monitoring angle for the RGB-D camera, the interaction process is fully recorded, and the color (RGB) and depth images of the human-machine interaction scene are collected in real time and input to the computer, where a human joint point detection module, a human model reconstruction network, and a safety detection network are deployed. Common devices for generating RGB-D images include: Microsoft Kinect, Intel RealSense, structured light cameras, and time of flight (ToF) cameras.

[0053] Step 2: extract the human joint point sequence from the RGB-D image using the joint point detection algorithm, and extract the user's body parameters using the human body model reconstruction network. It can be understood by those skilled in the art that, in order to ensure the efficiency and accuracy of subsequent image feature extraction, before extracting the RGB-D image, a series of preprocessing of the image is also included, which is not limited to standardizing the image data (such as normalization or mean normalization), resizing, graying, and other operations.

[0054] Step three, construct a safety detection network based on convolutional neural network, which receives input of human joint point sequence, user posture parameters and industrial robot joint positions, outputs the distance and contact status between humans and robots during the interaction process in real time, and detects whether a collision occurs or there is a potential collision risk.

[0055] Step 4: In response to a collision or a potential collision risk, a corresponding warning message or a corresponding operation instruction is sent to the industrial robot to implement a response such as shutdown or obstacle avoidance, and output a warning message to the user. The warning information includes sound and light prompts, message prompts, etc.

[0056] In this embodiment, if Figure 2As shown, the safety detection network includes a feature extraction module, a feature fusion module, an output optimization module and a task output module which are arranged in sequence.

[0057] The feature extraction module is one of a convolutional neural network, a graph convolutional neural network, an LSTM (Long Short-Term Memory) network and a Mamba model, which receives input human joint sequence, user posture parameters and industrial robot joint position; outputs human joint sequence features, user posture parameter features and industrial robot joint position sequence features. The human joint sequence is extracted from the RGB-D image by a human joint detection module deployed on a computer using a joint detection algorithm; the user posture parameters are extracted from the RGB-D image by a human model reconstruction network deployed on a computer; and the industrial robot joint position is provided by ROS (Robot Operating System) deployed on the industrial robot.

[0058] The sequence of human joints is generally composed of a series of coordinates (such as 2D or 3D coordinates). The sequence of joints can be represented as a time series or graph data. Convert the sequence of joints into a suitable input format, for example, represent the sequence as a two-dimensional feature matrix, where the rows represent the time step or the joint number, and the columns represent the coordinates (such as x, y, z). Normalization can be performed to ensure that the coordinate range is uniform. Apply 1D convolution to the time series or spatial sequence to extract the temporal features, or for the spatial topological relationship between the joints, use the graph convolutional network to extract the spatial structure information of the joints to enhance the representation.

[0059] User posture parameters can be composed of multiple posture angles, limb postures and other values, which can be regarded as multidimensional vectors. Combine user posture parameters into a feature vector, arrange all joint angles and posture information in a fixed order, and normalize the data to eliminate the dimensional differences between different posture values. Directly input the posture parameter vector into the fully connected layer for high-dimensional feature extraction. According to the sequence characteristics of posture parameters, use MLP or 1D convolutional layer to capture the local correlation between parameters and extract richer posture information.

[0060] The joint position data of industrial robots usually contains the angle and displacement information of multiple joints, which can be represented as a multi-dimensional feature vector. The joint position information is converted into a fixed-dimensional feature vector, which is input into the network after normalization. It can be input into the fully connected layer to learn the high-level features of the robot joint position, or for the joint position with time dependence, a 1D convolutional layer is used to extract the joint motion features that change over time.

[0061] The feature fusion module outputs human-robot interaction features, including a maximum pooling submodule, an average pooling submodule, a convolution submodule and a fully connected layer. The processing flow of the feature fusion module is specifically as follows:

[0062] The maximum pooling submodule and the average pooling submodule take the user's body parameter features as input data, and output the maximum features and average features of the user's body parameter; specifically, the user's body parameter features are represented as vectors, the user's body parameter feature vector is input into the maximum pooling module, and the maximum pooling feature vector is output, that is, the maximum value of each dimension of the user's body parameter feature vector; the user's body parameter feature vector is input into the average pooling module in parallel, and the average pooling feature vector is output, that is, the average value of each dimension of the user's body parameter feature vector.

[0063] The maximum feature and the average feature are concatenated and passed through a convolution submodule, and then processed through multiple convolution layers, activation functions (such as ReLU), and pooling layers to extract high-level features. A vector representing the posture weight coefficient is output in the last layer. These weight coefficients represent the importance of different posture parameters of the user in the feature, and are used to weight subsequent features to obtain the posture weight coefficient.

[0064] The joint position sequence features of the industrial robot are input in series to several fully connected layers for processing layer by layer. Each layer maps the features to different dimensions and outputs robot interaction features that are consistent with the dimensions of the human joint point sequence. That is, the output dimension of the last layer is the same as the dimension of the human joint point sequence features, thereby achieving the purpose of extracting the interaction features of the robot with each joint of the human body.

[0065] The human joint sequence features output by the feature extraction module are multiplied by the posture weight coefficient output by the convolution submodule to generate user mixed features, so that different user posture parameters have different features for the same joint position, so as to achieve the purpose of fusing posture parameters into joint features, and obtain user mixed features that consider human joint positions and posture parameters; then the user mixed features that consider human joint positions and posture parameters are subtracted from the robot interaction features output by several series fully connected layers, which represent the relative position relationship between the user and the robot in a unified coordinate system, and generate human-robot interaction features.

[0066] Through the feature fusion module, features that can characterize the interactive relationship between the human body and the industrial robot can be obtained, especially considering the influence of the user's body parameters on the joint point characteristics and the relative relationship between the robot joints and the human body joints, which helps to achieve more accurate human-robot collaboration.

[0067] The output optimization module includes a voting module and a fully connected layer, and the processing flow is specifically as follows:

[0068] The human-robot interaction features output by the feature fusion module are received, and the human-robot interaction features are respectively input into the voting module and two fully connected layers, one fully connected layer outputs the user virtual node, and the other fully connected layer outputs the robot virtual node. The voting module outputs the position compensation vector to obtain the user virtual node, the robot virtual node and the position compensation vector.

[0069] Specifically, the human-robot interaction features are passed to two independent fully connected layers. The first fully connected layer performs a linear transformation on the input features and processes them through an activation function (such as ReLU) to capture the high-level information of the user's joint features, generating an output vector as a user virtual node, which represents the abstract features of the user's key position points or joint state. The second fully connected layer is similar to the first layer structure, performing independent linear transformation and activation processing to extract high-level features corresponding to the robot's joints; generating an output vector as a robot virtual node, representing the abstract features of the robot's joint state.

[0070] The human-robot interaction features are also passed to the voting module, which can process input features based on a variety of strategies, such as multi-way classifier voting, decision trees, or heuristic weighted averages, and perform voting calculations on different components of the input features (such as weighted averages or classification voting between multiple categories of features) to determine the importance of each part of the features. It can also use simple linear transformations to calculate the position offset value; generate a position compensation vector for position compensation of the user and the robot in the coordinate system. After the above processing, the output optimization module obtains the following three results to the task output module: user virtual node, robot virtual node, and position compensation vector.

[0071] The fully connected layer in the above process is used to abstract the human-robot interaction features, so that the joint features of the user and the robot can be extracted independently, while the voting module adjusts the position deviation through position compensation to ensure that the interaction state between the user and the robot is accurately represented in a unified coordinate system. These features can be further used in the position, posture adjustment or control algorithm in human-robot collaboration.

[0072] The task output module performs a voting operation on the human-robot interaction features, and votes out the optimized user virtual nodes and robot virtual nodes along the direction of the feature compensation vector to achieve the purpose of optimizing the regression accuracy; the optimized user virtual nodes and robot virtual nodes are input into the distance calculation module, and the predicted minimum interaction distance is output.

[0073] In the task output module, the position compensation vector is used to fine-tune the positions of the user virtual node and the robot virtual node to optimize the prediction accuracy. The position compensation vector can be used as a fine-tuning direction and superimposed with the user virtual node and the robot virtual node. A weighting coefficient can be set to control the contribution of the compensation vector, and finally the optimized user virtual node and the optimized robot virtual node are output.

[0074] The calculation method of the minimum interaction distance is as follows:

[0075] According to the relative position characteristics of humans and robots in three-dimensional coordinates represented by user virtual nodes and robot virtual nodes, the distance formula between two points in space is used to calculate the value of the minimum distance to be estimated in the regression task.

[0076] Specifically, the user and robot virtual nodes can be regarded as two points in three-dimensional space, and their positions are and , using the distance formula between two points in three-dimensional space, calculate the minimum distance between the optimized user and robot virtual nodes :

[0077]

[0078] The above formula obtains the minimum interaction distance , represents the minimum interaction distance between the user and the robot, which is used for the final output of the regression task.

[0079] By voting along the compensation vector, the positions of the user and robot can be fine-tuned according to their respective virtual nodes, so as to more accurately estimate the minimum interaction distance between the two in three-dimensional space. This minimum interaction distance can effectively reflect the relative position relationship between the user and the robot, optimize the accuracy of human-robot interaction, and help control the interaction distance and safety management in actual tasks.

[0080] The construction process of the security detection network includes data set production, model architecture design, model training and model evaluation.

[0081] The dataset preparation includes four steps: obtaining the coordinates of human joints and user posture parameters, robot capsule modeling, coordinate system one, minimum distance and contact state label calculation.

[0082] The coordinates of human joint points and user body parameters can utilize existing data sets, select users with different actions and heights, weights and body types using the SMPL (Skinned Multi-Person Linear) model, and use the three-dimensional joint point information in space and body parameters representing the height, weight and body type of the human body as user input data.

[0083] The robot capsule modeling simulates the different postures of the robot in the interaction process by setting the angle sequence of each joint interaction process of the six-axis robot. The robot urdf (Unified Robot Description Format) file model generates an action sequence model according to the angle sequence, completes the reconstruction of the robot interaction model, and obtains the input data of the robot.

[0084] The coordinate system converts the user input data and robot input data in different coordinate systems and unifies them into the same coordinate system. The robot's joints are based on the robot base coordinates, and the user model is based on the world coordinate system. The robot coordinate system needs to be unified into the world coordinate system through the rotation matrix and translation matrix to achieve the effect of simulating the human-robot interaction process and obtain the input data of the overall data set.

[0085] The method for calculating the minimum distance and contact status label is specifically as follows:

[0086] The label of the dataset is formulated based on the minimum distance between the user's SMPL model and the robot's three-dimensional model. Each vertex in the user's SMPL model and each part of the reconstructed robot interaction model are traversed to calculate the minimum distance. The minimum distance uses the VTK (Visualization Toolkit) library to find the point closest to the robot model in the vertex of the user's SMPL model, and calculate the minimum distance and the point on the user's SMPL model closest to the robot; set the label of the dataset according to the calculation results, for example, the minimum distance is less than 0 for contact state; greater than 0 and less than 10 cm for danger state; greater than 10 and less than 30 cm for warning state; greater than 30 cm for safety state. Finally, the label of the dataset is obtained, which includes the minimum distance of interaction, the state of interaction contact, and the position of interaction contact; Finally, the label of the dataset is obtained, which includes the minimum distance of interaction, the state of interaction contact, and the position of interaction contact.

[0087] The above dataset can be used to train a safety detection method for human-robot interaction based on a convolutional neural network. The network is trained using the Pytorch version 1.12.1+cu116 deep learning framework on a computer equipped with an Intel® Core™ i7-9700 CPU@3.00GHz×8 CPU, an NVIDIA GeForce RTX2060 GPU, 16G memory, and an Ubuntu 18.04.6 operating system. The training is performed for 30 epochs with a batch size of 16.

[0088] Model architecture design Figure 2As shown in the figure, it includes a feature extraction module, a feature fusion module, an output optimization module and a task output module. Specifically, the model architecture design includes: defining the input layer according to the size of the input data; setting the convolution layer for extracting local features of the image, including parameters such as the size, number and stride of the convolution kernel; adding an activation function (such as ReLU) after each convolution layer to introduce nonlinearity so that the network can learn complex features; designing the maximum pooling (MaxPooling) or average pooling (Average Pooling) according to the needs to reduce the size of the feature map, reduce the amount of calculation, and prevent overfitting; designing the fully connected layer (Fully Connected Layer) to integrate the extracted features and perform classification or regression; the normalization layer (Normalization Layer) such as batch normalization (Batch Normalization) is used to accelerate model convergence and improve model stability. After designing the model architecture, the above-made data set is divided into a training set, a validation set and a test set to facilitate model evaluation and adjustment during the training process.

[0089] In terms of experimental results, the feasibility of the neural network was experimentally verified on the above-prepared data set, and the network optimized by the voting module was compared with it, focusing on the evaluation from the two aspects of accuracy and speed. In terms of accuracy, the minimum distance error prediction of the optimized network on the data set was reduced from 3.88cm to 3.47cm, a decrease of 10.57%; the accuracy of the two-class detection was increased from 84.3% to 86.3%, an increase of 2%; the accuracy of the four-class detection was increased from 70.9% to 74.5%, an increase of 3.4%; for the evaluation index of relaxed classification, the optimized classification effect can reach 95.4%, which is 2% higher than the 93.4% of the unoptimized network. In terms of speed, the processing speed of the safety detection method for human-robot interaction based on convolutional neural network in this embodiment can reach 662.4 frames per second, and the processing time for each frame is 1.51 milliseconds. After joining the voting network, the processing speed dropped to 657.5 frames per second, and the processing time for each frame was 1.52 milliseconds. Compared with the traditional collision interference detection method based on geometric model reconstruction, the speed of collision interference detection has been greatly improved, meeting the real-time requirements of safety detection tasks.

[0090] By collecting RGB-D images of the human-robot interaction environment and the joint positions of the industrial robot, the human joint point sequence and user posture parameters are extracted, and a safety detection network based on convolutional neural network is constructed to detect the distance and contact status between humans and robots during the interaction process. The minimum contact distance of the user-robot interaction status can be predicted in real time. While ensuring the accuracy of safety detection, it also meets the real-time nature of human-robot interaction safety detection.

[0091] In addition, when building a safety detection network, the postures of the user and the robot during human-machine interaction are simulated. After three steps of coordinate system conversion, capsule bounding box construction, and minimum distance calculation, a virtual data set for collision detection is prepared, so that the data set contains as many simulated interaction scenes between users of different postures and body shapes and robots of different postures as possible. The human posture morphological parameters and robot posture parameters are used as input data of the data set, and labels such as the closest distance, contact state, and contact position are produced according to different tasks. This can improve the accuracy and applicability of safety detection model predictions.

[0092] It can be understood by those skilled in the art that the above SMPL model is a digital human model based on vertex offset modeling of the human body developed by the Max Planck Institute, and is widely used in the field of three-dimensional reconstruction of digital humans or human characters. It supports various shapes and actions of the human body and can be simply understood as a character model obtained through training. As a vertex skinning-based model, it can accurately represent various body shapes in natural human postures. The parameters of the model are learned from data, and these parameters include standard posture templates, blending weights, posture-related blending shapes, identity-related blending shapes, and regressors from vertices to joint positions.

[0093] As for the above VTK library, VTK is an open source cross-platform software system for 3D computer graphics, image processing and visualization. It includes rich functions and tools, allowing users to create complex 3D visualization applications.

[0094] Embodiment 2

[0095] like Figure 3 As shown, this embodiment is a human-robot interaction safety detection system based on a convolutional neural network, and the system is used to execute the detection method as described in Example 1, including:

[0096] An imaging module is used to collect RGB-D images of the human-robot interaction environment and send them to the human joint point detection module and the human model reconstruction network; the imaging module in this embodiment is a depth camera 30, which completely records the interaction process by setting a reasonable monitoring angle of the depth camera 30, and collects color and depth images of the human-robot interaction scene in real time.

[0097] The human joint point detection module detects the human joint point sequence based on the received RGB-D image and outputs it to the safety detection network.

[0098] The human body model reconstruction network detects the user's body parameters based on the received RGB-D image and outputs them to the safety detection network.

[0099] The industrial robot 20 is equipped with a robot operating system, which acquires the joint position sequence of the industrial robot 20 in real time and sends it to the safety detection network.

[0100] The output module generates an operation instruction to the industrial robot 20 and / or generates a warning message for display to the user according to the minimum distance and contact status between the human and the industrial robot 20 during the interaction process output by the safety detection network.

[0101] The safety detection network further includes a feature extraction module, a feature fusion module, an output optimization module and a task output module.

[0102] The feature extraction module is one of a convolutional neural network, a graph convolutional neural network, a LSTM network and a Mamba model, receives input human joint point sequence and user posture parameters, and outputs human joint point sequence features, user posture parameter features and industrial robot joint position sequence features to a feature fusion module.

[0103] The feature fusion module includes a maximum pooling submodule, an average pooling submodule, a convolution submodule and a fully connected layer, and outputs human-robot interaction features to an output optimization module after processing the input features.

[0104] The output optimization module inputs the human-robot interaction features into the voting module and two fully connected layers respectively, and outputs the user virtual node, the robot virtual node and the position compensation vector.

[0105] The task output module performs a voting operation on the human-robot interaction features, and votes out optimized user virtual nodes and robot virtual nodes along the direction of the feature compensation vector; the optimized user virtual nodes and robot virtual nodes are input into the distance calculation module, and the predicted minimum interaction distance is output.

[0106] It should be noted that, in this embodiment, the human joint point detection module, the human model reconstruction network, the output module and the safety detection network are all deployed in Figure 3 In the computer 10 shown.

[0107] The above is only a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several modifications and improvements without departing from the inventive concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A human-robot interaction safety detection method based on convolutional neural network, characterized in that: The following steps are involved: Collect RGB-D images of the human-robot interaction environment and robot joint positions; Extracting a human joint point sequence from the RGB-D image using a joint point detection algorithm, and extracting a user's body posture parameters using a human body model reconstruction network; Construct a safety detection network based on a convolutional neural network, which receives input of a human joint point sequence, user posture parameters, and robot joint positions, outputs the distance and contact status between the human and the robot in real time during the interaction, and detects whether a collision occurs or there is a potential collision risk; In response to a collision or potential collision, send a corresponding warning message or send a corresponding operation instruction to the robot; The safety detection network includes a feature extraction module, a feature fusion module, an output optimization module and a task output module which are arranged in sequence; the feature fusion module outputs the human-robot interaction features, including a maximum pooling submodule, an average pooling submodule, a convolution submodule and a fully connected layer. The specific processing flow is as follows: The maximum pooling submodule and the average pooling submodule take the user's body parameter features as input data and output the maximum feature and the average feature of the user's body parameter; The maximum feature and the average feature are concatenated and passed through a convolution submodule to obtain a posture weight coefficient; The robot joint position sequence features are serially input into several fully connected layers, and the robot interaction features that are consistent with the human joint point sequence dimension are output; The human joint sequence features are multiplied by the posture weight coefficient to generate the user mixed features, and then the robot interaction features are subtracted from the user mixed features to generate the human-robot interaction features.

2. The human-robot interaction safety detection method based on convolutional neural network according to claim 1 is characterized in that: The feature extraction module outputs human joint point sequence features, user posture parameter features and robot joint position sequence features; the output optimization module outputs user virtual nodes, robot virtual nodes and position compensation vectors.

3. The human-robot interaction safety detection method based on convolutional neural network according to claim 2 is characterized in that: The feature extraction module is one of a convolutional neural network, a graph convolutional neural network, an LSTM network and a Mamba model, and receives input human joint point sequences, user posture parameters and robot joint point positions.

4. The human-robot interaction safety detection method based on convolutional neural network according to claim 2 is characterized in that: The output optimization module includes a voting module and a fully connected layer, and the processing flow is specifically as follows: The human-robot interaction features output by the feature fusion module are received, and the human-robot interaction features are respectively input into the voting module and two fully connected layers, one fully connected layer outputs the user virtual node, the other fully connected layer outputs the robot virtual node, and the voting module outputs the position compensation vector.

5. The method for human-robot interaction safety detection based on convolutional neural network according to claim 1 is characterized in that: The task output module performs a voting operation on the human-robot interaction features, and votes out optimized user virtual nodes and robot virtual nodes along the direction of the feature compensation vector; the optimized user virtual nodes and robot virtual nodes are input into the distance calculation module, and the predicted minimum interaction distance is output.

6. The method for human-robot interaction safety detection based on convolutional neural network according to claim 5 is characterized in that: The calculation method of the minimum interaction distance is: According to the relative position characteristics of humans and robots in three-dimensional coordinates represented by user virtual nodes and robot virtual nodes, the distance formula between two points in space is used to calculate the value of the minimum distance to be estimated in the regression task.

7. The method for detecting human-robot interaction safety based on convolutional neural network according to claim 1, characterized in that: The data set of the safety detection network is made as follows: The dataset preparation includes four steps: obtaining the coordinates of human joint points and user posture parameters, modeling the robot capsule, coordinate system I, minimum distance and contact state label calculation; Among them, the robot capsule modeling simulates the different postures of the robot in the interaction process by setting the angle sequence of each joint of the six-axis robot during the interaction process, generates an action sequence model based on the angle sequence of the robot urdf file model, completes the reconstruction of the robot interaction model, and obtains the input data of the robot.

8. The method for detecting human-robot interaction safety based on convolutional neural network according to claim 7, characterized in that: The method for calculating the minimum distance and contact status label is specifically as follows: Traverse each vertex in the user's SMPL model and each part in the reconstructed robot interaction model, calculate the minimum distance, and use the VTK library to find the point in the user's SMPL model vertex that is closest to the robot model, and calculate the minimum distance and the point on the user's SMPL model that is closest to the robot; set the label of the data set according to the calculation result; finally, obtain the label of the data set, which includes the minimum interaction distance, interaction contact state and interaction contact position.

9. A human-robot interaction safety detection system based on convolutional neural network, characterized in that: The system is used to perform the detection method according to any one of claims 1 to 8, comprising: An imaging module, which is used to collect RGB-D images of the human-robot interaction environment and send them to the human joint point detection module and the human body model reconstruction network; The human joint point detection module detects the human joint point sequence based on the received RGB-D image and outputs it to the safety detection network; The human body model reconstruction network detects the user's body posture parameters based on the received RGB-D image and outputs them to the safety detection network; A robot, wherein the robot is deployed with a robot operating system, and the robot operating system obtains the robot joint position sequence in real time and sends it to the safety detection network; An output module generates operation instructions to the robot and / or generates warning information for display to the user according to the minimum distance and contact status between the human and the robot during the interaction process output by the safety detection network; A safety detection network, wherein the safety detection network further comprises a feature extraction module, a feature fusion module, an output optimization module and a task output module; The feature extraction module is one of a convolutional neural network, a graph convolutional neural network, an LSTM network and a Mamba model, receives an input human joint point sequence and a user posture parameter, and outputs human joint point sequence features, user posture parameter features and robot joint position sequence features to a feature fusion module; The feature fusion module includes a maximum pooling submodule, an average pooling submodule, a convolution submodule and a fully connected layer, and outputs the human-robot interaction features to the output optimization module after processing the input features; The output optimization module inputs the human-robot interaction features into the voting module and two fully connected layers respectively, and outputs the user virtual node, the robot virtual node and the position compensation vector; The task output module performs a voting operation on the human-robot interaction features, and votes out optimized user virtual nodes and robot virtual nodes along the direction of the feature compensation vector; the optimized user virtual nodes and robot virtual nodes are input into the distance calculation module, and the predicted minimum interaction distance is output.

Images