Multi-sensor fusion human-machine distance estimation method based on digital space density grid

Abstract

The present invention discloses a multi-sensor fusion human-machine distance estimation method based on digital space density grid, which belongs to the field of robot intelligent perception, including: constructing a digital twin model of a human and a robot; constructing a density grid space, dividing the effective space of robot sensing detection into a set consisting of a certain number of grid units; developing a multimodal human body part recognition algorithm, estimating the human body part closest to the robot under each sensing channel in real time, and calculating the position of the part in the spatial density grid; constructing a data set based on the data detected by each sensing channel during the process of traversing all grid spaces; designing a density space neural network, and training a human-machine social distance estimation model. The present invention can improve the robot's distance perception accuracy to a human without modifying its own sensor, thereby realizing safer human-machine interaction based on distance.

Description

Technical Field

[0001] The present invention belongs to the field of robot intelligent perception, and in particular relates to a multi-sensor fusion human-machine distance estimation method based on digital space density grid. Background Art

[0002] As mobile robots, especially humanoid robots, are increasingly used in human-machine collaborative spaces, how to quickly and accurately identify humans based on the robot's existing sensor channels and calculate the distance between humans and machines in real time is an important prerequisite for ensuring safe human-machine collaboration.

[0003] However, mobile robots have a large workspace that may exceed the optimal detection range of a single sensor, making it difficult to accurately detect humans in the mission environment. In addition, the field of view of built-in camera sensors is usually narrow, which may cause the robot to miss important local feature information required to detect humans and maintain appropriate social distance. Finally, mobile robots usually use multiple sensors such as LiDAR and RGB-D cameras for human recognition. Due to the different detection ranges and limited fields of view, it becomes extremely challenging to fuse the information of these multimodal sensors to obtain more accurate human-machine distance detection.

[0004] Human-machine distance estimation has always been one of the research focuses of safety and state perception in the process of human-machine collaboration. Chinese patent application CN202011610213.8 discloses a human-machine distance perception method based on visual perception, but this method only relies on depth cameras, has a limited detection range, and is only suitable for fixed robots. Chinese patent application CN202110137126.3 discloses a cleaning robot motion control system and method based on multi-sensor fusion. This method is mainly used for obstacle detection, but cannot be directly applied to the perception of people in human-machine collaboration. Therefore, it is urgent to develop a multi-sensor fusion human-machine social distance estimation method by constructing a robot detection space density lattice. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a multi-sensor fusion human-machine distance estimation method based on digital space density grid, which calculates the position of the robot's digital space density lattice where the closest joint of the human-machine distance is located, and uses digital twin and multi-sensor information fusion technology to improve the accuracy of the robot's perception of people based on the robot's existing sensing level. Based on the method of updating the digital twin model by multi-sensor information fusion, the present invention improves the accuracy of the distance perception between the mobile robot and the human without changing the existing mobile robot sensor hardware, realizes efficient and safe human-machine collaborative operation, and is conducive to improving the efficiency of human-machine collaborative operation while ensuring human safety.

[0006] To achieve the above object, the present invention adopts the following technical solution:

[0007] The multi-sensor fusion human-machine distance estimation method based on digital space density grid includes the following steps:

[0008] S1. Build a digital twin model of humans and robots. The digital twin model includes the spatial geometric characteristics and kinematic relationship of the robot and humans.

[0009] S2, construct a density grid space, dividing the effective space for robot sensing detection into a set of a certain number of grid units;

[0010] S3. Design a multimodal human body part recognition algorithm. Based on the digital twin model and the data of each sensor channel, estimate the human body part closest to the robot in real time under each sensor channel, and calculate the position of the human body part in the dense grid space.

[0011] S4, constructing a data set based on the data detected by each sensor channel in the traversal density grid space by a person, and combining it with the human-machine distance recorded by the third-view scene camera in the scene, where the third-view scene camera is located at the perpendicular line of the plane where the center of the robot's detection range is located;

[0012] S5. Design a density space neural network, take the grid position of a person detected by each sensor channel when the person moves in the grid unit as input, take the human-machine distance detected by the third-person scene camera as output, train the human-machine social distance estimation model, and perform human-machine distance estimation.

[0013] Beneficial effects:

[0014] The present invention constructs a digital twin model of a person and a robot that includes the spatial geometric characteristics of the robot and the person and the kinematic relationship, and at the same time constructs a density grid space, divides the effective space of the robot's sensing detection into a certain number of grid units, and maps the human body part closest to the robot estimated in real time by the multimodal human body part recognition algorithm into the spatial density grid, thereby improving the accuracy of human-machine distance estimation. The present invention constructs a human-machine distance dataset in an offline spatial density grid, and uses it to train a human-machine social distance estimation model. Using this model, a mobile robot can improve its perception accuracy of people in the detection space, which is conducive to achieving efficient and safe human-machine collaboration. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A flowchart of a multi-sensor fusion human-machine distance estimation method based on digital space density grid provided by an embodiment of the present invention;

[0016] Figure 2 A schematic diagram of the spatial position relationship between a third-person perspective scene camera, a person, and a robot provided in an embodiment of the present invention;

[0017] Figure 3 A schematic diagram of the relationship between a spatial density grid, a person, and a robot in a digital space provided by an embodiment of the present invention;

[0018] Figure 4 This is a simulation effect diagram based on a multimodal human body recognition model in a digital space in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0020] like Figure 1 As shown, a multi-sensor fusion human-machine distance estimation method based on digital space density grid of the present invention comprises the following steps:

[0021] S1. Build a digital twin model of humans and robots. The digital twin model includes the spatial geometric characteristics and kinematic relationship of the robot and humans.

[0022] S2. Construct a dense grid space, dividing the effective space for robot sensing detection into a set of a certain number of grid units; the grid is the grid divided in the effective space for robot sensing detection, and the lattice is each smallest rectangular unit in the grid;

[0023] S3. Design a multimodal human body part recognition algorithm. Based on the digital twin model and the data of each sensor channel, estimate the human body part closest to the robot in real time under each sensor channel, and calculate the position of the part in the grid.

[0024] S4, based on the data detected by each sensor channel when a person traverses the dense grid space, and combined with the human-machine distance recorded by the third-person perspective scene camera in the scene, a data set is constructed;

[0025] S5. Design a density space neural network, take the grid position of the person detected by each channel moving in the grid as input, and take the human-machine distance detected by the third-person scene camera as output, and train the human-machine social distance estimation model.

[0026] Furthermore, the S1 includes:

[0027] Based on the three-dimensional geometric models of the human body and the robot, a collision bounding box model of the human and the robot is constructed, and the collision bounding box model is combined with the kinematic chain of the human and the robot to construct a kinematic relationship to achieve position information and corresponding joint angle mapping of the collision bounding box of the human and the robot to obtain the digital twin model.

[0028] Furthermore, the digital twin model uses the position information and joint angle information of the person recognized by the robot's vision, and the robot's joint angle information fed back by its own sensors as input to achieve real-time updates.

[0029] Furthermore, the kinematic relationship is constructed based on the kinematic chain of the human and the robot, and the front joint angle is known. , the three-dimensional coordinate information of the jth joint on the i-th kinematic chain for:

[0030] ;

[0031] Furthermore, the kinematic relationship It is expressed as:

[0032] ;

[0033] Among them, T represents the matrix transformation relationship between the joints solved based on the Denavit-Hartenberg parameters; i represents the kinematic chain, j represents the jth joint on the kinematic chain, and n represents the total number of joints on the i-th kinematic chain.

[0034] Furthermore, in S2, based on the detection performance characteristics of the sensor, the lattice length Δx along the x-axis and the lattice length Δy along the y-axis are gradually increased according to proportional constraints.

[0035] Furthermore, the lattice length Δx along the x-axis direction and the lattice length Δy along the y-axis direction follow the following formula:

[0036] ;

[0037] in, and Represents the default initial value of the lattice unit edge length. k x and k y are the adaptation coefficients in the x and y directions respectively. u and v are the encodings of the lattice units in the two directions. Indicates the preset increment in the y direction, Preset increment for edge length in the x direction.

[0038] Furthermore, the lattice length along the z-axis Divided in equal proportions according to sensor characteristics.

[0039] Furthermore, in step 3, when the multimodal human body part recognition algorithm calculates the human body joint closest to the robot in real time, the minimum distance between the human body joint and the robot is calculated as follows:

[0040] ;

[0041] in, represents the infimum, represents the Euclidean distance, is the minimum distance between the robot digital twin model and the virtual human bounding box corresponding to the jth joint, that is, the minimum distance between the human body part and the robot; for The point in for The point in For the The motion trajectory of the virtual human bounding box corresponding to each joint, is the motion trajectory of the robot’s digital twin model at time t.

[0042] According to the minimum distance between the human joint and the robot, the joint position where the minimum distance between the human and the robot is located is obtained.

[0043] Furthermore, after calculating the joint position where the human-machine minimum distance is located, the position of the joint position in the lattice can be calculated by the following formula:

[0044] ;

[0045] Among them, g, h, and l represent the lattice serial numbers in the three axes. (x, y, z) is the coordinate value corresponding to the center of the human body part. Once the lattice serial number is confirmed, the value of (x, y, z) of the human body part will be replaced by the coordinate value (x', y', z') corresponding to the lattice unit during the social distance calculation process.

[0046] Furthermore, the coordinate values ​​(x', y', z') are calculated as follows:

[0047] ;

[0048] Among them, k x and k y are the adaptation coefficients in the x and y directions respectively, is the default initial value for the lattice cell edge length along the z-axis.

[0049] Furthermore, the S4 includes: dividing the labeling machine based on the x-axis and y-axis grid on the ground of the scene, and placing a third-view scene camera in the scene, so that the human activity space (i.e., grid space) and the robot are both in the scene of the third-view scene camera, and letting the subject traverse all grids, recording the data detected by the sensor channels such as the laser radar and the airborne depth camera, and at the same time, the third-view camera records the distance between the human and the robot, and based on the grid coding, each sensor outputs the distance information of the human as the core parameter to construct a data set, such as Figure 2 shown.

[0050] Furthermore, in S5, by using the human-machine social distance estimation model, the robot can improve the accuracy of human-machine distance perception based on its own multimodal sensor data, thereby providing a guarantee for safe and efficient human-machine collaboration.

[0051] like Figure 3 As shown in the figure, the robot detection space is first divided into digital density grids to obtain the digital density grid of the robot detection space, and the movement of people in the physical space is mapped in the digital space. The specific implementation method is to use the robot's multimodal data, mainly the relevant data detected by sensors such as radar and depth camera, to calculate the human-machine distance information, and with the help of spatial density neural network training, obtain the estimation model of the social distance between humans and machines (i.e., human-machine distance), and realize the mapping of multimodal data to human-machine distance.

[0052] Specifically, the robot detection space is divided in the manner of a density grid space of the present invention, and the divided lattice is divided into cubes in equal parts. , , , , k x =1 and k y =1. ==0. The effective detection space of the robot is 1.4m×2.8m×2m (x, y, z directions). The embodiment uses the distance detection accuracy of the third perspective as a standard, and tests the human-machine distance detection accuracy using the method of the present invention and the Kalman filter for comparison.

[0053] Figure 4 The algorithm effect diagram of using laser radar data (people identified by the laser radar channel) and 3D visual point cloud data to identify the human body (people identified by the visual channel) is shown in the figure. The method of the present invention combines the results of the two recognitions and fuses them to obtain the bounding box of the human digital twin model, thereby calculating the minimum distance between the human and the machine and the human joints where the minimum distance is located, and mapping the results to the digital lattice of the robot detection space to achieve human-machine distance estimation. Experiments show that compared with the Kalman filter method, the present invention can improve the accuracy of human-machine distance detection by 31.4%.

Claims

1. A multi-sensor fusion human-machine distance estimation method based on digital space density grid, characterized in that: The steps include: S1. Build a digital twin model of humans and robots. The digital twin model contains the spatial geometric characteristics and kinematic relationship of the robot and humans, including: Based on the 3D geometric models of the human body and the robot, a collision bounding box model of the human and the robot is constructed; A kinematic relationship is constructed by combining the kinematic chains of the human and the robot to realize the mapping of the position information of the collision bounding box of the human and the robot and the corresponding joint angles, thereby obtaining the digital twin model; Using the position information and joint angle information of the person recognized by the robot vision and the robot joint angle information fed back by the robot sensor as input, the digital twin model is updated in real time; S2, construct a density grid space, dividing the effective space for robot sensing detection into a set of a certain number of grid units; S3. Design a multimodal human body part recognition algorithm. Based on the digital twin model and the data of each sensor channel, estimate the human body part closest to the robot in real time under each sensor channel, and calculate the position of the human body part in the dense grid space. S4, constructing a data set based on the data detected by each sensor channel in the traversal density grid space by a person, and combining it with the human-machine distance recorded by the third-view scene camera in the scene, where the third-view scene camera is located at the perpendicular line of the plane where the center of the robot's detection range is located; S5. Design a density space neural network, take the grid position of a person detected by each sensor channel when the person moves in the grid unit as input, take the human-machine distance detected by the third-person scene camera as output, train the human-machine social distance estimation model, and perform human-machine distance estimation.

2. The multi-sensor fusion human-machine distance estimation method based on digital space density grid as claimed in claim 1, characterized in that: In the kinematic relationship constructed by combining the kinematic chain of the human and the robot, the spatial position relationship of each joint point of the human and the robot is: ； in, Refers to the three-dimensional coordinate information of the jth joint on the i-th kinematic chain, is the current angle of the jth joint, f i Represents kinematic relationships; Kinematic relationship i It is expressed as: ； Among them, T represents the matrix transformation relationship between the joints solved based on the Denavit-Hartenberg parameters; i represents the sequence number of the kinematic chain, j represents the jth joint on the kinematic chain, and n represents the total number of joints on the i-th kinematic chain.

3. The multi-sensor fusion human-machine distance estimation method based on digital space density grid as claimed in claim 1, characterized in that: In S2, each smallest rectangular unit in the grid unit is a lattice. Based on the detection performance characteristics of the sensor, the lattice length along the x-axis direction is , lattice length along the y-axis Increase gradually according to the proportional constraints.

4. The multi-sensor fusion human-machine distance estimation method based on digital space density grid as claimed in claim 3 is characterized in that: The lattice length along the x-axis , lattice length along the y-axis Follow the formula below: ； in, and represents the default initial value of the lattice unit side length along the x-axis and the default initial value of the lattice unit side length along the y-axis. and are the adaptation coefficients in the x direction and the y direction, respectively; u and v are the IDs of the lattice cells in the x direction and the y direction; θ0 represents the preset increment in the y direction, Preset increment for edge length in x direction; ID indicates the name.

5. The multi-sensor fusion human-machine distance estimation method based on digital space density grid as claimed in claim 3 is characterized in that: The lattice length ∆z of the lattice along the z-axis direction is divided in equal proportion according to the sensor characteristics.

6. The multi-sensor fusion human-machine distance estimation method based on digital space density grid as claimed in claim 5, characterized in that: In the process of real-time estimation of the human body part closest to the robot under each sensor channel, the digital twin model of the robot is The minimum distance between the virtual human bounding boxes corresponding to the joints for: ； in, for The point in for The point in For the The motion trajectory of the virtual human bounding box corresponding to each joint, is the motion trajectory of the robot’s digital twin model at time t; represents the Euclidean distance; represents the infimum; According to the robot's digital twin model and The minimum distance between the virtual human bounding boxes corresponding to the joints Get the joint position where the human-machine minimum distance is located.

7. The multi-sensor fusion human-machine distance estimation method based on digital space density grid as claimed in claim 6, characterized in that: After S3 calculates the joint position where the human-machine minimum distance is located, the position of the joint position in the lattice is calculated by the following formula: ; Among them, g, h, and l represent the lattice serial numbers in the three axes; (x, y, z) is the coordinate value corresponding to the center of the human body part; once the lattice serial number is confirmed, the value of (x, y, z) of the human body part is replaced by the coordinate value (x', y', z') corresponding to the lattice unit during the social distance calculation process; Δz is the lattice length along the z-axis.

8. The multi-sensor fusion human-machine distance estimation method based on digital space density grid as claimed in claim 7, characterized in that: The coordinates (x', y', z') are calculated as follows: ； in, is the default initial value for the lattice cell edge length along the z-axis.

9. The multi-sensor fusion human-machine distance estimation method based on digital space density grid as claimed in claim 1, characterized in that: The S5 includes: using the human-machine social distance estimation model, the robot improves the accuracy of human-machine distance perception based on its own multimodal sensor data.

Images