A vision-based safe robot interaction method

Abstract

The present invention discloses a vision-based robot safe interaction method, comprising: obtaining a head camera image and an external camera image; extracting key points of a human body according to the head camera image and the external camera image, detecting the distance between the human body and the robot according to the key points of the human body; detecting whether the distance between the human body and the robot is less than a safe distance; and driving the robot to perform a corresponding obstacle avoidance action according to the detection result, and controlling the robot to maintain the safe distance and continue to work. The present invention controls the robot to maintain a safe distance from the human body and continue to work at all times, thereby improving the safety of the robot's operation.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a vision-based robot safety interaction method. Background Art

[0002] Safety is a key feature of collaborative robots. Ensuring safe interaction between robots and humans is a key goal to improve the quality and efficiency of industrial robot production. The promotion and development of robots depends largely on safe human-robot interaction. With the widespread application of mobile robots in various industries, ensuring their safe interaction with humans has become an important research area.

[0003] Traditional robot interaction methods mainly rely on predefined rules and algorithms, which usually fail to take into account the dynamic and unpredictable nature of human behavior. In response to this challenge, the field of embodied perception in safe mobile robot interaction has received increasing attention, aiming to overcome these limitations. Improving perception accuracy and real-time performance, addressing complex and dynamic environmental factors, and ensuring data security and privacy are all pressing issues.

[0004] Therefore, the prior art needs to be improved. Summary of the invention

[0005] The technical problem to be solved by the present invention is that, in view of the defects of the prior art, the present invention provides a vision-based robot safety interaction method to solve the problem of low safety in the existing robot interaction method.

[0006] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0007] In a first aspect, the present invention provides a vision-based robot safety interaction method, comprising:

[0008] Acquire head camera images and external camera images;

[0009] Extracting key points of a human body according to the head camera image and the external camera image, and detecting the distance between the human body and the robot according to the key points of the human body;

[0010] Detecting whether the distance between the human body and the robot is less than a safe distance;

[0011] The robot is driven to perform corresponding obstacle avoidance actions according to the detection results, and the robot is controlled to maintain the safe distance and continue working.

[0012] In one implementation, the step of acquiring the head camera image and the external camera image includes:

[0013] Acquire an image of the front scene of the robot through the head camera of the robot to obtain the head camera image;

[0014] Acquire an image of the entire scene in which the robot is working through an external camera to obtain the external camera image;

[0015] The head camera image and the external camera image are depth-aligned images containing the human body.

[0016] In one implementation, extracting key points of a human body according to the head camera image and the external camera image, and detecting the distance between the human body and the robot according to the key points of the human body includes:

[0017] Input the head camera image and the external camera image into a human body detection model, and output a video stream with key points of the human body;

[0018] Extracting the human body key points from a video stream of the human body key points;

[0019] Point cloud coordinate conversion is performed according to the key points of the human body and the corresponding camera external parameter matrix, and the distance between the human body and the robot is detected according to the converted point cloud data.

[0020] In one implementation, performing point cloud coordinate conversion according to the key points of the human body and the corresponding camera external parameter matrix, and detecting the distance between the human body and the robot according to the converted point cloud data, includes:

[0021] The dual-target positioning method is used to derive the extrinsic parameter matrix between the head camera and the external camera;

[0022] Converting the point cloud corresponding to the key points of the human body in the external camera image to the coordinate system of the head camera according to the external parameter matrix;

[0023] The distance between the human body and the robot is detected according to the converted point cloud data.

[0024] In one implementation, the detecting whether the distance between the human body and the robot is less than a safety distance comprises:

[0025] Obtain the complete point cloud of the robot's working scene and the complete coordinate information of the human body point cloud, and set the safety distance according to the complete point cloud of the working scene and the complete coordinate information of the human body point cloud; wherein the safety distance is the distance from the coordinate system of the robot's head camera as the origin to the preset position.

[0026] In one implementation, the detecting whether the distance between the human body and the robot is less than a safety distance also includes:

[0027] When the robot is detected to be moving, the posture transformation relationship of the robot between adjacent moments is obtained according to the positioning and map building algorithm;

[0028] The point cloud data of the corresponding key points of the human body are updated according to the posture transformation relationship between the adjacent moments, and the distance between the human body and the robot is detected according to the updated point cloud data.

[0029] In one implementation, the driving the robot to perform a corresponding obstacle avoidance action according to the detection result, and controlling the robot to maintain the safe distance and continue working, includes:

[0030] When it is detected that the distance between the human body and the robot is less than the safety distance, a corresponding obstacle avoidance trajectory is planned according to the obstacle avoidance strategy of the working scene where the robot is located, and the robot is controlled to perform corresponding obstacle avoidance actions according to the obstacle avoidance trajectory, so as to control the robot to maintain the safety distance and continue working.

[0031] In a second aspect, the present invention provides a vision-based robot safety interaction device:

[0032] An acquisition module, used for acquiring head camera images and external camera images;

[0033] A human body detection module, used to extract key points of the human body according to the head camera image and the external camera image, and detect the distance between the human body and the robot according to the key points of the human body;

[0034] The decision module is used to detect whether the distance between the human body and the robot is less than the safe distance; drive the robot to perform corresponding obstacle avoidance actions according to the detection result, and control the robot to maintain the safe distance and continue working.

[0035] In a third aspect, the present invention provides a terminal comprising: a processor and a memory, wherein the memory stores a vision-based robot safety interaction program, and when the vision-based robot safety interaction program is executed by the processor, it is used to implement the operation of the vision-based robot safety interaction method as described in the first aspect.

[0036] In a fourth aspect, the present invention further provides a medium, which is a computer-readable storage medium, and which stores a vision-based robot safety interaction program. When the vision-based robot safety interaction program is executed by a processor, it is used to implement the operation of the vision-based robot safety interaction method as described in the first aspect.

[0037] The present invention adopts the above technical solution to achieve the following effects:

[0038] The present invention can extract key points of the human body according to the head camera image and the external camera image by acquiring the head camera image and the external camera image, and detect the distance between the human body and the robot according to the key points of the human body in real time, so that when the distance between the human body and the robot is less than the safe distance, the robot can be automatically driven to perform the corresponding obstacle avoidance action, and the robot can be controlled to maintain a safe distance and continue to work. The present invention can always control the robot to maintain a safe distance from the human body and continue to work, thereby improving the safety of the robot's operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0040] Figure 1 It is a flow chart of the vision-based safe robot interaction method in the present invention.

[0041] Figure 2 It is a schematic diagram of a robot safety interaction device in an application scenario of the present invention.

[0042] Figure 3 It is a schematic diagram of binocular positioning and posture change in the present invention.

[0043] Figure 4 It is a functional principle diagram of a terminal in one implementation of the present invention.

[0044] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0046] Exemplary Methods

[0047] Traditional robot interaction methods mainly rely on predefined rules and algorithms, which usually fail to take into account the dynamic and unpredictable nature of human behavior. In response to this challenge, the field of embodied perception in safe mobile robot interaction has received increasing attention, aiming to overcome these limitations. Improving perception accuracy and real-time performance, addressing complex and dynamic environmental factors, and ensuring data security and privacy are all pressing issues.

[0048] In view of the above technical problems, an embodiment of the present invention provides a vision-based robot safe interaction method, which can extract key points of the human body according to the head camera image and the external camera image by acquiring the head camera image and the external camera image, and detect the distance between the human body and the robot according to the key points of the human body in real time, so that when the distance between the human body and the robot is less than the safe distance, the robot can be automatically driven to perform the corresponding obstacle avoidance action, and the robot can be controlled to maintain a safe distance and continue to work. The present invention can always control the robot to maintain a safe distance from the human body and continue to work, thereby improving the safety of the robot's operation.

[0049] like Figure 1 As shown, an embodiment of the present invention provides a vision-based robot safety interaction method, comprising the following steps:

[0050] Step S100, acquiring a head camera image and an external camera image.

[0051] In this embodiment, a vision-based robot safety interaction device is provided, such as Figure 2 As shown, the vision-based robot safety interaction device is mainly divided into three modules: an image input module, a human detection module and a decision-making module; wherein the image input module is used to collect images in the robot working scene, and fuse the collected images to obtain a fused image centered on the robot; the human detection module is used to detect the human body in the image, so as to obtain the key points of the human body and the distance between the robot and the human body; the decision-making module is used to plan and control the robot to perform obstacle avoidance actions based on the key points and distances of the human body detected by the human detection module, when it is detected that the distance between the human body and the robot is within the safety distance, so as to maintain a safe distance and continue working.

[0052] Specifically, in an implementation of this embodiment, step S100 includes the following steps:

[0053] Step S101, acquiring an image of the front scene of the robot through the head camera of the robot to obtain the head camera image;

[0054] Step S102, acquiring an image of the entire scene in which the robot is working through an external camera to obtain the external camera image;

[0055] The head camera image and the external camera image are depth-aligned images containing the human body.

[0056] In this embodiment, the image of the front scene of the robot can be obtained through the head camera of the robot, and the image of the entire scene in which the robot is working can be obtained through the external camera; it is worth mentioning that the head camera and the external camera in this embodiment can capture aligned depth maps and RGB maps, that is, the depth maps captured by the two devices are aligned images, and the captured RGB images are also aligned images.

[0057] Furthermore, among the images captured by the head camera image and the external camera image, at least one device can capture an image containing a human body. For example, when the human body is at the side or behind the robot, the image of the front scene captured by the external camera does not contain the human body, while the image captured by the external camera at this time contains the human body.

[0058] After the image input module acquires the scene image of the front of the robot and the entire scene, the image is transmitted to the human body detection module for processing, so as to extract key points of the human body through the human body detection module.

[0059] like Figure 1 As shown, in one implementation of the embodiment of the present invention, the vision-based robot safety interaction method further includes the following steps:

[0060] Step S200, extracting key points of the human body according to the head camera image and the external camera image, and detecting the distance between the human body and the robot according to the key points of the human body.

[0061] In this embodiment, the YOLOv8 model is used as the detection model in the human body detection module. Compared with the traditional two-stage detection method, the YOLOv8 model achieves higher real-time performance. Its wide application fields include intelligent monitoring, autonomous driving and industrial detection.

[0062] In this embodiment, after the image input module obtains the scene image of the front of the robot and the entire scene image, the images collected by the two camera devices are used as input, and after being detected by the YOLOv8 model, a video stream with human key points is output.

[0063] Specifically, in an implementation of this embodiment, step S200 includes the following steps:

[0064] Step S201, inputting the head camera image and the external camera image into a human body detection model, and outputting a video stream with the key points of the human body;

[0065] Step S202, extracting the human body key points from the video stream of the human body key points;

[0066] Step S203, performing point cloud coordinate conversion according to the key points of the human body and the corresponding camera external parameter matrix, and detecting the distance between the human body and the robot according to the converted point cloud data.

[0067] In this embodiment, the YOLOv8 model adopts a one-stage detection method and regards the target detection task as an end-to-end regression problem. Through a single neural network model, the bounding boxes and category information of all targets in the image are directly output; wherein the category information output by the YOLOv8 model includes: whether there is a human body in the image, and the coordinate position of the corresponding human body in the input image.

[0068] Specifically, in an implementation of the present embodiment, point cloud coordinates are converted according to the key points of the human body and the corresponding camera external parameter matrix, and the distance between the human body and the robot is detected according to the converted point cloud data, including: deriving the external parameter matrix between the head camera and the external camera using a dual-target positioning method; converting the point cloud corresponding to the key points of the human body in the external camera image to the coordinate system of the head camera according to the external parameter matrix; and detecting the distance between the human body and the robot according to the converted point cloud data.

[0069] In this embodiment, after receiving the image from the image input module, a corresponding video stream is formed according to the collected image, and then the human detection module performs a series of operations: the key points of the human body are extracted from the scene by the human detection module, and the external parameter matrix between the two cameras (head camera and external camera) is derived by using dual-target positioning.

[0070] Specifically, the calibration process starts with a large calibration paper with a chessboard pattern, e.g. Figure 3 As shown. Each checkerboard corner point seen by the head camera and the external camera can be considered the same point in space. By identifying the corner point coordinates in the two depth camera coordinate systems, the relationship between them is calculated using the least squares method. This enables the pose to be determined between the A and B steps of the calibration process.

[0071] When calculating the relationship between the two coordinate systems of the head camera and the external camera, the transformation relationship between step A (the location of the external camera) and step B (the location of the head camera) is calculated using the least squares method. This transformation relationship is represented by a 3×3 rotation matrix and a 1×3 translation matrix. This transformation relationship is the posture of step A relative to step B.

[0072] When calculating the posture between step A and step B, the positional relationship can be calculated through the point clouds of the same group of points in the space in the two coordinate systems, and a transformation matrix (i.e., external parameter matrix) that can convert the point cloud in the external camera coordinate system to the head camera coordinate system and coincide with the point cloud in the head camera coordinate system is solved. This transformation matrix is ​​the relationship between the two coordinate systems.

[0073] According to the transformation matrix of the coordinate systems of the two cameras directly obtained above, the point cloud corresponding to the key points of the human body in the external camera image can be converted to the coordinate system of the head camera, and then the distance between the human body and the robot can be detected based on the converted point cloud data; specifically, the coordinate origin of the head camera is set to the position of the robot, and then, by obtaining the coordinate position of the key points of the human body in the coordinate system of the head camera, the distance between the human body and the robot can be calculated based on the coordinate position of the key points of the human body.

[0074] like Figure 1 As shown, in one implementation of the embodiment of the present invention, the vision-based robot safety interaction method further includes the following steps:

[0075] Step S300, detecting whether the distance between the human body and the robot is less than a safety distance.

[0076] In this embodiment, after the scene image of the front of the robot and the entire scene image are input into the YOLOv8 model, the complete point cloud of the scene and the complete coordinate information of the human body point cloud are obtained from the image, and then a safety distance and a safety area are set according to the obtained point cloud information and coordinate information, so that the system can judge the safety status of the human body.

[0077] Specifically, in an implementation of this embodiment, the following steps are included before step S300:

[0078] Step S301a, obtaining the complete point cloud of the robot's working scene and the complete coordinate information of the human body point cloud, and setting the safety distance according to the complete point cloud of the working scene and the complete coordinate information of the human body point cloud.

[0079] In this embodiment, the working range of the robot is an approximately spherical area, and its maximum working radius is given in the user manual. However, it is difficult to measure the coordinate value of the center point of the restricted area in the depth camera coordinate system by conventional methods. Therefore, in this embodiment, the restricted area is set by determining the center point coordinates of the restricted area in the point cloud and defining the radius of the sphere, and the restricted area is set by the center point coordinates and radius; wherein, the center point of the restricted area is the origin of the head camera coordinate system, and a sphere with a radius of a preset length (for example, 1.5m) is drawn with this origin as the center of the sphere, and the entire sphere is the restricted area.

[0080] After determining the restricted area, the safety distance is defined as the distance from the coordinate system of the robot's head camera as the origin to outside the restricted area (i.e., the preset position); it can be understood that the safety distance is greater than or equal to the radius of the restricted area, and the safety area is the area outside the set restricted area.

[0081] Specifically, in an implementation of this embodiment, the following steps are further included before step S300:

[0082] Step S301b, when the robot is detected to be moving, obtaining the posture transformation relationship of the robot between adjacent moments according to the positioning and map building algorithm;

[0083] Step S301c, updating the point cloud data of the corresponding key points of the human body according to the posture transformation relationship between the adjacent moments, and detecting the distance between the human body and the robot according to the updated point cloud data.

[0084] In this embodiment, through calibration and external parameter matrix, the transformation posture of the robot when it is stationary can be determined, that is, the posture between step A and step B of the calibration process can be determined; once the robot starts to move, the posture transformation relationship in the motion scene is calculated through the localization and mapping (SLAM) framework.

[0085] SLAM stands for simultaneous localization and mapping, which means simultaneous positioning and mapping. It can know its position relative to the initial point in real time when the robot moves. When the robot moves, SLAM allows the system to obtain the adjacent steps (such as Figure 3 As shown, the pose transformation between step B, step C and step D). Then, these incremental poses obtained by SLAM are multiplied with the initial pose (the pose of step A) to update the poses between step A, step B, step C and step D, which ensures that the positional relationship between the depth cameras is adjusted in real time to adapt to the movement of the robot.

[0086] After updating the robot posture of the motion scene through SLAM, the point cloud data of the corresponding human body key points are updated according to the transformation relationship of the posture, and the distance between the human body and the robot is detected according to the updated point cloud data; when updating the distance, the position of the human body key points in the head camera coordinate system is updated according to the updated point cloud data, and then the distance between the human body and the robot can be updated according to the updated coordinate position.

[0087] like Figure 1 As shown, in one implementation of the embodiment of the present invention, the vision-based robot safety interaction method further includes the following steps:

[0088] Step S400, driving the robot to perform corresponding obstacle avoidance actions according to the detection results, and controlling the robot to maintain the safe distance and continue working.

[0089] Specifically, in an implementation of this embodiment, step S400 includes the following steps:

[0090] Step S401, when it is detected that the distance between the human body and the robot is less than the safety distance, a corresponding obstacle avoidance trajectory is planned according to the obstacle avoidance strategy of the working scene where the robot is located, and the robot is controlled to perform corresponding obstacle avoidance actions according to the obstacle avoidance trajectory, so as to control the robot to maintain the safety distance and continue working.

[0091] In this embodiment, after determining the distance between the human body and the robot, the distance between the human body and the robot is evaluated by the decision module (i.e., whether the position of the human body from the depth camera is less than the set distance of 1.5m), and this information is transmitted to the robot through topic communication.

[0092] Specifically, during the evaluation process of the decision module, the decision module detects whether the distance between the human body and the robot is less than the safety distance. When it is detected that the distance between the human body and the robot is less than the safety distance, the corresponding obstacle avoidance trajectory is planned according to the pre-set obstacle avoidance strategy to control the robot to perform obstacle avoidance actions, thereby controlling the robot to maintain the safety distance and continue working.

[0093] This embodiment uses the advanced YOLO V8 in the human body detection module to implement the human body detection function, which greatly reduces the hardware cost of the robot. At the same time, the YOLO human body detection function has become stable, which can increase the stability of the system. In addition, the SLAM framework is used to update the posture changes between the robot and the external camera in real time during the movement, thereby providing more accurate judgment, ensuring that even if the robot is far away from the initial working position, it can still use the external camera to judge the distance between the human body and the robot in real time.

[0094] The present embodiment is described below through actual application scenarios:

[0095] This embodiment provides a vision-based robot safety interaction system, which uses sensors including: a robot head camera and an external camera that monitors the entire working scene. When the working robot head camera and the human body detection module detect that the distance between the human body and the robot is less than the safe distance, the robot will automatically move away from the human body and continue to work at a safe distance. If the human body is in an area that the robot head camera cannot observe (for example, to the side or rear of the robot), the external camera and the human body detection module in the working scene will detect this dangerous behavior and also determine whether the distance between the human body and the robot is less than the safe distance, thereby driving the robot to make corresponding obstacle avoidance actions and continue to work after reaching the safe distance.

[0096] This embodiment achieves the following technical effects through the above technical solution:

[0097] This embodiment of the present invention can extract key points of the human body according to the head camera image and the external camera image by acquiring the head camera image and the external camera image, and detect the distance between the human body and the robot according to the key points of the human body in real time, so that when the distance between the human body and the robot is less than the safe distance, the robot can be automatically driven to perform the corresponding obstacle avoidance action, and the robot can be controlled to maintain a safe distance and continue to work. This embodiment can always control the robot to maintain a safe distance from the human body and continue to work, thereby improving the safety of the robot's operation.

[0098] Exemplary Devices

[0099] Based on the above embodiments, the present invention further provides a vision-based robot safety interaction device, comprising:

[0100] An acquisition module, used for acquiring head camera images and external camera images;

[0101] A human body detection module, used to extract key points of the human body according to the head camera image and the external camera image, and detect the distance between the human body and the robot according to the key points of the human body;

[0102] The decision module is used to detect whether the distance between the human body and the robot is less than the safe distance; drive the robot to perform corresponding obstacle avoidance actions according to the detection result, and control the robot to maintain the safe distance and continue working.

[0103] This embodiment achieves the following technical effects through the above technical solution:

[0104] This embodiment of the present invention can extract key points of the human body according to the head camera image and the external camera image by acquiring the head camera image and the external camera image, and detect the distance between the human body and the robot according to the key points of the human body in real time, so that when the distance between the human body and the robot is less than the safe distance, the robot can be automatically driven to perform the corresponding obstacle avoidance action, and the robot can be controlled to maintain a safe distance and continue to work. This embodiment can always control the robot to maintain a safe distance from the human body and continue to work, thereby improving the safety of the robot's operation.

[0105] Based on the above embodiment, the present invention further provides a terminal, whose principle block diagram can be as follows: Figure 4 shown.

[0106] The terminal includes: a processor, a memory, an interface, a display screen and a communication module connected through a system bus; wherein the processor of the terminal is used to provide computing and control capabilities; the memory of the terminal includes a storage medium and an internal memory; the storage medium stores an operating system and a computer program; the internal memory provides an environment for the operation of the operating system and the computer program in the storage medium; the interface is used to connect to external devices; the display screen is used to display corresponding information; and the communication module is used to communicate with a cloud server or other devices.

[0107] When the computer program is executed by a processor, it is used to implement the operation of the vision-based robot safety interaction method.

[0108] It can be understood by those skilled in the art that Figure 4 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal to which the scheme of the present invention is applied. The specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0109] In one embodiment, a terminal is provided, which includes: a processor and a memory, the memory storing a vision-based robot safety interaction program, and the vision-based robot safety interaction program is used to implement the operation of the above-mentioned vision-based robot safety interaction method when executed by the processor.

[0110] In one embodiment, a storage medium is provided, wherein the storage medium stores a vision-based robot safety interaction program, and when the vision-based robot safety interaction program is executed by a processor, it is used to implement the operations of the above-mentioned vision-based robot safety interaction method.

[0111] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and volatile memory.

[0112] In summary, the present invention provides a vision-based robot safe interaction method, the method comprising: acquiring a head camera image and an external camera image; extracting key points of the human body according to the head camera image and the external camera image, and detecting the distance between the human body and the robot according to the key points of the human body; detecting whether the distance between the human body and the robot is less than a safe distance; and driving the robot to perform corresponding obstacle avoidance actions according to the detection results, and controlling the robot to maintain the safe distance and continue to work. The present invention controls the robot to maintain a safe distance from the human body and continue to work at all times, thereby improving the safety of the robot's operation.

[0113] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A vision-based robot safety interaction method, characterized in that: include: Acquire head camera images and external camera images; Extracting key points of a human body according to the head camera image and the external camera image, and detecting the distance between the human body and the robot according to the key points of the human body; Detecting whether the distance between the human body and the robot is less than a safe distance; According to the detection result, the robot is driven to perform corresponding obstacle avoidance actions, and the robot is controlled to maintain the safe distance and continue working; The step of acquiring the head camera image and the external camera image comprises: Acquire an image of the front scene of the robot through the head camera of the robot to obtain the head camera image; Acquire an image of the entire scene in which the robot is working through an external camera to obtain the external camera image; Wherein, the head camera image and the external camera image are images containing the human body that are depth-aligned; The extracting of key points of the human body according to the head camera image and the external camera image, and detecting the distance between the human body and the robot according to the key points of the human body, comprises: Input the head camera image and the external camera image into a human body detection model, and output a video stream with key points of the human body; Extracting the human body key points from a video stream of the human body key points; Performing point cloud coordinate conversion according to the key points of the human body and the corresponding camera external parameter matrix, and detecting the distance between the human body and the robot according to the converted point cloud data; identifying the coordinates of the corner points in the coordinate system of the head camera and the external camera, and calculating the posture between the position of the external camera and the position of the head camera in the calibration process by using the least squares method; The detecting whether the distance between the human body and the robot is less than the safety distance comprises: When the robot is detected to be moving, the posture transformation relationship between adjacent moments of the robot is obtained according to the positioning and mapping algorithm; when the robot moves, the posture transformation between adjacent steps is obtained based on the SLAM system, and the incremental posture obtained by the SLAM system is multiplied by the posture at the location of the external camera, and the posture at the location of the external camera and between the adjacent steps is updated; updating the point cloud data of the corresponding key points of the human body according to the posture transformation relationship between the adjacent moments, and detecting the distance between the human body and the robot according to the updated point cloud data; The step of driving the robot to perform a corresponding obstacle avoidance action according to the detection result, and controlling the robot to maintain the safe distance and continue working, comprises: When it is detected that the distance between the human body and the robot is less than the safety distance, a corresponding obstacle avoidance trajectory is planned according to the obstacle avoidance strategy of the working scene where the robot is located, and the robot is controlled to perform corresponding obstacle avoidance actions according to the obstacle avoidance trajectory, so as to control the robot to maintain the safety distance and continue working.

2. The vision-based robot safe interaction method according to claim 1, characterized in that: The step of performing point cloud coordinate conversion according to the key points of the human body and the corresponding camera external parameter matrix, and detecting the distance between the human body and the robot according to the converted point cloud data, comprises: The dual-target positioning method is used to derive the extrinsic parameter matrix between the head camera and the external camera; Converting the point cloud corresponding to the key points of the human body in the external camera image to the coordinate system of the head camera according to the external parameter matrix; The distance between the human body and the robot is detected according to the converted point cloud data.

3. The vision-based robot safe interaction method according to claim 1, characterized in that: The detecting whether the distance between the human body and the robot is less than the safety distance comprises: Obtain the complete point cloud of the robot's working scene and the complete coordinate information of the human body point cloud, and set the safety distance according to the complete point cloud of the working scene and the complete coordinate information of the human body point cloud; wherein the safety distance is the distance from the coordinate system of the robot's head camera as the origin to the preset position.

4. A vision-based robot safety interaction device, used to implement the vision-based robot safety interaction method according to any one of claims 1 to 3, characterized in that: include: An acquisition module, used for acquiring head camera images and external camera images; A human body detection module, used to extract key points of the human body according to the head camera image and the external camera image, and detect the distance between the human body and the robot according to the key points of the human body; The decision module is used to detect whether the distance between the human body and the robot is less than the safe distance; drive the robot to perform corresponding obstacle avoidance actions according to the detection result, and control the robot to maintain the safe distance and continue working.

5. A terminal, characterized in that: include: A processor and a memory, wherein the memory stores a vision-based robot safety interaction program, and when the vision-based robot safety interaction program is executed by the processor, it is used to implement the operation of the vision-based robot safety interaction method as described in any one of claims 1-3.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a vision-based robot safety interaction program, and when the vision-based robot safety interaction program is executed by a processor, it is used to implement the operation of the vision-based robot safety interaction method as described in any one of claims 1-3.

Images