A robot obstacle avoidance method and device based on vision

Abstract

The present application provides a robot obstacle avoidance method and device based on vision, the method comprising: obtaining first path planning data, and controlling the robot's end-of-arm tool to move along the first path; obtaining a video stream of the end-of-arm tool, the video stream being captured by a camera installed at the end of the robot arm; based on the video stream, identifying obstacles in the direction of the end-of-arm tool; based on the obstacles, planning a second path, and controlling the end-of-arm tool to move along the second path. In the present application, a camera is set to obtain a real-time video stream in front of the robot arm, thereby obtaining obstacle information for timely obstacle avoidance, thereby avoiding the occurrence of collisions.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a vision-based robot obstacle avoidance method and device. Background Art

[0002] The spinal surgery robot is a medical robot system specially designed to assist spinal surgery. By combining image guidance technology and robotics technology, it can provide high-precision and high-stability surgical operations, helping doctors perform spinal surgery more accurately.

[0003] However, the human spine has a large number of vertebrae, which in many cases causes the spatial situation of the entire human spine to be complex and changeable. Relying solely on the early panoramic images for the movement navigation of the robotic arm is prone to collisions. Summary of the invention

[0004] The problem that this application solves is that current spinal surgery robots only rely on preliminary image navigation for robotic arm navigation, which is prone to errors.

[0005] To solve the above problems, the first aspect of the present application provides a robot obstacle avoidance method based on vision, comprising:

[0006] Acquire first path planning data, and control the end tool of the robot's mechanical arm to move along the first path;

[0007] Acquire a video stream of the tool at the end of the robotic arm, where the video stream is captured by a camera installed at the end of the robotic arm;

[0008] Based on the video stream, identifying obstacles in the advancing direction of the tool at the end of the robot arm;

[0009] Based on the obstacle, a second path is planned, and the end tool of the robot arm is controlled to move along the second path.

[0010] The second aspect of the present application provides a vision-based robot obstacle avoidance device, which includes:

[0011] A first path planning module, which is used to obtain first path planning data and control the robot's end-of-arm tool to move along the first path;

[0012] A video stream acquisition module, which is used to acquire a video stream of the tool at the end of the robotic arm, wherein the video stream is captured by a camera installed at the end of the robotic arm;

[0013] An obstacle recognition module, which is used to identify obstacles in the advancing direction of the tool at the end of the robot arm based on the video stream;

[0014] A secondary path planning module is used to plan a second path based on the obstacle and control the end tool of the robot arm to move along the second path.

[0015] A third aspect of the present application provides an electronic device, comprising: a memory and a processor;

[0016] The memory is used to store programs;

[0017] The processor, coupled to the memory, is configured to execute the program to:

[0018] Acquire first path planning data, and control the end tool of the robot's mechanical arm to move along the first path;

[0019] Acquire a video stream of the tool at the end of the robotic arm, where the video stream is captured by a camera installed at the end of the robotic arm;

[0020] Based on the video stream, identifying obstacles in the advancing direction of the tool at the end of the robot arm;

[0021] Based on the obstacle, a second path is planned, and the end tool of the robot arm is controlled to move along the second path.

[0022] A fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the vision-based robot obstacle avoidance method described above.

[0023] In this application, a camera is set up to obtain a real-time video stream in front of the robotic arm, so as to obtain obstacle information and avoid obstacles in time, thereby avoiding the occurrence of collisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flow chart of a robot obstacle avoidance method according to an embodiment of the present application;

[0025] Figure 2 A model architecture diagram of a robot obstacle avoidance method according to an embodiment of the present application;

[0026] Figure 3 A flowchart of obstacle recognition of a robot obstacle avoidance method according to an embodiment of the present application;

[0027] Figure 4 A flowchart of secondary path planning of a robot obstacle avoidance method according to an embodiment of the present application;

[0028] Figure 5 A flowchart of obstacle marking of a robot obstacle avoidance method according to an embodiment of the present application;

[0029] Figure 6It is an architecture diagram of an edge attention module of a robot obstacle avoidance method according to an embodiment of the present application;

[0030] Figure 7 is a structural block diagram of a robot obstacle avoidance device according to an embodiment of the present application;

[0031] Figure 8 is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described in detail below in conjunction with the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0033] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.

[0034] In response to the above problems, the present application provides a new vision-based robot obstacle avoidance solution, which can avoid obstacles by obtaining a real-time video stream of the robotic arm, thereby solving the problem that the current spinal surgery robot only relies on preliminary image navigation for robotic arm navigation, which is prone to errors.

[0035] The present application embodiment provides a robot obstacle avoidance method based on vision, and the specific scheme of the method is as follows: Figure 1-Figure 6 As shown, the method can be performed by a vision-based robot obstacle avoidance device, which can be integrated into electronic devices such as computers, servers, computers, server clusters, and data centers. Figure 1 , Figure 2 As shown, it is a flow chart of a vision-based robot obstacle avoidance method according to an embodiment of the present application; wherein the vision-based robot obstacle avoidance method includes:

[0036] S100, acquiring first path planning data, and controlling a robot end tool of a robot arm to move along the first path;

[0037] In the present application, the first path planning data is obtained by planning based on the panoramic three-dimensional image during the operation. When the panoramic three-dimensional image is known, planning can be performed based on a graph search algorithm or a sampling method.

[0038] Graph search-based methods: These methods model the three-dimensional space as a graph, where nodes represent locations in the space and edges represent the connections between locations. Common graph search algorithms include A* algorithm, Dijkstra algorithm, and minimum spanning tree algorithm.

[0039] Graph search-based methods: These methods model the three-dimensional space as a graph, where nodes represent locations in the space and edges represent the connections between locations. Common graph search algorithms include A* algorithm, Dijkstra algorithm, and minimum spanning tree algorithm.

[0040] In the present application, the process of controlling the robot's end tool of the robotic arm to move along the first path can be: taking the point on the first path as the desired posture, performing inverse operation and solving in combination with the current posture of the robotic arm to obtain the desired angles of each joint of the robotic arm, and controlling each joint of the robotic arm through a PID controller based on the desired angles to achieve the purpose of moving along the first path.

[0041] S200, obtaining a video stream of the tool at the end of the robot arm, where the video stream is captured by a camera installed at the end of the robot arm;

[0042] In the present application, a camera device is installed at the end of the robotic arm, and the camera device faces the target position or the front position of the robotic arm to obtain obstacle information on the moving path.

[0043] S300, identifying obstacles in the advancing direction of the tool at the end of the robot arm based on the video stream;

[0044] In this application, the obstacles are obstacles that do not appear in the panoramic 3D image, or objects that have changed their positions in the panoramic 3D image. By identifying obstacles that did not appear in the original plan, secondary planning can be performed to avoid collisions with the robot arm and other problems that affect the normal movement of the robot arm.

[0045] S400: planning a second path based on the obstacle, and controlling the robot arm end tool to move along the second path.

[0046] In the present application, the planning and control of the first path are similar to the planning and control of the second path and can be implemented in a similar manner.

[0047] In this application, a camera is set up to obtain a real-time video stream in front of the robotic arm, so as to obtain obstacle information and avoid obstacles in time, thereby avoiding the occurrence of collisions.

[0048] It should be noted that obstacle recognition through only a single image may result in recognition errors; recognition through continuous video images / video streams and by utilizing the position changes between different frame images can eliminate possible errors and correctly identify obstacles.

[0049] In one embodiment, in combination Figure 3 As shown, the identifying of obstacles in the advancing direction of the tool at the end of the robot arm based on the video stream includes:

[0050] S301, obtaining continuous video images of a video stream, wherein the continuous video images include a video image at a current moment;

[0051] In this application, the camera device will continuously shoot video streams. Each video stream is essentially composed of a series of video images. The frame rate refers to the number of frames displayed per second in a video or animation. It indicates the playback speed of continuous images per unit time. The latest frame of video image is the video image at the current moment.

[0052] In this way, by acquiring the video image at the current moment, the corresponding image is acquired in real time; and by using a plurality of video images before the current moment, the video image at the current moment is supplemented to improve the recognition accuracy.

[0053] Furthermore, the frame rate of the video stream is between 15fps and 30fps, thereby increasing the position difference between two adjacent frames of video images and facilitating obstacle recognition.

[0054] S302, inputting the continuous video image into an obstacle recognition model to mark obstacles in the current video image;

[0055] It should be noted that in this step, the identified obstacle is an obstacle in front of the robot arm or on the motion path. The obstacle may be an obstacle in the panoramic three-dimensional image or an obstacle outside the panoramic three-dimensional image.

[0056] S303, acquiring a panoramic three-dimensional image during the operation, wherein the first path is planned based on the panoramic three-dimensional image;

[0057] The panoramic three-dimensional image during the operation can be a three-dimensional image of the inside of the human body, which is obtained by segmenting and reconstructing the preoperative CT image; it can also be a three-dimensional image of the outside of the human body, which is obtained by a depth camera installed at a higher position; it can also include both the inside and outside of the human body. The specific content can be determined by the navigation purpose. If the navigation purpose is only to navigate to the human appendage, only the three-dimensional image of the outside of the human body needs to be included. If the navigation purpose is to navigate to the incised spinal surface, the three-dimensional image of the inside of the human body is also required.

[0058] S304: When the marked obstacle does not belong to the panoramic three-dimensional image, identify the obstacle as an obstacle in the advancing direction of the tool at the end of the robot arm.

[0059] In the present application, after identifying the obstacle in the video stream, it is also necessary to confirm whether the obstacle is an existing obstacle in the panoramic three-dimensional image. If there is an existing obstacle, a first path is planned based on the obstacle, and the obstacle will not affect the normal movement of the robot arm, so there is no need to re-plan; if the obstacle is not an existing obstacle in the panoramic three-dimensional image (including that the obstacle is newly added to the surgical scene, or the obstacle has just moved in the panoramic three-dimensional image), then the obstacle is likely to affect the normal operation of the robot arm, and the path of the robot arm needs to be re-planned.

[0060] In this application, obstacles in front of the tool at the end of the robot arm are identified through video stream, so as to prevent the obstacles from affecting the normal movement of the robot arm and achieve the purpose of timely avoidance.

[0061] It should be noted here that as long as the robot arm and the end tool no longer move along the first path, any other actions can be considered as movement along the second path, such as emergency pause, emergency ascent, emergency descent, re-planning of the path and control, etc.

[0062] In one implementation, the number of frames of the continuous video images is four frames, so as to limit the amount of data processed by the obstacle recognition model.

[0063] In this application, only four frames of continuous video images are used as the basis for recognition, thereby reducing the number of video images that need to be input and greatly reducing the amount of data that the corresponding convolution model needs to process.

[0064] In one implementation, the continuous video images are continuous video images selected by frame skipping.

[0065] It should be noted that the other video images before the current moment are set based on the fact that objects between the images have certain differences, so that the position of obstacles is identified based on the differences; however, if the difference between the video images of two adjacent frames is too small, the identification may be wrong.

[0066] In the present application, frame skipping selection refers to selecting a frame every certain number of frames, for example, selecting a frame at the current moment, and then selecting the second frame, the fourth frame, and the sixth frame at the previous moment (selecting at intervals of one frame), or selecting the fourth frame, the eighth frame, and the twelfth frame at the previous moment (selecting at intervals of three frames).

[0067] In the present application, frame skipping is performed to increase the interval between two adjacent frames in the selected continuous video images, thereby increasing the position difference between the two adjacent frames of the video images and increasing the accuracy of obstacle recognition.

[0068] In one embodiment, in combination Figure 4 As shown, planning a second path based on the obstacle includes:

[0069] S401, acquiring data of each waypoint of a first path, where the first path is composed of consecutively arranged waypoints;

[0070] In the present application, the planning algorithm for a path composed of waypoints may be an RRT algorithm or an RRM algorithm.

[0071] RRT (Rapidly-exploring Random Trees): RRT is a sampling-based path planning algorithm that gradually expands the search tree by randomly sampling in free space until the target state is found. The tree structure generated by the RRT algorithm can be used to represent the robot's motion trajectory in the environment, and the final path is the path in the tree connecting the starting state and the target state.

[0072] PRM (Probabilistic Roadmap): The PRM algorithm is also a sampling-based path planning algorithm. It first randomly samples some nodes, then establishes connections between these nodes to build a graph structure. Then, it uses a graph search algorithm (such as Dijkstra or A*) to find the shortest path between the starting state and the target state in the graph.

[0073] In the present application, the first path is composed of multiple waypoints in sequence, so that, on the one hand, a better control effect can be achieved, and on the other hand, the amount of calculation in the entire path planning process can be saved.

[0074] S402, mapping the identified obstacles into the panoramic three-dimensional image, and determining dangerous waypoints in each waypoint data;

[0075] In the present application, the camera device is a depth camera. After identifying an obstacle, the specific coordinates in the coordinate system of the camera device can be obtained; the depth camera is fixed at the end of the robotic arm, and according to the posture information of the end of the robotic arm, the mapping relationship between the coordinate system of the depth camera and the coordinate system of the robotic arm can be determined, and based on the mapping relationship, the coordinates of the obstacle can be mapped to the coordinate system of the robotic arm.

[0076] The panoramic three-dimensional image can be a three-dimensional image in the coordinate system of the robotic arm, or a three-dimensional image in other set coordinate systems. However, according to the planning principle, the mapping relationship between the other coordinate system and the coordinate system of the robotic arm is known, and the coordinates of the obstacle can be mapped into the panoramic three-dimensional image based on the mapping relationship.

[0077] After the obstacle is mapped to the panoramic three-dimensional image, the danger space area and the obstacle space area can be generated based on the shape of the obstacle, or the obstacle space area can be directly generated.

[0078] The dangerous space area refers to the space where a collision will definitely occur when the robot arm passes through it; the obstacle space area refers to the space where a collision will occur when the robot arm passes through it in a certain posture, but no collision will occur when the robot arm passes through it in other postures.

[0079] Among them, the dangerous space area and the obstacle space area are both three-dimensional, and can be extended outward from the outer boundary of the obstacle by a preset length to serve as the dangerous space area and the obstacle space area (with different extension lengths).

[0080] In the present application, if among the waypoints on the first path, there is a waypoint located in the dangerous space area or the obstacle space area, then the waypoint is a dangerous waypoint.

[0081] In the present application, since obstacles are generally small, dangerous waypoints are generally continuous waypoints. For example, waypoints 3 to 10 of the first path are all located in the obstacle space area and are dangerous waypoints.

[0082] S403, taking the previous waypoint of the dangerous waypoint as a starting point and the next waypoint as an ending point, generating multiple new waypoints in the mapped panoramic three-dimensional image;

[0083] In the present application, the waypoints are continuous waypoints, and the waypoints are arranged from front to back as the previous waypoint of the dangerous waypoint, the dangerous waypoint, and the next waypoint of the dangerous waypoint; as described in the previous example, if waypoints 3 to 10 are dangerous waypoints, then waypoint 2 is the previous waypoint of the dangerous waypoint, and waypoint 11 is the next waypoint of the dangerous waypoint. In the mapped panoramic three-dimensional image, multiple waypoints are generated with the starting point being waypoint 2 and the end point being waypoint 11, and these waypoints are used to replace the dangerous waypoints of waypoints 3 to 10.

[0084] S404: Combine the non-dangerous waypoints of the first path and the generated multiple waypoints to obtain the second path.

[0085] In the present application, the second path is obtained by combining the non-dangerous waypoints of the first path and the generated multiple waypoints; thus, only the dangerous part of the path needs to be replanned, and there is no need to replan the entire path, which greatly reduces the calculation amount of path planning.

[0086] In one embodiment, in combination Figure 5 and Figure 2 As shown, S302, inputting the continuous video image into an obstacle recognition model to mark obstacles in the current video image, including:

[0087] S201, down-sampling the continuous video image to obtain a down-sampled video feature map;

[0088] S202, inputting the downsampled video feature map into an edge attention module to obtain a first edge feature map;

[0089] S203, performing convolution processing on the first edge feature map and inputting the convolution processing into a second edge attention module to obtain a second edge feature map;

[0090] S204, performing convolution processing on the second edge feature map respectively to obtain two first edge branch feature maps;

[0091] S205, performing mean processing on the two first edge branch feature maps, and then performing convolution processing on each of them to obtain two second edge branch feature maps;

[0092] S206, performing mean processing on the two second edge branch feature maps and then performing convolution processing to obtain the marked position of the obstacle.

[0093] like Figure 2 As shown in the figure, the continuous video image is 4#512×512, the downsampled video feature map is 32#256×256, the first edge feature map is 64#128×128, and the first edge feature map is convolved to obtain a feature map of 64#128×128. The feature map is processed by the edge attention module to obtain a second edge feature map of 64#128×128. Through different convolutions, two first edge branch feature maps are obtained, and after averaging, two second edge branch feature maps are obtained through different convolutions. After averaging and convolution again, the marked position of the obstacle is obtained.

[0094] In this application, the traditional Unet architecture is abandoned in the obstacle recognition model because the Unet method requires multiple downsampling and multiple upsampling for feature extraction. This feature extraction method requires a lot of calculation and time. When the robot avoids obstacles, real-time performance is required. If the calculation time of the obstacle recognition process is too long, the robot will be unable to avoid obstacles after discovering the obstacle because the remaining obstacle avoidance time is too short. Therefore, the convolution and edge attention modules are used. Although the recognition accuracy will decrease, it will greatly reduce the recognition time, thereby avoiding obstacles in time.

[0095] In the present application, multiple continuous video images are input to compensate for the reduced accuracy of the obstacle recognition model with a simple structure, thereby avoiding recognition errors.

[0096] In the present application, the input video image is directly downsampled, thereby further reducing the amount of calculation and improving the calculation speed.

[0097] In the present application, the marked obstacles are marked in the video image at the current moment, thereby enhancing the real-time nature of the obstacle marking.

[0098] It should be noted that in the obstacle recognition model, this application does not indicate the fully connected layer, activation function and other information therein, but this does not mean that the above content does not exist in the obstacle recognition model, but that the corresponding marking is omitted. In the specific implementation, it is necessary to select the corresponding fully connected layer, activation function, etc. according to the actual situation.

[0099] In one embodiment, in combination Figure 6 As shown, the step S202, inputting the downsampled video feature map into the edge attention module to obtain a first edge feature map, includes:

[0100] Based on the horizontal operator and the vertical operator, feature extraction is performed on the video feature map to obtain a horizontal operator feature map and a vertical operator feature map respectively;

[0101] Fusing the horizontal operator feature map and the vertical operator feature map based on the weight to obtain a fused operator feature map;

[0102] The fusion operator feature map is added to the video feature map to obtain the first edge feature map.

[0103] In the present application, by setting the horizontal operator and the vertical operator, the edge features of the video feature map are extracted from different directions, and by fusing the horizontal and vertical directions, the integrity and adaptability of the edge features of the video feature map are improved.

[0104] Combination Figure 6 As shown, the horizontal canny operator on the left is the x-direction feature extraction operator, and the vertical canny operator on the right is the y-direction feature extraction operator; it can be seen that the x-direction feature extraction operator and the y-direction feature extraction operator are both 3x3 templates.

[0105] The horizontal canny operator and the vertical canny operator are respectively:

[0106]

[0107] Among them, when feature extraction is performed through the x-direction feature extraction operator and the y-direction feature extraction operator, the 3x3 template can be used as the kernel to perform convolution and operation with each pixel point in the bone joint image to be identified, and then a suitable threshold is selected to extract the edge; edge feature extraction can also be achieved through other methods, and the specific extraction method will not be repeated in this application.

[0108] In this way, the edge features are covered in the x direction and the y direction (with a larger angle), reducing the similarity of the features extracted in the two directions, thereby improving the coverage of the edge features.

[0109] In this application, different weights are assigned to different operator feature maps, and weighted processing is performed to solve problems such as blurred edges and low contrast in some images.

[0110] In this application, the addition is ADD processing.

[0111] In one implementation, before inputting the continuous video images into the obstacle recognition model and marking obstacles in the current video image, the method further includes:

[0112] Acquire a sample image, where the sample image is a continuous video image, and an obstacle is marked on the last video image;

[0113] Input the sample image into the obstacle recognition model to obtain the predicted obstacle;

[0114] Calculate the overall loss of the obstacle recognition model based on the marked obstacles and predicted obstacles;

[0115] The obstacle recognition model is iterated based on the overall loss until the overall loss converges.

[0116] In this way, the obstacle recognition model is trained through sample images, so that obstacle recognition can be performed directly through the trained obstacle recognition model.

[0117] In one embodiment, the calculation formula of the overall loss is:

[0118]

[0119] in, For the overall loss, Multi-class cross entropy loss for target classification, is the smooth L1 loss, and λ is a hyperparameter that balances the weights of the two.

[0120] An embodiment of the present application provides a vision-based robot obstacle avoidance device, which is used to execute the vision-based robot obstacle avoidance method described in the above content of the present application. The vision-based robot obstacle avoidance device is described in detail below.

[0121] like Figure 7 As shown, the vision-based robot obstacle avoidance device comprises:

[0122] A first path planning module 101, which is used to obtain first path planning data and control the robot's end-of-arm tool to move along the first path;

[0123] A video stream acquisition module 102 is used to acquire a video stream of the tool at the end of the robotic arm, wherein the video stream is captured by a camera installed at the end of the robotic arm;

[0124] An obstacle recognition module 103, which is used to recognize obstacles in the advancing direction of the tool at the end of the robot arm based on the video stream;

[0125] The secondary path planning module 104 is used to plan a second path based on the obstacle and control the robot end tool to move along the second path.

[0126] In one implementation, the obstacle identification module 103 is further configured to:

[0127] Acquire continuous video images of a video stream, wherein the continuous video images include a video image at a current moment; input the continuous video images into an obstacle recognition model to mark obstacles in the current video image; acquire a panoramic three-dimensional image during the operation, wherein the first path is planned based on the panoramic three-dimensional image; and when the marked obstacle does not belong to the panoramic three-dimensional image, identify the obstacle as an obstacle in the forward direction of the tool at the end of the robot arm.

[0128] In one implementation, the number of frames of the continuous video images is four frames, so as to limit the amount of data processed by the obstacle recognition model.

[0129] In one implementation, the continuous video images are continuous video images selected by frame skipping.

[0130] In one embodiment, the secondary path planning module 104 is further configured to:

[0131] The method comprises obtaining the data of each waypoint of a first path, wherein the first path is composed of the waypoints arranged continuously; mapping the identified obstacles into a panoramic three-dimensional image, and determining the dangerous waypoints in the data of each waypoint; taking the waypoint preceding the dangerous waypoint as a starting point and the waypoint following the dangerous waypoint as an ending point, and generating a plurality of new waypoints in the mapped panoramic three-dimensional image; and combining the non-dangerous waypoints of the first path and the generated plurality of waypoints to obtain the second path.

[0132] In one implementation, the obstacle identification module 103 is further configured to:

[0133] The continuous video image is downsampled to obtain a downsampled video feature map; the downsampled video feature map is input into an edge attention module to obtain a first edge feature map; the first edge feature map is convolved and input into a second edge attention module to obtain a second edge feature map; the second edge feature maps are convolved to obtain two first edge branch feature maps; the two first edge branch feature maps are averaged and then convolved to obtain two second edge branch feature maps; the two second edge branch feature maps are averaged and then convolved to obtain the marked position of the obstacle.

[0134] In one implementation, the obstacle identification module 103 is further configured to:

[0135] Based on the horizontal operator and the vertical operator, feature extraction is performed on the video feature map to obtain a horizontal operator feature map and a vertical operator feature map respectively; based on the weight, the horizontal operator feature map and the vertical operator feature map are fused to obtain a fused operator feature map; the fused operator feature map is added to the video feature map to obtain the first edge feature map.

[0136] The vision-based robot obstacle avoidance device provided in the above-mentioned embodiment of the present application has a corresponding relationship with the vision-based robot obstacle avoidance method provided in the embodiment of the present application. Therefore, the specific content in the device has a corresponding relationship with the robot obstacle avoidance method. The specific content can refer to the records in the robot obstacle avoidance method, and will not be repeated in this application.

[0137] The vision-based robot obstacle avoidance device provided in the above-mentioned embodiments of the present application and the vision-based robot obstacle avoidance method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0138] The above describes the internal functions and structure of the vision-based robot obstacle avoidance device, such as Figure 8 As shown, in practice, the vision-based robot obstacle avoidance device can be implemented as an electronic device, including: a memory 301 and a processor 303.

[0139] The memory 301 may be configured to store programs.

[0140] In addition, the memory 301 may also be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc.

[0141] The memory 301 can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0142] The processor 303 is coupled to the memory 301 and is configured to execute a program in the memory 301 to:

[0143] Acquire first path planning data, and control the end tool of the robot's mechanical arm to move along the first path;

[0144] Acquire a video stream of the tool at the end of the robotic arm, where the video stream is captured by a camera installed at the end of the robotic arm;

[0145] Based on the video stream, identifying obstacles in the advancing direction of the tool at the end of the robot arm;

[0146] Based on the obstacle, a second path is planned, and the end tool of the robot arm is controlled to move along the second path.

[0147] In one implementation, the obstacle identification module 103 is further configured to:

[0148] Acquire continuous video images of a video stream, wherein the continuous video images include a video image at a current moment; input the continuous video images into an obstacle recognition model to mark obstacles in the current video image; acquire a panoramic three-dimensional image during the operation, wherein the first path is planned based on the panoramic three-dimensional image; and when the marked obstacle does not belong to the panoramic three-dimensional image, identify the obstacle as an obstacle in the forward direction of the tool at the end of the robot arm.

[0149] In one implementation, the number of frames of the continuous video images is four frames, so as to limit the amount of data processed by the obstacle recognition model.

[0150] In one implementation, the continuous video images are continuous video images selected by frame skipping.

[0151] In one embodiment, the secondary path planning module 104 is further configured to:

[0152] The method comprises obtaining the data of each waypoint of a first path, wherein the first path is composed of the waypoints arranged continuously; mapping the identified obstacles into a panoramic three-dimensional image, and determining the dangerous waypoints in the data of each waypoint; taking the waypoint preceding the dangerous waypoint as a starting point and the waypoint following the dangerous waypoint as an ending point, and generating a plurality of new waypoints in the mapped panoramic three-dimensional image; and combining the non-dangerous waypoints of the first path and the generated plurality of waypoints to obtain the second path.

[0153] In one implementation, the obstacle identification module 103 is further configured to:

[0154] The continuous video image is downsampled to obtain a downsampled video feature map; the downsampled video feature map is input into an edge attention module to obtain a first edge feature map; the first edge feature map is convolved and input into a second edge attention module to obtain a second edge feature map; the second edge feature maps are convolved to obtain two first edge branch feature maps; the two first edge branch feature maps are averaged and then convolved to obtain two second edge branch feature maps; the two second edge branch feature maps are averaged and then convolved to obtain the marked position of the obstacle.

[0155] In one implementation, the obstacle identification module 103 is further configured to:

[0156] Based on the horizontal operator and the vertical operator, feature extraction is performed on the video feature map to obtain a horizontal operator feature map and a vertical operator feature map respectively; based on the weight, the horizontal operator feature map and the vertical operator feature map are fused to obtain a fused operator feature map; the fused operator feature map is added to the video feature map to obtain the first edge feature map.

[0157] In the present application, the processor is also specifically used to execute all processes and steps of the above-mentioned vision-based robot obstacle avoidance method. The specific content can be referred to the records in the robot obstacle avoidance method, and will not be repeated in this application.

[0158] In this application, Figure 8 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 8 Components shown.

[0159] The electronic device provided in this embodiment is based on the same inventive concept as the vision-based robot obstacle avoidance method provided in the embodiment of the present application, and has the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0160] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0161] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0162] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0163] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0164] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0165] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash memory (Flash RAM). Memory is an example of a computer-readable medium.

[0166] The present application also provides a computer-readable storage medium corresponding to the vision-based robot obstacle avoidance method provided in the aforementioned embodiment, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it will execute the vision-based robot obstacle avoidance method provided in any of the aforementioned embodiments.

[0167] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0168] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the vision-based robot obstacle avoidance method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0169] It should be noted that in the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

[0170] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0171] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A robot obstacle avoidance method based on vision, characterized in that: include: Acquire first path planning data, and control the end tool of the robot's mechanical arm to move along the first path; Acquire a video stream of the tool at the end of the robotic arm, where the video stream is captured by a camera installed at the end of the robotic arm; Based on the video stream, identifying obstacles in the advancing direction of the tool at the end of the robot arm; Based on the obstacle, planning a second path, and controlling the end tool of the robot arm to move along the second path; The step of identifying obstacles in the advancing direction of the tool at the end of the robot arm based on the video stream includes: Acquire continuous video images of a video stream, wherein the continuous video images include a video image at a current moment; Inputting the continuous video image into an obstacle recognition model to mark obstacles in the current video image; Acquire a panoramic three-dimensional image during surgery, wherein the first path is planned based on the panoramic three-dimensional image; In the case where the marked obstacle does not belong to the panoramic three-dimensional image, identifying the obstacle as an obstacle in the advancing direction of the tool at the end of the robot arm; Inputting the continuous video image into an obstacle recognition model to mark obstacles in the current video image includes: Downsampling the continuous video image to obtain a downsampled video feature map; The downsampled video feature map is input into the edge attention module to obtain a first edge feature map; After convolution processing, the first edge feature map is input into a second edge attention module to obtain a second edge feature map; Perform convolution processing on the second edge feature map respectively to obtain two first edge branch feature maps; After performing mean processing on the two first edge branch feature maps, convolution processing is performed on each of them to obtain two second edge branch feature maps; After averaging the two second edge branch feature maps, the marked position of the obstacle is obtained through convolution processing.

2. The robot obstacle avoidance method according to claim 1, characterized in that: The number of frames of the continuous video images is four, so as to limit the amount of data processed by the obstacle recognition model.

3. The robot obstacle avoidance method according to claim 1, characterized in that: The continuous video images are continuous video images selected by frame skipping.

4. The robot obstacle avoidance method according to any one of claims 1 to 3, characterized in that: The planning of the second path based on the obstacle includes: Acquire data of each waypoint of a first path, wherein the first path is composed of consecutively arranged waypoints; Mapping the identified obstacles into the panoramic 3D image and determining the dangerous waypoints in the individual waypoint data; Taking the previous waypoint of the dangerous waypoint as the starting point and the next waypoint as the ending point, a plurality of new waypoints are generated in the mapped panoramic three-dimensional image; The non-dangerous waypoints of the first path and the generated multiple waypoints are combined to obtain the second path.

5. The robot obstacle avoidance method according to any one of claims 1 to 3, characterized in that: The step of inputting the downsampled video feature map into the edge attention module to obtain a first edge feature map comprises: Based on the horizontal operator and the vertical operator, feature extraction is performed on the video feature map to obtain a horizontal operator feature map and a vertical operator feature map respectively; Fusing the horizontal operator feature map and the vertical operator feature map based on the weight to obtain a fused operator feature map; The fusion operator feature map is added to the video feature map to obtain the first edge feature map.

6. A vision-based robot obstacle avoidance device, characterized in that: include: A first path planning module, which is used to obtain first path planning data and control the robot's end-of-arm tool to move along the first path; A video stream acquisition module, which is used to acquire a video stream of the tool at the end of the robotic arm, wherein the video stream is captured by a camera installed at the end of the robotic arm; An obstacle recognition module, which is used to identify obstacles in the advancing direction of the tool at the end of the robot arm based on the video stream; A secondary path planning module, which is used to plan a second path based on the obstacle and control the end tool of the robot arm to move along the second path; The step of identifying obstacles in the advancing direction of the tool at the end of the robot arm based on the video stream includes: Acquire continuous video images of a video stream, wherein the continuous video images include a video image at a current moment; Inputting the continuous video image into an obstacle recognition model to mark obstacles in the current video image; Acquire a panoramic three-dimensional image during surgery, wherein the first path is planned based on the panoramic three-dimensional image; In the case where the marked obstacle does not belong to the panoramic three-dimensional image, identifying the obstacle as an obstacle in the advancing direction of the tool at the end of the robot arm; Inputting the continuous video image into an obstacle recognition model to mark obstacles in the current video image includes: Downsampling the continuous video image to obtain a downsampled video feature map; The downsampled video feature map is input into the edge attention module to obtain a first edge feature map; After convolution processing, the first edge feature map is input into a second edge attention module to obtain a second edge feature map; Perform convolution processing on the second edge feature map respectively to obtain two first edge branch feature maps; After performing mean processing on the two first edge branch feature maps, convolution processing is performed on each of them to obtain two second edge branch feature maps; After averaging the two second edge branch feature maps, the marked position of the obstacle is obtained through convolution processing.

7. An electronic device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor, coupled to the memory, is configured to execute the program to: Acquire first path planning data, and control the end tool of the robot's mechanical arm to move along the first path; Acquire a video stream of the tool at the end of the robotic arm, where the video stream is captured by a camera installed at the end of the robotic arm; Based on the video stream, identifying obstacles in the advancing direction of the tool at the end of the robot arm; Based on the obstacle, planning a second path, and controlling the end tool of the robot arm to move along the second path; The step of identifying obstacles in the advancing direction of the tool at the end of the robot arm based on the video stream includes: Acquire continuous video images of a video stream, wherein the continuous video images include a video image at a current moment; Inputting the continuous video image into an obstacle recognition model to mark obstacles in the current video image; Acquire a panoramic three-dimensional image during surgery, wherein the first path is planned based on the panoramic three-dimensional image; In the case where the marked obstacle does not belong to the panoramic three-dimensional image, identifying the obstacle as an obstacle in the advancing direction of the tool at the end of the robot arm; Inputting the continuous video image into an obstacle recognition model to mark obstacles in the current video image includes: Downsampling the continuous video image to obtain a downsampled video feature map; The downsampled video feature map is input into the edge attention module to obtain a first edge feature map; After convolution processing, the first edge feature map is input into a second edge attention module to obtain a second edge feature map; Perform convolution processing on the second edge feature map respectively to obtain two first edge branch feature maps; After performing mean processing on the two first edge branch feature maps, convolution processing is performed on each of them to obtain two second edge branch feature maps; After averaging the two second edge branch feature maps, the marked position of the obstacle is obtained through convolution processing.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the vision-based robot obstacle avoidance method described in any one of claims 1-5.

Images