Obstacle detection method and device, electronic equipment and readable storage medium

Abstract

The invention relates to an obstacle detection method and device, electronic equipment and a readable storage medium. The method comprises the steps of obtaining a to-be-detected image shot by a robot; inputting the to-be-detected image into an obstacle detection model, and determining key point information corresponding to a grounding wire in the to-be-detected image and semantic segmentation information corresponding to the to-be-detected image through the obstacle detection model; determining first pixel information belonging to the ground wire in the to-be-detected image based on the key point information, and determining second pixel information belonging to the ground wire in the to-be-detected image based on the semantic segmentation information; determining a target pixel belonging to the obstacle based on the first pixel information and the second pixel information; and determining a predicted position of the obstacle relative to the robot based on the target pixel, and determining a predicted category of the obstacle based on the semantic segmentation information corresponding to the target pixel. By adopting the method, the types of the detectable obstacles can be increased, so that the detection precision of the obstacles is improved.

Description

Technical Field

[0001] The present application relates to the field of robotics, and in particular to an obstacle detection method, device, electronic device, and computer-readable storage medium. Background Art

[0002] With the development of robotics technology, robots can not only work under human control, but also autonomously analyze the surrounding environment and perform some work based on the surrounding environment. For example, when encountering an obstacle, the robot can analyze the current distance to the obstacle and determine the driving route to bypass the obstacle. However, in the current obstacle avoidance method, the robot may not be able to bypass some special objects, such as objects with reflectivity within a specific range, which may cause the robot to collide with such obstacles. Summary of the invention

[0003] Based on this, it is necessary to provide an obstacle detection method, device, computer equipment, computer-readable storage medium and computer program product that can increase the types of detectable obstacles and thereby improve the obstacle detection accuracy in order to address the above technical problems.

[0004] In a first aspect, the present application provides a method for detecting an obstacle, comprising:

[0005] Obtain the image to be detected taken by the robot;

[0006] Inputting the image to be detected into an obstacle detection model, determining key point information corresponding to the ground line in the image to be detected and semantic segmentation information corresponding to each pixel in the image to be detected through the obstacle detection model, wherein the semantic segmentation information is used to characterize the category of each pixel in the image to be detected;

[0007] Determine first pixel information belonging to the ground line in the image to be detected based on the key point information, and determine second pixel information belonging to the ground line in the image to be detected based on the semantic segmentation information;

[0008] Determining a target pixel belonging to an obstacle based on the first pixel information and the second pixel information;

[0009] A predicted position of the obstacle relative to the robot is determined based on the target pixel, and a predicted category of the obstacle is determined based on semantic segmentation information corresponding to the target pixel.

[0010] In one embodiment, after determining the predicted position of the obstacle relative to the robot based on the target pixel and determining the predicted category of the obstacle based on semantic segmentation information corresponding to the target pixel, the method further comprises:

[0011] Obtaining the actual position of the obstacle on the obstacle map based on the mapping table and the predicted position, obtaining the actual distance between the robot and the obstacle based on the actual position, and determining the driving area of ​​the robot based on the actual distance and the obstacle map, wherein the mapping table is used to represent the mapping relationship between the pixel coordinate system where the position information is located and the world coordinate system where the actual distance is located;

[0012] If the obstacle is determined to be a dynamic obstacle based on the predicted category, the attenuation coefficient of the obstacle on the obstacle map is set to a first attenuation coefficient; if the obstacle is determined to be a static obstacle based on the predicted category, the attenuation coefficient of the obstacle on the obstacle map is set to a second attenuation coefficient, and the first attenuation coefficient is greater than the second attenuation coefficient.

[0013] In one embodiment, the key point information includes detection frame information and key pixel points, and determining the first ground line pixel information in the image to be detected based on the key point information includes:

[0014] Connecting key pixel points in the detection frame corresponding to the detection frame information in sequence to obtain a connection line;

[0015] The first pixel information is obtained based on the pixel points of the connecting line.

[0016] In one embodiment, before obtaining the image to be detected, the process includes:

[0017] Acquire a training sample, wherein the training sample includes a labeled image and labeled information of a ground line in the labeled image;

[0018] Based on the ground line annotation points and the ground line annotation categories in the annotation information, the annotation category of each pixel in the annotated image is obtained to obtain semantic segmentation annotation information of the annotated image, and based on the ground line annotation points, key point annotation information of the ground line in the annotated image is obtained;

[0019] The semantic network of the model to be trained is trained based on the annotated image and the corresponding semantic segmentation annotation information, and the key point network of the model to be trained is trained based on the annotated image and the corresponding key point annotation information to obtain the obstacle detection model.

[0020] In one embodiment, the training of the semantic network of the to-be-trained model based on the annotated image and the corresponding semantic segmentation information includes:

[0021] Obtaining semantic segmentation prediction information of the annotated image based on the semantic network of the network to be trained;

[0022] Substituting the semantic segmentation prediction information and the semantic segmentation annotation information into a semantic loss function, wherein the semantic loss function includes a first loss function and a second loss function, wherein the first loss function is used to adjust the similarity between the semantic segmentation prediction information and the semantic segmentation annotation information, and the second loss function is used to adjust the difficulty balance of the training sample;

[0023] The semantic network is trained based on the semantic loss function.

[0024] In one embodiment, the key point annotation information includes a first type of key point annotation information and a second type of key point annotation information, the first type of key point annotation information is used to annotate the key point information of the first type of grounding wire, the number of key points on the first type of grounding wire is a first number, the second type of key point annotation information is used to annotate the key point information of the second type of grounding wire, the number of key points on the second type of grounding wire is a second number, the first number is less than the second number, the key point network includes a first key point network and a second key point network, the first key point network is used to train the first type of grounding wire, and the second key point network is used to train the second type of grounding wire, and the training of the key point network of the model to be trained based on the annotated image and the corresponding key point annotation information includes:

[0025] Training the first key point network based on the first type of key point annotation information and the corresponding annotated image;

[0026] The second key point network is trained based on the second type of key point annotation information and the corresponding annotated images.

[0027] In one embodiment, the training of the first key point network based on the first type of key point annotation information and the corresponding annotated image includes:

[0028] Predicting a first type of key points in the labeled image based on the first key point network to obtain first type of key point prediction information;

[0029] Substituting the first-category key point prediction information and the first-category key point annotation information into the first-category key point loss function;

[0030] Training the first key point network based on the first type of key point loss function;

[0031] The training of the second key point network based on the second type of key point annotation information and the corresponding annotated image includes:

[0032] Predicting the second type of key points in the labeled image based on the second key point network to obtain second type of key point prediction information;

[0033] Substituting the second-category key point prediction information and the second-category key point annotation information into the second-category key point loss function;

[0034] The second key point network is trained based on the second type of key point loss function.

[0035] In a second aspect, the present application also provides an obstacle detection device, comprising:

[0036] An image acquisition module is used to obtain the image to be detected taken by the robot;

[0037] An information prediction module, used for inputting the image to be detected into an obstacle detection model, and determining key point information corresponding to the ground line in the image to be detected and semantic segmentation information corresponding to each pixel in the image to be detected through the obstacle detection model, wherein the semantic segmentation information is used to characterize the category of each pixel in the image to be detected;

[0038] An information processing module, configured to determine first pixel information belonging to a ground line in the image to be detected based on the key point information, determine second pixel information belonging to the ground line in the image to be detected based on the semantic segmentation information; and determine a target pixel belonging to an obstacle based on the first pixel information and the second pixel information;

[0039] The obstacle information detection module is used to determine the predicted position of the obstacle relative to the robot based on the target pixel, and to determine the predicted category of the obstacle based on the semantic segmentation information corresponding to the target pixel.

[0040] In a third aspect, the present application further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0041] Obtain the image to be detected taken by the robot;

[0042] Inputting the image to be detected into an obstacle detection model, determining key point information corresponding to the ground line in the image to be detected and semantic segmentation information corresponding to each pixel in the image to be detected through the obstacle detection model, wherein the semantic segmentation information is used to characterize the category of each pixel in the image to be detected;

[0043] Determine first pixel information belonging to the ground line in the image to be detected based on the key point information, and determine second pixel information belonging to the ground line in the image to be detected based on the semantic segmentation information;

[0044] Determining a target pixel belonging to an obstacle based on the first pixel information and the second pixel information;

[0045] A predicted position of the obstacle relative to the robot is determined based on the target pixel, and a predicted category of the obstacle is determined based on semantic segmentation information corresponding to the target pixel.

[0046] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0047] Obtain the image to be detected taken by the robot;

[0048] Inputting the image to be detected into an obstacle detection model, determining key point information corresponding to the ground line in the image to be detected and semantic segmentation information corresponding to each pixel in the image to be detected through the obstacle detection model, wherein the semantic segmentation information is used to characterize the category of each pixel in the image to be detected;

[0049] Determine first pixel information belonging to the ground line in the image to be detected based on the key point information, and determine second pixel information belonging to the ground line in the image to be detected based on the semantic segmentation information;

[0050] Determining a target pixel belonging to an obstacle based on the first pixel information and the second pixel information;

[0051] A predicted position of the obstacle relative to the robot is determined based on the target pixel, and a predicted category of the obstacle is determined based on semantic segmentation information corresponding to the target pixel.

[0052] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0053] Obtain the image to be detected taken by the robot;

[0054] Inputting the image to be detected into an obstacle detection model, determining key point information corresponding to the ground line in the image to be detected and semantic segmentation information corresponding to each pixel in the image to be detected through the obstacle detection model, wherein the semantic segmentation information is used to characterize the category of each pixel in the image to be detected;

[0055] Determine first pixel information belonging to the ground line in the image to be detected based on the key point information, and determine second pixel information belonging to the ground line in the image to be detected based on the semantic segmentation information;

[0056] Determining a target pixel belonging to an obstacle based on the first pixel information and the second pixel information;

[0057] A predicted position of the obstacle relative to the robot is determined based on the target pixel, and a predicted category of the obstacle is determined based on semantic segmentation information corresponding to the target pixel.

[0058] The obstacle detection method, device, computer equipment, computer-readable storage medium and computer program product described above obtain an image to be detected taken by a robot, input the image to be detected into an obstacle detection model, and determine the key point information corresponding to the grounding line in the image to be detected and the semantic segmentation information corresponding to each pixel in the image to be detected through the obstacle detection model. In this way, the first pixel information belonging to the grounding line in the image to be detected can be determined based on the key point information, and the second pixel information belonging to the grounding line in the image to be detected can be determined based on the semantic segmentation information. That is to say, the first pixel information and the second pixel information both correspond to the information of the grounding line in the image to be detected, and then based on the first pixel information, the grounding line information of the first pixel information and the second pixel information are determined. The target pixel belonging to the obstacle is determined by the grounding line information determined by the first pixel information and the second pixel information, so that the reliability of obstacle detection can be improved. Since the target pixel corresponds to the point where the obstacle contacts the ground, the predicted position of the obstacle relative to the robot can be determined based on the target pixel, and since the semantic segmentation information corresponding to the image to be detected can be obtained through the obstacle detection model, the category of each pixel in the image to be detected can be obtained based on the semantic segmentation information, so the predicted category of the obstacle can be obtained based on the semantic segmentation information corresponding to the target pixel, and then the predicted category and predicted position can be input into the downstream task to realize the versatility of the robot, thereby improving the obstacle avoidance function of the robot. In addition, since the robot processes the captured image and determines the obstacles around the robot by analyzing the grounding line in the image, the robot can identify more types of obstacles, thereby improving the robot's detection accuracy of obstacles. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0060] FIG1 is a schematic flow chart of a method for detecting an obstacle in one embodiment;

[0061] FIG2 is a schematic diagram of a flow chart of the training steps of an obstacle detection model in one embodiment;

[0062] FIG3 is a structural block diagram of an obstacle detection device in one embodiment;

[0063] FIG4 is a diagram showing the internal structure of a computer device in one embodiment;

[0064] FIG. 5 is a diagram showing the internal structure of an electronic device in one embodiment. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0066] In an exemplary embodiment, a method for detecting obstacles is provided. This embodiment uses the method applied to a terminal as an example. It is understandable that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and can be implemented through the interaction between the terminal and the server, or can be implemented through the interaction between the terminal and the terminal. Among them, the terminal can be a computer device, such as a robot, a smart wearable device, a self-moving balancing car, etc.; it is understandable that the robot can be a delivery robot (commercial delivery robot, industrial delivery robot, AGV (Automated Guided Vehicle, automatic navigation vehicle) robot), a guide robot, a disinfection robot, a cleaning robot, an inspection robot, and other self-moving robots, which are not specifically limited here. The server can be an independent physical server or a service node in a blockchain system. The service nodes in the blockchain system form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on the Transmission Control Protocol (TCP) protocol. In addition, the server can also be a server cluster composed of multiple physical servers, and can be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms. The robot and the server can be connected through Bluetooth, USB (Universal Serial Bus) or network communication connection methods, and this application does not limit this.

[0067] In the prior art, RGBD sensors are usually used to achieve obstacle avoidance between autonomous mobile robots and obstacles. The robot can use the RGBD sensor to obtain the depth information between itself and the obstacle, thereby obtaining the distance between itself and the obstacle, and then determine its own moving route. In this implementation, the ability to avoid obstacles mainly depends on the performance of the RGBD sensor hardware itself. However, the RGBD sensor may not be able to detect certain special objects, such as objects with high reflectivity (such as metal bases), objects with low reflectivity (such as glass, black objects), and obstacles with small volumes (such as pens and keys). This is because there may be noise in the depth image obtained by the RGBD sensor, and small pens, keys, etc. may be mistaken for noise, which leads to the inability to detect such small obstacles. In addition, the field of view of the RGBD sensor in the prior art may be small, with a maximum horizontal field of view of about 120 degrees and a maximum vertical field of view of about 70 degrees, which may result in a single RGBD sensor being unable to monitor obstacles in the 180-degree three-dimensional space in front of the robot, and multiple RGBD sensors need to be superimposed to achieve a certain degree of coverage. However, installing multiple RGBD sensors on the robot will make the cost of the robot too high. Therefore, in order to solve or improve the above technical problems, the present application proposes the following embodiments.

[0068] In an exemplary embodiment, as shown in FIG1 , a method for detecting obstacles is provided, and the method is applied to a robot, and can be executed by a processor of the robot, for example, and includes the following steps 102 to 110. Among them:

[0069] Step S102, obtaining an image to be detected taken by the robot.

[0070] The image to be detected taken by the robot may be an image obtained by taking a fisheye camera. A fisheye camera is a camera with a fisheye lens, which is characterized by an extremely short focal length and a viewing angle close to or equal to 180 degrees, and usually uses a 16mm or shorter lens. In this way, the image to be detected taken by the robot can monitor obstacles in the 180-degree three-dimensional space in front of the robot, which can improve the detection range of obstacles.

[0071] Exemplarily, the robot's camera device sends the captured image to be detected to the robot's processor so that the robot's processor can process the image to be detected. This can reduce the time for uploading the image to the cloud for analysis, thereby improving the robot's real-time detection efficiency.

[0072] Step S104: input the image to be detected into the obstacle detection model, and determine the key point information corresponding to the ground line in the image to be detected and the semantic segmentation information corresponding to the image to be detected through the obstacle detection model.

[0073] In order to improve the detection rate of obstacles in the image to be detected, an obstacle detection model is proposed. The obstacle detection model uses deep learning technology to accurately identify the ground wire in the image. In this way, the obstacles around the robot can be determined based on the ground wire in the image. The ground wire refers to the line between the obstacle and the ground. In the obstacle detection model, the model includes two networks, which are used to analyze the key point information of the ground wire in the image and the semantic segmentation information corresponding to the image. The semantic segmentation information can be used to characterize the category of each pixel in the image to be detected, that is, the semantic segmentation information can be used to determine the category of each pixel belonging to the ground wire in the image to be detected, and then the category of the ground wire can be determined based on the category of each pixel belonging to the ground wire. The pixels in the image to be detected can be divided into ground wire pixels and background pixels. For background pixels, their category is background. For ground wire pixels, since there may be multiple categories of ground wires, there may also be multiple categories of ground wire pixels, and they correspond to the categories of ground wires. For example, the category of ground wire pixels can be human feet, metal, glass, pen, key, etc. The category of ground wire is used to indicate the object to which the ground wire belongs, such as human feet, metal, glass, pen, key, etc. That is to say, the two networks included in the obstacle detection model can be used to predict the key point information and category information of the ground line in the image to be detected respectively.

[0074] Among them, the key point information may include the position of the key point on the ground line (such as which pixel in the image is the key point of the ground line) and the bounding box information of the ground line. The bounding box information of the ground line is a rectangular box used to describe the position and range of the ground line in the image. The rectangular box can be defined by the coordinates of the upper left corner and the lower right corner of the box, or by the coordinates of the center point, width and height of the box. The bounding box information can be used to better mark and locate each ground line in the image.

[0075] Optionally, for different types of grounding wires, the number of key points corresponding thereto may be different. For example, the number of key points may be determined by the volume, complexity, and shape characteristics of the obstacle. For example, there are 15 key points for the grounding wires of a metal base, glass, or a human foot, and there are 5 key points for the grounding wires of a wheel, a thin stool leg, a pen, or a key. The shape characteristics of the obstacle can be determined by the number of key points.

[0076] Exemplarily, the processor inputs the image to be detected into the obstacle detection model, and determines the key point information corresponding to the ground line in the image to be detected and the semantic segmentation information corresponding to the image to be detected through the obstacle detection model.

[0077] Step S106, determining first pixel information belonging to the ground line in the image to be detected based on the key point information, and determining second pixel information belonging to the ground line in the image to be detected based on the semantic segmentation information.

[0078] The first pixel information may include the pixel coordinate position in the image corresponding to each pixel point on each ground line in the image to be detected. For example, the first pixel information corresponding to a ground line in the image is "(234,234), (235,234), (236,235), ...". The second pixel information may include the corresponding category of each pixel point on each ground line in the image to be detected. For example, the category of pixel point A1 on ground line A is metal, the category of pixel point A2 is metal, the category of pixel point A3 is glass, the category of pixel point A4 is metal, and so on...

[0079] For example, since the key point information can describe the pixel positions of the key points of each grounding line in the image, the first pixel information belonging to the grounding line in the image to be detected can be determined based on the key point information. And since the semantic segmentation information can describe the category information of each pixel in the image, for the pixel corresponding to the non-grounding line, the corresponding category is background, so the second pixel information belonging to the grounding line in the image to be detected can be determined based on the semantic segmentation information.

[0080] In one implementation, after obtaining the key point information, the key points of each ground line may be connected in sequence in a certain order, so that a whole ground line may be obtained, and then the first pixel information belonging to the ground line may be obtained based on the ground line.

[0081] In one implementation, after obtaining the semantic segmentation information, the category of each pixel is obtained, and the second pixel information belonging to the ground line is obtained based on each pixel that does not belong to the background and its corresponding category.

[0082] Step S108: determining a target pixel belonging to the obstacle based on the first pixel information and the second pixel information.

[0083] The target pixel belonging to the obstacle refers to a pixel that belongs to the obstacle in the image.

[0084] In this step, the target pixel belonging to the obstacle is determined by the first pixel information and the second pixel information. Exemplarily, the first pixel information stores the pixels belonging to the ground line, and the second pixel information stores the categories of the pixels belonging to the ground line. If for a certain pixel, the first pixel information does not store the information of the pixel or the second pixel information does not store the category of the pixel, then the pixel does not belong to the obstacle. When the first pixel information and the second pixel information both store the information of a certain pixel, the pixel is determined to be the target pixel belonging to the obstacle.

[0085] Step S110, determining a predicted position of the obstacle relative to the robot based on the target pixel, and determining a predicted category of the obstacle based on semantic segmentation information corresponding to the target pixel.

[0086] For example, the pixel position of the obstacle in the image can be determined based on the target pixels attributable to the obstacle, and then the pixel position of the obstacle in the image can be converted by distance measurement to obtain the predicted position of the obstacle relative to the robot in the world coordinate system.

[0087] In some embodiments, the target pixel in the pixel coordinate system may be processed based on the mapping table to obtain the predicted position of the obstacle in the world coordinate system. The mapping table is used to represent the mapping relationship between the pixel coordinate system and the world coordinate system.

[0088] Furthermore, based on the semantic segmentation information corresponding to the target pixel, the category corresponding to the target pixel can be determined, thereby determining the predicted category of the obstacle.

[0089] In some embodiments, for a certain ground line, the ground line includes multiple target pixels, and the semantic segmentation information corresponding to each target pixel may be different. For example, the ground line includes 100 target pixels, of which 60 target pixels are predicted to be glass, 30 target pixels are predicted to be metal, and 10 target pixels are predicted to be wheels. Then, when determining the predicted category of the obstacle based on the semantic segmentation information determined by the target pixels, the predicted category of the obstacle can be determined based on voting, that is, the category with the largest number is selected as the predicted category of the obstacle. In the above example, the predicted category of the obstacle corresponding to the ground line is glass.

[0090] In the above obstacle detection method, by obtaining the image to be detected taken by the robot, the image to be detected is input into the obstacle detection model, and the key point information corresponding to the ground line in the image to be detected and the semantic segmentation information corresponding to each pixel in the image to be detected are determined by the obstacle detection model. In this way, the first pixel information belonging to the ground line in the image to be detected can be determined based on the key point information, and the second pixel information belonging to the ground line in the image to be detected can be determined based on the semantic segmentation information. That is to say, the first pixel information and the second pixel information both correspond to the information of the ground line in the image to be detected, and then the target pixel belonging to the obstacle is determined based on the ground line information determined based on the first pixel information and the second pixel information. In this way, the reliability of obstacle detection can be improved. Since the target pixel corresponds to the point where the obstacle contacts the ground, the predicted position of the obstacle relative to the robot can be determined based on the target pixel, and since the semantic segmentation information corresponding to the image to be detected can be obtained by the obstacle detection model, the category of each pixel in the image to be detected can be obtained based on the semantic segmentation information, so the predicted category of the obstacle can be obtained based on the semantic segmentation information corresponding to the target pixel, and then the predicted category and predicted position can be input into the downstream task to realize the versatility of the robot, thereby improving the obstacle avoidance function of the robot. In addition, since the robot processes the captured images and determines the obstacles around the robot by analyzing the ground lines in the images, the robot can recognize more types of obstacles, thereby improving the robot's obstacle detection accuracy.

[0091] After determining the category and position of the obstacle based on the obstacle detection model, the robot's obstacle avoidance strategy or drivable area can be determined based on the category and position of the obstacle. In one embodiment, the robot determines the predicted position of the obstacle relative to the robot based on the target pixel, and after determining the predicted category of the obstacle based on the semantic segmentation information corresponding to the target pixel, the process includes steps 202 to 204, wherein:

[0092] Step S202: determining the position of the obstacle on the obstacle map based on the predicted position, so as to determine the movement area of ​​the robot based on the obstacle map.

[0093] This step is a post-processing step for the predicted position.

[0094] For example, after obtaining the predicted positions of obstacles around the robot, the robot can mark these obstacles on the obstacle map based on the predicted positions, that is, determine the positions of the obstacles on the obstacle map based on the predicted positions. In this way, the distribution of obstacles around the robot can be obtained based on the obstacle map, and then the movement area of ​​the robot can be determined based on the obstacle map. Among them, when the robot moves in the movement area, it will not collide with obstacles.

[0095] Step S204: If the obstacle is determined to be a dynamic obstacle based on the predicted category, the attenuation coefficient of the obstacle on the obstacle map is set to a first attenuation coefficient; if the obstacle is determined to be a static obstacle based on the predicted category, the attenuation coefficient of the obstacle on the obstacle map is set to a second attenuation coefficient, and the first attenuation coefficient is greater than the second attenuation coefficient.

[0096] This step is a post-processing step for the predicted category.

[0097] Among them, dynamic obstacles refer to objects that can move autonomously, such as people. Static obstacles refer to objects that are generally in a stationary state, such as fixed tables, fixed glass doors, etc. The attenuation coefficient is used to characterize the time it takes for an obstacle to disappear on the obstacle map. If the attenuation coefficient is high, it means that the corresponding obstacle stays on the obstacle map for a short time. If the attenuation coefficient is low, it means that the corresponding obstacle stays on the obstacle map for a long time. The establishment of the obstacle map is to enable the robot to fully understand the obstacles in the preset area. This is because the images captured by the camera device will continue to change during the movement of the robot. For example, after the robot passes an obstacle, the camera device of the robot may no longer capture the obstacle. At this time, if the position of the obstacle can be recorded, the obstacles encountered by the robot in the preset area can be fully understood, and then the robot's movement route can be reasonably planned to reduce the probability of collision between the robot and the obstacle.

[0098] Since dynamic obstacles and static obstacles have different moving speeds, different attenuation coefficients need to be set for dynamic obstacles and static obstacles on the obstacle map. In other words, a dynamic obstacle may only pass through a preset area and will not appear around the robot for a long time in the future. Therefore, the attenuation coefficient of a dynamic obstacle is higher, that is, the time it stays on the obstacle map is shorter. For static obstacles, such as a door, after the robot passes through this door, although the glass door may not appear in the field of view captured in real time for a period of time, since the door is stationary, the subsequent robot needs to bypass this door when it goes back to avoid collision. Therefore, the attenuation coefficient of static obstacles on the obstacle map is lower.

[0099] Therefore, after obtaining the predicted position and predicted category of the obstacle, the robot can also perform the above-mentioned post-processing to improve the robot's obstacle avoidance accuracy.

[0100] In some embodiments, in order to reduce the misjudgment of static obstacles, the category of the obstacle can also be verified. Optionally, the real-time position of the robot is used to use multiple frames to confirm whether the obstacle is a static obstacle to reduce the false detection rate. For example, the robot can process two frames of images obtained successively, and map the obstacles identified in the two frames of images to the obstacle map. If the position of the static obstacle in the two frames in the obstacle map does not change, it can be determined that the obstacle is a static obstacle, and then the corresponding attenuation coefficient can be set for the obstacle.

[0101] In this embodiment, by post-processing the predicted category and predicted position after obtaining the obstacle, the information of the obstacle on the obstacle map, such as the position and attenuation coefficient, can be determined based on the predicted category and predicted position of the obstacle. In this way, the robot can determine the movable area based on the obstacle map, which can not only improve the accuracy of obstacle avoidance, but also improve the efficiency of the robot in determining the moving route.

[0102] In some embodiments, the attenuation coefficient of a dynamic obstacle can be determined according to the moving speed of the dynamic obstacle. For example, the moving speed and the attenuation coefficient can be in a linear relationship. The faster the moving speed, the greater the attenuation coefficient. The slower the moving speed, the smaller the attenuation coefficient. The moving speed can be determined according to the predicted category of the obstacle and the distance between the obstacles after two frames of image mapping. If the predicted category of the obstacle is a wheel and the distance between the location of the obstacle after the first frame of image mapping and the location of the obstacle after the second frame of image mapping is large, it means that the obstacle is a dynamic obstacle and the moving speed is fast, and it can be quickly attenuated on the obstacle map. If the predicted category of the obstacle is a human foot and the distance between the location of the obstacle after the first frame of image mapping and the location of the obstacle after the second frame of image mapping is small, it means that the obstacle is a dynamic obstacle and the moving speed is slow, and it can be slowly attenuated on the obstacle map. In this way, determining the attenuation coefficient of a dynamic obstacle by moving speed can improve the accuracy and rationality of setting the attenuation coefficient, thereby improving the accuracy of the obstacle map, and then planning an efficient and correct driving path.

[0103] In some embodiments, the obstacle detection model outputs key point information of the grounding line. In order to obtain the corresponding grounding line information, the key point information needs to be processed. Exemplarily, the processor can determine the first pixel information belonging to the grounding line in the image to be detected based on the key point information.

[0104] Among them, the first pixel information may refer to the pixel information of the grounding wire in the image to be detected, for example, which pixels in the image to be detected constitute a certain grounding wire. The key point information is the key point information corresponding to the grounding wire obtained by the obstacle detection model. For example, for the grounding wire of a piece of glass in the image, the key point information obtained may be 15 pixel coordinates, the upper left corner coordinates and the lower right corner coordinates of the bounding box. It can be understood that for an image, the key point information may include multiple groups, each group corresponding to the key point information of a grounding wire. Optionally, the key point information may include bounding box information and key pixel points. The bounding box corresponding to the bounding box information may be the minimum bounding rectangle surrounding the key pixel points, and the bounding box information may be the coordinates of the upper left corner and the lower right corner of the minimum bounding rectangle, or the coordinates and width and height of the center point of the minimum bounding rectangle. The key pixel points may be the pixel coordinates of the key points corresponding to the grounding wire. The number of key pixel points corresponding to different types of grounding wires is different.

[0105] For example, after obtaining the key point information, multiple bounding boxes in the image to be detected can be obtained based on the bounding box information. Each bounding box is used to locate the corresponding ground line, so the key pixel points in the bounding box corresponding to the bounding box information can be connected in sequence to obtain a connecting line, that is, the ground line; and then the first pixel information is obtained based on the pixel points of the connecting line.

[0106] In this embodiment, by sequentially connecting key pixel points in the bounding box corresponding to the bounding box information to obtain connecting lines, and obtaining the grounding line through connecting lines based on key points, the grounding line recognition process can be simplified and the grounding line recognition efficiency can be improved.

[0107] It is understandable that before using the obstacle detection model to detect the ground wire in the image to be detected, it is necessary to train an initial model using training samples to obtain the target obstacle detection model. Among them, it is also necessary to use a loss function to constrain the learning results of the model so that the generated obstacle detection model can correctly predict the category and position of the ground wire in the image, and then can be used to determine the obstacle information around the robot. Therefore, before obtaining the image to be detected, it is necessary to train an initial model to obtain an obstacle detection model. In a possible implementation, as shown in Figure 2, before obtaining the image to be detected, steps 302 to 306 are included. Among them:

[0108] Step S302, obtaining training samples.

[0109] The training sample includes the annotated image and the annotation information of the grounding wire in the annotated image. The annotated image may be a fisheye image taken by a fisheye camera. In order to be able to predict the position and category of the grounding wire after training, the training sample also needs to include the annotation information of the position and category of the annotated grounding wire corresponding to the annotated image.

[0110] Step S304 , obtaining the annotation category of each pixel in the annotated image based on the ground line annotation category to obtain semantic segmentation annotation information of the annotated image, and obtaining key point annotation information of the ground line in the annotated image based on the ground line annotation points.

[0111] Among them, the grounding line annotation points are the pixel coordinates of the key points of the grounding line in the image. Corresponding to the key point information predicted by the model, for different categories of grounding lines, the number of key points of the corresponding grounding line annotation points is different. For example, there are multiple groups of two-dimensional pixel coordinates in the grounding line annotation points corresponding to the annotated image. The number of each group of two-dimensional pixel coordinates may be different, some may be 5, and some may be 15. The key point annotation information of the grounding line can be obtained by processing the grounding line annotation points. The key point annotation information includes the pixel coordinates and visibility of the key points of the grounding line, and the bounding box information corresponding to the bounding box of the grounding line. Each pixel coordinate of the grounding line annotation point is the pixel coordinate of the key point in the key point annotation information.

[0112] Among them, the grounding wire annotation category is the category of the grounding wire in the image. For example, the grounding wire annotation category corresponding to a certain image can be "the object in the bounding box A is metal; the object in the bounding box B is glass; the object in the bounding box C is a key". In order to obtain the semantic segmentation annotation information of the annotated image, each group of two-dimensional pixel coordinates can be connected in sequence, and then the obtained grounding wires can be matched with the grounding wire annotation categories to obtain the category to which each grounding wire belongs. After that, each pixel in the annotated image is analyzed to determine the category to which each pixel belongs, and the categories of each pixel are integrated to obtain the semantic segmentation information. The semantic segmentation information stores the category corresponding to each pixel in the image. Optionally, the category of pixels that are not grounding wires can be set to background.

[0113] Step S306, training the semantic network of the model to be trained based on the annotated image and the corresponding semantic segmentation annotation information, and training the key point network of the model to be trained based on the annotated image and the corresponding key point annotation information, so as to obtain an obstacle detection model.

[0114] In the obstacle detection model, since the model needs to predict both the key point information of the grounding wire and the category of the grounding wire, it is a multi-task model. In order to improve the prediction accuracy of each task, the model is set to include two network branches, which are used to predict key points and categories respectively. Exemplarily, the model to be trained may include a semantic network and a key point network. For the semantic network, the semantic network can be trained based on the annotated image and the corresponding semantic segmentation annotation information. For the key point network, the key point network can be trained based on the annotated image and the corresponding key point annotation information. In this way, the multi-task model is trained separately, which can not only achieve the convergence of the loss function, but also improve the detection accuracy of each network branch. Subsequently, the final grounding wire information can be determined in combination with the prediction results of each branch, thereby improving the detection rate of the model.

[0115] In this embodiment, by obtaining training samples and training the semantic network of the model to be trained based on the annotated images and corresponding semantic segmentation annotation information in the training samples, and training the key point network of the model to be trained based on the annotated images and corresponding key point annotation information, a multi-task detection model can be obtained and the prediction accuracy of the model can be improved through separate training.

[0116] From the above content, we can know that the learning network in the obstacle detection model can be divided into a semantic network and a key point network. The semantic network is used to learn the category of the ground wire in the image based on the image features, and the key point network is used to learn the key point information of the ground wire in the image based on the image features. The key point information can be, for example, the number of key points, key point positions, bounding box information, etc.

[0117] In the process of training the semantic network in the training model, the semantic segmentation information generated by the semantic network can be constrained by the loss function, so that the semantic network in the obstacle detection model can correctly identify the category of the ground wire in the image. In one embodiment, training the semantic network of the training model based on the annotated image and the corresponding semantic segmentation information includes steps 402 to 406. Among them:

[0118] Step S402: obtaining semantic segmentation prediction information of the annotated image based on the semantic network of the model to be trained.

[0119] Step S404: Substitute the semantic segmentation prediction information and the semantic segmentation annotation information into the semantic loss function.

[0120] Exemplarily, the semantic loss function includes a first loss function and a second loss function, the first loss function is used to adjust the similarity between the semantic segmentation prediction information and the semantic segmentation annotation information, and the second loss function is used to adjust the difficulty balance of the training samples. The first loss function may be, for example, a diceloss function. The diceloss function is a loss function suitable for segmentation problems, which measures the prediction accuracy by calculating the intersection over union (Intersection over Union) between the predicted value and the true value. The second loss function may be, for example, a focalloss function. In this way, the parameters of the semantic network trained based on the semantic loss function established by combining multiple loss functions can be more accurate, thereby improving the accuracy and efficiency of the semantic network predicting the semantic segmentation true value of the ground line.

[0121] Step S406: training the semantic network based on the semantic loss function.

[0122] In this embodiment, the semantic segmentation prediction information of the annotated image is obtained based on the semantic network of the model to be trained, and then the semantic segmentation prediction information and the semantic segmentation annotation information corresponding to the annotated image are substituted into a preset semantic loss function, and the parameters of the semantic network are adjusted based on the result of the semantic loss function, that is, the semantic network is trained based on the semantic loss function, so that the difference between the semantic segmentation prediction information and the semantic segmentation annotation information can be reduced, making the semantic segmentation information predicted by the semantic network more accurate. And through the joint semantic loss function, the semantic network can solve the problem of imbalance between difficult and easy samples, and pay more attention to difficult samples.

[0123] As mentioned above, for different types of obstacles, the number of key points of the corresponding grounding wire may be different. In one possible implementation, the number of key points in the key point information can be divided into 5 key points and 15 key points. When the model to be trained learns to predict the key point information of the grounding wire, in order to make the key point network predict the key point information more accurately, different numbers of key points can be trained separately. That is, in one embodiment, the key point annotation information includes a first type of key point annotation information and a second type of key point annotation information, the first type of key point annotation information is used to annotate the key point information of the first type of grounding wire, the number of key points on the first type of grounding wire is the first number, the second type of key point annotation information is used to annotate the key point information of the second type of grounding wire, the number of key points on the second type of grounding wire is the second number, the first number is less than the second number, the key point network includes a first key point network and a second key point network, the first key point network is used to train the first type of grounding wire, and the second key point network is used to train the second type of grounding wire. Therefore, in this embodiment, when the key point network of the training model is trained based on the annotated images and the corresponding key point annotation information, the first key point network is trained based on the first type of key point annotation information and the corresponding annotated images; and the second key point network is trained based on the second type of key point annotation information and the corresponding annotated images.

[0124] For example, the key point annotation information can be divided into the annotation information of 5 key points and the annotation information of 15 key points. The annotation information of 5 key points and the annotation information of 15 key points are respectively input into two key point networks to train and obtain the corresponding network parameters. That is, for the annotation information of 5 key points and the annotation information of 15 key points, two branches are separated to construct the key point loss function, so that the detection efficiency of each can be improved, so that the two generated key point networks can better detect the key points of the grounding wire.

[0125] In this embodiment, by training the first key point network based on the first category of key point annotation information and the corresponding annotation images, and training the second key point network based on the second category of key point annotation information and the corresponding annotation images, the model can learn different types of key points, thereby improving the prediction accuracy of the model.

[0126] Regarding the step of training the first key point network based on the first type of key point annotation information and the corresponding annotated image, this step can be achieved by the following steps: predicting the first type of key points in the annotated image based on the first key point network to obtain the first type of key point prediction information; substituting the first type of key point prediction information and the first type of key point annotation information into the first type of key point loss function; training the first key point network based on the first type of key point loss function.

[0127] Regarding the step of training the second key point network based on the second type of key point annotation information and the corresponding annotated image, this step can be achieved by the following steps: predicting the second type of key points in the annotated image based on the second key point network to obtain second type of key point prediction information; substituting the second type of key point prediction information and the second type of key point annotation information into the second type of key point loss function; training the second key point network based on the second type of key point loss function.

[0128] It is understandable that the structures of the first key point network and the second key point network may be the same, but due to the difference in training samples, the network parameters of the first key point network and the second key point network may be different or the same.

[0129] In some embodiments, the first type of key point loss function and the second type of key point loss function may be composed of at least one of a third loss function, a fourth loss function, a fifth loss function, and a sixth loss function. Among them, the third loss function may be a CIoU loss function. The fourth loss function may be a BCE loss function. The fifth loss function may be a DFL loss function. The sixth loss function may be a Pose loss function. Among them, the CIoU (Complete IoU Loss) loss function is a loss function for target detection tasks, which is an improved version of IoU (Intersection over Union). CIoU not only considers the overlapping area between the predicted box and the true box, that is, the overlapping area between the bounding box in the key point annotation information and the bounding box in the key point prediction information, but also considers the center point distance and aspect ratio between them. The BCE (BinaryCross-Entropy) loss function is a binary cross entropy loss function. It measures the difference between the actual output and the target output, that is, the difference between the key point pixel coordinates in the key point annotation information and the key point pixel coordinates in the key point prediction information. The DFL (Distance-IoU) loss function aims to directly minimize the distance between the predicted box and the true box, thereby improving positioning accuracy. The DFL loss takes into account the distance loss, IoU (overlapping area) loss, and aspect ratio loss between the predicted box and the true box. Pose loss (pose estimation loss) is a loss function used for pose estimation tasks. Pose loss can calculate the average Euclidean distance between the predicted key points and the true key points, and can also measure the proportion of the error between the predicted key points and the true key points within a certain threshold, as well as consider the importance of the key points and the correlation between the key points.

[0130] In some embodiments, the key point loss function can be constructed by combining the above four loss functions. The corresponding key point loss value can be obtained by weighting the loss values ​​obtained by the four loss functions.

[0131] Through the above steps, grounding lines with different levels of complexity of key points can be well learned, thereby improving the recognition rate of key points of grounding lines in images.

[0132] In an exemplary embodiment, in order to be able to comprehensively determine the obstacles around the robot by identifying the ground wire in the image, an obstacle detection model can be trained. The obstacle detection model is a multi-task model based on semantic segmentation and key point detection. Among them, the encoding part of the model can use the mobileone network. This can meet the high detection rate and high real-time requirements on the embedded end. The encoding part of the model is used to extract features from the image to better learn the deep features of the image. Therefore, the obstacle detection model can be divided into a semantic network and a key point network.

[0133] In the process of training the model, a training sample may be obtained. The training sample includes a labeled image and labeling information of a ground line in the labeled image. Based on the labeling category of the ground line in the labeling information, the labeling category of each pixel in the labeled image is obtained to obtain semantic segmentation labeling information of the labeled image, and based on the labeling points of the ground line in the labeling information, key point labeling information of the ground line in the labeled image is obtained.

[0134] In this embodiment, the grounding lines of various obstacles are defined as line segments of separate instances, including but not limited to the following categories: metal base, glass, human feet, wheels, and thin stool legs. In combination with the shape characteristics of each obstacle, the line segments of metal base, glass, and human feet are fixed to 15 points; the line segments of wheels and thin stool legs are fixed to 5 points. The annotation information of the annotated image can annotate the grounding line based on the above definition.

[0135] For the semantic network, the semantic segmentation prediction information of the annotated image can be obtained based on the semantic network of the model to be trained; the semantic segmentation prediction information and the semantic segmentation annotation information are substituted into the semantic loss function; and the semantic network is trained based on the semantic loss function. For example, the ground line semantic segmentation truth data of each image is generated using the manually annotated labels (i.e., annotation information); and then the diceloss function and the focalloss function are used to construct a joint semantic loss function for learning.

[0136] For the key point network, the key point network can be divided into a first key point network and a second key point network. The first type of key points in the labeled image can be predicted based on the first key point network to obtain the first type of key point prediction information; the first type of key point prediction information and the first type of key point labeling information are substituted into the first type of key point loss function; the first key point network is trained based on the first type of key point loss function;

[0137] Based on the second key point network, the second type of key points in the labeled image are predicted to obtain the second type of key point prediction information; the second type of key point prediction information and the second type of key point labeling information are substituted into the second type of key point loss function; based on the second type of key point loss function, the second key point network is trained.

[0138] For example, first use manually annotated labels to generate the true value of the key points of each image, including the x, y, w, h (x, y represent the coordinates of the center point of the bounding box, w represents the width of the bounding box, and h represents the height of the bounding box) of each grounding wire box, as well as the position and visibility of the key points of the grounding wire. In addition, the first key point network is used to predict the key points of the grounding wire with 5 key points, and the second key point network is used to predict the key points of the grounding wire with 15 key points. And the categories of 5 key points and 15 key points are separated into two branches for anchor-free loss construction. By dividing different types of grounding wires into grounding wires with different numbers of key points, the key point network can determine the number of key points of the grounding wire based on the shape characteristics of the grounding wire, thereby improving the accuracy of key point recognition.

[0139] The loss function of a multi-task model is usually difficult to converge. In this embodiment, a pre-trained model is first obtained based on big data training. Then, the model is trained using an adaptive multi-task loss weight.

[0140] After the obstacle detection model is trained, the robot can obtain the image to be detected taken by the robot. The image to be detected is input into the obstacle detection model, and the key point information corresponding to the grounding line in the image to be detected and the semantic segmentation information corresponding to the image to be detected are determined by the obstacle detection model. The semantic segmentation information is used to characterize the category of each pixel in the image to be detected. The first pixel information belonging to the grounding line in the image to be detected is determined based on the key point information, and the second pixel information belonging to the grounding line in the image to be detected is determined based on the semantic segmentation information. Based on the first pixel information and the second pixel information, the target pixel belonging to the obstacle is determined. The predicted position of the obstacle relative to the robot is determined based on the target pixel, and the predicted category of the obstacle is determined based on the semantic segmentation information corresponding to the target pixel.

[0141] The design and use of multiple branches is based on minimizing the false detection of the algorithm. After obtaining the semantic segmentation results and key point detection results, further judgment will be made on the semantic segmentation results and key point detection results. Only the pixel positions where both branches believe there are obstacles will be confirmed as obstacle points. And according to the category distribution of the pixel area, a voting-based obstacle category is finally assigned. For example, the semantic segmentation information corresponding to each pixel point in the bounding box can be analyzed, so as to obtain the non-background category with the largest number of pixels in the bounding box, such as glass and metal base.

[0142] The following post-processing may then be performed: determining the position of the obstacle on the obstacle map based on the predicted position, so as to determine the movement area of ​​the robot based on the obstacle map.

[0143] The following post-processing may also be performed: if the obstacle is determined to be a dynamic obstacle based on the predicted category, the attenuation coefficient of the obstacle on the obstacle map is set to a first attenuation coefficient; if the obstacle is determined to be a static obstacle based on the predicted category, the attenuation coefficient of the obstacle on the obstacle map is set to a second attenuation coefficient, and the first attenuation coefficient is greater than the second attenuation coefficient.

[0144] For example, different obstacle categories are divided into two categories: dynamic obstacles and static obstacles, and different post-processing strategies are used. For dynamic obstacles, Kalman filtering can be used to filter and track dynamic obstacles in multiple frames, and a certain life cycle can be set. For example, it can be set to 5s. And the attenuation coefficient on the obstacle map is higher. For static obstacles, the real-time position of the robot can be used, and two frames can be used for confirmation to reduce algorithm misdetection. And the attenuation coefficient on the obstacle map is lower.

[0145] In this embodiment, the robot's perception field of view can be improved by using a fisheye camera. Within a certain budget, two fisheye cameras can be set to cover the target range, such as the detection of a 2m range of 180 degrees in the horizontal and vertical directions in front of the robot. In addition, the use of fisheye cameras can also reduce the number of missed objects, such as highly reflective metal bases, wires, and glass. In addition, the detected obstacles can also carry specific semantic information, which is convenient for subsequent processing to perform dynamic and static post-processing strategies based on semantics, such as retaining and attenuating static objects to a certain extent, thereby improving the feasibility and convenience of downstream tasks.

[0146] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0147] Based on the same inventive concept, the embodiment of the present application also provides a detection device for implementing the above-mentioned obstacle detection method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in the embodiments of the detection device for one or more obstacles provided below can refer to the limitations of the obstacle detection method above, and will not be repeated here.

[0148] In an exemplary embodiment, as shown in FIG3 , an obstacle detection device 300 is provided, comprising: an image acquisition module 310, an information prediction module 320, an information processing module 330 and an obstacle information detection module 340, wherein:

[0149] An image acquisition module 310 is used to obtain an image to be detected taken by the robot;

[0150] An information prediction module 320 is used to input the image to be detected into an obstacle detection model, and determine key point information corresponding to the ground line in the image to be detected and semantic segmentation information corresponding to each pixel in the image to be detected through the obstacle detection model, wherein the semantic segmentation information is used to characterize the category of each pixel in the image to be detected;

[0151] An information processing module 330 is used to determine first pixel information belonging to the ground line in the image to be detected based on the key point information, determine second pixel information belonging to the ground line in the image to be detected based on the semantic segmentation information; and determine a target pixel belonging to an obstacle based on the first pixel information and the second pixel information;

[0152] The obstacle information detection module 340 is used to determine the predicted position of the obstacle relative to the robot based on the target pixel, and determine the predicted category of the obstacle based on the semantic segmentation information corresponding to the target pixel.

[0153] In some embodiments, the obstacle detection device may also include a downstream task module, wherein the downstream task module is used to obtain the actual position of the obstacle on the obstacle map based on a mapping table and the predicted position, obtain the actual distance between the robot and the obstacle based on the actual position, and determine the driving area of ​​the robot based on the actual distance and the obstacle map, and the mapping table is used to characterize the mapping relationship between the pixel coordinate system where the position information is located and the world coordinate system where the actual distance is located.

[0154] In some embodiments, the downstream task module is also used to set the attenuation coefficient of the obstacle on the obstacle map to a first attenuation coefficient if the obstacle is determined to be a dynamic obstacle based on the prediction category; if the obstacle is determined to be a static obstacle based on the prediction category, set the attenuation coefficient of the obstacle on the obstacle map to a second attenuation coefficient, and the first attenuation coefficient is greater than the second attenuation coefficient.

[0155] In some embodiments, the information processing module 330 is used to sequentially connect key pixel points in the detection frame corresponding to the detection frame information to obtain a connection line; and obtain the first pixel information based on the pixel points of the connection line.

[0156] In some embodiments, the robot may further include an obstacle training device, wherein the obstacle training device may include a sample acquisition module, a labeling information processing module and a training module.

[0157] The sample acquisition module is used to acquire a training sample, wherein the training sample includes a labeled image and labeled information of a ground line in the labeled image;

[0158] The annotation information processing module is used to obtain the annotation category of each pixel in the annotated image based on the grounding line annotation points and the grounding line annotation category in the annotation information to obtain semantic segmentation annotation information of the annotated image, and obtain key point annotation information of the grounding line in the annotated image based on the grounding line annotation points;

[0159] The training module is used to train the semantic network of the model to be trained based on the annotated image and the corresponding semantic segmentation annotation information, and to train the key point network of the model to be trained based on the annotated image and the corresponding key point annotation information to obtain the obstacle detection model.

[0160] In some embodiments, the training module also includes a semantic network training module, wherein the semantic network training module is used to obtain semantic segmentation prediction information of the annotated image based on the semantic network of the network to be trained; the semantic segmentation prediction information and the semantic segmentation annotation information are substituted into a semantic loss function, the semantic loss function includes a first loss function and a second loss function, the first loss function is used to adjust the similarity between the semantic segmentation prediction information and the semantic segmentation annotation information, and the second loss function is used to adjust the difficulty balance of the training sample; the semantic network is trained based on the semantic loss function.

[0161] In some embodiments, the key point annotation information includes a first type of key point annotation information and a second type of key point annotation information, the first type of key point annotation information is used to annotate the key point information of a first type of grounding wire, the number of key points on the first type of grounding wire is a first number, the second type of key point annotation information is used to annotate the key point information of a second type of grounding wire, the number of key points on the second type of grounding wire is a second number, the first number is less than the second number, the key point network includes a first key point network and a second key point network, the first key point network is used to train the first type of grounding wire, and the second key point network is used to train the second type of grounding wire.

[0162] In some embodiments, the training module also includes a key point network training module, wherein the key point network training module is used to train the first key point network based on the first category of key point annotation information and the corresponding annotation image; and to train the second key point network based on the second category of key point annotation information and the corresponding annotation image.

[0163] In some embodiments, the key point network training module is further used to predict the first type of key points in the labeled image based on the first key point network to obtain the first type of key point prediction information; substitute the first type of key point prediction information and the first type of key point labeling information into the first type of key point loss function; train the first key point network based on the first type of key point loss function;

[0164] In some embodiments, the key point network training module is also used to predict the second type of key points in the annotated image based on the second key point network to obtain second type of key point prediction information; substitute the second type of key point prediction information and the second type of key point annotation information into the second type of key point loss function; and train the second key point network based on the second type of key point loss function.

[0165] Each module in the obstacle detection device and the obstacle training device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0166] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG4. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data involved in the application process and training process of the obstacle detection model. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for detecting an obstacle is implemented.

[0167] In an exemplary embodiment, an electronic device is provided, which may be a robot, and its internal structure diagram may be shown in FIG5. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected via a system bus, and the communication interface, the display unit, and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a method for detecting obstacles is implemented. The display unit of the computer device is used to form a visually visible picture, which may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0168] Those skilled in the art will appreciate that the structure shown in FIG. 4 or 5 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0169] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0170] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0171] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0172] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0173] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0174] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0175] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for detecting an obstacle, characterized in that: The method comprises: Obtain the image to be detected taken by the robot; Inputting the image to be detected into an obstacle detection model, determining key point information corresponding to the ground line in the image to be detected and semantic segmentation information corresponding to the image to be detected through the obstacle detection model, wherein the semantic segmentation information is used to characterize the category of each pixel in the image to be detected; Determine first pixel information belonging to the ground line in the image to be detected based on the key point information, and determine second pixel information belonging to the ground line in the image to be detected based on the semantic segmentation information; Determining a target pixel belonging to an obstacle based on the first pixel information and the second pixel information; A predicted position of the obstacle relative to the robot is determined based on the target pixel, and a predicted category of the obstacle is determined based on semantic segmentation information corresponding to the target pixel.

2. The detection method according to claim 1, characterized in that: After determining the predicted position of the obstacle relative to the robot based on the target pixel and determining the predicted category of the obstacle based on the semantic segmentation information corresponding to the target pixel, the method further comprises: determining a position of the obstacle on an obstacle map based on the predicted position, so as to determine a movement area of ​​the robot based on the obstacle map; If the obstacle is determined to be a dynamic obstacle based on the predicted category, the attenuation coefficient of the obstacle on the obstacle map is set to a first attenuation coefficient; if the obstacle is determined to be a static obstacle based on the predicted category, the attenuation coefficient of the obstacle on the obstacle map is set to a second attenuation coefficient, and the first attenuation coefficient is greater than the second attenuation coefficient.

3. The detection method according to claim 1, characterized in that: The key point information includes bounding box information and key pixel points, and determining first pixel information belonging to the ground line in the image to be detected based on the key point information includes: Connecting key pixel points in the bounding box corresponding to the bounding box information in sequence to obtain a connecting line; The first pixel information is obtained based on the pixel points of the connecting line.

4. The detection method according to claim 1, characterized in that: Before obtaining the image to be detected, the method includes: Acquire a training sample, wherein the training sample includes a labeled image and labeled information of a ground line in the labeled image; Based on the ground line annotation category in the annotation information, obtaining the annotation category of each pixel in the annotated image to obtain semantic segmentation annotation information of the annotated image, and based on the ground line annotation points in the annotation information, obtaining key point annotation information of the ground line in the annotated image; The semantic network of the model to be trained is trained based on the annotated image and the corresponding semantic segmentation annotation information, and the key point network of the model to be trained is trained based on the annotated image and the corresponding key point annotation information to obtain the obstacle detection model.

5. The detection method according to claim 4, characterized in that: The training of the semantic network of the to-be-trained model based on the annotated image and the corresponding semantic segmentation information comprises: Obtaining semantic segmentation prediction information of the annotated image based on the semantic network of the model to be trained; Substituting the semantic segmentation prediction information and the semantic segmentation annotation information into a semantic loss function; The semantic network is trained based on the semantic loss function.

6. The detection method according to claim 4, characterized in that: The key point annotation information includes a first type of key point annotation information and a second type of key point annotation information, the first type of key point annotation information is used to annotate key point information of a first type of grounding wire, the number of key points on the first type of grounding wire is a first number, the second type of key point annotation information is used to annotate key point information of a second type of grounding wire, the number of key points on the second type of grounding wire is a second number, the first number is less than the second number, the key point network includes a first key point network and a second key point network, the first key point network is used to train the first type of grounding wire, and the second key point network is used to train the second type of grounding wire, and the training of the key point network of the model to be trained based on the annotated image and the corresponding key point annotation information includes: Training the first key point network based on the first type of key point annotation information and the corresponding annotated image; The second key point network is trained based on the second type of key point annotation information and the corresponding annotated images.

7. The detection method according to claim 6, characterized in that: The training of the first key point network based on the first type of key point annotation information and the corresponding annotated image includes: Predicting a first type of key points in the labeled image based on the first key point network to obtain first type of key point prediction information; Substituting the first-category key point prediction information and the first-category key point annotation information into the first-category key point loss function; Training the first key point network based on the first type of key point loss function; The training of the second key point network based on the second type of key point annotation information and the corresponding annotated image includes: Predicting the second type of key points in the labeled image based on the second key point network to obtain second type of key point prediction information; Substituting the second-category key point prediction information and the second-category key point annotation information into the second-category key point loss function; The second key point network is trained based on the second type of key point loss function.

8. An obstacle detection device, characterized in that: The device comprises: An image acquisition module is used to obtain the image to be detected taken by the robot; An information prediction module, used for inputting the image to be detected into an obstacle detection model, and determining key point information corresponding to the ground line in the image to be detected and semantic segmentation information corresponding to each pixel in the image to be detected through the obstacle detection model, wherein the semantic segmentation information is used to characterize the category of each pixel in the image to be detected; An information processing module, configured to determine first pixel information belonging to a ground line in the image to be detected based on the key point information, determine second pixel information belonging to the ground line in the image to be detected based on the semantic segmentation information; and determine a target pixel belonging to an obstacle based on the first pixel information and the second pixel information; The obstacle information detection module is used to determine the predicted position of the obstacle relative to the robot based on the target pixel, and to determine the predicted category of the obstacle based on the semantic segmentation information corresponding to the target pixel.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images