Obstacle detection method and device, electronic equipment and readable storage medium

Abstract

The application relates to an obstacle detection method and device, electronic equipment and a readable storage medium. The method comprises the following steps: obtaining an image to be detected photographed by a robot; inputting the image to be detected into an obstacle detection model, determining key point information corresponding to a ground wire in the image to be detected and semantic segmentation information corresponding to the image to be detected through the obstacle detection model; determining first pixel information belonging to the ground wire in the image to be detected based on the key point information, and determining second pixel information belonging to the ground wire in the image to be detected based on the semantic segmentation information; determining target pixels belonging to obstacles based on the first pixel information and the second pixel information; determining a predicted position of the obstacles relative to the robot based on the target pixels, and determining a predicted category of the obstacles based on semantic segmentation information corresponding to the target pixels. The method can increase the types of detectable obstacles, thereby improving the detection accuracy of the obstacles.

Description

Technical Field

[0001] This application relates to the field of robotics, and in particular to a method, apparatus, electronic device, and computer-readable storage medium for detecting obstacles. Background Technology

[0002] With the development of robotics technology, robots can not only work under human control, but also autonomously analyze their surroundings and perform tasks based on those environments. For example, when encountering an obstacle, a robot can analyze its current distance to the obstacle to determine a route to avoid it. However, current obstacle avoidance methods may not be able to avoid certain special objects, such as those with reflectivity within a specific range, potentially leading to collisions. Summary of the Invention

[0003] Therefore, it is necessary to provide an obstacle detection method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can increase the types of obstacles that can be detected and thus improve the accuracy of obstacle detection, in order to address the above-mentioned technical problems.

[0004] In a first aspect, this application provides a method for detecting obstacles, including:

[0005] Obtain the image to be inspected captured by the robot;

[0006] The image to be detected is input into an obstacle detection model, and the obstacle detection model determines the key point information corresponding to the grounding wire in the image to be detected, and the semantic segmentation information corresponding to each pixel in the image to be detected. The semantic segmentation information is used to characterize the category of each pixel in the image to be detected.

[0007] Based on the key point information, the first pixel information belonging to the grounding wire in the image to be detected is determined, and based on the semantic segmentation information, the second pixel information belonging to the grounding wire in the image to be detected is determined.

[0008] Based on the first pixel information and the second pixel information, the target pixel belonging to the obstacle is determined;

[0009] The predicted position of the obstacle relative to the robot is determined based on the target pixel, and the predicted category of the obstacle is determined based on the semantic segmentation information corresponding to the target pixel.

[0010] In one embodiment, after determining the predicted position of the obstacle relative to the robot based on the target pixel and determining the predicted category of the obstacle based on the semantic segmentation information corresponding to the target pixel, the process includes:

[0011] The actual position of the obstacle on the obstacle map is obtained based on the mapping table and the predicted position. The actual distance between the robot and the obstacle is obtained based on the actual position. The driving area of ​​the robot is determined based on the actual distance and the obstacle map. The mapping table is used to represent the mapping relationship between the pixel coordinate system where the position information is located and the world coordinate system where the actual distance is located.

[0012] If the obstacle is determined to be a dynamic obstacle based on the predicted category, then the attenuation coefficient of the obstacle on the obstacle map is set to a first attenuation coefficient; if the obstacle is determined to be a static obstacle based on the predicted category, then the attenuation coefficient of the obstacle on the obstacle map is set to a second attenuation coefficient, wherein the first attenuation coefficient is greater than the second attenuation coefficient.

[0013] In one embodiment, the key point information includes detection box information and key pixels, and determining the first grounding line pixel information in the image to be detected based on the key point information includes:

[0014] Connect the key pixels within the detection box corresponding to the detection box information in sequence to obtain the connecting line;

[0015] The first pixel information is obtained based on the pixels of the connecting line.

[0016] In one embodiment, prior to obtaining the image to be detected, the process includes:

[0017] Obtain training samples, which include labeled images and annotation information of grounding lines in the labeled images;

[0018] Based on the grounding wire annotation points and grounding wire annotation categories in the annotation information, the annotation category of each pixel in the annotation image is obtained to obtain the semantic segmentation annotation information of the annotation image. Based on the grounding wire annotation points, the key point annotation information of the grounding wire in the annotation image is obtained.

[0019] The semantic network of the model to be trained is trained based on the labeled images and the corresponding semantic segmentation annotation information, and the key point network of the model to be trained is trained based on the labeled images and the corresponding key point annotation information to obtain the obstacle detection model.

[0020] In one embodiment, training the semantic network of the model to be trained based on the labeled image and the corresponding semantic segmentation information includes:

[0021] The semantic network based on the network to be trained obtains the semantic segmentation prediction information of the labeled image;

[0022] The semantic segmentation prediction information and the semantic segmentation annotation information are substituted into the semantic loss function, which includes a first loss function and a second loss function. The first loss function is used to adjust the similarity between the semantic segmentation prediction information and the semantic segmentation annotation information, and the second loss function is used to adjust the balance between the difficulty of the training samples.

[0023] The semantic network is trained based on the semantic loss function.

[0024] In one embodiment, the keypoint annotation information includes a first type of keypoint annotation information and a second type of keypoint annotation information. The first type of keypoint annotation information is used to annotate the keypoint information of a first type of grounding wire, and the number of keypoints on the first type of grounding wire is a first quantity. The second type of keypoint annotation information is used to annotate the keypoint information of a second type of grounding wire, and the number of keypoints on the second type of grounding wire is a second quantity. The first quantity is less than the second quantity. The keypoint network includes a first keypoint network and a second keypoint network. The first keypoint network is used to train the first type of grounding wire, and the second keypoint network is used to train the second type of grounding wire. The step of training the keypoint network of the model to be trained based on the annotated image and the corresponding keypoint annotation information includes:

[0025] The first keypoint network is trained based on the first type of keypoint annotation information and the corresponding annotation image;

[0026] The second keypoint network is trained based on the second type of keypoint annotation information and the corresponding annotation images.

[0027] In one embodiment, training the first keypoint network based on the first type of keypoint annotation information and the corresponding annotated image includes:

[0028] Based on the first keypoint network, the first type of keypoints in the labeled image are predicted to obtain the first type of keypoint prediction information.

[0029] Substitute the first type of key point prediction information and the first type of key point annotation information into the first type of key point loss function;

[0030] The first keypoint network is trained based on the first type of keypoint loss function;

[0031] The training of the second keypoint network based on the second type of keypoint annotation information and the corresponding annotated image includes:

[0032] Based on the second keypoint network, the second type of keypoints in the labeled image are predicted to obtain the second type of keypoint prediction information.

[0033] Substitute the second type of keypoint prediction information and the second type of keypoint annotation information into the second type of keypoint loss function;

[0034] The second keypoint network is trained based on the second type of keypoint loss function.

[0035] Secondly, this application also provides an obstacle detection device, comprising:

[0036] The image acquisition module is used to acquire the image to be detected captured by the robot;

[0037] The information prediction module is used to input the image to be detected into the obstacle detection model, and to determine the key point information corresponding to the grounding wire in the image to be detected and the semantic segmentation information corresponding to each pixel in the image to be detected through the obstacle detection model. The semantic segmentation information is used to characterize the category of each pixel in the image to be detected.

[0038] The information processing module is used to determine, based on the key point information, the first pixel information belonging to the grounding wire in the image to be detected; and to determine, based on the semantic segmentation information, the second pixel information belonging to the grounding wire in the image to be detected; and to determine, based on the first pixel information and the second pixel information, the target pixel belonging to the obstacle.

[0039] An obstacle information detection module is used to determine the predicted position of the obstacle relative to the robot based on the target pixel, and to determine the predicted category of the obstacle based on the semantic segmentation information corresponding to the target pixel.

[0040] Thirdly, this application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0041] Obtain the image to be inspected captured by the robot;

[0042] The image to be detected is input into an obstacle detection model, and the obstacle detection model determines the key point information corresponding to the grounding wire in the image to be detected, and the semantic segmentation information corresponding to each pixel in the image to be detected. The semantic segmentation information is used to characterize the category of each pixel in the image to be detected.

[0043] Based on the key point information, the first pixel information belonging to the grounding wire in the image to be detected is determined, and based on the semantic segmentation information, the second pixel information belonging to the grounding wire in the image to be detected is determined.

[0044] Based on the first pixel information and the second pixel information, the target pixel belonging to the obstacle is determined;

[0045] The predicted position of the obstacle relative to the robot is determined based on the target pixel, and the predicted category of the obstacle is determined based on the semantic segmentation information corresponding to the target pixel.

[0046] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0047] Obtain the image to be inspected captured by the robot;

[0048] The image to be detected is input into an obstacle detection model, and the obstacle detection model determines the key point information corresponding to the grounding wire in the image to be detected, and the semantic segmentation information corresponding to each pixel in the image to be detected. The semantic segmentation information is used to characterize the category of each pixel in the image to be detected.

[0049] Based on the key point information, the first pixel information belonging to the grounding wire in the image to be detected is determined, and based on the semantic segmentation information, the second pixel information belonging to the grounding wire in the image to be detected is determined.

[0050] Based on the first pixel information and the second pixel information, the target pixel belonging to the obstacle is determined;

[0051] The predicted position of the obstacle relative to the robot is determined based on the target pixel, and the predicted category of the obstacle is determined based on the semantic segmentation information corresponding to the target pixel.

[0052] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0053] Obtain the image to be inspected captured by the robot;

[0054] The image to be detected is input into an obstacle detection model, and the obstacle detection model determines the key point information corresponding to the grounding wire in the image to be detected, and the semantic segmentation information corresponding to each pixel in the image to be detected. The semantic segmentation information is used to characterize the category of each pixel in the image to be detected.

[0055] Based on the key point information, the first pixel information belonging to the grounding wire in the image to be detected is determined, and based on the semantic segmentation information, the second pixel information belonging to the grounding wire in the image to be detected is determined.

[0056] Based on the first pixel information and the second pixel information, the target pixel belonging to the obstacle is determined;

[0057] The predicted position of the obstacle relative to the robot is determined based on the target pixel, and the predicted category of the obstacle is determined based on the semantic segmentation information corresponding to the target pixel.

[0058] The aforementioned obstacle detection method, apparatus, computer equipment, computer-readable storage medium, and computer program product acquire an image to be detected captured by a robot, input the image into an obstacle detection model, and determine key point information corresponding to the grounding wire in the image and semantic segmentation information corresponding to each pixel in the image through the obstacle detection model. Thus, based on the key point information, the first pixel information belonging to the grounding wire in the image can be determined, and based on the semantic segmentation information, the second pixel information belonging to the grounding wire in the image can be determined. In other words, both the first and second pixel information correspond to the grounding wire information in the image. Furthermore, based on the first pixel... The grounding information determined by the second pixel information is used to identify the target pixel belonging to the obstacle, thus improving the reliability of obstacle detection. Since the target pixel corresponds to the point where the obstacle contacts the ground, the predicted position of the obstacle relative to the robot can be determined based on the target pixel. Furthermore, since the obstacle detection model can obtain semantic segmentation information corresponding to the image to be detected, and the category of each pixel in the image to be detected can be obtained based on the semantic segmentation information corresponding to the target pixel, the predicted category of the obstacle can be obtained. The predicted category and predicted position can then be input into downstream tasks to achieve the robot's versatility and improve its obstacle avoidance capabilities. In addition, since the robot processes the captured image and determines obstacles around the robot by analyzing the grounding wire in the image, the robot can identify more types of obstacles, thereby improving the robot's obstacle detection accuracy. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 This is a flowchart illustrating an obstacle detection method in one embodiment;

[0061] Figure 2 This is a flowchart illustrating the training steps of an obstacle detection model in one embodiment;

[0062] Figure 3 This is a structural block diagram of an obstacle detection device in one embodiment;

[0063] Figure 4 This is an internal structural diagram of a computer device in one embodiment;

[0064] Figure 5 This is a diagram of the internal structure of an electronic device in one embodiment. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0066] In one exemplary embodiment, an obstacle detection method is provided. This embodiment illustrates the application of this method to a terminal, but it is understood that the method can also be applied to a server, or to a system including both a terminal and a server, and can be implemented through interaction between the terminal and the server, or through interaction between terminals. The terminal can be a computer device, such as a robot, a smart wearable device, or a self-balancing scooter. The robot can be a delivery robot (commercial delivery robot, industrial delivery robot, AGV (Automated Guided Vehicle) robot), a guide robot, a disinfection robot, a cleaning robot, an inspection robot, or other self-moving robots, without specific limitations. The server can be an independent physical server or a service node in a blockchain system. The service nodes in this blockchain system form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP). Furthermore, the server can also be a server cluster consisting of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The robot and the server can connect via Bluetooth, USB (Universal Serial Bus), or network communication methods; this application does not impose any restrictions on these methods.

[0067] In existing technologies, RGBD sensors are typically used to achieve obstacle avoidance for autonomous mobile robots. The robot can use RGBD sensors to obtain depth information between itself and obstacles, thereby determining the distance between itself and the obstacles and ultimately deciding its movement path. In this approach, obstacle avoidance capability primarily depends on the performance of the RGBD sensor hardware itself. However, RGBD sensors may fail to detect certain special objects, such as highly reflective objects (e.g., metal bases), low-reflective objects (e.g., glass, black objects), and small obstacles (e.g., pens, keys). This is because the depth image obtained using RGBD sensors may contain noise, which may be mistaken for noise in small objects like pens and keys, thus preventing detection. Furthermore, the field of view of existing RGBD sensors may be relatively small, with a maximum lateral field of view of around 120 degrees and a maximum vertical field of view of around 70 degrees. This means that a single RGBD sensor may not be able to detect obstacles in a 180-degree three-dimensional space in front of the robot, requiring multiple RGBD sensors to achieve adequate coverage. However, equipping a robot with multiple RGBD sensors would make the robot too expensive. Therefore, to solve or improve the above-mentioned technical problems, this application proposes the following embodiments.

[0068] In one exemplary embodiment, such as Figure 1 As shown, an obstacle detection method is provided, which can be applied to a robot, for example, executed by the robot's processor, and includes the following steps 102 to 110. Wherein:

[0069] Step S102: Obtain the image to be detected captured by the robot.

[0070] The images captured by the robot to be inspected can be obtained using a fisheye camera. A fisheye camera is a camera with a fisheye lens, characterized by an extremely short focal length and a field of view close to or equal to 180 degrees, typically using a 16mm or shorter lens. Thus, the images captured by the robot can detect obstacles in a 180-degree three-dimensional space in front of it, improving the obstacle detection range.

[0071] For example, the robot's camera device sends the captured image to be inspected to the robot's processor, so that the robot's processor can process the image to be inspected. This reduces the time required to upload the image to the cloud for analysis, thereby improving the robot's real-time detection efficiency.

[0072] Step S104: Input the image to be detected into the obstacle detection model, and use the obstacle detection model to determine the key point information corresponding to the grounding wire in the image to be detected, and the semantic segmentation information corresponding to the image to be detected.

[0073] To improve the obstacle detection rate in images, an obstacle detection model is proposed. This model uses deep learning technology to accurately identify grounding wires in the image. This allows for the identification of obstacles around the robot based on the grounding wires in the image. A grounding wire is the line connecting the obstacle to the ground. The obstacle detection model includes two networks: one for analyzing key point information of the grounding wires in the image, and the other for analyzing the corresponding semantic segmentation information. The semantic segmentation information characterizes the category of each pixel in the image, determining the category of each pixel belonging to the grounding wire, and thus determining the category of the grounding wire itself. Pixels in the image can be divided into grounding wire pixels and background pixels. Background pixels are categorized as "background." Grounding wire pixels can have multiple categories, corresponding to the grounding wire category. For example, a grounding wire pixel could be categorized as a human foot, metal, glass, pen, or key. The grounding wire category represents the object to which it belongs, such as a human foot, metal, glass, pen, or key. In other words, the obstacle detection model includes two networks that can be used to predict key point information and category information of grounding wires in the image to be detected, respectively.

[0074] The key point information can include the location of key points on the grounding wire (e.g., which pixel in the image is the key point of the grounding wire) and the bounding box information of the grounding wire. The bounding box information of the grounding wire is a rectangular box used to describe the position and extent of the grounding wire in the image. This rectangle can be defined by the coordinates of the top left and bottom right corners of the box, or by the coordinates of the center point, width, and height of the box. Using the bounding box information, the various grounding wires in the image can be better marked and located.

[0075] Optionally, the number of key points may vary depending on the type of grounding wire. For example, the number of key points can be determined by the volume, complexity, and shape of the obstacle. For instance, a grounding wire for a metal base, glass, or a person's foot might have 15 key points, while a grounding wire for a wheel, a thin stool leg, a pen, or a key might have 5 key points. The shape of the obstacle can be determined by the number of key points.

[0076] For example, the processor inputs the image to be detected into the obstacle detection model, and the obstacle detection model determines the key point information corresponding to the grounding wire in the image to be detected and the semantic segmentation information corresponding to the image to be detected.

[0077] Step S106: Determine the first pixel information belonging to the grounding wire in the image to be detected based on key point information, and determine the second pixel information belonging to the grounding wire in the image to be detected based on semantic segmentation information.

[0078] The first pixel information may include the pixel coordinates of each pixel on each grounding wire in the image to be detected. For example, the first pixel information for a certain grounding wire in the image might be "(234,234), (235,234), (236,235),...". The second pixel information may include the category of each pixel on each grounding wire in the image to be detected. For example, pixel A1 on grounding wire A is classified as metal, pixel A2 as metal, pixel A3 as glass, pixel A4 as metal, and so on...

[0079] For example, since key point information can describe the pixel positions of key points of each grounding wire in the image, the first pixel information belonging to the grounding wire in the image to be detected can be determined based on the key point information. Furthermore, since semantic segmentation information can describe the category information of each pixel in the image, for pixels that are not corresponding to grounding wires, the corresponding category is background; therefore, the second pixel information belonging to the grounding wire in the image to be detected can be determined based on the semantic segmentation information.

[0080] In one implementation, after obtaining the key point information, the key points of each grounding wire can be connected sequentially in a certain order to obtain a complete grounding wire, and then the first pixel information belonging to the grounding wire can be obtained based on the grounding wire.

[0081] In one implementation, after obtaining semantic segmentation information, the category of each pixel is obtained, and second pixel information belonging to the ground line is obtained based on each pixel that does not belong to the background and its corresponding category.

[0082] Step S108: Based on the first pixel information and the second pixel information, determine the target pixel belonging to the obstacle.

[0083] Among them, the target pixel belonging to the obstacle refers to a pixel in the image that belongs to the obstacle.

[0084] In this step, the target pixel belonging to the obstacle is determined by jointly using the first pixel information and the second pixel information. For example, the first pixel information stores each pixel belonging to the grounding wire, and the second pixel information stores the category of each pixel belonging to the grounding wire. If, for a certain pixel, neither the first pixel information nor the second pixel information stores its category, then that pixel does not belong to the obstacle. When both the first and second pixel information store information for a certain pixel, that pixel is determined to be the target pixel belonging to the obstacle.

[0085] Step S110: Determine the predicted position of the obstacle relative to the robot based on the target pixel, and determine the predicted category of the obstacle based on the semantic segmentation information corresponding to the target pixel.

[0086] For example, the pixel position of an obstacle in an image can be determined based on the target pixel belonging to the obstacle, and then the distance of the pixel position of the obstacle in the image can be calculated to obtain the predicted position of the obstacle relative to the robot in the world coordinate system.

[0087] In some embodiments, target pixels in the pixel coordinate system can be processed based on a mapping table to obtain the predicted position of the obstacle in the world coordinate system. The mapping table is used to represent the mapping relationship between the pixel coordinate system and the world coordinate system.

[0088] Furthermore, based on the semantic segmentation information corresponding to the target pixel, the category corresponding to the target pixel can be determined, thereby determining the predicted category of the obstacle.

[0089] In some embodiments, a grounding wire may include multiple target pixels, and the semantic segmentation information corresponding to each target pixel may be different. For example, if the grounding wire includes 100 target pixels, of which 60 target pixels are predicted as glass, 30 target pixels are predicted as metal, and 10 target pixels are predicted as wheels, then when determining the predicted category of an obstacle based on the semantic segmentation information determined by the target pixels, the predicted category of the obstacle can be determined by voting, that is, selecting the category with the most votes as the predicted category of the obstacle. In the above example, the predicted category of the obstacle corresponding to the grounding wire is glass.

[0090] In the aforementioned obstacle detection method, an image to be detected is acquired by the robot and input into an obstacle detection model. The obstacle detection model determines the key point information corresponding to the grounding wire in the image and the semantic segmentation information corresponding to each pixel in the image. Based on the key point information, the first pixel information belonging to the grounding wire in the image can be determined, and based on the semantic segmentation information, the second pixel information belonging to the grounding wire in the image can be determined. In other words, both the first and second pixel information correspond to the grounding wire information in the image. Then, based on the grounding wire information determined by the first and second pixel information, the target pixel belonging to the obstacle is determined. This improves the reliability of obstacle detection. Since the target pixel corresponds to the point where the obstacle contacts the ground, the predicted position of the obstacle relative to the robot can be determined based on the target pixel. Furthermore, since the obstacle detection model can obtain the semantic segmentation information corresponding to the image, and based on the semantic segmentation information, the category of each pixel in the image can be obtained, the predicted category of the obstacle can be obtained based on the semantic segmentation information corresponding to the target pixel. The predicted category and predicted position can then be input into downstream tasks to achieve the robot's multi-functionality and improve its obstacle avoidance capabilities. Furthermore, because the robot processes the captured images and identifies obstacles around it by analyzing the grounding wires in the images, it can recognize more types of obstacles, thereby improving the robot's obstacle detection accuracy.

[0091] After determining the category and location of obstacles based on the obstacle detection model, the robot's obstacle avoidance strategy or drivable area can be determined based on the obstacle category and location. In one embodiment, the robot determines the predicted position of the obstacle relative to the robot based on the target pixel, and after determining the predicted category of the obstacle based on the semantic segmentation information corresponding to the target pixel, steps 202 to 204 are included, wherein:

[0092] Step S202: Determine the position of the obstacle on the obstacle map based on the predicted position, so as to determine the robot's movement area based on the obstacle map.

[0093] This step is a post-processing step performed on the predicted location.

[0094] For example, after obtaining the predicted positions of obstacles around itself, the robot can mark these obstacles on an obstacle map based on their predicted positions, that is, determine the positions of the obstacles on the obstacle map based on their predicted positions. Thus, based on the obstacle map, the distribution of obstacles around the robot can be obtained, and the robot's movement area can be determined based on this obstacle map. When the robot moves within this movement area, it will not collide with any obstacles.

[0095] Step S204: If the obstacle is determined to be a dynamic obstacle based on the predicted category, then the attenuation coefficient of the obstacle on the obstacle map is set to the first attenuation coefficient; if the obstacle is determined to be a static obstacle based on the predicted category, then the attenuation coefficient of the obstacle on the obstacle map is set to the second attenuation coefficient, wherein the first attenuation coefficient is greater than the second attenuation coefficient.

[0096] This step is a post-processing step performed on the predicted category.

[0097] Dynamic obstacles refer to objects that move autonomously, such as people. Static obstacles refer to objects that are generally stationary, such as fixed tables or glass doors. The attenuation coefficient characterizes the time it takes for an obstacle to disappear from the obstacle map. A high attenuation coefficient indicates that the obstacle remains on the map for a short time, while a low attenuation coefficient indicates that it remains on the map for a longer time. The obstacle map is created to enable the robot to fully understand the obstacles within a pre-defined area. This is because the images captured by the camera constantly change during the robot's movement. For example, after the robot passes an obstacle, its camera may no longer be able to capture it. Recording the obstacle's position allows for a comprehensive understanding of the obstacles the robot encounters within the pre-defined area, enabling the robot to plan its movement route more effectively and reduce the probability of collisions.

[0098] Because dynamic and static obstacles differ in their movement speed, different attenuation coefficients need to be set for them on the obstacle map. That is, dynamic obstacles may only pass through a predetermined area and will not reappear around the robot for a considerable period of time; therefore, dynamic obstacles have a higher attenuation coefficient, meaning they remain on the obstacle map for a shorter time. For static obstacles, such as a door, although the door may not reappear in the real-time captured field of view for a period of time after the robot passes through it, because the door is stationary, the robot still needs to avoid it when circling back to prevent collisions. Therefore, static obstacles have a lower attenuation coefficient on the obstacle map.

[0099] Therefore, after obtaining the predicted location and category of obstacles, the robot can perform the above post-processing to improve the robot's obstacle avoidance accuracy.

[0100] In some embodiments, to reduce false positives for static obstacles, the obstacle category can be verified. Optionally, the robot's real-time pose can be used to confirm whether the obstacle is a static obstacle using multiple frames, thereby reducing the false detection rate. For example, the robot can process two consecutively acquired image frames, mapping the obstacles identified in both frames onto an obstacle map. If the position of a static obstacle in the obstacle map does not change in these two frames, it can be determined that the obstacle is a static obstacle, and a corresponding attenuation coefficient can then be set for that obstacle.

[0101] In this embodiment, by post-processing the predicted category and predicted location after obtaining the obstacle, the information of the obstacle on the obstacle map, such as its position and attenuation coefficient, can be determined based on the predicted category and predicted location of the obstacle. This allows the robot to determine the movable area based on the obstacle map, which can improve both the accuracy of obstacle avoidance and the efficiency of the robot in determining the movement route.

[0102] In some embodiments, the attenuation coefficient of a dynamic obstacle can be determined based on its moving speed. For example, the moving speed and the attenuation coefficient can have a linear relationship: the faster the moving speed, the larger the attenuation coefficient; the slower the moving speed, the smaller the attenuation coefficient. The moving speed can be determined based on the predicted category of the obstacle and the distance between the obstacle after mapping two frames of images. If the predicted category of the obstacle is "wheel" and the distance between the location of the obstacle after mapping the first frame and the location after mapping the second frame is large, it indicates that the obstacle is a dynamic obstacle with a fast moving speed, and it can attenuate quickly on the obstacle map. If the predicted category of the obstacle is "foot" and the distance between the location of the obstacle after mapping the first frame and the location after mapping the second frame is small, it indicates that the obstacle is a dynamic obstacle with a slow moving speed, and it can attenuate slowly on the obstacle map. Determining the attenuation coefficient of a dynamic obstacle based on its moving speed can improve the accuracy and rationality of setting the attenuation coefficient, thereby improving the accuracy of the obstacle map and enabling the planning of efficient and correct driving paths.

[0103] In some embodiments, the obstacle detection model outputs key point information of the grounding wire. To obtain the corresponding grounding wire information, this key point information needs to be processed. For example, the processor can determine the first pixel information in the image to be detected that belongs to the grounding wire based on the key point information.

[0104] The first pixel information can refer to the pixel information of the grounding wire in the image to be detected, such as which pixels in the image constitute a certain grounding wire. Keypoint information is the keypoint information corresponding to the grounding wire obtained by the obstacle detection model. For example, for the grounding wire of a piece of glass in an image, the obtained keypoint information could be 15 pixel coordinates, the coordinates of the top-left corner and the bottom-right corner of the bounding box. It can be understood that for an image, the keypoint information can include multiple sets, each set corresponding to the keypoint information of one grounding wire. Optionally, the keypoint information can include bounding box information and key pixels. The bounding box information can be the smallest bounding rectangle surrounding the key pixels, and the bounding box information can be the coordinates of the top-left corner and the bottom-right corner of this smallest bounding rectangle, or the coordinates of the center point, width, and height of this smallest bounding rectangle. Key pixels can be the pixel coordinates of the key points corresponding to the grounding wire. The number of key pixels varies depending on the type of grounding wire.

[0105] For example, after obtaining the key point information, multiple bounding boxes in the image to be detected can be obtained based on the bounding box information. Each bounding box is used to locate the corresponding grounding line. Therefore, the key pixels within the bounding box corresponding to the bounding box information can be connected sequentially to obtain the connecting line, i.e., the grounding line; and then the first pixel information is obtained based on the pixels of the connecting line.

[0106] In this embodiment, by sequentially connecting the key pixels within the bounding box corresponding to the bounding box information, a connecting line is obtained. The grounding line is obtained by connecting the key points, which simplifies the grounding line identification process and improves the grounding line identification efficiency.

[0107] Understandably, before using an obstacle detection model to detect ground wires in an image, it's necessary to train an initial model using training samples to obtain the target obstacle detection model. This involves using a loss function to constrain the model's learning results, ensuring the generated obstacle detection model can correctly predict the type and location of ground wires in the image, thus determining obstacles around the robot. Therefore, an initial model needs to be trained before obtaining the image to be detected to obtain the obstacle detection model. One possible implementation is as follows: Figure 2 As shown, before obtaining the image to be detected, steps 302 to 306 are included. Wherein:

[0108] Step S302: Obtain training samples.

[0109] The training samples include labeled images and annotation information for the grounding wires within those images. The labeled images can be fisheye images taken with a fisheye camera. To enable training to predict the location and category of the grounding wires, the training samples also need to include annotation information for the location and category of the grounding wires corresponding to the labeled images.

[0110] Step S304: Obtain the annotation category of each pixel in the annotation image based on the grounding wire annotation category to obtain the semantic segmentation annotation information of the annotation image, and obtain the key point annotation information of the grounding wire in the annotation image based on the grounding wire annotation points.

[0111] In this context, grounding wire annotation points refer to the pixel coordinates of key points on the grounding wire in the image. Corresponding to the key point information predicted by the model, the number of key points in the corresponding grounding wire annotation points varies for different types of grounding wires. For example, the grounding wire annotation points in an annotated image contain multiple sets of two-dimensional pixel coordinates. The number of pixels in each set may differ; some may have 5, while others may have 15. Processing the grounding wire annotation points yields the key point annotation information. This key point annotation information includes the pixel coordinates and visibility of the key points, as well as the bounding box information corresponding to the grounding wire's bounding box. Each pixel coordinate of a grounding wire annotation point is essentially the pixel coordinate of the key point in the key point annotation information.

[0112] The grounding wire label category refers to the category of the grounding wires in the image. For example, the grounding wire label category for a certain image could be "the object within bounding box A is metal; the object within bounding box B is glass; the object within bounding box C is a key." To obtain the semantic segmentation annotation information of the labeled image, each set of two-dimensional pixel coordinates can be connected sequentially, and then the resulting grounding wires can be mapped to the grounding wire label categories to obtain the category to which each grounding wire belongs. Then, each pixel in the labeled image is analyzed to determine the category to which each pixel belongs, and the categories of each pixel are integrated to obtain the semantic segmentation information. The semantic segmentation information stores the category corresponding to each pixel in the image. Optionally, the category of pixels that are not grounding wires can be set to background.

[0113] Step S306: Train the semantic network of the model to be trained based on the labeled images and the corresponding semantic segmentation annotation information, and train the key point network of the model to be trained based on the labeled images and the corresponding key point annotation information to obtain the obstacle detection model.

[0114] In obstacle detection models, because the model needs to predict both the keypoint information and the category of the grounding wire, it is a multi-task model. To improve the prediction accuracy of each task, the model is configured to include two network branches, one for predicting keypoints and the other for predicting the category. For example, the model to be trained can include a semantic network and a keypoint network. The semantic network can be trained based on labeled images and corresponding semantic segmentation annotations. The keypoint network can be trained based on labeled images and corresponding keypoint annotations. Training the multi-task model separately in this way achieves convergence of the loss function and improves the detection accuracy of each network branch. Furthermore, the prediction results of each branch can be combined to determine the final grounding wire information, thereby improving the model's detection rate.

[0115] In this embodiment, by acquiring training samples and training the semantic network of the model to be trained based on the labeled images and corresponding semantic segmentation annotation information in the training samples, and training the key point network of the model to be trained based on the labeled images and corresponding key point annotation information, a multi-task detection model can be obtained, and the prediction accuracy of the model can be improved by training separately.

[0116] As can be seen from the above, the learning network in the obstacle detection model can be divided into semantic networks and keypoint networks. Semantic networks are used to learn the category of grounding wires in an image based on image features, while keypoint networks are used to learn keypoint information of the grounding wires in an image based on image features. Keypoint information can include, for example, the number of keypoints, keypoint locations, and bounding box information.

[0117] During the training of the semantic network in the training model, a loss function can be used to constrain the semantic segmentation information generated by the semantic network, thereby enabling the semantic network in the obstacle detection model to correctly identify the category of grounding wires in the image. In one embodiment, training the semantic network of the training model based on the labeled image and the corresponding semantic segmentation information includes steps 402 to 406. Wherein:

[0118] Step S402: Obtain semantic segmentation prediction information of the labeled image based on the semantic network of the model to be trained.

[0119] Step S404: Substitute the semantic segmentation prediction information and semantic segmentation annotation information into the semantic loss function.

[0120] For example, the semantic loss function includes a first loss function and a second loss function. The first loss function adjusts the similarity between the semantic segmentation prediction information and the semantic segmentation annotation information, while the second loss function adjusts the balance between the difficulty and ease of training samples. The first loss function could be, for example, the diceloss function. The diceloss function is a loss function suitable for segmentation problems; it measures prediction accuracy by calculating the intersection-over-union ratio (IoU) between the predicted and ground truth values. The second loss function could be, for example, the focal loss function. Training the semantic network based on a semantic loss function established by combining multiple loss functions can yield more accurate parameters, thereby improving the accuracy and efficiency of the semantic network in predicting ground truth semantic segmentation values.

[0121] Step S406: Train the semantic network based on the semantic loss function.

[0122] In this embodiment, semantic segmentation prediction information of labeled images is obtained based on the semantic network of the model to be trained. Then, the semantic segmentation prediction information and the corresponding semantic segmentation annotation information of the labeled images are substituted into a preset semantic loss function. The parameters of the semantic network are adjusted based on the result of the semantic loss function. That is, the semantic network is trained based on the semantic loss function, thereby reducing the difference between the semantic segmentation prediction information and the semantic segmentation annotation information, making the semantic segmentation information predicted by the semantic network more accurate. Furthermore, through the simultaneous semantic loss function, the semantic network can solve the problem of imbalance between easy and difficult samples, giving more attention to difficult samples.

[0123] As mentioned earlier, the number of key points for the grounding wire may differ depending on the type of obstacle. In one possible implementation, the number of key points in the key point information can be divided into 5 key points and 15 key points. When the model to be trained learns to predict the key point information of the grounding wire, in order to make the key point network's prediction of the key point information more accurate, the different numbers of key points can be trained separately. That is, in one embodiment, the key point annotation information includes a first type of key point annotation information and a second type of key point annotation information. The first type of key point annotation information is used to annotate the key point information of the first type of grounding wire, and the number of key points on the first type of grounding wire is a first number. The second type of key point annotation information is used to annotate the key point information of the second type of grounding wire, and the number of key points on the second type of grounding wire is a second number. The first number is less than the second number. The key point network includes a first key point network and a second key point network. The first key point network is used to train the first type of grounding wire, and the second key point network is used to train the second type of grounding wire. Therefore, in this embodiment, when training the keypoint network of the model to be trained based on the labeled images and the corresponding keypoint annotation information, the first keypoint network is trained based on the first type of keypoint annotation information and the corresponding labeled images; and the second keypoint network is trained based on the second type of keypoint annotation information and the corresponding labeled images.

[0124] For example, keypoint annotation information can be divided into annotations of 5 keypoints and annotations of 15 keypoints. The annotations of 5 keypoints and 15 keypoints are input separately into two keypoint networks to train the corresponding network parameters. That is, for the annotations of 5 keypoints and 15 keypoints, the keypoint loss function is constructed in two separate branches. This improves the detection efficiency of each branch, enabling the two generated keypoint networks to better detect the keypoints of the grounding wire.

[0125] In this embodiment, by training the first keypoint network based on the first type of keypoint annotation information and the corresponding annotation image, and by training the second keypoint network based on the second type of keypoint annotation information and the corresponding annotation image, the model can learn different types of keypoints, thereby improving the prediction accuracy of the model.

[0126] The step of training the first keypoint network based on the first type of keypoint annotation information and the corresponding annotation image can be implemented through the following steps: predicting the first type of keypoints in the annotation image based on the first keypoint network to obtain the first type of keypoint prediction information; substituting the first type of keypoint prediction information and the first type of keypoint annotation information into the first type of keypoint loss function; and training the first keypoint network based on the first type of keypoint loss function.

[0127] The step of training a second keypoint network based on the annotation information of the second type of keypoints and the corresponding annotated images can be achieved through the following steps: predicting the second type of keypoints in the annotated images based on the second keypoint network to obtain the prediction information of the second type of keypoints; substituting the prediction information of the second type of keypoints and the annotation information of the second type of keypoints into the loss function of the second type of keypoints; and training the second keypoint network based on the loss function of the second type of keypoints.

[0128] It is understandable that the first keypoint network and the second keypoint network can have the same structure, but due to the different training samples, the network parameters of the first keypoint network and the second keypoint network may be different or the same.

[0129] In some embodiments, the first and second keypoint loss functions can be composed of at least one of a third, fourth, fifth, and sixth loss function. The third loss function can be the CIoU loss function. The fourth loss function can be the BCE loss function. The fifth loss function can be the DFL loss function. The sixth loss function can be the Pose loss function. The CIoU (Complete IoU Loss) loss function is a loss function used for object detection tasks; it is an improved version of IoU (Intersection over Union). CIoU considers not only the overlap area between the predicted and ground truth boxes (i.e., the overlap area between the bounding boxes in the keypoint annotation information and the bounding boxes in the keypoint prediction information), but also the distance between their center points and their aspect ratio. The BCE (Binary Cross-Entropy) loss function measures the difference between the actual output and the target output (i.e., the difference between the keypoint pixel coordinates in the keypoint annotation information and the keypoint pixel coordinates in the keypoint prediction information). The Distance-In-U (DFL) loss function aims to directly minimize the distance between the predicted bounding box and the ground truth bounding box, thereby improving localization accuracy. DFL loss considers the distance loss between the predicted and ground truth bounding boxes, the IoU (overlap area) loss, and their aspect ratio loss. Pose loss (pose estimation loss) is a loss function used for pose estimation tasks. Pose loss can calculate the average Euclidean distance between predicted and ground truth keypoints, measure the proportion of errors between predicted and ground truth keypoints within a certain threshold, and consider the importance of keypoints and the correlation between them.

[0130] In some embodiments, the keypoint loss function can be constructed by combining the four loss functions described above. The corresponding keypoint loss value can be obtained by weighting the loss values ​​obtained from the four loss functions.

[0131] By following the steps above, we can effectively learn about grounding wires with varying degrees of complexity at key points, thereby improving the recognition rate of key points of grounding wires in images.

[0132] In an exemplary embodiment, to comprehensively determine obstacles around a robot by identifying grounding wires in an image, an obstacle detection model can be trained. The obstacle detection model is a multi-task model based on semantic segmentation and keypoint detection. The encoding part of the model can use a mobile network. This meets the requirements of high detection rate and high real-time performance on embedded devices. The encoding part of the model is used to extract features from the image to better learn the deep features of the image. Therefore, the obstacle detection model can be divided into a semantic network and a keypoint network.

[0133] During model training, training samples can be obtained. These training samples include labeled images and annotation information of grounding wires within the labeled images. Based on the grounding wire annotation categories in the annotation information, the annotation category of each pixel in the labeled image is obtained to acquire semantic segmentation annotation information for the labeled image. Based on the grounding wire annotation points in the annotation information, the key point annotation information of the grounding wires in the labeled image is obtained.

[0134] In this embodiment, the grounding wires of various obstacles are defined as line segments of individual instances, including but not limited to the following categories: metal bases, glass, human feet, wheels, and thin stool legs. Based on the shape characteristics of each obstacle, the line segments for metal bases, glass, and human feet are fixed at 15 points; the line segments for wheels and thin stool legs are fixed at 5 points. The annotation information for the labeled image can be based on the above definitions to annotate the grounding wires.

[0135] For semantic networks, semantic segmentation prediction information of labeled images can be obtained based on the semantic network of the model to be trained; the semantic segmentation prediction information and semantic segmentation annotation information are substituted into the semantic loss function; and the semantic network is trained based on the semantic loss function. For example, ground truth semantic segmentation data of ground lines for each image can be generated using manually annotated labels (i.e., annotation information); then, a joint semantic loss function is constructed using the diceloss function and the focalloss function for learning.

[0136] For keypoint networks, they can be divided into first keypoint networks and second keypoint networks. First keypoint networks can be used to predict first-class keypoints in an labeled image, obtaining first-class keypoint prediction information. This first-class keypoint prediction information and the first-class keypoint annotation information are then substituted into a first-class keypoint loss function. The first keypoint network is then trained based on this first-class keypoint loss function.

[0137] The second keypoint network is used to predict the second type of keypoints in the labeled image to obtain the prediction information of the second type of keypoints; the prediction information of the second type of keypoints and the labeling information of the second type of keypoints are substituted into the loss function of the second type of keypoints; the second keypoint network is trained based on the loss function of the second type of keypoints.

[0138] For example, firstly, ground truth values ​​for keypoints in each image are generated using manually labeled tags. These include the x, y, w, and h coordinates of each grounding line's box (i.e., bounding box) (x and y represent the center coordinates of the bounding box, w represents the width of the bounding box, and h represents the height of the bounding box), as well as the location and visibility of the keypoints for the grounding lines. Additionally, a first keypoint network is used to predict the keypoints of grounding lines with 5 keypoints, and a second keypoint network is used to predict the keypoints of grounding lines with 15 keypoints. Furthermore, the 5-keypoint and 15-keypoint categories are treated as separate branches for anchor-free loss construction. By classifying different types of grounding lines into those with varying numbers of keypoints, the keypoint network can determine the number of keypoints for each grounding line based on its shape characteristics, thereby improving the accuracy of keypoint recognition.

[0139] Multi-task models often suffer from loss function convergence difficulties. This embodiment first obtains a pre-trained model based on large datasets. Then, adaptive multi-task loss weights are used to train the model.

[0140] After training the obstacle detection model, the robot can obtain images to be detected captured by the robot. The images are input into the obstacle detection model, which determines key point information corresponding to the grounding wire in the image and semantic segmentation information corresponding to the image. The semantic segmentation information characterizes the category of each pixel in the image. Based on the key point information, a first pixel belonging to the grounding wire is determined, and based on the semantic segmentation information, a second pixel belonging to the grounding wire is determined. Based on the first and second pixel information, a target pixel belonging to an obstacle is determined. Based on the target pixel, the predicted position of the obstacle relative to the robot is determined, and based on the semantic segmentation information corresponding to the target pixel, the predicted category of the obstacle is determined.

[0141] The multi-branch design and usage are based on minimizing false detections in the algorithm. After obtaining the semantic segmentation and keypoint detection results, further evaluation is performed on these results. Only pixels where both branches agree there is an obstacle are confirmed as obstacle points. Finally, based on the category distribution of the pixel region, a voting-based obstacle category is assigned. For example, this could involve analyzing the semantic segmentation information corresponding to each pixel within the bounding box to determine the non-background category with the most pixels within the bounding box, such as glass or a metal base.

[0142] The following post-processing can then be performed: determine the position of the obstacle on the obstacle map based on the predicted position, so as to determine the robot's movement area based on the obstacle map.

[0143] The following post-processing can also be performed: if the obstacle is determined to be a dynamic obstacle based on the predicted category, then the attenuation coefficient of the obstacle on the obstacle map is set to a first attenuation coefficient; if the obstacle is determined to be a static obstacle based on the predicted category, then the attenuation coefficient of the obstacle on the obstacle map is set to a second attenuation coefficient, wherein the first attenuation coefficient is greater than the second attenuation coefficient.

[0144] For example, obstacles are categorized into dynamic and static obstacles, and different post-processing strategies are used. For dynamic obstacles, Kalman filtering can be used to filter and track dynamic obstacles across multiple frames, with a set lifespan, such as 5 seconds. The attenuation coefficient on the obstacle map is also higher. For static obstacles, the robot's real-time pose can be utilized, with two frames used for confirmation to reduce false positives. The attenuation coefficient on the obstacle map is also lower.

[0145] In this embodiment, fisheye cameras can improve the robot's field of vision. Within a certain budget, two fisheye cameras can cover the target area, such as a 2m range covering 180 degrees horizontally and vertically in front of the robot. Furthermore, using fisheye cameras can reduce the number of missed objects, such as highly reflective metal bases, wires, and glass. In addition, detected obstacles can carry specific semantic information, facilitating subsequent dynamic and static post-processing based on semantics. For example, static objects can be preserved or attenuated to a certain extent, thereby improving the feasibility and convenience of downstream tasks.

[0146] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0147] Based on the same inventive concept, this application also provides a detection device for implementing the obstacle detection method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more obstacle detection device embodiments provided below can be found in the limitations of the obstacle detection method described above, and will not be repeated here.

[0148] In one exemplary embodiment, such as Figure 3 As shown, an obstacle detection device 300 is provided, including: an image acquisition module 310, an information prediction module 320, an information processing module 330, and an obstacle information detection module 340, wherein:

[0149] Image acquisition module 310 is used to acquire the image to be detected captured by the robot;

[0150] The information prediction module 320 is used to input the image to be detected into the obstacle detection model, and determine the key point information corresponding to the grounding wire in the image to be detected and the semantic segmentation information corresponding to each pixel in the image to be detected through the obstacle detection model. The semantic segmentation information is used to characterize the category of each pixel in the image to be detected.

[0151] The information processing module 330 is used to determine, based on the key point information, the first pixel information belonging to the grounding wire in the image to be detected; and to determine, based on the semantic segmentation information, the second pixel information belonging to the grounding wire in the image to be detected; and to determine, based on the first pixel information and the second pixel information, the target pixel belonging to the obstacle.

[0152] The obstacle information detection module 340 is used to determine the predicted position of the obstacle relative to the robot based on the target pixel, and to determine the predicted category of the obstacle based on the semantic segmentation information corresponding to the target pixel.

[0153] In some embodiments, the obstacle detection device may further include a downstream task module, wherein the downstream task module is configured to obtain the actual position of the obstacle on the obstacle map based on the mapping table and the predicted position, obtain the actual distance between the robot and the obstacle based on the actual position, and determine the robot's driving area based on the actual distance and the obstacle map, wherein the mapping table is used to characterize the mapping relationship between the pixel coordinate system where the position information is located and the world coordinate system where the actual distance is located.

[0154] In some embodiments, the downstream task module is further configured to, if the obstacle is determined to be a dynamic obstacle based on the prediction category, set the attenuation coefficient of the obstacle on the obstacle map to a first attenuation coefficient; and if the obstacle is determined to be a static obstacle based on the prediction category, set the attenuation coefficient of the obstacle on the obstacle map to a second attenuation coefficient, wherein the first attenuation coefficient is greater than the second attenuation coefficient.

[0155] In some embodiments, the information processing module 330 is used to sequentially connect the key pixels within the detection box corresponding to the detection box information to obtain a connecting line; and obtain the first pixel information based on the pixels of the connecting line.

[0156] In some embodiments, the robot may further include an obstacle training device, wherein the obstacle training device may include a sample acquisition module, an annotation information processing module, and a training module. Wherein:

[0157] The sample acquisition module is used to acquire training samples, which include labeled images and the annotation information of grounding lines in the labeled images;

[0158] The annotation information processing module is used to obtain the annotation category of each pixel in the annotation image based on the grounding wire annotation points and grounding wire annotation categories in the annotation information to obtain the semantic segmentation annotation information of the annotation image, and to obtain the key point annotation information of the grounding wire in the annotation image based on the grounding wire annotation points;

[0159] The training module is used to train the semantic network of the model to be trained based on the labeled image and the corresponding semantic segmentation label information, and to train the key point network of the model to be trained based on the labeled image and the corresponding key point label information, so as to obtain the obstacle detection model.

[0160] In some embodiments, the training module further includes a semantic network training module, wherein the semantic network training module is used to obtain semantic segmentation prediction information of the labeled image based on the semantic network of the network to be trained; substitute the semantic segmentation prediction information and the semantic segmentation annotation information into a semantic loss function, the semantic loss function including a first loss function and a second loss function, the first loss function being used to adjust the similarity between the semantic segmentation prediction information and the semantic segmentation annotation information, and the second loss function being used to adjust the difficulty balance of the training samples; and train the semantic network based on the semantic loss function.

[0161] In some embodiments, the key point annotation information includes a first type of key point annotation information and a second type of key point annotation information. The first type of key point annotation information is used to annotate the key point information of a first type of grounding wire, and the number of key points on the first type of grounding wire is a first quantity. The second type of key point annotation information is used to annotate the key point information of a second type of grounding wire, and the number of key points on the second type of grounding wire is a second quantity. The first quantity is less than the second quantity. The key point network includes a first key point network and a second key point network. The first key point network is used to train the first type of grounding wire, and the second key point network is used to train the second type of grounding wire.

[0162] In some embodiments, the training module further includes a keypoint network training module, wherein the keypoint network training module is used to train the first keypoint network based on the first type of keypoint annotation information and the corresponding annotation image; and to train the second keypoint network based on the second type of keypoint annotation information and the corresponding annotation image.

[0163] In some embodiments, the keypoint network training module is further configured to predict a first type of keypoints in the labeled image based on the first keypoint network to obtain first type of keypoint prediction information; substitute the first type of keypoint prediction information and the first type of keypoint labeling information into a first type of keypoint loss function; and train the first keypoint network based on the first type of keypoint loss function.

[0164] In some embodiments, the keypoint network training module is further configured to predict a second type of keypoints in the labeled image based on the second keypoint network to obtain second type of keypoint prediction information; substitute the second type of keypoint prediction information and the second type of keypoint labeling information into the second type of keypoint loss function; and train the second keypoint network based on the second type of keypoint loss function.

[0165] The modules in the obstacle detection device and obstacle training device described above can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0166] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data related to the application and training processes of the obstacle detection model. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements an obstacle detection method.

[0167] In one exemplary embodiment, an electronic device is provided, which may be a robot, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements an obstacle detection method. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0168] Those skilled in the art will understand that Figure 4 or Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0169] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0170] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0171] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0172] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0173] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0174] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0175] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for detecting obstacles, characterized in that, The method includes: Obtain training samples, which include labeled images and annotation information of grounding lines in the labeled images; Based on the grounding wire annotation category in the annotation information, the annotation category of each pixel in the annotation image is obtained to obtain the semantic segmentation annotation information of the annotation image; and based on the grounding wire annotation points in the annotation information, the key point annotation information of the grounding wire in the annotation image is obtained. The semantic network of the model to be trained is trained based on the labeled image and the corresponding semantic segmentation label information, and the key point network of the model to be trained is trained based on the labeled image and the corresponding key point label information to obtain an obstacle detection model. Obtain the image to be inspected captured by the robot; The image to be detected is input into an obstacle detection model, and the obstacle detection model determines the key point information corresponding to the grounding wire in the image to be detected, and the semantic segmentation information corresponding to the image to be detected. The semantic segmentation information is used to characterize the category of each pixel in the image to be detected. Based on the key point information, the first pixel information belonging to the grounding wire in the image to be detected is determined, and based on the semantic segmentation information, the second pixel information belonging to the grounding wire in the image to be detected is determined. Based on the first pixel information and the second pixel information, the target pixel belonging to the obstacle is determined; The predicted position of the obstacle relative to the robot is determined based on the target pixel, and the predicted category of the obstacle is determined based on the semantic segmentation information corresponding to the target pixel.

2. The detection method according to claim 1, characterized in that, After determining the predicted position of the obstacle relative to the robot based on the target pixel, and determining the predicted category of the obstacle based on the semantic segmentation information corresponding to the target pixel, the process includes: The position of the obstacle on the obstacle map is determined based on the predicted position, so as to determine the robot's movement area based on the obstacle map; If the obstacle is determined to be a dynamic obstacle based on the predicted category, then the attenuation coefficient of the obstacle on the obstacle map is set to a first attenuation coefficient; if the obstacle is determined to be a static obstacle based on the predicted category, then the attenuation coefficient of the obstacle on the obstacle map is set to a second attenuation coefficient, wherein the first attenuation coefficient is greater than the second attenuation coefficient.

3. The detection method according to claim 1, characterized in that, The key point information includes bounding box information and key pixels. Determining the first pixel information belonging to the grounding line in the image to be detected based on the key point information includes: Connect the key pixels within the bounding box corresponding to the bounding box information in sequence to obtain the connecting line; The first pixel information is obtained based on the pixels of the connecting line.

4. The detection method according to claim 1, characterized in that, The step of training the semantic network of the model to be trained based on the labeled image and the corresponding semantic segmentation annotation information includes: Semantic segmentation prediction information of the labeled image is obtained based on the semantic network of the model to be trained; Substitute the semantic segmentation prediction information and the semantic segmentation annotation information into the semantic loss function; The semantic network is trained based on the semantic loss function.

5. The detection method according to claim 1, characterized in that, The keypoint annotation information includes a first type of keypoint annotation information and a second type of keypoint annotation information. The first type of keypoint annotation information is used to annotate the keypoint information of a first type of grounding wire, and the number of keypoints on the first type of grounding wire is a first quantity. The second type of keypoint annotation information is used to annotate the keypoint information of a second type of grounding wire, and the number of keypoints on the second type of grounding wire is a second quantity. The first quantity is less than the second quantity. The keypoint network includes a first keypoint network and a second keypoint network. The first keypoint network is used to train the first type of grounding wire, and the second keypoint network is used to train the second type of grounding wire. The step of training the keypoint network of the model to be trained based on the annotated image and the corresponding keypoint annotation information includes: The first keypoint network is trained based on the first type of keypoint annotation information and the corresponding annotation image; The second keypoint network is trained based on the second type of keypoint annotation information and the corresponding annotation images.

6. The detection method according to claim 5, characterized in that, The step of training the first keypoint network based on the first type of keypoint annotation information and the corresponding annotated image includes: Based on the first keypoint network, the first type of keypoints in the labeled image are predicted to obtain the first type of keypoint prediction information. Substitute the first type of key point prediction information and the first type of key point annotation information into the first type of key point loss function; The first keypoint network is trained based on the first type of keypoint loss function; The training of the second keypoint network based on the second type of keypoint annotation information and the corresponding annotated image includes: Based on the second keypoint network, the second type of keypoints in the labeled image are predicted to obtain the second type of keypoint prediction information. Substitute the second type of keypoint prediction information and the second type of keypoint annotation information into the second type of keypoint loss function; The second keypoint network is trained based on the second type of keypoint loss function.

7. An obstacle detection device, characterized in that, The device includes: The image acquisition module is used to acquire the image to be detected captured by the robot; The information prediction module is used to input the image to be detected into the obstacle detection model, and to determine the key point information corresponding to the grounding wire in the image to be detected and the semantic segmentation information corresponding to each pixel in the image to be detected through the obstacle detection model. The semantic segmentation information is used to characterize the category of each pixel in the image to be detected. The information processing module is used to determine, based on the key point information, the first pixel information belonging to the grounding wire in the image to be detected; and to determine, based on the semantic segmentation information, the second pixel information belonging to the grounding wire in the image to be detected; and to determine, based on the first pixel information and the second pixel information, the target pixel belonging to the obstacle. An obstacle information detection module is used to determine the predicted position of the obstacle relative to the robot based on the target pixel, and to determine the predicted category of the obstacle based on the semantic segmentation information corresponding to the target pixel; A sample acquisition module is used to acquire training samples, the training samples including labeled images and the annotation information of grounding lines in the labeled images; The annotation information processing module is used to obtain the annotation category of each pixel in the annotation image based on the grounding wire annotation points and grounding wire annotation categories in the annotation information to obtain the semantic segmentation annotation information of the annotation image, and to obtain the key point annotation information of the grounding wire in the annotation image based on the grounding wire annotation points; The training module is used to train the semantic network of the model to be trained based on the labeled images and the corresponding semantic segmentation annotation information, and to train the key point network of the model to be trained based on the labeled images and the corresponding key point annotation information, so as to obtain the obstacle detection model.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images