Obstacle avoidance method, device, equipment and medium

Abstract

The invention relates to the field of robot obstacle avoidance, in particular to an obstacle avoidance method, device and equipment and a medium. The method comprises the following steps: acquiring a single-frame image acquired by a monocular camera arranged on a walking device; performing image recognition according to the single-frame image, and determining a first pixel coordinate of a target point in the drivable area; acquiring a rolling angle, a first pitch angle and a horizontal distance of the walkable equipment; according to the roll angle, the first pitch angle, the horizontal distance and the first pixel coordinate, the actual distance between the target obstacle and the monocular camera is determined, and the actual distance is used for determining an obstacle avoidance strategy. Image recognition is performed according to the single frame image, the point with the maximum pixel coordinate v value of the edge of the target obstacle is selected as the target point, the actual distance between the target obstacle and the monocular camera is jointly determined according to the roll angle, the first pitch angle, the horizontal distance and the first pixel coordinate, and the distance between the target obstacle and the monocular camera is determined only by using the camera and the segmentation algorithm. Therefore, accurate distance measurement can be completed, and accurate obstacle avoidance can be realized.

Description

Technical Field

[0001] The present application relates to the field of robot obstacle avoidance, and in particular to an obstacle avoidance method, device, equipment and medium. Background Art

[0002] In today's intelligent scenarios, accurate and efficient distance measurement technology is essential to realize various application functions. In robot navigation, path planning, such as inspection robots or lawn mowing robots, requires accurate measurement of the distance between objects and equipment.

[0003] Among the existing distance measurement solutions, a common method is to use the camera's elevation for monocular distance measurement. This method uses the image information captured by the monocular camera and combines the monocular camera's internal and external parameters to calculate the actual distance between the object and the camera. This method can achieve high distance measurement accuracy under ideal conditions, and the calculation process is relatively simple.

[0004] However, with the complexity and diversification of application scenarios, the ground in the actual environment is often uneven, and this method is difficult to accurately measure its distance, which limits its scope of application.

[0005] Therefore, how to accurately measure distance to adapt to more complex application scenarios has become a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the invention

[0006] The purpose of this application is to provide an obstacle avoidance method, device, equipment and medium, which can complete accurate distance measurement and thus achieve accurate obstacle avoidance using only a camera and a segmentation algorithm.

[0007] In a first aspect, an obstacle avoidance method is provided, comprising:

[0008] Obtain a single-frame image captured by a monocular camera set on a walkable device;

[0009] Perform image recognition based on the single-frame image to determine the first pixel coordinate of a target point in the drivable area, wherein the target point is a point with the largest pixel coordinate v value at the edge of the target obstacle in the single-frame image;

[0010] Obtaining a roll angle, a first pitch angle, and a horizontal distance of the walkable device, wherein the horizontal distance is a distance between an intersection of a monocular camera center perpendicular to the ground and a point where the wheel touches the ground;

[0011] The actual distance between the target obstacle and the monocular camera is determined according to the roll angle, the first pitch angle, the horizontal distance and the first pixel coordinate, and the actual distance is used to determine an obstacle avoidance strategy.

[0012] In a preferred example, the present application may be further configured to: determine the actual distance of the target obstacle from the monocular camera according to the roll angle, the first pitch angle, the horizontal distance and the first pixel coordinate, including:

[0013] Correcting the first pixel coordinate according to the roll angle to obtain a second pixel coordinate;

[0014] Determining a true height of the monocular camera according to the first pitch angle and the horizontal distance;

[0015] Determining a second pitch angle according to the first pitch angle and an installation angle of the monocular camera;

[0016] Determine the actual distance of the target obstacle from the monocular camera according to the second pixel coordinate, the true height, the second pitch angle and the horizontal distance.

[0017] In a preferred example, the present application may be further configured as follows: after determining the actual distance of the target obstacle from the monocular camera according to the roll angle, the first pitch angle, the horizontal distance and the first pixel coordinate, the application may further include:

[0018] When the actual distance is less than a set threshold, determining the type of the monocular camera;

[0019] When the type of the monocular camera is a rear-view camera, controlling the walkable device to stop moving backward;

[0020] When the type of the monocular camera is a forward-looking camera, the motion mode of the walkable device is obtained, and the walkable device is controlled to perform an obstacle avoidance operation according to the motion mode.

[0021] In a preferred example, the present application may be further configured as follows: controlling the walkable device to perform obstacle avoidance operations according to the motion mode, including:

[0022] When the movement mode is a U-turn mode upon encountering an obstacle, controlling the walkable device to stop moving forward and then turn around;

[0023] When the motion mode is the detour mode, the obstacle area in the left area and the obstacle area in the right area of ​​the drivable area in the single-frame image are determined; and the walkable device is controlled to detour in a target direction, wherein the target direction is a direction with a smaller area between the obstacle area in the left area and the obstacle area in the right area.

[0024] In a preferred example, the present application may be further configured to: obtain the motion mode of the walkable device, including:

[0025] According to the target obstacle area in the single frame image;

[0026] Identify the target obstacle area and determine the obstacle category;

[0027] When the obstacle category is a boundary obstacle, the movement mode is determined to be a U-turn mode upon encountering an obstacle; otherwise, the movement mode is determined to be a detour mode.

[0028] In a preferred example, the present application may be further configured as follows: performing image recognition according to a single frame image to determine the first pixel coordinate of a target point in the drivable area includes:

[0029] Inputting the single frame image into a segmentation model to obtain a segmentation mask;

[0030] According to the segmentation mask, the obstacle contour points in the drivable area are screened to determine the first pixel coordinates of the target point in the drivable area.

[0031] In a preferred example, the present application can be further configured as follows: the segmentation model is a pidnet model using depthwise separable convolution; the depthwise separable convolution includes: depthwise convolution and pointwise convolution.

[0032] In a second aspect, an obstacle avoidance device is provided, comprising:

[0033] A first acquisition module is used to acquire a single-frame image captured by a monocular camera arranged on the walkable device;

[0034] A pixel coordinate determination module, used to perform image recognition based on a single-frame image to determine a first pixel coordinate of a target point in a drivable area, wherein the target point is a point with a maximum pixel coordinate v value at the edge of a target obstacle in the single-frame image;

[0035] A second acquisition module is used to acquire a roll angle, a first pitch angle and a horizontal distance of the walkable device, where the horizontal distance is a distance between an intersection of a monocular camera center perpendicular to the ground and a point where the wheel touches the ground;

[0036] A distance determination module is used to determine the actual distance between the target obstacle and the monocular camera according to the roll angle, the first pitch angle, the horizontal distance and the first pixel coordinate, and the actual distance is used to determine the obstacle avoidance strategy.

[0037] In a third aspect, a walking device is provided, comprising:

[0038] one or more processors;

[0039] Memory;

[0040] One or more applications, wherein one or more applications are stored in a memory and configured to be executed by one or more processors, and one or more programs are configured to: perform operations corresponding to the method shown in any possible implementation of the first aspect.

[0041] In a fourth aspect, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded by a processor and executes the steps of the method shown in any possible implementation of the first aspect.

[0042] In a fifth aspect, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, operations corresponding to the method shown in any possible implementation manner in the first aspect are implemented.

[0043] In summary, the method provided by this application includes the following beneficial technical effects:

[0044] In this application, a single-frame image captured by a monocular camera installed on a walkable device is obtained, image recognition is performed based on the single-frame image, and the point with the largest edge pixel coordinate v value is selected as the target point. Based on the roll angle, the first pitch angle, the horizontal distance and the first pixel coordinate, the actual distance of the target obstacle from the monocular camera is jointly determined by real-time compensation of the camera center elevation, pitch angle and roll angle. Using only the camera and the segmentation algorithm, accurate distance measurement can be completed, thereby achieving accurate obstacle avoidance.

[0045] In addition, the present application also provides an obstacle avoidance device, equipment and medium, all of which have the above-mentioned beneficial technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions of the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 It is a flowchart of an obstacle avoidance method provided in an embodiment of the present application;

[0048] Figure 2 is a schematic diagram of a target point provided in an embodiment of the present application;

[0049] Figure 3 is a schematic diagram of a process for determining an actual distance provided in an embodiment of the present application;

[0050] Figure 4 It is a schematic diagram of a situation in which the right wheel of a lawn mower falls during the movement of a lawn mower provided by an embodiment of the present application;

[0051] Figure 5 It is a schematic diagram of a situation in which a front wheel of a lawn mower falls into a pothole provided by an embodiment of the present application;

[0052] Figure 6 It is a side view of a lawn mower provided in an embodiment of the present application, in which a point on the ground is projected onto a virtual imaging plane when a camera is imaging during the motion process;

[0053] Figure 7 It is a structural diagram of a pidnet real-time semantic segmentation network provided in an embodiment of the present application;

[0054] Figure 8 It is a schematic diagram of a Depthwise structure provided in an embodiment of the present application;

[0055] Fig. 9 It is a schematic diagram of a Depthwise data processing flow provided in an embodiment of the present application;

[0056] Fig.10 It is a schematic diagram of a Pointwise structure provided in an embodiment of the present application;

[0057] Fig.11 is a schematic diagram of a Pointwise data processing flow provided in an embodiment of the present application;

[0058] Fig.12 is a structural schematic diagram of an obstacle avoidance device provided in an embodiment of the present application;

[0059] Fig.13 It is a structural schematic diagram of a walkable device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] This specific embodiment is merely an explanation of the present application and is not a limitation of the present application. After reading this specification, a person skilled in the art may make non-creative modifications to the present embodiment as needed, but such modifications are protected by the patent law as long as they are within the scope of the present application.

[0061] It should be noted that in the optional embodiments of the present application, the object information and other related data involved, when the embodiments in the present application are applied to specific products or technologies, need to obtain the permission or consent of the object, and the collection, use and processing of the relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions. In other words, if the embodiments of the present application involve data related to the object, it needs to be obtained with the authorization and consent of the object, the authorization and consent of the relevant departments, and in compliance with the relevant laws, regulations and standards of the country and region. If personal information is involved in the embodiments, the acquisition of all personal information needs to obtain the consent of the individual. If sensitive information is involved, the separate consent of the information subject needs to be obtained. The embodiments also need to be implemented with the authorization and consent of the object.

[0062] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0063] In addition, the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article, unless otherwise specified, generally means that the associated objects before and after are in an "or" relationship.

[0064] At present, the commonly used ranging solutions mainly include ultrasonic ranging, infrared ranging, laser ranging, monocular depth ranging, etc. Ultrasonic ranging calculates the distance of an object by measuring the round-trip time of ultrasonic waves. The detection distance is relatively short and there are detection blind spots. Infrared ranging is based on the principle of triangulation, but the measurement is affected by the color, direction and surrounding light of the object, and there are certain errors. Laser ranging uses laser to measure the distance or displacement of the measured object. It is expensive and not suitable for high-precision measurement. Monocular depth estimation ranging is a type of visual ranging. Its deployment is affected by the computing power of edge devices. Generally, only small models can be deployed. Its depth map is extremely unstable and has large errors.

[0065] Monocular ranging using camera elevation. This method measures distance on an ideal plane. However, real scenes are often uneven, and the pitch angle and roll angle will affect the results of the ranging. This method is not applicable in this case.

[0066] Based on this, in order to use lower cost to perform more stable distance measurement and make up for the limitation of using camera elevation for monocular distance measurement, the embodiment of the present application provides a method that can complete the distance measurement using only a camera and a segmentation algorithm, which not only has stable results but also low cost and has a very broad application prospect.

[0067] The present application embodiment provides an obstacle avoidance method, such as Figure 1 As shown, the method provided in the embodiment of the present application can be executed by an electronic device, and the electronic device is a server. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services. The terminal device can be a smart phone, a tablet computer, a laptop computer, a desktop computer, an edge computing device, a walkable device, etc., but is not limited to this. The terminal device and the electronic device can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiment of the present application. The method includes:

[0068] S101, obtaining a single-frame image captured by a monocular camera disposed on a walkable device;

[0069] Among them, walking equipment refers to equipment that can move, such as lawn mowing robots, inspection robots, etc.

[0070] When the walkable device is a lawn mowing robot, the lawn mowing robot may include a monocular camera and other sensors in addition to the edge device that can implement the obstacle avoidance method, which is not limited in this embodiment. The single-frame image is an image captured by the walkable device. Specifically, a monocular camera with a height h and a pitch angle p1 (the angle between the central axis of the camera and the horizontal plane) is installed at the front end of the robot. The Zhang Zhengyou camera calibration method is used to calibrate the internal and external parameters of the camera to obtain the internal and external parameter matrix of the front camera. The front image of the lawn mowing robot is obtained through the monocular camera, and the image resolution is (wc, hc). Then opencv is used to deframe the video stream to obtain a single-frame image.

[0071] In some embodiments, a capture instruction is sent to the monocular camera, and the monocular camera captures an image after receiving the instruction; or, the monocular camera is automatically triggered to capture an image when a preset time is reached.

[0072] S102, performing image recognition according to the single-frame image to determine the first pixel coordinate of a target point in the drivable area, where the target point is a point with the largest pixel coordinate v value at the edge of the target obstacle in the single-frame image;

[0073] Among them, obstacles in a single frame image are identified, obstacles in the drivable area are determined, and the first pixel coordinate of the point with the largest pixel coordinate v value at the edge of the target obstacle in the drivable area is determined, wherein the target point is the point closest to the walkable device. Among them, the drivable area is a preset drivable area of ​​the walkable device.

[0074] In one feasible way, a single frame image is sent to the segmentation model network for inference, and the obtained output is post-processed to obtain a segmentation mask that matches the resolution of the input image; then OpenCV is used to filter the obstacle contour points in the drivable area, and the pixel coordinates (u1, v1) of the point with the largest pixel coordinate v value on the edge of the target obstacle are obtained. For example, see Figure 2 , the target point is the dot in the figure.

[0075] S103, obtaining a roll angle, a first pitch angle, and a horizontal distance of the walkable device, where the horizontal distance is the distance between the intersection of the center of the monocular camera perpendicular to the ground and the point where the wheel touches the ground;

[0076] Get the roll angle and first pitch angle output by the imu (inertial measurement unit) of the walkable device, where the roll angle (Roll): rotation around the X axis, describes the roll state of the robot. The pitch angle (Pitch): rotation around the Y axis, describes the up and down tilt state of the robot. The horizontal distance is the distance between the intersection of the center of the monocular camera perpendicular to the ground and the point where the wheels of the walkable device touch the ground.

[0077] S104. Determine the actual distance of the target obstacle from the monocular camera according to the roll angle, the first pitch angle, the horizontal distance, and the first pixel coordinate, where the actual distance is used to determine an obstacle avoidance strategy.

[0078] In this step, by combining the roll angle, pitch angle, horizontal distance and first pixel coordinate, the shortest distance between the target obstacle and the camera is determined more accurately.

[0079] It can be seen that in the embodiment of the present application, a single-frame image captured by a monocular camera arranged on a walkable device is obtained, image recognition is performed based on the single-frame image, and the point with the largest edge pixel coordinate v value is selected as the target point, and based on the roll angle, the first pitch angle, the horizontal distance and the first pixel coordinate, the actual distance of the target obstacle from the monocular camera is jointly determined by real-time compensation of the camera center elevation, pitch angle and roll angle. Only by using the camera and the segmentation algorithm, accurate ranging can be completed, thereby achieving accurate obstacle avoidance.

[0080] A possible implementation of the embodiment of the present application is, S104, determining the actual distance of the target obstacle from the monocular camera according to the roll angle, the first pitch angle, the horizontal distance and the first pixel coordinate, see Figure 3 ,include:

[0081] S1041, correcting the first pixel coordinate according to the rolling angle to obtain a second pixel coordinate;

[0082] Using the roll angle r output by the imu, the second pixel coordinate (u1, v1) of the target obstacle is converted from the coordinate system with a roll angle not equal to 0 (H1 coordinate system) to the coordinate system with a roll angle of 0 (H coordinate system) to obtain the coordinate (u, v).

[0083] in:

[0084] u=h c *tanh(r)-(h c / cos(r)-h c ) / sin(r)+((h c / cos(r)-h c ) / tanh(r)+u1-v1*tanh(r))*cos(r);

[0085] v=h c -(h c -v1-u1*tanh(r))*cos(r);

[0086] h c is the height of the image; r is the roll angle.

[0087] For example, see 4. Figure 4 This is the case where the right wheel of the lawn mower falls during its motion, which is equivalent to the rectangle IJKH rotating r degrees around point I to rectangle IJ1K1H1. The coordinates of point Q in the coordinate system with H as the origin are (u,v), and the coordinates of point Q in the coordinate system with H1 as the origin are (u1,v1). It should be noted that when the left wheel of the lawn mower falls, the same as above, r can be negative.

[0088] The derivation process of u and v is as follows:

[0089] v=HW=HI-WI,

[0090] WI=IT*cos(r),

[0091] IT=IH1-H1X-XT,

[0092] v=h c -(h c -v1-u1*tanh(r))*cos(r),

[0093] u=HM+MX,

[0094] HM=HN-MN,

[0095] HN=HI*tanh(r)=h c *tanh(r),

[0096] MN=NH1 / sin(r)=(IN-IH1) / sin(r)=(HI / cos(r)-IH1) / sin(r)=(h c / cos(r)-h c ) / sin(r)

[0097] MX=MR*cos(r),

[0098] MR=MH1+H1R,

[0099] MH1=MN*cos(r)=(h c / cos(r)-h c ) / tanh(r),

[0100] H1R=H1S-RS=u1-v1*tanh(r),

[0101] u=h c *tanh(r)-(h c / cos(r)-h c ) / sin(r)+((h c / cos(r)-h c ) / tanh(r)+u1-v1*tanh(r))*cos(r).

[0102] S1042, determining a real height of the monocular camera according to the first pitch angle and the horizontal distance;

[0103] The pitch angle p2 of the lawn mower body is obtained through imu. At this time, the actual height of the camera center h1=h*cos(p2)-L*sin(p2).

[0104] L is the horizontal distance. h is the distance between the center of the camera and the ground when the robot is in a horizontal position;

[0105] Exemplary, participating Figure 5 , Figure 5 This is the situation when the front wheel of the lawn mower sinks when it encounters a pothole. At this time, the angle between the body and the horizontal is p2. When the robot is horizontal, the horizontal distance between the intersection of the camera center perpendicular to the ground and the point where the rear wheel touches the ground is L. At this time, the front wheel of the robot sinks, so X1X2 is L, and the actual height of the camera center at this time is h1.

[0106] The pitch angle p2 of the lawn mower body is obtained through imu. At this time, the actual height of the camera center h1=h*cos(p2)-L*sin(p2).

[0107] S1043, determining a second pitch angle according to the first pitch angle and the installation angle of the monocular camera;

[0108] Wherein, the second pitch angle p=p1+p2 is determined according to the first pitch angle p2 and the installation angle p1 of the monocular camera.

[0109] S1044. Determine the actual distance of the target obstacle from the monocular camera according to the second pixel coordinate, the true height, the second pitch angle, and the horizontal distance.

[0110] Among them, the second pixel coordinate (u, v), the real height h1, the second pitch angle p and the horizontal distance L are used to determine the actual distance, where the actual distance is ;

[0111] f: camera focal length; c: y : The vertical optical center coordinate in the camera intrinsic parameter matrix (usually in pixel units), which indicates the pixel offset of the image center in the vertical direction (i.e., the y-axis direction). x : Focal length parameter in the camera intrinsic matrix (usually in pixels), which represents the ratio of the camera lens focal length to the image sensor pixel size, and the equivalent focal length in the horizontal direction (i.e., the x-axis direction). For details, see Figure 6 , Figure 6 It is the side view of the lawn mower when the camera is imaging and the point on the ground is projected onto the virtual imaging plane. The angle p is the sum of the camera installation angle p1 and the vehicle body pitch angle p2. The camera center height h1 is the value after considering the pitch angle of the lawn mower. AD is the virtual imaging plane, and AD is parallel to OO2.

[0112] It is easy to find that triangle BOC is similar to triangle C1OO2. We can get:

[0113] ,

[0114] p=p1+p2,

[0115] BO=f / cos(p),

[0116] C1O2=C1O1+O1O2,

[0117] O1O2=h1*tanh(p)=(h*cos(p)-L*sin(p))*tanh(p),

[0118] BC=BO1+O1C=f*tanh(p)+(vc y ) / f x *f,

[0119] OO2=h1 / cos(p),

[0120] Obstacle distance C1O1=C1O2-O1O2=

[0121] .

[0122] It can be seen that in the embodiment of the present application, the second pixel coordinate is obtained by correcting the first pixel coordinate according to the roll angle; the real height of the monocular camera is determined according to the first pitch angle and the horizontal distance; the second pitch angle is determined according to the first pitch angle and the installation angle of the monocular camera; the actual distance of the target obstacle from the monocular camera is determined according to the second pixel coordinate, the real height, the second pitch angle and the horizontal distance, which provides a specific process for determining the actual distance, which is simple in method and small in calculation.

[0123] A possible implementation of the embodiment of the present application, after determining the actual distance of the target obstacle from the monocular camera according to the roll angle, the first pitch angle, the horizontal distance and the first pixel coordinate, further includes:

[0124] When the actual distance is less than the set threshold, the walking device is controlled to perform obstacle avoidance operations.

[0125] A drivable area is planned in front of the robot. If an obstacle appears in the drivable area and the obstacle distance is less than the set threshold, the robot stops moving and needs to perform obstacle avoidance operations.

[0126] Specifically, when the actual distance is less than the set threshold, the walkable device is controlled to perform obstacle avoidance operations, including:

[0127] When the actual distance is less than the set threshold, the type of the monocular camera is determined;

[0128] When the type of the monocular camera is a rear-view camera, the walkable device is controlled to stop moving backward;

[0129] When the type of the monocular camera is a forward-looking camera, a motion mode of the walkable device is obtained, and the walkable device is controlled to perform an obstacle avoidance operation according to the motion mode.

[0130] The threshold value can be set according to actual needs.

[0131] When the movement mode is a U-turn mode when encountering an obstacle, the walking device is controlled to stop moving forward and then turn around;

[0132] When the motion mode is the detour mode, the obstacle areas in the left area and the right area within the drivable area in the single-frame image are determined; the walkable device is controlled to detour in a target direction, wherein the target direction is a direction with a smaller area between the obstacle areas in the left area and the obstacle areas in the right area.

[0133] Specifically, a single frame image may be input into a segmentation model to obtain a segmentation mask, and then the obstacle area in the left region and the obstacle area in the right region may be determined by the segmentation mask.

[0134] For the front-view camera, if the motion mode is to turn around immediately when encountering an obstacle, you can turn around at this moment; if the motion mode is the detour mode, at this moment, the robot's central axis is used to divide the traversable area into two parts, left and right, and calculate the area of ​​the obstacle in the left and right traversable areas. If the area on the left is larger, turn right and detour; if the area on the right is larger, turn left and detour; if the areas are the same, both directions are acceptable. For the rear-view camera, the world coordinates of the obstacle can be obtained in the same way. When an obstacle appears in the traversable area during the robot's retreat and the distance is less than the set threshold, the robot stops retreating immediately.

[0135] It can be seen that in the embodiments of the present application, the obstacle avoidance operation is performed by controlling the walkable device according to the motion mode; when the motion mode is determined to be a U-turn mode when encountering an obstacle, the device is controlled to stop moving forward and perform a U-turn, which effectively avoids direct collision with the obstacle and improves the safety of the device; when the motion mode is a detour mode, by comparing the obstacle areas of the left and right areas, the side with a smaller area is intelligently selected as the detour direction, thereby improving the obstacle avoidance efficiency and enabling the device to more flexibly cope with complex environments.

[0136] A possible implementation of the embodiment of the present application is to obtain a motion mode of a walkable device, including:

[0137] According to the target obstacle area in a single frame image;

[0138] Identify the target obstacle area and determine the obstacle category;

[0139] When the obstacle category is a boundary obstacle, the movement mode is determined to be a U-turn mode upon encountering an obstacle; otherwise, the movement mode is determined to be a detour mode.

[0140] In the embodiment of the present application, specifically, a single frame image can be input into a segmentation model to obtain a segmentation mask, and then the area where the target obstacle in the single frame image is located is determined by the segmentation mask, and the obstacle category is determined. Among them, boundary obstacles include but are not limited to: walls and fences, which serve as boundaries in geography and restrict the passage of traffic.

[0141] It can be seen that in the embodiment of the present application, the obstacle category is identified and determined. If the obstacle is identified as a boundary obstacle, it means that the device has reached the preset boundary or an inaccessible area. At this time, the movement mode is determined to be a U-turn mode when encountering an obstacle; for other types of obstacles, a detour mode is adopted, which can more accurately respond to different obstacle avoidance scenarios and improve the intelligence level of obstacle avoidance.

[0142] A possible implementation of the embodiment of the present application is to perform image recognition based on a single frame image to determine the first pixel coordinate of a target point in the drivable area, including:

[0143] Input a single frame image into the segmentation model to obtain a segmentation mask;

[0144] According to the segmentation mask, the obstacle contour points in the drivable area are screened to determine the first pixel coordinates of the target point in the drivable area.

[0145] Specifically, OpenCV is used to deframe the video stream to obtain a single frame image, which is resized to the model input resolution and input into the segmentation model to obtain a segmentation mask.

[0146] In the semantic segmentation task, the segmentation mask output by the segmentation model is a two-dimensional array of the same size as the input image, but in this array, the value of each element represents the category to which the pixel belongs. For example, there are two categories: non-obstacles and obstacles. A pixel with a value of 0 indicates that the position belongs to the "non-obstacle" category, and a pixel with a value of 1 indicates that the position belongs to the "obstacle" category. Of course, the actual number of categories and corresponding values ​​will be set according to actual conditions.

[0147] The segmentation mask provides detailed information about the category to which each pixel in the image belongs. Through the mask, it is easy to identify different objects or areas where objects are located in the image. Further analysis or processing can be performed, such as calculating the area of ​​a certain category, extracting the outline of a specific object, etc.

[0148] In one achievable manner, an obstacle is determined based on a segmentation mask, and then it is determined whether the obstacle is in a drivable area. If so, the first pixel coordinates of a target point of the obstacle are determined. In another achievable manner, it is determined whether there is an obstacle in the segmentation mask in the drivable area. If so, the first pixel coordinates of the target point of the obstacle are determined.

[0149] It can be seen that in the embodiment of the present application, by inputting a single-frame image into the segmentation model, an accurate segmentation mask can be obtained. Based on the segmentation mask, the obstacle contour points in the drivable area can be efficiently screened out, and the first pixel coordinates of the target point can be determined, thereby greatly improving the accuracy of target point positioning.

[0150] In a possible implementation manner of an embodiment of the present application, the segmentation model is a PidNet model using depthwise separable convolution; the depthwise separable convolution includes: depthwise convolution and pointwise convolution.

[0151] Among them, the pidnet real-time semantic segmentation network is used, such as Figure 7As shown in the figure, it is a three-branch network architecture that parses details, context, and boundary information respectively, and designs a boundary attention guided fusion module (Bag) to fuse the features of the three branches.

[0152] In order to further make the model lightweight, the embodiment of the present application replaces the ordinary convolution with the depth-separable convolution based on pidnet. The depth-separable convolution consists of two parts: 1. Depth-wise convolution: Each input channel is convolved with a separate convolution kernel, which reduces the number of parameters; 2. Point-by-point convolution: The output of the depth-wise convolution is convolved with a 1×N×1 convolution kernel to combine the features of different channels. In this way, the separation of image regions and channels is achieved.

[0153] Comparison between ordinary convolution and depth-separable convolution: parameter amount and computational amount: Ordinary convolution has a large number of parameters and computational amount because it learns independent weights for each output channel. In contrast, depth-separable convolution significantly reduces the number of parameters and computational amount by reducing the number of independent convolution kernels per channel and using smaller 1×N×1 convolution kernels.

[0154] Depthwise separable convolution divides ordinary convolution into two processes: Depthwise process and Pointwise process.

[0155] Depthwise: Figure 8 As shown in the figure, the N*H*W*C input is divided into group=C groups, and each group is subjected to 3*3 convolution. In this way, the spatial features of each channel, namely the depthwise features, are obtained. The process is as follows Fig. 9 shown.

[0156] Pointwise: Fig.10 As shown in the figure, k 1*1 convolutions are performed on the N*H*W*C input, so that the features of each pixel are obtained, namely the Pointwise features. Fig.11 shown.

[0157] The final output of Depthwise+Pointwise is N*H*W*k.

[0158] Expressive power and precision: Ordinary convolution has strong expressive power and precision, but it has a large number of parameters and high computational complexity. Although the expressive power and precision of depthwise separable convolution are relatively weak, it has a small number of parameters and fast computational speed. Without losing too much precision, it greatly improves the inference speed and is suitable for scenarios with limited computing resources such as lightweight and mobile terminals.

[0159] Furthermore, the training process of the segmentation model includes: obtaining training images, which are images of target scenes, and annotating the training images to obtain training samples, wherein the target scene corresponds to the usage scenario of the walkable device. If the walkable device is a lawn mowing robot, the target scene is a lawn mowing scene; inputting the pidnet model according to the training images to obtain semantic segmentation results; and iteratively training the pidnet model according to the annotation and semantic segmentation results to obtain a trained pidnet model.

[0160] It can be seen that in the embodiment of the present application, the segmentation model adopts the PidNet model that replaces the ordinary convolution with the depthwise separable convolution; the depthwise separable convolution includes two stages: depthwise convolution and pointwise convolution, which not only reduces the number of parameters and calculation amount of the model, but also improves the operation efficiency of the model.

[0161] Based on any of the above embodiments, the embodiment of the present application provides a specific obstacle avoidance process. Taking the scene of the mowing robot moving during mowing as an example, specifically, a camera is used to obtain a video stream, and a single frame image is obtained by splitting the frame with OpenCV, which is sent to the segmentation model for reasoning to obtain a segmentation mask; OpenCV is then used to process and obtain the pixel coordinates of the edge point set under the obstacle; in reality, the lawn is uneven, the body posture of the mowing robot is not horizontal, and its camera center height and camera pitch angle are always changing. The body pitch angle and roll angle compensation output by the imu are used to more accurately measure the obstacle distance in the real scene; the actual distance of the obstacle from the camera is obtained, and the obstacle is judged to be in the drivable area and the distance is less than the set threshold, and it stops immediately. When the camera is forward-looking, if the motion mode is in the U-turn mode when encountering an obstacle, it will turn around directly after stopping; if the motion mode is in the detour mode, the drivable area in front is divided into left and right parts. If the area of ​​the obstacle in the left half of the area is larger than the area in the right half of the area, turn right to detour, otherwise turn left to detour. When the camera is in rear-view mode, if an obstacle appears in the drivable area during the robot's backward movement and the distance is less than the set threshold, the robot will stop moving backward immediately.

[0162] Furthermore, in terms of segmentation models, in order to meet the needs of edge device deployment, the ordinary convolution of PidNet is changed to depthwise separable convolution, which reduces the amount of calculation and improves the inference speed while ensuring accuracy.

[0163] Compared with ultrasonic ranging, the embodiment of the present application has a wider ranging range and higher accuracy; compared with infrared ranging, it is not affected by the color, direction and surrounding light of the object; compared with laser ranging, it is inexpensive while ensuring that the accuracy difference is not large; compared with monocular depth estimation, the invention is convenient for edge device deployment and the ranging is stable and accurate; in monocular ranging based on camera elevation, the camera center height, pitch angle and roll angle are all assumed to be fixed, which is impossible in reality. The embodiment of the present application uses the real-time pitch angle and roll angle output by the imu to compensate for the camera center elevation, pitch angle and roll angle in real time, so as to more accurately measure the obstacle distance in real scenes, making it possible to implement the solution.

[0164] The following is an introduction to a device provided in an embodiment of the present application. The device described below and the method described above can be referred to each other. The device of this embodiment is set in a walkable device. Fig.12 , Fig.12 It is a structural block diagram of a device of one embodiment of the present application, comprising:

[0165] A first acquisition module 210 is used to acquire a single frame image captured by a monocular camera disposed on the walkable device;

[0166] The pixel coordinate determination module 220 is used to perform image recognition based on the single-frame image to determine the first pixel coordinate of the target point in the drivable area, where the target point is the point with the largest pixel coordinate v value at the edge of the target obstacle in the single-frame image;

[0167] The second acquisition module 230 is used to acquire the rolling angle, the first pitch angle and the horizontal distance of the lawn mower, where the horizontal distance is the distance between the intersection of the center of the monocular camera perpendicular to the ground and the point where the wheel touches the ground;

[0168] The distance determination module 240 is used to determine the actual distance between the target obstacle and the monocular camera according to the roll angle, the first pitch angle, the horizontal distance and the first pixel coordinate, and the actual distance is used to determine the obstacle avoidance strategy.

[0169] It can be seen that in the embodiment of the present application, the first acquisition module 210 acquires a single-frame image captured by a monocular camera arranged on a walkable device, the pixel coordinate determination module 220 performs image recognition based on the single-frame image, and selects the point with the largest edge pixel coordinate v value as the target point, and the distance determination module 240 determines the actual distance of the target obstacle from the monocular camera based on the roll angle, the first pitch angle, the horizontal distance and the first pixel coordinate of the first acquisition module 210 obtained by the second acquisition module 230, by real-time compensation of the camera center elevation, pitch angle and roll angle. Only using the camera and the segmentation algorithm, accurate distance measurement can be completed, thereby achieving accurate obstacle avoidance.

[0170] In one practicable manner, the distance determination module 240 is used to correct the first pixel coordinate according to the roll angle to obtain the second pixel coordinate;

[0171] Determine the actual height of the monocular camera according to the first pitch angle and the horizontal distance;

[0172] Determine a second pitch angle according to the first pitch angle and the installation angle of the monocular camera;

[0173] The actual distance between the target obstacle and the monocular camera is determined according to the second pixel coordinate, the true height, the second pitch angle and the horizontal distance.

[0174] In one achievable manner, the present invention further includes:

[0175] The obstacle avoidance module 250 is used to determine the type of the monocular camera when the actual distance is less than a set threshold;

[0176] When the type of the monocular camera is a rear-view camera, the walkable device is controlled to stop moving backward;

[0177] When the type of the monocular camera is a forward-looking camera, a motion mode of the walkable device is obtained, and the walkable device is controlled to perform an obstacle avoidance operation according to the motion mode.

[0178] In one achievable manner, the obstacle avoidance module is used to control the walkable device to stop moving forward and then turn around when the movement mode is a U-turn mode upon encountering an obstacle;

[0179] When the motion mode is the detour mode, the obstacle areas in the left area and the right area within the drivable area in the single-frame image are determined; the walkable device is controlled to detour in a target direction, wherein the target direction is a direction with a smaller area between the obstacle areas in the left area and the obstacle areas in the right area.

[0180] In one achievable manner, the obstacle avoidance module is used to determine a target obstacle area in a single frame image;

[0181] Identify the target obstacle area and determine the obstacle category;

[0182] When the obstacle category is a boundary obstacle, the movement mode is determined to be a U-turn mode upon encountering an obstacle; otherwise, the movement mode is determined to be a detour mode.

[0183] In one achievable manner, the second acquisition module is used to input the single frame image into the segmentation model to obtain a segmentation mask;

[0184] According to the segmentation mask, the obstacle contour points in the drivable area are screened to determine the first pixel coordinates of the target point in the drivable area.

[0185] In one achievable manner, the segmentation model is a pidnet model using depthwise separable convolution; the depthwise separable convolution includes: depthwise convolution and pointwise convolution.

[0186] In the embodiment of the present application, a walking device is provided, such as Fig.13 As shown, Fig.13 The walkable device 300 shown includes: a processor 301 and a memory 303. The processor 301 and the memory 303 are connected, such as through a bus 302. Optionally, the walkable device 300 may also include a transceiver 304. It should be noted that in actual applications, the transceiver 304 is not limited to one, and the structure of the walkable device 300 does not constitute a limitation on the embodiments of the present application.

[0187] The processor 301 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor 301 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0188] The bus 302 may include a path to transmit information between the above components. The bus 302 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 302 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.13 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0189] The memory 303 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0190] The memory 303 is used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 301. The processor 301 is used to execute the application code stored in the memory 303 to implement the contents shown in the above method embodiment.

[0191] Fig.13 The walkable device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0192] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding content in the aforementioned method embodiment.

[0193] An embodiment of the present application provides a computer program product, including a computer program, which implements the corresponding contents of the aforementioned method embodiment when the computer program is executed by a processor.

[0194] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0195] The above are only some implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An obstacle avoidance method, characterized in that: include: Obtain a single-frame image captured by a monocular camera set on a walkable device; Perform image recognition based on the single-frame image to determine the first pixel coordinate of a target point in the drivable area, wherein the target point is a point with the largest pixel coordinate v value at the edge of the target obstacle in the single-frame image; Obtaining a roll angle, a first pitch angle, and a horizontal distance of the walkable device, wherein the horizontal distance is a distance between an intersection of a monocular camera center perpendicular to the ground and a point where the wheel touches the ground; The actual distance between the target obstacle and the monocular camera is determined according to the roll angle, the first pitch angle, the horizontal distance and the first pixel coordinate, and the actual distance is used to determine an obstacle avoidance strategy.

2. The method according to claim 1, characterized in that Determining an actual distance between a target obstacle and the monocular camera according to the roll angle, the first pitch angle, the horizontal distance, and the first pixel coordinate includes: Correcting the first pixel coordinate according to the roll angle to obtain a second pixel coordinate; Determining a true height of the monocular camera according to the first pitch angle and the horizontal distance; Determining a second pitch angle according to the first pitch angle and an installation angle of the monocular camera; Determine the actual distance of the target obstacle from the monocular camera according to the second pixel coordinate, the true height, the second pitch angle and the horizontal distance.

3. The method according to claim 1, characterized in that After determining the actual distance between the target obstacle and the monocular camera according to the roll angle, the first pitch angle, the horizontal distance and the first pixel coordinate, the method further includes: When the actual distance is less than a set threshold, determining the type of the monocular camera; When the type of the monocular camera is a rear-view camera, controlling the walkable device to stop moving backward; When the type of the monocular camera is a forward-looking camera, the motion mode of the walkable device is obtained, and the walkable device is controlled to perform an obstacle avoidance operation according to the motion mode.

4. The method according to claim 3, characterized in that Controlling the walkable device to perform obstacle avoidance operations according to the motion mode includes: When the movement mode is a U-turn mode upon encountering an obstacle, controlling the walkable device to stop moving forward and then turn around; When the motion mode is the detour mode, the obstacle area in the left area and the obstacle area in the right area of ​​the drivable area in the single-frame image are determined; and the walkable device is controlled to detour in a target direction, wherein the target direction is a direction with a smaller area between the obstacle area in the left area and the obstacle area in the right area.

5. The method according to claim 3, characterized in that: Obtaining the motion mode of the walkable device includes: According to the target obstacle area in the single frame image; Identify the target obstacle area and determine the obstacle category; When the obstacle category is a boundary obstacle, the movement mode is determined to be a U-turn mode upon encountering an obstacle; otherwise, the movement mode is determined to be a detour mode.

6. The method according to any one of claims 1 to 5, characterized in that: The method of performing image recognition according to a single-frame image to determine the first pixel coordinate of a target point in the drivable area includes: Inputting the single frame image into a segmentation model to obtain a segmentation mask; According to the segmentation mask, the obstacle contour points in the drivable area are screened to determine the first pixel coordinates of the target point in the drivable area.

7. The method according to claim 6, characterized in that The segmentation model is a pidnet model using depthwise separable convolution; the depthwise separable convolution includes: depthwise convolution and pointwise convolution.

8. An obstacle avoidance device, characterized in that: include: A first acquisition module is used to acquire a single-frame image captured by a monocular camera arranged on the walkable device; A pixel coordinate determination module, used to perform image recognition based on a single-frame image to determine a first pixel coordinate of a target point in a drivable area, wherein the target point is a point with a maximum pixel coordinate v value at the edge of a target obstacle in the single-frame image; A second acquisition module is used to acquire a roll angle, a first pitch angle and a horizontal distance of the walkable device, where the horizontal distance is a distance between an intersection of a monocular camera center perpendicular to the ground and a point where the wheel touches the ground; A distance determination module is used to determine the actual distance between the target obstacle and the monocular camera according to the roll angle, the first pitch angle, the horizontal distance and the first pixel coordinate, and the actual distance is used to determine the obstacle avoidance strategy.

9. A walking device, characterized in that: include: one or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to: perform the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded by the processor and executes the steps of the method according to any one of claims 1 to 7.

Images