An obstacle avoidance method, device, equipment and medium of a vehicle

Abstract

The application discloses an obstacle avoidance method, device and equipment of a vehicle and a medium, and relates to the technical field of vehicle obstacle avoidance. The method comprises the following steps: acquiring a detection frame of a target obstacle, size information of the detection frame and a driving direction of the vehicle; wherein the target obstacle is an obstacle detected by the vehicle in real time; determining a pixel center point coordinate position of the target obstacle according to the size information of the detection frame, and constructing a depth sampling region; performing extended sampling of obstacle depth on a depth image based on the center of the depth sampling region, and obtaining at least one obstacle sampling depth; determining a relative distance between the target obstacle and the vehicle according to the obstacle sampling depth; and determining a target obstacle avoidance instruction to be executed by the vehicle according to a horizontal position, the relative distance, the driving direction of the vehicle, a set distance threshold and a set monitoring angle. The application realizes distance identification between the obstacle and the vehicle, and further realizes automatic obstacle avoidance of the vehicle, thereby improving the safety of vehicle driving.

Description

Technical Field

[0001] This invention relates to the field of vehicle technology, and in particular to a method, device, equipment and medium for obstacle avoidance in vehicles. Background Technology

[0002] An agricultural machinery automatic driving system is a device system composed of a vehicle attitude acquisition and environmental information perception module, a task planning module, and a steering control module. Based on a high-precision global satellite navigation system, unmanned agricultural machinery can already meet the needs of operations such as fertilization, harvesting, and rice transplanting.

[0003] However, currently, agricultural machinery based on global satellite navigation cannot meet the requirements for operation in unknown and complex environments, especially environments where unknown obstacles are prone to occur, which can lead to operational safety accidents. Obstacle identification and avoidance are the core issues that current unmanned agricultural machinery navigation systems need to solve. Summary of the Invention

[0004] This invention provides a method, apparatus, device, and medium for vehicle obstacle avoidance, to identify obstacles and avoid them, thereby improving vehicle safety.

[0005] In a first aspect, the present invention provides a method for obstacle avoidance of a vehicle, comprising:

[0006] The system acquires the detection bounding box of the target obstacle, the size information of the detection bounding box, and the vehicle's driving direction; the target obstacle is an obstacle detected by the vehicle in real time.

[0007] Based on the size information of the detection box, determine the coordinates of the pixel center point of the target obstacle and construct the depth sampling region;

[0008] Based on the center of the depth sampling region, the depth image is augmented with obstacle depth to obtain at least one obstacle sampling depth;

[0009] The relative distance between the target obstacle and the vehicle is determined based on the sampling depth of each obstacle.

[0010] Based on the horizontal position, relative distance, vehicle direction of travel, set distance threshold, and set monitoring angle, the target obstacle avoidance command that the vehicle needs to execute is determined.

[0011] Secondly, the present invention also provides a vehicle obstacle avoidance device, comprising:

[0012] The information acquisition module is used to acquire the detection frame of the target obstacle, the size information of the detection frame, and the vehicle's driving direction; among which, the target obstacle is an obstacle detected by the vehicle in real time;

[0013] The region construction module is used to determine the pixel center point coordinates of the target obstacle based on the size information of the detection box, and to construct the depth sampling region;

[0014] The sampling depth determination module is used to perform extended sampling of the obstacle depth on the depth image based on the center of the depth sampling region to obtain at least one obstacle sampling depth;

[0015] The relative distance determination module is used to determine the relative distance between the target obstacle and the vehicle based on the sampling depth of each obstacle;

[0016] The instruction determination module is used to determine the target obstacle avoidance instruction that the vehicle needs to execute based on the horizontal position, relative distance, vehicle driving direction, set distance threshold, and set monitoring angle.

[0017] Thirdly, embodiments of the present invention also provide an electronic device, comprising:

[0018] At least one processor; and

[0019] A memory that is communicatively connected to at least one processor; wherein

[0020] The memory stores instructions that can be executed by at least one processor, which enables the at least one processor to perform the obstacle avoidance method for a vehicle provided in any embodiment of the present invention.

[0021] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the obstacle avoidance method for a vehicle according to any embodiment of the present invention.

[0022] This invention provides an embodiment of the invention that acquires the detection frame of a target obstacle, the size information of the detection frame, and the vehicle's driving direction. The target obstacle is an obstacle detected by the vehicle in real time. Based on the size information of the detection frame, the coordinates of the pixel center point of the target obstacle are determined, and a depth sampling region is constructed. Using the center of the depth sampling region as a reference, the depth image is expanded by sampling the obstacle depth to obtain at least one obstacle sampling depth. Based on each obstacle sampling depth, the relative distance between the target obstacle and the vehicle is determined. Based on the horizontal position, relative distance, vehicle driving direction, a set distance threshold, and a set monitoring angle, the target obstacle avoidance command to be executed by the vehicle is determined. This invention can accurately identify the distance between the obstacle and the vehicle through the obstacle detection frame and its size information, and automatically determine whether obstacle avoidance is necessary based on the distance. This achieves obstacle identification and automatic obstacle avoidance, improving the safety of vehicle operation.

[0023] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart of a vehicle obstacle avoidance method according to Embodiment 1 of the present invention;

[0026] Figure 2 This is a flowchart of a vehicle obstacle avoidance method according to Embodiment 2 of the present invention;

[0027] Figure 3A This is a flowchart of a vehicle obstacle avoidance method according to Embodiment 3 of the present invention;

[0028] Figure 3B This is a schematic diagram of an obstacle detection area provided according to Embodiment 3 of the present invention;

[0029] Figure 3C This is a schematic diagram of a vehicle coordinate system and a global coordinate system provided according to Embodiment 3 of the present invention;

[0030] Figure 3D This is a schematic diagram of an azimuth angle provided according to Embodiment 3 of the present invention;

[0031] Figure 4 This is a schematic diagram of the structure of a vehicle obstacle avoidance device according to Embodiment 4 of the present invention;

[0032] Figure 5 This is a schematic diagram of the structure of an electronic device that implements the obstacle avoidance method for vehicles according to embodiments of the present invention. Detailed Implementation

[0033] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0034] It should be noted that the terms "first" and "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0035] In the technical solutions of this invention, the acquisition, storage, and application of the target obstacle detection frame, the size information of the detection frame, and the vehicle driving direction, etc., all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0036] Example 1

[0037] Figure 1 This is a flowchart of a vehicle obstacle avoidance method provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where a vehicle is controlled to avoid obstacles. The method can be executed by a vehicle obstacle avoidance device, which can be implemented in hardware and / or software and specifically configured in an electronic device, such as a server or vehicle.

[0038] See Figure 1 The obstacle avoidance methods shown for the vehicle include:

[0039] S101. Obtain the detection frame of the target obstacle, the size information of the detection frame, and the vehicle's driving direction; wherein, the target obstacle is an obstacle detected by the vehicle in real time.

[0040] In this embodiment, the detection bounding box can be an image region containing the target obstacle. The size information of the detection bounding box may include, but is not limited to, its height, width, and shape. The vehicle's travel direction can be the direction the vehicle is moving in real time. The vehicle can be, for example, agricultural machinery, i.e., a vehicle used for agricultural operations.

[0041] In one optional embodiment, obstacles can be detected in real time using an obstacle detector, and the detection boxes of the target obstacles detected by the obstacle detector can be obtained. The obstacle detector can employ a neural network model, such as the YOLOv8 object detection model, the Faster R-CNN detection model, or the SSD (Single Shot MultiBox Detector) model. The obstacle detector can be trained using a dataset of obstacles commonly encountered during vehicle operations, and the optimal trained model can be deployed in the vehicle based on the TensorRT (deep learning inference engine) framework.

[0042] Furthermore, an RGB-D (color-depth) camera can be deployed in the vehicle. The RGB-D camera acquires color images of the vehicle's surroundings in real time, and an obstacle detector performs target recognition on these images to obtain bounding boxes for the obstacles. The RGB-D camera can also acquire depth images, and the true distance between objects in the image and the camera can be calculated based on the depth image information. For example, the true position of an obstacle can be calculated using the following formula:

[0043]

[0044] Where (u, v) represents the pixel coordinates of the obstacle in the image captured by the camera; d represents the sampling depth value; PPx and PPy are camera intrinsic parameters; and (x0, y0, z0) represents the three-dimensional coordinates of the obstacle relative to the camera. It should be noted that the relative distance between the target obstacle and the vehicle mentioned below refers to the relative distance between the target obstacle and the camera on the vehicle.

[0045] In one optional implementation, the vehicle is equipped with a GNSS (Global Navigation Satellite System) antenna. When capturing color images using an RGB-D camera, the lateral offset of the RGB-D camera relative to the GNSS antenna can be measured, and the RGB-D camera can be adjusted based on this lateral offset. The vehicle can obtain real-time vehicle latitude and longitude information via the GNSS antenna. First, the vehicle initializes and accepts the work path. After entering the working state, it executes the obstacle avoidance method provided in any embodiment of this invention.

[0046] S102. Based on the size information of the detection box, determine the coordinates of the pixel center point of the target obstacle and construct the depth sampling area.

[0047] In this embodiment, the pixel center coordinates of the target obstacle can be obtained from the obstacle detection bounding box position in the image captured by the camera. The depth sampling region can be a region used to sample the depth between the obstacle and the vehicle. Specifically, a certain algorithm is used to determine the pixel center coordinates of the target obstacle based on the size information of the detection bounding box, and a depth sampling region is constructed.

[0048] S103. Based on the center of the depth sampling region, perform obstacle depth augmentation sampling on the depth image to obtain at least one obstacle sampling depth.

[0049] In this embodiment, the obstacle sampling depth can be the sampled depth between the obstacle and the vehicle. Specifically, using a certain algorithm, multiple location points are randomly expanded based on the center of the depth sampling area, and obstacle depth sampling is performed on the depth image according to the expanded location points to obtain at least one obstacle sampling depth.

[0050] S104. Determine the relative distance between the target obstacle and the vehicle based on the sampling depth of each obstacle.

[0051] Specifically, a certain algorithm is used to determine the relative distance between the target obstacle and the vehicle based on the sampling depth of each obstacle.

[0052] Optionally, the relative distance between the target obstacle and the vehicle is determined based on the sampling depth of each obstacle, including: calculating the median of the sampling depths of each obstacle; determining the median as the relative depth between the target obstacle and the vehicle; and determining the relative distance between the target obstacle and the vehicle based on the camera parameters and the relative depth between the target obstacle and the vehicle. The camera parameters refer to the camera's intrinsic and extrinsic parameters. It is understood that by employing the above technical solution, and by calculating the median of the sampling depths of each obstacle and converting the sampling depth corresponding to the median to obtain the relative distance between the target obstacle and the vehicle, large and small deviations in the obstacle sampling depths can be avoided, thus improving the accuracy of the relative distance.

[0053] In one optional implementation, the median of the obstacle sampling depths in each consecutive frame can be used; and the average of the medians across consecutive frames can be converted to obtain the relative distance between the target obstacle and the vehicle. For example, the relative depth can be determined using the following formula:

[0054]

[0055] Where, d z Indicates relative depth; n represents the number of consecutive frames containing the target obstacle; d j This represents the median of the sampling depths of all obstacles corresponding to the j-th frame.

[0056] S105. Based on the horizontal position, relative distance, vehicle driving direction, set distance threshold, and set monitoring angle, determine the target obstacle avoidance command that the vehicle needs to execute.

[0057] In this embodiment, the monitoring angle can be set as the range angle centered on the vehicle for detecting obstacles. For example, with the straight line along the vehicle's direction of travel as 0 degrees, if the obstacle detection range is set to be from 45 degrees to the left and 45 degrees to the right of the vehicle's direction of travel, then the monitoring angle is set to 90°. Target obstacle avoidance commands can include, but are not limited to, empty commands, deceleration commands, and obstacle bypass commands. Among these, obstacle bypass commands control the vehicle to avoid obstacles; deceleration commands control the vehicle to decelerate; empty commands do not instruct the vehicle to perform any operation.

[0058] It should be noted that the distance threshold and monitoring angle can be set independently by technicians based on actual needs or practical experience, and this invention does not impose any limitations on this. Specifically, a certain algorithm is used to determine the target obstacle avoidance command that the vehicle needs to execute based on the horizontal position, relative distance, vehicle driving direction, set distance threshold, and set monitoring angle.

[0059] This invention provides an embodiment of the invention that acquires the detection frame of a target obstacle, the size information of the detection frame, and the vehicle's driving direction. The target obstacle is an obstacle detected by the vehicle in real time. Based on the size information of the detection frame, the coordinates of the pixel center point of the target obstacle are determined, and a depth sampling region is constructed. Based on the center of the depth sampling region, the depth image is augmented with obstacle depth sampling to obtain at least one obstacle sampling depth. Based on each obstacle sampling depth, the relative distance between the target obstacle and the vehicle is determined. Based on the horizontal position, relative distance, vehicle driving direction, a set distance threshold, and a set monitoring angle, the target obstacle avoidance command that the vehicle needs to execute is determined. This invention can accurately identify the distance between the obstacle and the vehicle through the obstacle detection frame and its size information, and automatically determine whether obstacle avoidance is necessary based on the distance. This achieves obstacle identification and automatic obstacle avoidance, improving the safety of vehicle operation.

[0060] Example 2

[0061] Figure 2 This is a flowchart of a vehicle obstacle avoidance method provided in Embodiment 2 of the present invention. The present invention is an optimization and improvement based on the technical solution of the above embodiments.

[0062] Furthermore, the process of "determining the pixel center coordinates of the target obstacle based on the size information of the detection box and constructing the depth sampling region" is further refined into "determining the center coordinates of the detection box based on the coordinates of the upper left and lower right vertices of the detection box; and constructing the depth sampling region with the center coordinates as the origin based on the set radius".

[0063] It should be noted that for any parts not described in detail in the embodiments of the present invention, please refer to the description in the foregoing embodiments.

[0064] See Figure 2 The obstacle avoidance methods shown for the vehicle include:

[0065] S201. Obtain the detection frame of the target obstacle, the size information of the detection frame, and the vehicle's driving direction; wherein, the target obstacle is an obstacle detected by the vehicle in real time.

[0066] S202. Determine the coordinates of the center point of the detection box based on the coordinates of the top left and bottom right vertices of the detection box.

[0067] In this embodiment, the size information includes the coordinates of the top left vertex and the bottom right vertex of the detection box.

[0068] Specifically, the x-coordinate of the center point is determined based on the x-coordinates of the top-left and bottom-right vertices; the y-coordinate of the center point is determined based on the x-coordinates of the top-left and bottom-right vertices. For example, the following formula can be used to determine the coordinates of the center point:

[0069]

[0070]

[0071] Where, x m The x-coordinate represents the center point. l The x-coordinate represents the top-left vertex; r The x-coordinate of the bottom right vertex; y m The y-coordinate represents the center point. l The y-coordinate represents the top-left vertex. r This represents the y-coordinate of the bottom right vertex.

[0072] S203. Based on the set radius, construct the depth sampling area with the center point coordinates as the origin.

[0073] In this embodiment, it should be noted that the set radius can be set independently by technicians based on actual needs or practical experience, and this invention does not limit this. For example, the depth sampling area can be represented by the following formula:

[0074] (x′-x m )+(y′-ym )≤r 2 ;

[0075] Where x' represents the horizontal coordinate range of the depth sampling region; y' represents the vertical coordinate range of the depth sampling region; and r represents the set radius.

[0076] S204. Based on the center of the depth sampling region, perform obstacle depth augmentation sampling on the depth image to obtain at least one obstacle sampling depth.

[0077] Optionally, the depth image is augmented with obstacle depth based on the center of the depth sampling region, including: generating at least one random coordinate offset; adjusting the center point coordinates according to the random coordinate offset for each random coordinate offset to obtain adjusted coordinates; determining the adjusted coordinates located in the depth sampling region as target sampling coordinates; and sampling the depth between the object and the vehicle at the target sampling coordinate from the depth image for each target sampling coordinate to obtain the obstacle sampling depth corresponding to the target sampling coordinate.

[0078] The random offset can include, but is not limited to, random horizontal coordinate offset and random vertical coordinate offset. The adjusted coordinates are the adjusted center point coordinates; the target sampling coordinates are the adjusted coordinates located within the depth sampling area.

[0079] Specifically, for each random coordinate offset, the random coordinate offset is added to or subtracted from the center point coordinate to obtain the adjusted coordinate; the adjusted coordinates located in the depth sampling region are filtered out, and the adjusted coordinates located in the depth sampling region are determined as the target sampling coordinates; for each target sampling coordinate, the depth between the object and the vehicle at the target sampling coordinate is sampled from the depth image to obtain the obstacle sampling depth corresponding to the target sampling coordinate.

[0080] It is understandable that by adopting the above technical solution, at least one random coordinate offset can be generated; for each random coordinate offset, the coordinates of the center point are adjusted according to the random coordinate offset to obtain the adjusted coordinates; the adjusted coordinates located in the depth sampling area are determined as the target sampling coordinates; for each target sampling coordinate, the depth between the object and the vehicle at the target sampling coordinate is sampled from the depth image to obtain the obstacle sampling depth corresponding to the target sampling coordinate, which improves the richness of obstacle sampling depth and thus improves the accuracy of the relative distance between the target obstacle and the vehicle.

[0081] S205. Determine the relative distance between the target obstacle and the vehicle based on the sampling depth of each obstacle.

[0082] S206. Based on the horizontal position, relative distance, vehicle driving direction, set distance threshold, and set monitoring angle, determine the target obstacle avoidance command that the vehicle needs to execute.

[0083] In this embodiment of the invention, the center point coordinates of the detection frame are determined based on the coordinates of the upper left and lower right vertices of the detection frame; the center point coordinates are determined as the horizontal position of the target obstacle; and a depth sampling area is constructed with the center point coordinates as the origin according to a set radius. This improves the matching degree between the depth sampling area and the position of the target obstacle, thereby improving the accuracy of determining the relative distance between the target obstacle and the vehicle from the depth sampling area.

[0084] Example 3

[0085] Figure 3A This is a flowchart of a vehicle obstacle avoidance method provided in Embodiment 3 of the present invention. The present invention is an optimization and improvement based on the technical solutions of the above embodiments.

[0086] Furthermore, the process of "determining the target obstacle avoidance command to be executed by the vehicle based on the horizontal position, relative distance, vehicle driving direction, set distance threshold, and set monitoring angle" is further refined into "determining the angle between the first straight line and the straight line where the vehicle driving direction is located based on the relative distance and horizontal position; the first straight line is the connecting straight line between the vehicle and the obstacle; comparing the angle with the set angle threshold to obtain the first comparison result; comparing the relative distance with the first distance threshold to obtain the second comparison result; comparing the relative distance with the second distance threshold to obtain the third comparison result; and determining the target obstacle avoidance command to be executed by the vehicle based on the first comparison result, the second comparison result, and the third comparison result."

[0087] It should be noted that for any parts not described in detail in the embodiments of the present invention, please refer to the description in the foregoing embodiments.

[0088] See Figure 3A The obstacle avoidance methods shown for the vehicle include:

[0089] S301. Obtain the detection frame of the target obstacle, the size information of the detection frame, and the vehicle's driving direction; wherein, the target obstacle is an obstacle detected by the vehicle in real time.

[0090] S302. Based on the size information of the detection box, determine the coordinates of the pixel center point of the target obstacle and construct the depth sampling area.

[0091] S303. Based on the center of the depth sampling region, perform obstacle depth augmentation sampling on the depth image.

[0092] S304. Determine the relative distance between the target obstacle and the vehicle based on the sampling depth of each obstacle.

[0093] S305. Determine the angle between the first straight line and the straight line in the direction of vehicle travel based on the relative distance and horizontal position; the first straight line is the connecting line between the vehicle and the obstacle.

[0094] For example, trigonometric functions can be used to take the relative distance as the hypotenuse of a triangle, the position component value perpendicular to the side of the vehicle in the horizontal position as the opposite side of the triangle, and the angle between the first straight line and the straight line in the direction of vehicle travel can be determined by the sine function.

[0095] S306. Compare the included angle with the set angle threshold to obtain the first comparison result.

[0096] In this embodiment, the angle threshold can be set independently by technicians based on actual needs or practical experience, and the present invention does not limit this.

[0097] S307. Compare the relative distance with the first distance threshold to obtain the second comparison result.

[0098] In this embodiment, the first distance threshold can be set independently by technicians based on actual needs or practical experience, and the present invention does not limit this.

[0099] S308. Compare the relative distance with the second distance threshold to obtain the third comparison result.

[0100] In this embodiment, the second distance threshold can be set independently by technicians based on actual needs or practical experience, and this invention does not limit this. The first distance threshold is greater than the second distance threshold;

[0101] S309. Based on the first comparison result, the second comparison result, and the third comparison result, determine the target obstacle avoidance command that the vehicle needs to execute.

[0102] Specifically, a certain algorithm is used to determine the target obstacle avoidance command that the vehicle needs to execute based on the first comparison result, the second comparison result, and the third comparison result.

[0103] Optionally, based on the first comparison result, the second comparison result, and the third comparison result, the target obstacle avoidance command to be executed by the vehicle is determined, including: if the first comparison result is that the included angle is greater than a set angle threshold, then the target obstacle avoidance command is determined to be empty; if the first comparison result is that the included angle is less than or equal to the set angle threshold, and the second comparison result is that the relative distance is greater than a first distance threshold, then the target obstacle avoidance command is determined to be empty; if the first comparison result is that the included angle is less than or equal to the set angle threshold, the second comparison result is that the relative distance is less than or equal to the first distance threshold, and the third comparison result is that the relative distance is greater than the second distance threshold, then the vehicle deceleration command is determined to be the target obstacle avoidance command; if the first comparison result is that the included angle is less than or equal to the set angle threshold, and the third comparison result is that the relative distance is less than or equal to the second distance threshold, then the obstacle avoidance command is determined to be the target obstacle avoidance command.

[0104] It is understandable that by adopting the above technical solution, the relationship between the included angle and the set angle threshold, the relationship between the relative distance and the first distance threshold, and the relationship between the relative distance and the second distance threshold can be used to flexibly determine the target obstacle avoidance command, thereby improving the flexibility of the target obstacle avoidance command.

[0105] Optional, Figure 3B This is a schematic diagram of an obstacle detection area. (Example) Figure 3B As shown, the vehicle's forward direction is the x-axis, and the direction perpendicular to the left side of the vehicle is the y-axis. θ is the set monitoring angle; r1 is the first distance threshold; and r2 is the second distance threshold. Within the angle θ, the area between the first and second distance thresholds is the obstacle warning zone; the area within the second distance threshold is the emergency obstacle avoidance zone.

[0106] Optionally, after determining the obstacle avoidance command as the target obstacle avoidance command, the method further includes: obtaining the coordinates of the obstacle avoidance path point in the vehicle coordinate system, the coordinate rotation matrix, and the latitude and longitude coordinates of the vehicle's real-time position; for each obstacle avoidance path point, determining the latitude and longitude coordinates of the obstacle avoidance path point based on the coordinates of the obstacle avoidance path point in the vehicle coordinate system, the coordinate rotation matrix, the latitude and longitude coordinates of the vehicle's real-time position, and the Earth's radius; and controlling the vehicle to drive according to the latitude and longitude coordinates of the obstacle avoidance path point to avoid obstacles.

[0107] The obstacle avoidance path points can be path points within the vehicle's obstacle avoidance path. The obstacle avoidance path can be determined using any existing technology, and this invention is not limited to this; for example, it can be generated using an obstacle avoidance path planning algorithm. The obstacle avoidance path can be determined in the vehicle's coordinate system. The latitude and longitude coordinates of the vehicle's real-time position can be determined using a GNSS antenna.

[0108] Specifically, based on the coordinate rotation matrix, the obstacle avoidance path points are transformed from the vehicle coordinate system to the global coordinate system, obtaining the coordinates of the obstacle avoidance path points in the global coordinate system, and determining the coordinates of the vehicle's real-time position in the global coordinate system.

[0109] Optional, Figure 3C This is a schematic diagram of a vehicle coordinate system and a global coordinate system; as shown in the figure, the x1y1 coordinate system is the vehicle coordinate system with the vehicle as the origin; the x0y0 coordinate system is the global coordinate system with a fixed position other than the vehicle as the origin.

[0110] Based on the vehicle's real-time position coordinates in the global coordinate system and the coordinates of the obstacle avoidance path point in the global coordinate system, determine the distance between the obstacle avoidance path point and the vehicle's real-time position, and determine the azimuth angle of the obstacle avoidance path point relative to the vehicle's real-time position. The azimuth angle can be the angle between the line connecting the obstacle avoidance path point and the vehicle's real-time position and a fixed direction, such as true north. Based on the azimuth angle, the vehicle's real-time position's latitude and longitude coordinates, and the distance between the vehicle's real-time position and the obstacle avoidance path point, determine the latitude and longitude coordinates of the obstacle avoidance path point. For example, the latitude and longitude coordinates of the obstacle avoidance path point can be determined using the following formula:

[0111]

[0112]

[0113] Where lon1 represents the longitude value of the vehicle's real-time location; lon2 represents the longitude value of the obstacle avoidance path point; d represents the distance between the vehicle's real-time location and the obstacle avoidance path point; α represents the azimuth angle value; ARC represents the Earth's radius value; lat1 represents the latitude value of the vehicle's real-time location; and lat2 represents the longitude value of the obstacle avoidance path point.

[0114] Optional, Figure 3D This is a schematic diagram of an azimuth angle, as shown in the figure. (lon1, lat1) are the latitude and longitude coordinates of the vehicle's real-time position; (lon2, lat2) are the latitude and longitude coordinates of the vehicle's real-time position; the distance between the vehicle's real-time position and the obstacle avoidance path point is d; the azimuth angle is the angle between the line connecting the vehicle's real-time position and the obstacle avoidance path point and the straight line in the due north direction, with a magnitude of α.

[0115] Understandably, by adopting the above technical solution, the distance between the obstacle avoidance path point and the vehicle's real-time position, as well as the azimuth angle, coordinate rotation matrix, and latitude and longitude coordinates of the vehicle's real-time position, are obtained. For each obstacle avoidance path point, the latitude and longitude coordinates of the obstacle avoidance path point are determined based on the distance between the obstacle avoidance path point and the vehicle's real-time position, the azimuth angle, the coordinate rotation matrix, the latitude and longitude coordinates of the vehicle's real-time position, and the Earth's radius. By controlling the vehicle to travel according to the latitude and longitude coordinates of the obstacle avoidance path point, the success rate of obstacle avoidance can be improved.

[0116] In this embodiment of the invention, the angle between a first straight line and the straight line in the vehicle's direction of travel is determined based on the relative distance and horizontal position; the first straight line is the connecting line between the vehicle and the obstacle; the angle is compared with a set angle threshold to obtain a first comparison result; the relative distance is compared with a first distance threshold to obtain a second comparison result; the relative distance is compared with a second distance threshold to obtain a third comparison result; based on the first comparison result, the second comparison result, and the third comparison result, the target obstacle avoidance command that the vehicle needs to execute is determined, thereby improving the accuracy of the target obstacle avoidance command.

[0117] Example 4

[0118] Figure 4 This is a schematic diagram of a vehicle obstacle avoidance device according to Embodiment 4 of the present invention. This embodiment is applicable to situations where a vehicle is controlled to avoid obstacles. The device can execute the vehicle's obstacle avoidance method and can be implemented in hardware and / or software. The device can be configured in electronic devices, such as servers or vehicles.

[0119] See Figure 4 The obstacle avoidance device for the vehicle shown includes an information acquisition module 401, a region construction module 402, a sampling depth determination module 403, a relative distance determination module 404, and a command determination module 405, wherein...

[0120] The information acquisition module 401 is used to acquire the detection frame of the target obstacle, the size information of the detection frame, and the vehicle's driving direction; wherein, the target obstacle is an obstacle detected by the vehicle in real time;

[0121] The region construction module 402 is used to determine the pixel center point coordinates of the target obstacle based on the size information of the detection box, and to construct the depth sampling region;

[0122] The sampling depth determination module 403 is used to perform extended sampling of the obstacle depth on the depth image based on the center of the depth sampling region to obtain at least one obstacle sampling depth.

[0123] The relative distance determination module 404 is used to determine the relative distance between the target obstacle and the vehicle based on the sampling depth of each obstacle;

[0124] The instruction determination module 405 is used to determine the target obstacle avoidance instruction that the vehicle needs to execute based on the horizontal position, relative distance, vehicle driving direction, set distance threshold and set monitoring angle.

[0125] This invention, through an information acquisition module, acquires the detection frame of a target obstacle, the size information of the detection frame, and the vehicle's driving direction; wherein, the target obstacle is an obstacle detected by the vehicle in real time; through a region construction module, the pixel center point coordinates of the target obstacle are determined based on the size information of the detection frame, and a depth sampling region is constructed; through a sampling depth determination module, the obstacle depth is extended and sampled in the depth image based on the center of the depth sampling region to obtain at least one obstacle sampling depth; through a relative distance determination module, the relative distance between the target obstacle and the vehicle is determined based on the sampling depths of each obstacle; through an instruction determination module, the target obstacle avoidance instruction to be executed by the vehicle is determined based on the horizontal position, relative distance, vehicle driving direction, set distance threshold, and set monitoring angle. This invention can accurately identify the distance between the obstacle and the vehicle through the obstacle detection frame and its size information, and automatically determine whether obstacle avoidance is required based on the distance, thus achieving obstacle identification and automatic obstacle avoidance, improving the safety of vehicle operation.

[0126] Optionally, the size information includes the coordinates of the top-left and bottom-right vertices of the detection box;

[0127] Region building module 402 includes:

[0128] The center point coordinate determination unit is used to determine the center point coordinates of the detection box based on the coordinates of the top left and bottom right vertices of the detection box.

[0129] The horizontal position determination unit is used to determine the horizontal position of the target obstacle by the coordinates of the center point;

[0130] The sampling area determination unit is used to construct a depth sampling area based on a set radius, with the coordinates of the center point as the origin.

[0131] Optionally, the sampling depth determination module 403 includes:

[0132] Offset determination unit, used to generate at least one random coordinate offset;

[0133] The coordinate adjustment unit is used to adjust the center point coordinates according to each random coordinate offset to obtain the adjusted coordinates.

[0134] The sampling coordinate determination unit is used to determine the adjustment coordinates located in the depth sampling area as the target sampling coordinates;

[0135] The sampling depth determination unit is used to sample the depth between the object and the vehicle at each target sampling coordinate to obtain the obstacle sampling depth corresponding to the target sampling coordinate.

[0136] Optionally, the relative distance determination module 404 includes:

[0137] The median determination unit is used to calculate the median of the sampling depths of each obstacle;

[0138] The relative depth determination unit is used to determine the median as the relative depth between the target obstacle and the vehicle;

[0139] The relative distance determination unit is used to determine the relative distance between the target obstacle and the vehicle based on the parameters of the camera and the relative depth between the target obstacle and the vehicle.

[0140] Optionally, the distance threshold can be set to include a first distance threshold and a second distance threshold; the first distance threshold is greater than the second distance threshold.

[0141] Instruction determination module 405 includes:

[0142] Angle determination unit is used to determine the angle between the first straight line and the straight line in the direction of vehicle travel based on the relative distance and horizontal position; the first straight line is the connecting line between the vehicle and the obstacle;

[0143] The first comparison unit is used to compare the included angle with a set angle threshold to obtain the first comparison result;

[0144] The second comparison unit is used to compare the relative distance with the first distance threshold to obtain a second comparison result;

[0145] The third comparison unit is used to compare the relative distance with the second distance threshold to obtain the third comparison result;

[0146] The instruction determination unit is used to determine the target obstacle avoidance instruction that the vehicle needs to execute based on the first comparison result, the second comparison result, and the third comparison result.

[0147] Optional, instruction determination unit, specifically used for:

[0148] If the first comparison result is that the included angle is greater than the set angle threshold, then the target obstacle avoidance command is determined to be empty;

[0149] If the first comparison result is that the included angle is less than or equal to the set angle threshold, and the second comparison result is that the relative distance is greater than the first distance threshold, then the target obstacle avoidance command is determined to be empty;

[0150] If the first comparison result is that the included angle is less than or equal to the set angle threshold, the second comparison result is that the relative distance is less than or equal to the first distance threshold, and the third comparison result is that the relative distance is greater than the second distance threshold, then the vehicle deceleration command is determined as the target obstacle avoidance command.

[0151] If the first comparison result is that the included angle is less than or equal to the set angle threshold, and the third comparison result is that the relative distance is less than or equal to the second distance threshold, then the obstacle avoidance command will be determined as the target obstacle avoidance command.

[0152] Optionally, the device may also include:

[0153] The data acquisition module is used to acquire the coordinates of the obstacle avoidance path points in the vehicle coordinate system, the coordinate rotation matrix, and the latitude and longitude coordinates of the vehicle's real-time position.

[0154] The latitude and longitude coordinate determination module is used to determine the latitude and longitude coordinates of each obstacle avoidance path point based on the coordinates of the obstacle avoidance path point in the vehicle coordinate system, the coordinate rotation matrix, the latitude and longitude coordinates of the vehicle's real-time position, and the Earth's radius.

[0155] The driving control module is used to control the vehicle to drive according to the latitude and longitude coordinates of the obstacle avoidance path points in order to avoid obstacles.

[0156] The vehicle obstacle avoidance device provided in the embodiments of the present invention can execute the vehicle obstacle avoidance method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the vehicle obstacle avoidance method.

[0157] Example 5

[0158] Figure 5 A schematic diagram of an electronic device 500 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0159] like Figure 5As shown, the electronic device 500 includes at least one processor 501 and a memory, such as a read-only memory (ROM) 502 or a random access memory (RAM) 503, communicatively connected to the at least one processor 501. The memory stores computer programs executable by the at least one processor. The processor 501 can perform various appropriate actions and processes based on the computer program stored in the ROM 502 or loaded into the RAM 503 from storage unit 508. The RAM 503 can also store various programs and data required for the operation of the electronic device 500. The processor 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0160] Multiple components in electronic device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows electronic device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0161] Processor 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 501 performs the various methods and processes described above, such as obstacle avoidance methods for vehicles.

[0162] In some embodiments, the obstacle avoidance method for a vehicle may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded into and / or installed on electronic device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by processor 501, one or more steps of the obstacle avoidance method for a vehicle described above may be performed. Alternatively, in other embodiments, processor 501 may be configured to perform the obstacle avoidance method for a vehicle by any other suitable means (e.g., by means of firmware).

[0163] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0164] Computer programs used to implement the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable vehicle obstacle avoidance device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0165] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0166] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0167] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0168] A computing system can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system. It addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability.

[0169] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0170] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for obstacle avoidance by a vehicle, characterized in that, The method includes: The detection bounding box of the target obstacle, the size information of the detection bounding box, and the vehicle's driving direction are obtained; wherein, the target obstacle is an obstacle detected by the vehicle in real time, the detection bounding box is an image region containing the target obstacle, and the size information includes the coordinates of the upper left vertex and the lower right vertex of the detection bounding box; Based on the size information of the detection frame, the pixel center point coordinates of the target obstacle are determined, and a depth sampling region is constructed; Based on the center of the depth sampling region, the depth image is augmented with obstacle depth to obtain at least one obstacle sampling depth; The relative distance between the target obstacle and the vehicle is determined based on the sampling depth of each obstacle. Based on the horizontal position, the relative distance, the vehicle's direction of travel, the set distance threshold, and the set monitoring angle, the target obstacle avoidance command that the vehicle needs to execute is determined. The target obstacle avoidance command includes an empty command, a deceleration command, and an obstacle bypass command. The obstacle bypass command is used to control the vehicle to bypass the obstacle, and the deceleration command is used to control the vehicle to decelerate. The step of determining the pixel center coordinates of the target obstacle based on the size information of the detection frame and constructing a depth sampling region includes: The coordinates of the center point of the detection box are determined based on the coordinates of the top left and bottom right vertices of the detection box. Based on the set radius, a depth sampling region is constructed with the center point coordinates as the origin; The step of augmenting the depth image based on the center of the depth sampling region to obtain at least one obstacle sampling depth includes: Generate at least one random coordinate offset; For each of the aforementioned random coordinate offsets, the coordinates of the center point are adjusted according to the random coordinate offsets to obtain the adjusted coordinates; The adjustment coordinates located in the depth sampling area are determined as the target sampling coordinates; For each target sampling coordinate, the depth between the object and the vehicle at the target sampling coordinate is sampled from the depth image to obtain the obstacle sampling depth corresponding to the target sampling coordinate.

2. The method according to claim 1, characterized in that, Determining the relative distance between the target obstacle and the vehicle based on the sampling depth of each obstacle includes: Calculate the median of the sampling depths for each of the aforementioned obstacles; The median is determined as the relative depth between the target obstacle and the vehicle; The relative distance between the target obstacle and the vehicle is determined based on the camera parameters and the relative depth between the target obstacle and the vehicle.

3. The method according to claim 1, characterized in that, The set distance threshold includes a first distance threshold and a second distance threshold; the first distance threshold is greater than the second distance threshold. The process of determining the target obstacle avoidance command that the vehicle needs to execute based on the horizontal position, the relative distance, the vehicle's driving direction, a set distance threshold, and a set monitoring angle includes: Based on the relative distance and the horizontal position, the angle between the first straight line and the straight line containing the vehicle's direction of travel is determined; the first straight line is the connecting line between the vehicle and the obstacle. By comparing the included angle with the set angle threshold, a first comparison result is obtained; By comparing the relative distance with the first distance threshold, a second comparison result is obtained; A third comparison result is obtained by comparing the relative distance with the second distance threshold; Based on the first comparison result, the second comparison result, and the third comparison result, the target obstacle avoidance command that the vehicle needs to execute is determined.

4. The method according to claim 3, characterized in that, The step of determining the target obstacle avoidance command that the vehicle needs to execute based on the first comparison result, the second comparison result, and the third comparison result includes: If the first comparison result indicates that the included angle is greater than the set angle threshold, then the target obstacle avoidance command is determined to be empty; If the first comparison result is that the included angle is less than or equal to the set angle threshold, and the second comparison result is that the relative distance is greater than the first distance threshold, then the target obstacle avoidance command is determined to be empty; If the first comparison result is that the included angle is less than or equal to the set angle threshold, the second comparison result is that the relative distance is less than or equal to the first distance threshold, and the third comparison result is that the relative distance is greater than the second distance threshold, then the vehicle deceleration command is determined as the target obstacle avoidance command. If the first comparison result is that the included angle is less than or equal to the set angle threshold, and the third comparison result is that the relative distance is less than or equal to the second distance threshold, then the obstacle avoidance command is determined as the target obstacle avoidance command.

5. The method according to claim 4, characterized in that, After determining the obstacle avoidance command as the target obstacle avoidance command, it also includes: Obtain the coordinates of the obstacle avoidance path points in the vehicle coordinate system, the coordinate rotation matrix, and the latitude and longitude coordinates of the vehicle's real-time position; For each obstacle avoidance path point, the latitude and longitude coordinates of the obstacle avoidance path point are determined based on the coordinates of the obstacle avoidance path point in the vehicle coordinate system, the coordinate rotation matrix, the latitude and longitude coordinates of the vehicle's real-time position, and the Earth's radius. The vehicle is controlled to travel according to the latitude and longitude coordinates of the obstacle avoidance path points in order to avoid obstacles.

6. An obstacle avoidance device for a vehicle, characterized in that, The device includes: The information acquisition module is used to acquire the detection box of the target obstacle, the size information of the detection box, and the vehicle's driving direction; wherein, the target obstacle is an obstacle detected by the vehicle in real time, the detection box is an image area containing the target obstacle, and the size information includes the coordinates of the upper left vertex and the lower right vertex of the detection box; The region construction module is used to determine the pixel center point coordinates of the target obstacle based on the size information of the detection box, and to construct the depth sampling region; The sampling depth determination module is used to perform extended sampling of obstacle depth in the depth image based on the center of the depth sampling region to obtain at least one obstacle sampling depth; A relative distance determination module is used to determine the relative distance between the target obstacle and the vehicle based on the sampling depth of each obstacle. The instruction determination module is used to determine the target obstacle avoidance instruction that the vehicle needs to execute based on the horizontal position, the relative distance, the vehicle's driving direction, a set distance threshold, and a set monitoring angle. The target obstacle avoidance instruction includes an empty instruction, a deceleration instruction, and an obstacle bypass instruction. The obstacle bypass instruction is used to control the vehicle to bypass the obstacle, and the deceleration instruction is used to control the vehicle to decelerate. The region building module includes: The center point coordinate determination unit is used to determine the center point coordinates of the detection box based on the coordinates of the top left and bottom right vertices of the detection box. The sampling area determination unit is used to construct a depth sampling area based on a set radius, with the coordinates of the center point as the origin; The sampling depth determination module includes: Offset determination unit, used to generate at least one random coordinate offset; A coordinate adjustment unit is used to adjust the coordinates of the center point according to each random coordinate offset to obtain the adjusted coordinates. The sampling coordinate determination unit is used to determine the adjustment coordinates located in the depth sampling area as the target sampling coordinates; The sampling depth determination unit is used to sample the depth between the object and the vehicle at each target sampling coordinate from the depth image to obtain the obstacle sampling depth corresponding to the target sampling coordinate.

7. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the obstacle avoidance method of the vehicle according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the obstacle avoidance method of the vehicle according to any one of claims 1-5.

Images