Automatic driving method, device and storage medium

Abstract

The present invention discloses an autonomous driving method, device, and storage medium, comprising obtaining obstacle perception information and multiple perception images; obtaining driving boundary information corresponding to the multiple perception images based on the multiple perception images; optimizing the multiple driving boundary information based on the obstacle perception information to obtain multiple optimized driving boundary information; and fusing the multiple optimized driving boundary information to obtain drivable area information. By optimizing each driving boundary information based on the obstacle perception information and fusing the multiple optimized driving boundary information, the present invention can significantly improve the accuracy and reliability of the drivable area, avoid analysis anomalies, and expand the range of the drivable area.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to an autonomous driving method, device, and storage medium. Background Art

[0002] FreeSpace is a feature of advanced driver assistance systems (ADAS) in the autonomous driving field. Currently, the more common surround-view freespace method is to obtain raw images from multiple sensing devices, dedistort them, and splice them into an IPM map for segmentation. However, the distance that can be represented by the boundaries of the IPM map is much smaller than that of the original image (the dedistortion and splicing operation will lose some image information), and the semantic segmentation method is time-consuming. In addition, the freespace results of existing technologies are not accurate enough, and the freespace results of multiple sensing devices overlap, which can easily lead to analysis anomalies. Summary of the Invention

[0003] To address the problems in the prior art, the present invention provides an autonomous driving method, device, and storage medium that can significantly improve the accuracy and reliability of drivable areas. The technical solution is as follows:

[0004] In one aspect, the present invention provides an autonomous driving method, comprising:

[0005] Obtaining obstacle perception information and a plurality of perception images; the perception images are acquired by a vehicle-mounted perception device corresponding to the perception images;

[0006] obtaining driving boundary information corresponding to the plurality of perception images according to the plurality of perception images;

[0007] Optimizing the plurality of driving boundary information respectively according to the obstacle perception information to obtain a plurality of optimized driving boundary information;

[0008] A plurality of optimized driving boundary information are fused to obtain driving area information; the driving area information represents the driving area of ​​the vehicle.

[0009] Furthermore, obtaining the driving boundary information corresponding to the plurality of perception images according to the plurality of perception images includes:

[0010] Performing regression processing on the plurality of perception images to obtain a plurality of driving boundary information; or

[0011] Performing regression processing on the plurality of perception images to obtain a plurality of regression boundary information;

[0012] A first interpolation process is performed on the plurality of regression boundary information to obtain a plurality of driving boundary information.

[0013] Furthermore, the optimizing the plurality of driving boundary information respectively according to the obstacle perception information includes:

[0014] If at least part of the obstacle perception area corresponding to the obstacle perception information is located within the drivable area corresponding to the driving boundary information, the driving boundary information is updated according to the obstacle perception information to obtain the optimized driving boundary information.

[0015] Furthermore, updating the driving boundary information according to the obstacle perception information includes:

[0016] Obtaining a boundary area to be updated according to the obstacle perception area and the driving boundary information;

[0017] The driving boundary information corresponding to the to-be-updated boundary area is updated according to the obstacle perception information.

[0018] Furthermore, the boundary area to be updated includes the obstacle perception area and the jump area.

[0019] Furthermore, the updating of the driving boundary information corresponding to the boundary area to be updated further includes:

[0020] A second interpolation process is performed in the jump region to update the boundary information corresponding to the jump region.

[0021] Furthermore, the fusing of the plurality of optimized driving boundary information includes:

[0022] performing a first filtering process on the plurality of optimized driving boundary information to obtain a plurality of filtered boundary information;

[0023] The fusion process is performed on the plurality of filtered boundary information.

[0024] Furthermore, the fusion process includes:

[0025] Determining, based on the positions of the plurality of vehicle-mounted sensing devices, preset areas corresponding to two adjacent vehicle-mounted sensing devices in a vehicle coordinate system;

[0026] fusing the optimized driving boundary information in the plurality of preset areas respectively to obtain a plurality of first fused information;

[0027] The drivable area information is obtained according to the plurality of first fusion information.

[0028] In another aspect, the present invention provides an automatic driving device, comprising at least:

[0029] An acquisition module, configured to acquire obstacle perception information and a plurality of perception images; the perception images are acquired by an on-vehicle perception device corresponding to the perception images;

[0030] An image processing module, configured to obtain driving boundary information corresponding to the plurality of perception images based on the plurality of perception images;

[0031] an optimization module, configured to optimize the plurality of driving boundary information respectively according to the obstacle perception information to obtain a plurality of optimized driving boundary information;

[0032] The fusion module is used to fuse the plurality of optimized driving boundary information to obtain driving area information; the driving area information represents the driving area of ​​the vehicle.

[0033] On the other hand, the present invention provides a storage medium, which stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by a processor to implement the autonomous driving method as described above.

[0034] The implementation of the present invention has the following beneficial effects:

[0035] 1. The present invention optimizes the driving boundary information corresponding to each perception image through obstacle perception information to improve the accuracy and reliability of the driving boundary information corresponding to each perception image; and through fusion processing, multiple optimized driving boundary information are merged to obtain a longer range of drivable areas, and overlap between multiple perception images can be avoided, thereby avoiding the situation where multiple optimized driving boundary information in the same area are inconsistent, further improving the accuracy and reliability of the drivable area.

[0036] 2. The present invention regresses each perceived image through regression processing. Compared with IPM map segmentation, it can improve the accuracy of driving boundary information, greatly reduce time consumption, and improve real-time performance.

[0037] 3. The present invention adopts interpolation and sampling methods to perform a first interpolation process on the regression boundary information after regression processing, and a first filtering process on the optimized driving boundary information, which can greatly improve the continuity and accuracy of the information and greatly reduce the amount of calculation, which is conducive to improving the real-time performance of autonomous driving.

[0038] 4. In the fusion process, a second filtering process is performed based on the fusion points to obtain the first fusion information within the preset area, which can eliminate the overlap between the corresponding two perception images, thereby improving the consistency of the drivable area information within the preset area, with good fusion effect and high accuracy of the drivable area. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings used in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be derived from these drawings without inventive effort.

[0040] Figure 1 A logical structure diagram of an autonomous driving method provided by an embodiment of the present invention;

[0041] Figure 2 A logical structure diagram of a method for obtaining driving boundary information provided by an embodiment of the present invention;

[0042] Figure 3 A logical structure diagram of a processing method for a single perceived image provided by an embodiment of the present invention;

[0043] Figure 4 Schematic diagram of the interpolation method of the first interpolation process in the present invention;

[0044] Figure 5 An example diagram of an optimization process provided by an embodiment of the present invention;

[0045] Figure 6 A logical structure diagram of optimization processing in a possible implementation manner provided by an embodiment of the present invention;

[0046] Figure 7 A logical structure diagram of the first filtering process in a possible implementation manner provided in an embodiment of the present invention;

[0047] Figure 8 A logical structure diagram of the fusion processing of multiple preset areas in a possible implementation manner provided in an embodiment of the present invention;

[0048] Figure 9 This is an example diagram of the optimized driving boundary information in the vehicle body coordinate system of the present invention;

[0049] Figure 10 A logical structure diagram of the fusion processing of a single preset area in a possible implementation manner provided in an embodiment of the present invention;

[0050] Figure 11 An example diagram of fusion processing of a single preset area in a possible implementation manner provided in an embodiment of the present invention;

[0051] Figure 12 A schematic diagram of an optimization process for a drivable area provided in an embodiment of the present invention;

[0052] Figure 13Schematic diagram of an automatic driving device in a possible embodiment of the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments, and therefore should not be understood as limiting the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0054] It should be noted that the terms "first", "second", etc. in the specification, claims, and drawings of the present invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way are interchangeable where appropriate so that the embodiments of the present invention can be implemented in an order other than the following diagrams or descriptions. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or inherent to these processes, methods, products, or devices.

[0055] Taking into account the insufficient accuracy of existing drivable areas and the excessively long planning time, this embodiment provides an autonomous driving method, which can be applied to the autonomous driving device provided in the embodiment of the present invention. First, obstacle perception information and multiple perception images are acquired; wherein the perception images are collected by the vehicle-mounted perception device corresponding to the perception images; then, based on the multiple perception images, driving boundary information corresponding to the multiple perception images is obtained; then, based on the obstacle perception information, the multiple driving boundary information are optimized separately to obtain multiple optimized driving boundary information; the multiple optimized driving boundary information are fused to obtain drivable area information, which represents the drivable area of ​​the vehicle; through optimization and fusion processing, this method can obtain a drivable area with a farther range than the IPM graph segmentation model, thereby improving the accuracy, reliability and real-time performance of the final drivable area information.

[0056] The technical solutions of the embodiments of the present invention are described in detail below. Figure 1 , the method comprising:

[0057] S101, obtaining obstacle perception information and multiple perception images.

[0058] Among them, the perceived image is collected by the vehicle-mounted perception device corresponding to the perceived image; the vehicle-mounted perception device is a device that uses the perceived initial data to identify road information around the vehicle; the vehicle-mounted perception device includes but is not limited to any one of a lidar, a camera and a millimeter-wave radar, and the initial data of the vehicle-mounted perception device includes but is not limited to any one of point cloud data, image data and millimeter-wave radar data. Accordingly, the perceived image is obtained from any one of the point cloud data, image data and millimeter-wave radar data.

[0059] Obstacle perception information is obtained through an obstacle perception device; the obstacle perception device is a device that uses perceived data to identify information about obstacle objects around the vehicle. The obstacle perception device includes but is not limited to any one of a lidar, a camera, and a millimeter-wave radar. For example, in one possible embodiment, the on-board perception device is a camera, and four cameras installed on the vehicle respectively capture four fisheye images as four perception images, while the obstacle perception device is a lidar, which obtains obstacle perception information through point cloud data.

[0060] Furthermore, in one possible embodiment, within the region of the same perceived image, the obstacle sensing device corresponding to the perceived image and the vehicle-mounted sensing device corresponding to the perceived image may be the same, thereby reducing the number of obstacle sensing devices and saving costs. Step S101 may further specifically include:

[0061] Acquiring multiple perception images;

[0062] Obstacle perception information corresponding to the plurality of perception images is obtained according to the plurality of perception images.

[0063] In this process, obstacle perception information is obtained by detecting and analyzing the corresponding perception images, which is more conducive to information analysis, reduces time consumption, and improves real-time performance.

[0064] In one possible embodiment, obstacle perception information is obtained through an obstacle monitoring model. By parsing information about obstacle objects present in the perception image, obstacle perception information of different obstacle objects within the range of the perception image is obtained, so that in the subsequent step S105, the driving boundary information can be optimized according to the obstacle perception information, thereby improving the accuracy and reliability of the final drivable area information and improving the safety of autonomous driving.

[0065] S103: Obtain driving boundary information corresponding to the plurality of perception images according to the plurality of perception images.

[0066] The driving boundary information is the boundary information of the driving area corresponding to the perceived image and obtained by parsing the perceived image. The driving boundary information includes any one of the boundary point information of the driving area, the boundary line information of the driving area, and the boundary vector information of the driving area.

[0067] S105 , optimizing the plurality of driving boundary information respectively according to the obstacle perception information to obtain a plurality of optimized driving boundary information.

[0068] Obstacle perception information is information about obstacle objects within the area corresponding to the perceived image. By parsing the information about the corresponding obstacle objects, obstacle perception areas for different obstacle objects on the perceived image are obtained. In this embodiment, the obstacle perception area is a frame-shaped area, that is, the coordinate point of any corner of the obstacle object and the width and height corresponding to the obstacle object are used to represent the obstacle perception area corresponding to the obstacle object. On the one hand, the shape of the obstacle object is simplified to a regular shape, which helps to reduce time consumption. On the other hand, the area of ​​the obstacle object in the perceived image is increased, which is conducive to avoiding the obstacle object in the final drivable area, reducing the risk of collision, further improving the reliability of the optimized driving boundary information and the final drivable area, and improving the reliability and safety of autonomous driving.

[0069] Optimization processing is an avoidance process that corrects the drivable area corresponding to the driving boundary information according to the position of the obstacle perception area. The position of the boundary of the drivable area is modified to eliminate the obstacle perception area, so that the drivable area corresponding to the optimized driving boundary information does not contain any obstacle objects, greatly reducing the risk of collision with obstacle objects and improving the safety of autonomous driving.

[0070] S107 , fusing the plurality of optimized driving boundary information to obtain driving area information; the driving area information represents the driving area of ​​the vehicle.

[0071] In steps S103-S105, each perception image is processed to obtain the driving boundary information corresponding to each perception image, and then the driving boundary information corresponding to each perception image is optimized to obtain the optimized driving boundary information corresponding to each perception image. In step S107, multiple optimized driving boundary information are aggregated and fused to filter out overlapping information in multiple optimized driving boundary information to obtain overall drivable area information with high reliability and good safety.

[0072] Specifically, in a possible implementation, as Figure 2 As shown, in step S103, obtaining the driving boundary information corresponding to the plurality of perception images according to the plurality of perception images includes:

[0073] S202: Perform regression processing on the plurality of perception images to obtain a plurality of driving boundary information.

[0074] or

[0075] S204a: Perform regression processing on the plurality of perceived images to obtain a plurality of regression boundary information.

[0076] S204b: Perform a first interpolation process on the plurality of regression boundary information to obtain a plurality of driving boundary information.

[0077] Among them, regression processing is a type of supervised learning algorithm, which obtains a mathematical model from continuous statistical data, and then uses the generated regression model for prediction and classification; in this specification, regression processing is a processing method for a single perceived image using a regression model to obtain discrete boundary point information, and the regression processing is any one of linear regression, least squares regression, stepwise regression and multivariate adaptive regression, and the regression model can be any one of a linear regression model, a nonlinear regression model and a neural network model.

[0078] In a possible implementation, in step S202, each perceived image is regressed using the trained regression model to obtain a series of discrete boundary points, such as Figure 3 As shown in FIG. 5( a ), the discrete boundary points constitute the driving boundary information corresponding to the perception image, and the optimization process of the subsequent step S105 is continued.

[0079] In another possible implementation, that is, in step S204a, each perceived image is regressed using the trained regression model to obtain a series of discrete boundary points, such as Figure 3 As shown in Figure (a), discrete boundary points are combined into regression boundary information; and if the number of boundary points returned in the regression boundary information is much smaller than the width (or size) corresponding to the perceived image, then after step S204a, step S204b is continued to be executed to perform a first interpolation process on the boundary points in the regression boundary information to obtain more boundary points within the area corresponding to the perceived image, thereby ensuring the continuity of the boundary points in the driving boundary information, especially at the junction of the boundary points and the obstacle perception area. The interpolated series of boundary points are combined into the driving boundary information corresponding to the perceived image, and the subsequent optimization process of step S105 is continued to further ensure the continuity of the boundary points in the optimized driving boundary information.

[0080] Specifically, after obtaining the regression boundary information, that is, after step S204 or step S204a, the method further includes:

[0081] Initialize the regression boundary information to obtain the information of each boundary point in the image coordinate system and the information of each boundary point in the vehicle coordinate system.

[0082] Among them, the regression processing is performed based on the perceived image, and the subsequent optimization processing and the first interpolation processing are both performed based on the coordinate system. Through initialization, the information of each boundary point in the image coordinate system of the corresponding perceived image and the information of each boundary point in the vehicle coordinate system can be obtained, so that in subsequent processing, the coordinate conversion can be performed according to the position of the on-board perception device; and the information of the boundary point includes the position information of the boundary point, and through the position information of the boundary point, the angle information and distance information of the boundary point in the vehicle coordinate system can be further obtained.

[0083] Specifically, after the regression process (after initialization), performing the first interpolation process on the plurality of regression boundary information to obtain the plurality of driving boundary information includes:

[0084] Effectively processing the multiple regression boundary information to obtain multiple valid boundary information;

[0085] The first interpolation processing is performed on the multiple valid boundary information to obtain the multiple driving boundary information.

[0086] This effective processing includes:

[0087] Obtaining the positions of the boundary points in the regression boundary information;

[0088] When the relative position of the boundary point with respect to the vanishing point satisfies a first preset condition, the position of the boundary point is adjusted to a preset position to obtain the valid boundary information; the relative position is the position of the boundary point with respect to the vanishing point.

[0089] Before performing the first interpolation process, effective processing is first performed to ensure that the interpolation objects of the first interpolation process are all valid, thereby ensuring the accuracy and reliability of the driving boundary information after subsequent interpolation. When the position of the boundary point in the regression boundary information is obtained, a vanishing point determination is performed on the boundary point. The vanishing point refers to the position information of a point at infinity in the image. In a specific embodiment, the position of the vanishing point in the image coordinate system can be calculated using camera extrinsics. If the result of the vanishing point determination is that the relative position of the boundary point with respect to the vanishing point does not meet a first preset condition, the boundary point is considered valid and reliable and does not require effective processing. The boundary point can be directly used as one of the boundary points in the driving boundary information for the subsequent step S204b. If the result of the vanishing point determination is that the relative position of the boundary point with respect to the vanishing point does not meet the first preset condition, the boundary point is considered impossible to exist, and the position of the boundary point is adjusted to a preset position and used as one of the boundary points in the valid boundary information for the subsequent step S204b.

[0090] In one possible implementation, the first preset condition is that the relative position of the boundary point with respect to the vanishing point is a negative value, indicating that the boundary point is below the vanishing point, or that the value corresponding to the boundary point is smaller than the value corresponding to the vanishing point. If the first preset condition is not met, it means that the boundary point is above the vanishing point, or that the value corresponding to the boundary point is larger than the value corresponding to the vanishing point. In this case, the boundary point is considered to be at infinity, which is obviously unrealistic. In another possible implementation, the preset position may be selected as a position at least 5 pixels below the vanishing point. In a specific embodiment, the preset position may be selected as 10 pixels below the vanishing point.

[0091] The first interpolation process is an interpolation method that uses known, discrete boundary points to infer new boundary points in the area corresponding to the perceived image. In a possible implementation, Figure 4 As shown, after initialization, the first interpolation process uses linear interpolation, that is, using the position information of the two boundary points, calculating the equation of a straight line y=kx+b passing through the two boundary points, finding the y value corresponding to the middle specified x, and using the point (x, y) as a boundary point after interpolation; for example, Figure 4 The boundary point information at the middle edge is extrapolated using the equation of the straight line where the first (or last) two boundary points of the edge are located. Figure 4 The midpoint P3 can be obtained by linear interpolation between points P1 and P2. The straight line between points P1 and P2 is

[0092]

[0093] Here, y3 is expressed as k(x3-x1)+y1, and x3 is known.

[0094] Linear interpolation can reduce the amount of calculation and time consumption, which is conducive to improving the real-time performance of autonomous driving. The optimization object in the subsequent step S105 using obstacle perception information for optimization processing is the driving boundary information including the boundary points after interpolation.

[0095] Specifically, the optimizing the plurality of driving boundary information respectively according to the obstacle perception information includes:

[0096] If at least part of the obstacle perception area corresponding to the obstacle perception information is located within the drivable area corresponding to the driving boundary information, the driving boundary information is updated according to the obstacle perception information to obtain the optimized driving boundary information.

[0097] Among them, the perceived obstacle area represents the area where the obstacle object corresponding to the perceived obstacle information is located, such as Figure 5As shown in the figure, p represents the position of the vanishing point in the perceived image. In the image coordinate system, the driving boundary information is compared with the area where the obstacle object is located pixel by pixel in the x direction. The position between the obstacle perception area and the drivable area corresponding to the driving boundary information includes the following three cases: A) Figure 5 As shown in the obstacle perception area A, part of the obstacle perception area is located in the drivable area corresponding to the driving boundary information; Figure 5 As shown in the obstacle perception area B, all obstacle perception areas are located within the drivable area corresponding to the driving boundary information; C) Figure 5 As shown in the obstacle perception area C, all obstacle perception areas are located outside the drivable area corresponding to the driving boundary information, and there is no boundary between the obstacle perception area C and the drivable area.

[0098] The optimization processing targets the driving boundary information in Case A) and Case B). That is, in these two cases, at least part of the obstacle perception area corresponding to the obstacle perception information is located within the drivable area corresponding to the driving boundary information. Only obstacles that intersect with or are below the boundary points need to be considered, thereby updating the driving boundary information in the obstacle perception areas A and B to obtain the optimized driving boundary information. The driving boundary information in Case C) does not need to be updated and can be directly used as the optimized driving boundary information for subsequent fusion processing to reduce the amount of calculation.

[0099] Specifically, if Figure 6 As shown, updating the driving boundary information according to the obstacle perception information includes:

[0100] S602: Obtain a boundary area to be updated according to the obstacle perception area and the driving boundary information.

[0101] S604: Update the driving boundary information corresponding to the to-be-updated boundary area according to the obstacle perception information.

[0102] like Figure 5 As shown, the boundary area to be updated includes an obstacle perception area and a jump area. In the x-direction of the image coordinate system, after the boundary points corresponding to the obstacle perception area are optimized, boundary point jumps may occur at the edge of the obstacle perception area. The jump area is an area of ​​a preset width in the x-direction located before and / or after the obstacle perception area. The boundary points of the jump area are synchronously optimized, and the driving boundary information corresponding to the jump area is updated. That is, when updating the driving boundary information, not only the boundary points within the x-interval corresponding to the obstacle perception area are updated, but also the boundary points within an x-interval outside the obstacle perception area are updated to avoid boundary point jumps and improve the continuity of the optimized driving boundary information.

[0103] In one possible implementation, the preset width of the jump area is 5 to 15 pixels, which requires little computation and has high optimization efficiency. Furthermore, the preset width of the jump area can be optionally 8 to 12 pixels. In a specific embodiment, the preset width of the jump area is 10 pixels.

[0104] In step S604, the obstacle perception information includes at least one of an obstacle perception area and an obstacle perception boundary. This step may further include:

[0105] updating the driving boundary information corresponding to the to-be-updated boundary area according to the obstacle perception area; or

[0106] The driving boundary information corresponding to the to-be-updated boundary area is updated according to the obstacle perception boundary corresponding to the obstacle perception information.

[0107] Among them, updating according to the obstacle perception area is updating according to the position of the obstacle perception area and the area occupied in the image coordinate system. It can be further understood as updating and optimizing the driving boundary information according to the area of ​​the obstacle perception area; and updating according to the obstacle perception boundary is updating according to the boundary of the perceived object in the obstacle perception information. It only needs to use the boundary information corresponding to the obstacle perception area.

[0108] In the boundary area to be updated, for the interval corresponding to the obstacle perception area, the bottom boundary of the obstacle perception area can be directly optimized using the bottom boundary of the obstacle perception area, and the boundary point in the driving boundary information is updated to a preset optimization position to obtain optimized driving boundary information. The preset optimization position is located below the bottom boundary of the obstacle perception area and is at a distance from the bottom boundary of the obstacle perception area by a preset optimization distance. In one possible implementation, the preset optimization distance is greater than or equal to zero pixels. In a specific embodiment, the preset optimization distance is equal to zero, and then after optimization, the position of the boundary point is updated to the position corresponding to the bottom boundary of the obstacle perception area, that is, the preset optimization position is at the bottom boundary of the interval corresponding to the obstacle perception area. In another specific embodiment, the preset optimization distance is greater than zero, and then the position of the boundary point is updated to a position at a preset distance below the bottom boundary of the obstacle perception area. Optionally, the preset optimization distance is 1 to 10 pixels, and then after optimization, the position of the boundary point in the driving boundary information is updated to 1 to 10 pixels below the bottom boundary of the obstacle perception area.

[0109] For the boundary points within the jump area, updating the driving boundary information corresponding to the boundary area to be updated further includes:

[0110] A second interpolation process is performed in the jump region to update the boundary information corresponding to the jump region.

[0111] Similar to the first interpolation process, the second interpolation process can also be selected as linear interpolation to reduce the amount of calculation while ensuring the continuity of the boundary points; in this step, if Figure 5 As shown in (for analysis only, not in actual application scenarios), the jump area has two interval edges in the x direction, one of which is the boundary between the obstacle perception area and the jump area. The second interpolation process can select the pixel point corresponding to the bottom boundary of the current obstacle perception area as the first jump point for interpolation; and at the other interval edge, select the boundary point in the driving boundary information away from the first jump point by a preset width as the second jump point, that is, the second jump point is a boundary point corresponding to the interval edge of the jump area, and interpolation update is performed between the position corresponding to the first jump point and the position corresponding to the second jump point, as shown in FIG. Figure 5 As shown in the jump area on the right side of the obstacle perception area A and the jump areas on both sides of the obstacle perception area B, the information of the optimized boundary points in the jump area is obtained, that is, the optimized driving boundary information.

[0112] Specifically, if Figure 7 As shown, in step S107, the fusion processing of the plurality of optimized driving boundary information includes:

[0113] S701: Perform a first filtering process on the plurality of optimized driving boundary information to obtain a plurality of filtered boundary information.

[0114] Before performing step S107, the boundary points in the driving boundary information have undergone a first interpolation process, and the number of boundary points has greatly increased, that is, the number of boundary points in the optimized driving boundary information has also greatly increased. At this time, the optimized driving boundary information corresponding to each perceived image is subjected to a first filtering process, that is, sampling is performed on a large number of boundary points contained in the optimized driving boundary information to reduce the number of boundary points in the subsequent fusion process, greatly reduce the amount of calculation, and simplify unnecessary time consumption.

[0115] In one possible implementation, based on the optimized driving boundary information, the sampling rate in the optimized area to be updated is greater than the sampling rate in the drivable area that does not require optimization. That is, more sampling is performed in the obstacle perception area and the jump area, while less sampling is performed in the non-optimized area, to obtain the final filtered boundary information; for example, Figure 5 The sampling rates in the obstacle perception area A and its transition area and the obstacle perception area B and its transition area are greater than the sampling rate in the corresponding interval of the obstacle perception area C.

[0116] S703: Perform the fusion process on the plurality of filtered boundary information.

[0117] Specifically, in step S107 (or S703), the fusion process includes:

[0118] S802: Determine, based on the positions of the plurality of vehicle-mounted sensing devices, the preset areas corresponding to two adjacent vehicle-mounted sensing devices in the vehicle body coordinate system.

[0119] When the optimized boundary information corresponding to multiple perception images is fused, they are fused in pairs. The perception images obtained by two adjacent on-board perception devices may overlap, and the corresponding two optimized driving boundary information may also overlap. In this step, the object to be fused is first determined according to the position of the on-board perception device corresponding to the perception image, that is, only two on-board perception devices among the multiple on-board perception devices need to be determined, reducing the corresponding calculation amount. Then, within the preset area where the corresponding two optimized driving boundary information exist, the two optimized driving boundary information are first fused to eliminate the overlapping parts within the preset area and improve the accuracy of the fusion processing.

[0120] Moreover, the processing object of the fusion process includes multiple optimized driving boundary information. In the above steps, they are all performed in the image coordinate system corresponding to each perception image. Before fusion, Figure 9 As shown, based on the positions of multiple vehicle-mounted sensing devices, coordinate transformation processing is performed on the optimized driving boundary information (or filtered boundary information) corresponding to each of the multiple sensing images to obtain the optimized driving boundary information located in the vehicle body coordinate system, so as to perform fusion processing of the optimized driving boundary information in the vehicle body coordinate system.

[0121] In a possible implementation, the range of the preset area may be a preset quadrant in the vehicle coordinate system, and the preset quadrant only includes optimized driving boundary information corresponding to at most two vehicle-mounted sensing devices; Figure 9 For example (for illustration only, not the actual coordinate conversion result), in the vehicle body coordinate system, the optimized driving boundary information in the vehicle body coordinate system is first obtained, wherein the four curves respectively represent the filtered boundary information corresponding to the drivable area after optimized sampling of the four cameras in the front, rear, left and right directions, wherein the first quadrant and the fourth quadrant are the results of the front camera, the first quadrant and the second quadrant are the results of the left camera, the second quadrant and the third quadrant are the results of the rear camera, and the third quadrant and the fourth quadrant are the results of the right camera; in another possible implementation manner, the preset area can also be selected to include only the area where the boundary points of the two optimized driving boundary information intersect, further narrowing the scope of the fusion processing and achieving better targeted fusion effect.

[0122] S804: Fusing the optimized driving boundary information in the plurality of preset areas respectively to obtain a plurality of first fused information.

[0123] S806: Obtain the drivable area information based on the multiple first fusion information.

[0124] Among them, each preset area corresponds to two optimized driving boundary information, which are the first optimization information and the second optimization information respectively. By fusing the first optimization information and the second optimization information, the first fusion information corresponding to the preset area is obtained; then the first fusion information corresponding to multiple preset areas is further fused to obtain the final drivable area information, which has high accuracy and good reliability.

[0125] Specifically, in step S804, the optimized driving boundary information in the plurality of preset areas is fused respectively to obtain a plurality of first fused information including:

[0126] S1002: Obtain distance information between a boundary point in the first optimization information and a boundary point in the second optimization information based on the optimized driving boundary information.

[0127] It should be noted that steps S1002 to S1008 are described based on a single preset area, and steps S1002 to S1008 are performed on the optimized driving boundary information in each preset area to obtain the first fusion information corresponding to each preset area.

[0128] by Figure 11 As an example, the dotted lines in the figure represent the boundary points in the first optimization information and the boundary points in the second optimization information, respectively. According to the first optimization information, the position of each boundary point in the first optimization information can be obtained, and according to the second optimization information, the position of each boundary point in the second optimization information can be obtained. Then, according to the position of each boundary point, the distance between the boundary points in the first optimization information and the boundary points in the second optimization information is calculated pairwise to obtain distance information, which is a collection of a series of distance values. In a possible implementation, the distance values ​​in the distance information are sorted so that the distance values ​​are arranged in ascending order to facilitate the subsequent first fusion processing. In the distance information, the index information corresponding to each distance value can be obtained, that is, the index information of the corresponding two boundary points in the optimized driving boundary information. Through the index information, the distance value corresponding to the index information and the position information of the two boundary points corresponding to the distance value can be obtained.

[0129] S1004: Determine a plurality of candidate boundary points according to the distance information and the boundary points corresponding to the distance information.

[0130] According to the corresponding index information in the distance information, a corresponding combination of boundary points can be obtained, and among multiple boundary points, multiple boundary points with distance values ​​less than a first preset distance are selected as candidate boundary points; wherein the distance information includes a minimum distance value, and when the distance information is sorted, the minimum distance value is the first distance value in the distance information; in a possible implementation manner, the first preset distance is greater than the minimum distance value; further, the first preset distance is greater than the minimum distance value, and the difference between the first preset distance and the minimum distance value is in the range of 5 to 15 pixels; in a specific embodiment, the first preset distance is equal to the minimum distance value plus 10 pixels.

[0131] In a possible implementation, determining a plurality of candidate boundary points further includes:

[0132] determining a plurality of preselected boundary points according to the distance information and the boundary points corresponding to the distance information;

[0133] According to the position of the pre-selected boundary point, the boundary point to be selected is obtained from the pre-selected boundary points.

[0134] Among them, among multiple boundary points, a boundary point with a distance value less than a second preset distance is selected as a pre-selected boundary point, the second preset distance is greater than the minimum distance value, and the second preset distance is greater than the first preset distance; further, the difference between the second preset distance and the minimum distance value ranges from 10 to 30 pixels; in a specific embodiment, the first preset distance is equal to the minimum distance value plus 20 pixels.

[0135] Optionally, after obtaining a plurality of preselected boundary points, fitting processing is performed according to the position (coordinate) information corresponding to the preselected boundary points to obtain a fitting straight line, such as Figure 11 The straight line between the middle boundary points is the shortest distance between each pre-selected boundary point and the fitting line, and the visualization is good.

[0136] Then, among the multiple pre-selected boundary points, boundary points with distance values ​​less than the first preset distance are further determined as candidate boundary points. Through two-step screening, the reliability of the candidate boundary points can be improved.

[0137] S1006 , obtaining a fusion point from the plurality of boundary points to be selected according to positions of the plurality of boundary points to be selected; wherein the distance from the fusion point to the coordinate origin is the maximum value of the distances from the boundary points to the coordinate origin.

[0138] S1008: Based on the fusion point, perform a second filtering process on the first optimization information and the second optimization information respectively to obtain the first fusion information.

[0139] That is, among the multiple candidate boundary points obtained, the one farthest from the coordinate origin is selected as the final fusion point; Figure 11 Taking the point in as an example, assuming that the boundary point on the lower left side of the fitting straight line is the first optimization information, and the boundary point on the upper right side of the fitting straight line is the second optimization information, then in this preset area, the second filtering process is: for the first optimization information, the boundary points corresponding to the optimized driving boundary information above the fusion point in the first optimization information are filtered, and only the boundary points below the fusion point are retained to obtain the first filtered optimization information; and for the second optimization information, the boundary points corresponding to the optimized driving boundary information on the left side of the fusion point in the second optimization information are filtered, and only the boundary points on the right side of the fusion point are retained to obtain the second filtered optimization information.

[0140] Finally, the first filtered optimization information is fused with the second filtered optimization information to obtain the first fused information corresponding to the preset area. The first fused information in different preset areas is further fused to obtain complete drivable area information in the entire vehicle coordinate system.

[0141] In a specific embodiment, the autonomous driving method of the present invention refers to performing target detection, segmentation and post-processing operations based on the perception images captured by four cameras, front, back, left and right, to obtain obstacle perception information and drivable area information, thereby achieving the effect of assisting autonomous driving; wherein, the regression boundary information corresponding to each perception image includes 224 boundary points, and the size of the perception image is 1920*1080. The actual results are as follows Figure 3 As shown in Figure (a), the vanishing point is first determined for the 224 returned boundary points. If there is a boundary point above the vanishing point, the value of the corresponding boundary point is modified to 10 pixels below the vanishing point. After that, the first interpolation process is performed. In this way, 1920 boundary points can be obtained for each perceived image. The actual results are as follows: Figure 3 As shown in Figure (b), the continuity of the boundary points in the optimized driving boundary information can be greatly improved; then the information of the boundary points is combined into the driving boundary information for subsequent optimization processing. The actual result after optimization processing is as follows Figure 3 As shown in Figure (c), after the optimization process, there is no boundary between the drivable area and the obstacle perception area corresponding to the optimized driving boundary information. The obstacle perception area is located outside the drivable area, which can effectively avoid obstacles and collisions, further improving the reliability of the final drivable area information and the safety of autonomous driving. The optimized boundary information is then filtered to obtain the following: Figure 3 The filtered boundary information shown in Figure (d) reduces the number of boundary points in the subsequent cross-camera fusion processing, reducing the amount of calculation and unnecessary time consumption.

[0142] During the fusion process, the filtered boundary information is first converted from the image coordinate system to the vehicle coordinate system through the camera's internal and external parameter coefficients, and the optimized driving boundary information corresponding to multiple perception images is fused in the vehicle coordinate system, such as Figure 12 As shown, Figure 12 Figure (a) shows the optimized driving boundary information corresponding to the four cameras in the vehicle coordinate system. Figure 12 The middle (b) figure shows the drivable area information after fusion processing. The information fusion effect of different preset areas is good, there is no overlap, and the accuracy and reliability of the drivable area information are good; in addition, Figure 12 The middle (c) picture shows Figure 12 The result after connecting adjacent boundary points in Figure (b) can obtain a more complete drivable area and better visualization effect.

[0143] It can be seen from the above embodiments that the autonomous driving method in the embodiments of the present invention has the following beneficial effects:

[0144] 1. The present invention optimizes the driving boundary information corresponding to each perception image through obstacle perception information to improve the accuracy and reliability of the driving boundary information corresponding to each perception image; and through fusion processing, multiple optimized driving boundary information are merged to obtain a longer range of drivable areas, and overlap between multiple perception images can be avoided, thereby avoiding the situation where multiple optimized driving boundary information in the same area are inconsistent, further improving the accuracy and reliability of the drivable area.

[0145] 2. The present invention regresses each perceived image through regression processing. Compared with IPM map segmentation, it can improve the accuracy of driving boundary information, greatly reduce time consumption, and improve real-time performance.

[0146] 3. The present invention adopts interpolation and sampling methods to perform a first interpolation process on the regression boundary information after regression processing, and a first filtering process on the optimized driving boundary information, which can greatly improve the continuity and accuracy of the information and greatly reduce the amount of calculation, which is conducive to improving the real-time performance of autonomous driving.

[0147] 4. In the fusion process, a second filtering process is performed based on the fusion points to obtain the first fusion information within the preset area, which can eliminate the overlap between the corresponding two perception images, thereby improving the consistency of the drivable area information within the preset area, with good fusion effect and high accuracy of the drivable area.

[0148] Corresponding to the autonomous driving method provided in the above-mentioned embodiment, an embodiment of the present invention also provides an autonomous driving device. Since the autonomous driving device provided in the embodiment of the present invention corresponds to the autonomous driving methods provided in the above-mentioned implementation methods, the implementation methods of the aforementioned autonomous driving methods are also applicable to the autonomous driving device provided in this embodiment and will not be described in detail in this embodiment.

[0149] The automatic driving device provided by the embodiment of the present invention can implement the automatic driving method in the above method embodiment, as shown in the attached manual. Figure 13 As shown, the device may include:

[0150] An acquisition module 1310 is configured to acquire obstacle perception information and a plurality of perception images; the perception images are acquired by an onboard perception device corresponding to the perception images;

[0151] An image processing module 1320 is configured to obtain driving boundary information corresponding to the plurality of perception images based on the plurality of perception images;

[0152] An optimization module 1330 is configured to optimize the plurality of driving boundary information respectively according to the obstacle perception information to obtain a plurality of optimized driving boundary information;

[0153] The fusion module 1340 is configured to fuse the plurality of optimized driving boundary information to obtain driving area information; the driving area information represents the driving area of ​​the vehicle.

[0154] In one possible embodiment, the device may further include:

[0155] A regression module, configured to perform regression processing on the plurality of perception images to obtain a plurality of regression boundary information;

[0156] The first interpolation module is used to perform a first interpolation process on the plurality of regression boundary information to obtain a plurality of driving boundary information.

[0157] In another possible embodiment, the device may further include:

[0158] The first filtering module is used to perform a first filtering process on the plurality of optimized driving boundary information to obtain a plurality of filtered boundary information.

[0159] It should be noted that the apparatus provided in the above embodiments, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0160] An embodiment of the present invention also provides an electronic device, including a processor and a memory, wherein the processor (or CPU (Central Processing Unit)) is the core component of the autonomous driving device, and its main function is to interpret memory instructions and process data fed back by each module; the structure of the processor is roughly divided into an arithmetic logic component and a register component, etc. The arithmetic logic component mainly performs related logical calculations (such as shift operations, logical operations, fixed-point or floating-point arithmetic operations and address operations, etc.), and the register component is used to temporarily store instructions, data and addresses.

[0161] The memory is a storage device that can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory mainly includes a program storage area and a data storage area. The program storage area can store an operating system, which may include but is not limited to: Windows system (an operating system), Linux (an operating system), etc., and the present invention is not limited to this. In addition, it can also store application programs required for functions. For example, the storage space of the memory also stores at least one instruction suitable for being loaded and executed by the processor. These instructions can be one or more computer programs (including program codes); and the data storage area can store data created according to the use of the device. Accordingly, the memory can also include a memory controller to provide the processor with access to the memory.

[0162] An embodiment of the present invention also provides a storage medium, in which at least one instruction or at least one program is stored, and the at least one instruction or the at least one program is loaded and executed by a processor to implement the above-mentioned automatic driving method; optionally, the storage medium can be located in at least one network server among multiple network servers of a computer network; in addition, the storage medium can include but is not limited to random access memory (RAM), read-only memory (ROM), non-volatile memory (NVM), U disk, mobile hard disk, disk storage device, flash memory device, other volatile solid-state storage devices and other storage media that can store program code.

[0163] It should be noted that the order of the embodiments of the present invention described above is for illustrative purposes only and does not represent the superiority or inferiority of the embodiments. The above description is of specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0164] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences from other embodiments. In particular, the device embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0165] What is described above are only some embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art should understand that the present invention may be subject to various changes and improvements, and any modifications, equivalent substitutions and improvements made in accordance with the present invention shall fall within the scope of protection required by the present invention.

Claims

1. An automatic driving method, characterized in that: include: Obtain obstacle perception information and multiple perception images; The perceived image is acquired by a vehicle-mounted perception device corresponding to the perceived image, and the obstacle perception information is obtained by analyzing information of obstacle objects present in the perceived image; Performing regression processing on the plurality of perception images to obtain a plurality of driving boundary information; or performing regression processing on the plurality of perception images to obtain a plurality of regression boundary information; performing first interpolation processing on the plurality of regression boundary information to obtain a plurality of driving boundary information; If at least part of the obstacle perception area corresponding to the obstacle perception information is located within the drivable area corresponding to the driving boundary information, obtaining a boundary area to be updated based on the obstacle perception area and the driving boundary information, wherein the boundary area to be updated includes a jump area; updating the driving boundary information corresponding to the to-be-updated boundary area according to the obstacle perception information to obtain a plurality of optimized driving boundary information; performing a first filtering process on the plurality of optimized driving boundary information to obtain a plurality of filtered boundary information; Fusing the plurality of filtered boundary information to obtain drivable area information; The drivable area information represents the drivable area of ​​the vehicle; The updating of the driving boundary information corresponding to the boundary area to be updated includes: A second interpolation process is performed in the jump region to update the boundary information corresponding to the jump region.

2. The automatic driving method according to claim 1, wherein: The boundary area to be updated also includes the obstacle perception area.

3. The automatic driving method according to claim 1, wherein: The fusion process includes: Determining, based on the positions of the plurality of vehicle-mounted sensing devices, preset areas corresponding to two adjacent vehicle-mounted sensing devices in a vehicle coordinate system; fusing the optimized driving boundary information in the plurality of preset areas respectively to obtain a plurality of first fused information; The drivable area information is obtained according to the plurality of first fusion information.

4. An automatic driving device, characterized in that: At least: An acquisition module, used to acquire obstacle perception information and multiple perception images; The perceived image is acquired by a vehicle-mounted perception device corresponding to the perceived image, and the obstacle perception information is obtained by analyzing information of obstacle objects present in the perceived image; An image processing module, configured to perform regression processing on the plurality of perception images to obtain a plurality of driving boundary information; or perform regression processing on the plurality of perception images to obtain a plurality of regression boundary information; and perform first interpolation processing on the plurality of regression boundary information to obtain a plurality of driving boundary information; a boundary region to be updated determining module, configured to, if at least a portion of the obstacle perception region corresponding to the obstacle perception information is within the drivable region corresponding to the driving boundary information, determine a boundary region to be updated based on the obstacle perception region and the driving boundary information, wherein the boundary region to be updated includes a transition region; an updating module, configured to update the driving boundary information corresponding to the to-be-updated boundary area according to the obstacle perception information, to obtain a plurality of optimized driving boundary information; a filtering module, configured to perform a first filtering process on the plurality of optimized driving boundary information to obtain a plurality of filtered boundary information; A fusion module, configured to fuse the plurality of filtered boundary information to obtain drivable area information; The drivable area information represents the drivable area of ​​the vehicle; The updating module includes: performing a second interpolation process in the jump area to update the boundary information corresponding to the jump area.

5. A storage medium, characterized in that The storage medium stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the automatic driving method as described in any one of claims 1 to 3.

Images