Vehicle drivable area detection method, device, equipment and storage medium

By projecting three-dimensional ray sampling points onto the image and combining with the convolutional neural network, the vehicle can be directly predicted, which solves the problem of insufficient detection accuracy in the prior art, and significantly improves the detection accuracy of suspended objects and distant objects.

CN115063760BActive Publication Date: 2024-11-12GUANGZHOU WERIDE TECH LTD CO
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210554342.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-20
Publication Date
2024-11-12
Estimated Expiration
2042-05-20

AI Technical Summary

Technical Problem

The existing vehicle driving area detection methods have insufficient detection accuracy for hanging objects and far-distance objects, and cannot effectively deal with hanging objects and long-distance objects.

Method used

By obtaining the image to be detected in the driving scene and the three-dimensional ray sampling points emitted by the target vehicle, using preset camera parameters to project the sampling points to the image, extracting ray characteristics and performing boundary detection, and directly predicting the travelable area with convolutional neural network. Improve detection accuracy.

Benefits of technology

It improves the accuracy of driving area detection of suspended objects and distant objects, realizes direct end-to-end prediction, and improves the accuracy of object detection in autonomous driving scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115063760B_ABST
    Figure CN115063760B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of autonomous driving technology, and discloses a vehicle drivable area detection method, device, equipment and storage medium. The method comprises: obtaining an image to be detected corresponding to a driving scene, and collecting sampling points corresponding to each three-dimensional ray emitted by a target vehicle in the driving scene; using preset camera parameters, projecting the sampling points to the image to be detected, and obtaining the ray features corresponding to each three-dimensional ray; based on the ray features, performing boundary detection on each three-dimensional ray to obtain a detection result, and determining the drivable area of ​​the target vehicle in the driving scene based on the detection result. The present invention improves the detection accuracy of suspended objects and objects at a long distance in an autonomous driving scene.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a vehicle drivable area detection method, device, equipment and storage medium. Background Art

[0002] Detecting 3D drivable areas from images is an important part of vision-based unmanned driving solutions. The so-called 3D drivable area refers to the distance in 3D space from any direction before encountering obstacles.

[0003] The current detection schemes include two schemes: "image segmentation + projection" and "direct segmentation under overhead view". For image road segmentation + projection, the image is first segmented at the pixel level to obtain the separation of ground and non-ground. Then the edge of the ground is directly projected into the 3D space to form the edge of the road in the 3D space. This method requires a large amount of calculation, requires segmentation of all pixels in the entire image, and cannot handle suspended objects (such as ETC crossbars, etc.). The drivable area detected at the suspended object will be farther than the actual drivable area. For direct road segmentation under overhead view, the image information is first projected into 3D space, and then the road / non-road segmentation is performed in 3D space (overhead view). This method is limited by the depth estimation error and the limitations of the method itself. Its performance deteriorates rapidly at a long distance (such as >50m) and cannot provide effective detection. In short, the detection method of the vehicle's drivable area has the problem of insufficient detection accuracy for suspended objects and objects at a distance. Summary of the invention

[0004] The main purpose of the present invention is to solve the technical problem that the detection method of the vehicle drivable area has insufficient detection accuracy for suspended objects and objects at a relatively far distance.

[0005] A first aspect of the present invention provides a method for detecting a vehicle's drivable area, comprising: acquiring an image to be detected corresponding to a driving scene, and collecting sampling points corresponding to each three-dimensional ray emitted by a target vehicle in the driving scene; using preset camera parameters, projecting the sampling points onto the image to be detected to obtain ray features corresponding to each three-dimensional ray; based on the ray features, performing boundary detection on each three-dimensional ray to obtain a detection result, and determining the drivable area of ​​the target vehicle in the driving scene based on the detection result.

[0006] Optionally, in a first implementation method of the first aspect of the present invention, the sampling points corresponding to each three-dimensional ray emitted by the target vehicle in the driving scene are collected, including: based on a preset resolution, determining a corresponding number of three-dimensional rays emitted by the target vehicle in the driving scene, and determining the three-dimensional coordinates of each sampling point on each three-dimensional ray; according to the three-dimensional coordinates, selecting a preset number of sampling points from the sampling points on the three-dimensional ray from near to far.

[0007] Optionally, in a second implementation manner of the first aspect of the present invention, the sampling points are projected onto the image to be detected using preset camera parameters to obtain ray features corresponding to each three-dimensional ray, including: extracting multi-scale features of the image to be detected, and extracting a position tensor of a preset resolution of the sampling points; the sampling points are projected onto the image to be detected using preset camera parameters, and based on a result of the projection, the multi-scale features and the position tensor are fused to obtain ray features corresponding to each three-dimensional ray.

[0008] Optionally, in a third implementation method of the first aspect of the present invention, the multi-scale features and the position tensor are fused according to the projection result to obtain the ray features corresponding to each three-dimensional ray, including: determining the projection position information of each sampling point in the image to be detected according to the projection result; and fusing the multi-scale features and the position tensor whose projection positions overlap according to the projection position information to obtain the ray features corresponding to each three-dimensional ray.

[0009] Optionally, in a fourth implementation manner of the first aspect of the present invention, the multi-scale features and position tensors that overlap the projection positions are fused according to the projection position information to obtain ray features corresponding to each three-dimensional ray, including: according to the projection position information, the multi-scale features are combined to corresponding sampling points to obtain content features of each sampling point; the content features of each sampling point are fused with the position tensor to obtain a final feature tensor of each sampling point; and based on the feature tensors of each sampling point in each three-dimensional ray, a ray feature corresponding to each three-dimensional ray is generated.

[0010] Optionally, in a fifth implementation manner of the first aspect of the present invention, the boundary detection is performed on each three-dimensional ray based on the ray characteristics, and the detection results obtained include: based on the ray characteristics, detecting the drivable boundary points corresponding to each three-dimensional ray, and predicting the drivable distance and boundary parameters of each three-dimensional ray based on the drivable boundary points; based on the drivable distance and the boundary parameters, obtaining the detection results corresponding to each three-dimensional ray.

[0011] Optionally, in the sixth implementation method of the first aspect of the present invention, after detecting the drivable boundary points corresponding to each three-dimensional ray based on the ray characteristics, it also includes: obtaining the marked boundary points corresponding to each three-dimensional ray; calculating the relative distance between the marked boundary points and the drivable boundary points corresponding to each three-dimensional ray; and adjusting the drivable boundary points corresponding to each three-dimensional ray according to the relative distance to obtain new drivable boundary points.

[0012] The second aspect of the present invention provides a vehicle drivable area detection device, including: an acquisition module, used to obtain an image to be detected corresponding to a driving scene, and to collect sampling points corresponding to each three-dimensional ray emitted by a target vehicle in the driving scene; a projection module, used to project the sampling points onto the image to be detected using preset camera parameters, and obtain ray features corresponding to each three-dimensional ray; a detection module, used to perform boundary detection on each three-dimensional ray based on the ray features, obtain detection results, and determine the drivable area of ​​the target vehicle in the driving scene based on the detection results.

[0013] Optionally, in a first implementation method of the second aspect of the present invention, the acquisition module includes: a determination unit, used to determine, based on a preset resolution, a corresponding number of three-dimensional rays emitted by a target vehicle in the driving scene, and determine the three-dimensional coordinates of each sampling point on each three-dimensional ray; a selection unit, used to select a preset number of sampling points from the sampling points on the three-dimensional ray according to the three-dimensional coordinates and from near to far for each three-dimensional ray.

[0014] Optionally, in a second implementation manner of the second aspect of the present invention, the projection module includes: an extraction unit, used to extract multi-scale features of the image to be detected, and to extract a position tensor of a preset resolution of the sampling point; a projection unit, used to project the sampling point onto the image to be detected using preset camera parameters, and based on the projection result, fuse the multi-scale features and the position tensor to obtain a ray feature corresponding to each three-dimensional ray.

[0015] Optionally, in a third implementation method of the second aspect of the present invention, the projection unit is also used to: determine the projection position information of each sampling point in the image to be detected based on the projection result; and fuse the multi-scale features and position tensors that overlap the projection positions based on the projection position information to obtain the ray features corresponding to each three-dimensional ray.

[0016] Optionally, in a fourth implementation manner of the second aspect of the present invention, the projection unit is further used to: combine the multi-scale features to corresponding sampling points according to the projection position information to obtain content features of each sampling point; fuse the content features of each sampling point with the position tensor to obtain a final feature tensor of each sampling point; and generate ray features corresponding to each three-dimensional ray based on the feature tensors of each sampling point in each three-dimensional ray.

[0017] Optionally, in a fifth implementation of the second aspect of the present invention, the detection module includes: a prediction unit, used to detect the drivable boundary points corresponding to each three-dimensional ray based on the ray characteristics, and predict the drivable distance and boundary parameters of each three-dimensional ray based on the drivable boundary points; a generation unit, used to obtain the detection results corresponding to each three-dimensional ray based on the drivable distance and the boundary parameters.

[0018] Optionally, in a sixth implementation method of the second aspect of the present invention, the detection module also includes an adjustment unit, which is used to: obtain the marked boundary points corresponding to each of the three-dimensional rays; calculate the relative distance between the marked boundary points corresponding to each of the three-dimensional rays and the drivable boundary points; and adjust the drivable boundary points corresponding to each of the three-dimensional rays according to the relative distance to obtain new drivable boundary points.

[0019] A third aspect of the present invention provides a vehicle drivable area detection device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the vehicle drivable area detection device executes the above-mentioned vehicle drivable area detection method.

[0020] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the above-mentioned vehicle drivable area detection method.

[0021] In the technical solution provided by the present invention, a new neural network architecture is designed, image features and 3D ray projection are combined, and the sampling points of the 3D space are directly projected onto the image using the internal and external parameter information of the camera to obtain the overall features of the 3D ray. Based on the overall features, the distance of a drivable area is regressed for the ray in each direction. Direct end-to-end prediction of the drivable area is achieved, and the accuracy of the prediction of the drivable area of ​​suspended objects and distant objects is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A schematic diagram of an embodiment of a method for detecting a vehicle drivable area in an embodiment of the present invention;

[0023] Figure 2 A schematic diagram of another embodiment of a method for detecting a vehicle drivable area according to an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of an embodiment of a vehicle drivable area detection device in an embodiment of the present invention;

[0025] Figure 4 A schematic diagram of another embodiment of a vehicle drivable area detection device according to an embodiment of the present invention;

[0026] Figure 5 The figure is a schematic diagram of an embodiment of a vehicle drivable area detection device in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The embodiment of the present invention provides a vehicle drivable area detection method, device, equipment and storage medium, which obtains the image to be detected corresponding to the driving scene, and collects the sampling points corresponding to each three-dimensional ray emitted by the target vehicle in the driving scene; uses the preset camera parameters to project the sampling points to the image to be detected, and obtains the ray features corresponding to each three-dimensional ray; based on the ray features, performs boundary detection on each three-dimensional ray to obtain the detection result, and determines the drivable area of ​​the target vehicle in the driving scene according to the detection result. The present invention improves the detection accuracy of suspended objects and objects at a long distance in the automatic driving scene.

[0028] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 The first embodiment of the vehicle drivable area detection method in the embodiment of the present invention includes:

[0030] 101. Acquire an image to be detected corresponding to the driving scene, and collect sampling points corresponding to each three-dimensional ray emitted by the target vehicle in the driving scene;

[0031] It is understandable that the execution subject of the present invention may be a vehicle drivable area detection device, or a terminal or a server, which is not limited here. The embodiment of the present invention is described by taking a server as the execution subject as an example.

[0032] In this embodiment, during the driving process of the vehicle, the current driving scene is photographed by a camera installed on the vehicle to obtain an image to be detected; or an image to be detected corresponding to a driving scene photographed by the vehicle in the past; or an image to be detected generated by simulating a driving scene. The image to be detected can be a multi-frame image taken by different cameras and different directions of the driving scene, or a single-frame image.

[0033] In this embodiment, the vehicle is used as the origin, and multiple three-dimensional rays are emitted outward to detect physical objects in the driving scene. According to the detection results, sampling points in the driving scene can be determined, and the detected sampling points are collected. The sampling points here are the feedback points corresponding to the physical objects in the driving scene scanned by the laser radar rays, so that the physical three-dimensional spatial distribution of the collected driving scene can be restored.

[0034] Specifically, the target vehicle here uses a multi-line laser radar to emit three-dimensional rays to collect the overall spatial contour of the entire driving scene. A multi-line laser radar with 4, 8, 16, 32, 64, or 128 lines can be used, that is, a plurality of transmitters and receivers are provided in the vertical direction, and multiple rays are obtained through the rotation of the motor. The more rays there are, the more perfect the surface contour of the object is.

[0035] 102. Projecting the sampling points onto the image to be detected using preset camera parameters to obtain ray features corresponding to each three-dimensional ray;

[0036] In this embodiment, in order to increase the information depth of the relevant features of the drivable area, the laser radar and the camera are combined according to the camera parameters, and the spatial sampling points collected by three-dimensional rays and the two-dimensional information of the image to be seen are fused, so that the obtained ray features contain richer information for subsequent detection of distant objects and suspended objects to predict the drivable area of ​​the vehicle.

[0037] In this embodiment, according to the camera parameters of the image to be detected, including internal and external parameters, the sampling points are converted from three-dimensional space coordinates to two-dimensional plane coordinates relative to the image to be detected, and a projection relationship between the two is constructed. The sampling points are projected into the image to be detected according to the projection relationship. Then, the image information of the sampling points and the projection positions are fused and combined in the same three-dimensional ray unit to obtain the ray features corresponding to each three-dimensional ray.

[0038] It should be noted that a single three-dimensional ray is used as the minimum unit for subsequent detection. After the sampling points and the projection of the image to be detected are completed, the three-dimensional rays to which each sampling point belongs need to be combined to obtain the ray characteristics of each three-dimensional ray.

[0039] In addition, after projecting the sampling points onto the image to be detected, the corresponding two-dimensional information can be combined to obtain the ray features of each three-dimensional ray; the sampling points can also be projected onto the image to be detected, and the image plane information can be obtained by fusion, and then converted into three-dimensional information. That is, the ray features here can be two-dimensional information or three-dimensional information.

[0040] 103. Based on the ray features, perform boundary detection on each of the three-dimensional rays to obtain a detection result, and determine a drivable area of ​​the target vehicle in the driving scene according to the detection result.

[0041] In this embodiment, a convolutional neural network is used to perform convolution processing and dimensionality reduction on the ray features, gradually extracting and reducing the dimensionality of the ray features related to the drivable area, so as to determine the boundary of the three-dimensional ray and predict the drivable area for each three-dimensional ray.

[0042] Specifically, the ray features of each 3D ray are sequentially input into a convolutional neural network to detect the boundary of each 3D ray, and the detection results obtained include the boundary point position, boundary point category, and boundary point speed, etc. The detection results obtained for each 3D ray are combined to draw the boundary contour, and the drivable area can be obtained.

[0043] Specifically, if the output here is a two-dimensional ray feature, the boundary of each three-dimensional ray on the image plane is finally detected to predict the drivable area of ​​the driving scene on the image plane; the drivable area on the image plane can also be further converted into a drivable area in three-dimensional space as the final drivable area.

[0044] Specifically, if a three-dimensional ray feature is input here, the boundary of each three-dimensional ray in the three-dimensional space is detected, and the drivable area of ​​the driving scene in the three-dimensional space is subsequently predicted; the drivable area in the three-dimensional space can be further converted into a drivable area in the two-dimensional space as the final drivable area.

[0045] In the embodiment of the present invention, a new neural network architecture is designed, image features and 3D ray projection are combined, and the sampling points of the 3D space are directly projected onto the image using the internal and external parameter information of the camera to obtain the overall features of the 3D ray. Based on the overall features, the distance of a drivable area is regressed for the ray in each direction. Direct end-to-end prediction of the drivable area is achieved, and the accuracy of the prediction of the drivable area of ​​suspended objects and distant objects is improved.

[0046] See also Figure 2 The second embodiment of the vehicle drivable area detection method in the embodiment of the present invention includes:

[0047] 201. Acquire an image to be detected corresponding to a driving scene, and determine a corresponding number of three-dimensional rays emitted by a target vehicle in the driving scene based on a preset resolution, and determine the three-dimensional coordinates of each sampling point on each three-dimensional ray;

[0048] 202. According to the three-dimensional coordinates, a preset number of sampling points are selected from the sampling points on each three-dimensional ray from near to far;

[0049] In this embodiment, the resolution of the three-dimensional rays scanned by the laser radar is set to control the number of three-dimensional rays emitted, thereby controlling the scanning efficiency and scanning detail of the three-dimensional rays. The lower the resolution, the fewer the number of three-dimensional rays, the higher the scanning efficiency, and the lower the scanning detail, and vice versa. Finally, the corresponding sampling points with three-dimensional coordinates are collected according to the physical objects existing in the area scanned by the three-dimensional rays.

[0050] Specifically, the resolution set here includes distance, azimuth, speed and angle. The number of emitted three-dimensional rays can be controlled by setting the angular resolution and azimuth resolution. For example, if the azimuth resolution is set to 360° and the angular resolution is set to 0.2°, a total of 1800 rays can be emitted (360° / 0.2°=1800).

[0051] In this embodiment, the distance range of 3D ray collection is controlled by distance resolution. For each 3D ray, the farther the sampling point is from the vehicle, the lower the predicted availability of the sampling point for the current drivable area, and vice versa. In order to reduce the amount of calculation, only available sampling points with high availability are taken. The number K can be pre-set, and K sampling points are selected from near to far as sampling points for subsequent detection.

[0052] 203. Extracting multi-scale features of the image to be detected, and extracting a position tensor of a preset resolution of the sampling point;

[0053] 204. Project the sampling point onto the image to be detected using preset camera parameters, and fuse the multi-scale feature and the position tensor according to the projection result to obtain a ray feature corresponding to each three-dimensional ray;

[0054] In this embodiment, when the sampling points are projected onto the image to be detected, the essence is to convert the three-dimensional information on the sampling points into two-dimensional information and fuse it with the image information in the image to be detected, so as to enrich the information richness of each sampling point, so that the subsequent prediction of the drivable area for each three-dimensional ray is more accurate. Therefore, the multi-scale features of the image to be detected are first extracted here, and the position tensor of the sampling points is extracted.

[0055] Specifically, a convolutional neural network, such as a residual neural network and a feature pyramid network, is used to extract multi-scale image features, including multi-scale features such as size scale, plane scale, and semantic scale. Then, the position tensor of the sampling point is extracted, and position tensors of different sizes are extracted according to different resolution settings. For example, the azimuth resolution is set to 360°, the angle resolution is set to 0.2°, and the preset number is K. The position tensor extracted here has a size of N x1800x K x3, where N is the batch size and 3 represents the x, y, and z three-dimensional coordinates of each sampling point.

[0056] Specifically, the external parameter matrix and the internal parameter matrix of the camera parameters can be directly cascaded to directly realize the projection of the sampling points on the image to be detected. In the projection process, the specific coordinate system conversion process includes: lidar world coordinate system-camera coordinate system-image plane coordinate system-pixel coordinate system, and the corresponding coordinate system is converted to: Euclidean coordinate system -> homogeneous coordinate system -> Euclidean coordinate system.

[0057] In addition, the fusion of the multi-scale features and the position tensor may specifically include the following steps:

[0058] 1) According to the projection result, determine the projection position information of each sampling point in the image to be detected;

[0059] 2) According to the projection position information, the multi-scale features and position tensors that overlap the projection positions are fused to obtain the ray features corresponding to each three-dimensional ray.

[0060] In this embodiment, it is necessary to first determine the projection of the sampling point in the image to be detected, and then fuse the corresponding multi-scale features and position tensors. The conversion of the sampling point to the two-dimensional pixels on the image to be detected can be achieved through the intrinsic parameters of the camera, and then the overall external coordinate system of the driving scene and the external coordinate system of the camera are converted into coordinate systems through the extrinsic parameters of the camera. The two-dimensional pixels after the conversion of the sampling point are projected onto the coordinate system where the camera is located, and the projection of the sampling point in the image to be detected is realized. Finally, the corresponding multi-scale features and position tensors can be directly fused according to the pixel points of the projection where each sampling point is located.

[0061] Furthermore, when fusing the multi-scale features and the position tensor, the multi-scale features can be combined to corresponding sampling points according to the projection position information to obtain the content features of each sampling point; the content features of each sampling point are fused with the position tensor to obtain the final feature tensor of each sampling point; based on the feature tensors of each sampling point in each three-dimensional ray, the ray features corresponding to each three-dimensional ray are generated.

[0062] In this embodiment, after the sampling point is projected onto the image to be detected, the row and column coordinates of the sampling point on the image plane are obtained. The size is N x1800x K x3, where 3 represents u, v, and flag, where u represents the column coordinate, v represents the row coordinate, and flag represents whether it is within the range of the image to be detected.

[0063] Then, we can use the gathering operator, such as gather_nd, to collect the multi-scale features at each sampling point and combine them to finally obtain the content features of N x1800x K x C dimensions. At the same time, we superimpose the position tensor of each sampling point on the content features to finally obtain the feature tensors of each sampling point of N x1800x K x C dimensions, which is the ray feature of each three-dimensional ray.

[0064] Specifically, for example, when using gather_nd to collect multi-scale features for each sampling point, params can be used to determine the tensor to be searched, indices determines the search, which can be obtained through tf / where(), axis determines the search axis, name determines the operation name, and batch_dims determines the batch dimension that controls indices.

[0065] 205. Detecting a drivable boundary point corresponding to each three-dimensional ray based on the ray feature, and predicting a drivable distance and boundary parameters of each three-dimensional ray according to the drivable boundary point;

[0066] 206. Based on the drivable distance and the boundary parameter, obtain a detection result corresponding to each three-dimensional ray, and determine a drivable area of ​​the target vehicle in the driving scene according to the detection result.

[0067] In this embodiment, the convolutional neural network such as the residual neural network and the feature pyramid network mentioned above is used here to detect the drivable boundary points of each three-dimensional ray, and the ray features are gradually extracted and reduced in dimension, and finally an N x 1800x1x1 dimensional tensor can be obtained to detect the drivable boundary points of each three-dimensional ray.

[0068] In this embodiment, based on the coordinate information of the plane or space where the drivable boundary point is located, the target vehicle is taken as the origin to predict the formal distance from the target vehicle to the boundary point of the drivable area, and based on the Nx1800x1x1 dimensional tensor corresponding to the drivable boundary point, the boundary parameters of each three-dimensional ray, such as the boundary category (static object or dynamic object) and the boundary speed (can be stationary, i.e. 0), are predicted to obtain the detection result. Finally, the boundary contour of the drivable area is reversely drawn for the predicted drivable distance and boundary parameters, and the drivable area of ​​the target vehicle in the driving scene is determined based on the position information of the boundary contour.

[0069] In addition, if the image to be detected is obtained from a historical driving scene of a vehicle and is used for training the convolutional neural network mentioned above, the convolutional neural network is iteratively updated here, as shown below:

[0070] 1) Obtaining the marked boundary points corresponding to each of the three-dimensional rays;

[0071] 2) Calculating the relative distance between the marked boundary point and the drivable boundary point corresponding to each three-dimensional ray;

[0072] 3) According to the relative distance, the drivable boundary point corresponding to each three-dimensional ray is adjusted to obtain a new drivable boundary point.

[0073] In this embodiment, since each 3D ray will detect a drivable boundary point in each direction, we can regard all drivable boundary points as a predicted point set. At the same time, the marked boundary points are also regarded as a marked point set. The two point sets are compared and the relative distance is calculated to adjust the drivable boundary points.

[0074] Specifically, the relative distance between two point sets is used as the loss function. For any point P in the prediction point set, i , take the minimum L between it and all the true value points in the annotation point set i The distance is used as the loss, and the total loss between the predicted point set and the labeled point set is finally obtained. At the same time, the loss threshold is set to determine whether to update the convolutional neural network to adjust the drivable boundary points. Here, the relative distance between point sets is used as the loss function, which allows the network to tolerate the impact of internal and external parameter noise.

[0075] The above describes the vehicle drivable area detection method in the embodiment of the present invention. The following describes the vehicle drivable area detection device in the embodiment of the present invention. Figure 3 In one embodiment of the present invention, a vehicle drivable area detection device includes:

[0076] The acquisition module 301 is used to acquire the image to be detected corresponding to the driving scene, and to acquire the sampling points corresponding to each three-dimensional ray emitted by the target vehicle in the driving scene;

[0077] A projection module 302 is used to project the sampling points onto the image to be detected using preset camera parameters to obtain ray features corresponding to each three-dimensional ray;

[0078] The detection module 303 is used to perform boundary detection on each of the three-dimensional rays based on the ray characteristics to obtain a detection result, and determine the drivable area of ​​the target vehicle in the driving scene according to the detection result.

[0079] In the embodiment of the present invention, a new neural network architecture is designed, image features and 3D ray projection are combined, and the sampling points of the 3D space are directly projected onto the image using the internal and external parameter information of the camera to obtain the overall features of the 3D ray. Based on the overall features, the distance of a drivable area is regressed for the ray in each direction. Direct end-to-end prediction of the drivable area is achieved, and the accuracy of the prediction of the drivable area of ​​suspended objects and distant objects is improved.

[0080] See also Figure 4 Another embodiment of the vehicle drivable area detection device in the embodiment of the present invention includes:

[0081] The acquisition module 301 is used to acquire the image to be detected corresponding to the driving scene, and to acquire the sampling points corresponding to each three-dimensional ray emitted by the target vehicle in the driving scene;

[0082] A projection module 302 is used to project the sampling points onto the image to be detected using preset camera parameters to obtain ray features corresponding to each three-dimensional ray;

[0083] The detection module 303 is used to perform boundary detection on each of the three-dimensional rays based on the ray characteristics to obtain a detection result, and determine the drivable area of ​​the target vehicle in the driving scene according to the detection result.

[0084] Specifically, the acquisition module 301 includes:

[0085] A determination unit 3011 is used to determine a corresponding number of three-dimensional rays emitted by a target vehicle in a driving scene based on a preset resolution, and to determine the three-dimensional coordinates of each sampling point on each of the three-dimensional rays;

[0086] The selection unit 3012 is used to select a preset number of sampling points from the sampling points on the three-dimensional ray according to the three-dimensional coordinates and from near to far on each three-dimensional ray.

[0087] Specifically, the projection module 302 includes:

[0088] An extraction unit 3021 is used to extract multi-scale features of the image to be detected and to extract a position tensor of a preset resolution of the sampling point;

[0089] The projection unit 3022 is used to project the sampling points onto the image to be detected using preset camera parameters, and fuse the multi-scale features and the position tensor according to the projection result to obtain the ray features corresponding to each three-dimensional ray.

[0090] Specifically, the projection unit 3022 is further used for:

[0091] Determine the projection position information of each sampling point in the image to be detected according to the projection result;

[0092] According to the projection position information, the multi-scale features and the position tensor that overlap the projection positions are fused to obtain the ray features corresponding to each three-dimensional ray.

[0093] Specifically, the projection unit 3022 is further used for:

[0094] According to the projection position information, the multi-scale features are combined to corresponding sampling points to obtain content features of each sampling point;

[0095] Fusing the content feature of each sampling point with the position tensor to obtain a final feature tensor of each sampling point;

[0096] Based on the feature tensor of each sampling point in each three-dimensional ray, a ray feature corresponding to each three-dimensional ray is generated.

[0097] Specifically, the detection module 303 includes:

[0098] The prediction unit 3031 is used to detect the drivable boundary points corresponding to each three-dimensional ray based on the ray characteristics, and predict the drivable distance and boundary parameters of each three-dimensional ray according to the drivable boundary points;

[0099] The generating unit 3032 is used to obtain the detection result corresponding to each three-dimensional ray based on the drivable distance and the boundary parameter.

[0100] Specifically, the detection module 303 further includes an adjustment unit 3033, which is used to:

[0101] Obtaining the marked boundary points corresponding to each of the three-dimensional rays;

[0102] Calculating the relative distance between the marked boundary point and the drivable boundary point corresponding to each three-dimensional ray;

[0103] According to the relative distance, the drivable boundary point corresponding to each three-dimensional ray is adjusted to obtain a new drivable boundary point.

[0104] above Figure 3 and Figure 4 The vehicle drivable area detection device in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The vehicle drivable area detection device in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0105] Figure 5 is a schematic diagram of the structure of a vehicle drivable area detection device provided in an embodiment of the present invention. The vehicle drivable area detection device 500 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 510 (for example, one or more processors) and a memory 520, and one or more storage media 530 (for example, one or more mass storage devices) storing application programs 533 or data 532. Among them, the memory 520 and the storage medium 530 can be short-term storage or permanent storage. The program stored in the storage medium 530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the vehicle drivable area detection device 500. Furthermore, the processor 510 can be configured to communicate with the storage medium 530 to execute a series of instruction operations in the storage medium 530 on the vehicle drivable area detection device 500.

[0106] The vehicle drivable area detection device 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input and output interfaces 560, and / or one or more operating systems 531, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will appreciate that Figure 5 The structure of the vehicle drivable area detection device shown does not constitute a limitation on the vehicle drivable area detection device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0107] The present invention also provides a vehicle drivable area detection device, the computer device includes a memory and a processor, the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the vehicle drivable area detection method in the above-mentioned embodiments.

[0108] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the vehicle drivable area detection method.

[0109] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0110] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0111] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting a vehicle drivable area, characterized in that: The vehicle drivable area detection method comprises: Acquire an image to be detected corresponding to the driving scene, and determine a corresponding number of three-dimensional rays emitted by a target vehicle in the driving scene based on a preset resolution, and determine the three-dimensional coordinates of each sampling point on each three-dimensional ray; according to the three-dimensional coordinates, select a preset number of sampling points from the sampling points on each three-dimensional ray from near to far; Extracting multi-scale features of the image to be detected, and extracting the position tensor of the preset resolution of the sampling point; using preset camera parameters, projecting the sampling point to the image to be detected, and determining the projection position information of each sampling point in the image to be detected according to the projection result; combining the multi-scale features to the corresponding sampling points according to the projection position information to obtain the content features of each sampling point; fusing the content features of each sampling point with the position tensor to obtain the final feature tensor of each sampling point; generating the ray feature corresponding to each three-dimensional ray based on the feature tensor of each sampling point in each three-dimensional ray; Based on the ray features, boundary detection is performed on each of the three-dimensional rays to obtain a detection result, and according to the detection result, a drivable area of ​​the target vehicle in the driving scene is determined.

2. The vehicle drivable area detection method according to claim 1, characterized in that: The performing boundary detection on each three-dimensional ray based on the ray feature to obtain the detection result includes: Based on the ray features, detecting the drivable boundary points corresponding to each of the three-dimensional rays, and predicting the drivable distance and boundary parameters of each of the three-dimensional rays according to the drivable boundary points; Based on the drivable distance and the boundary parameter, a detection result corresponding to each three-dimensional ray is obtained.

3. The vehicle drivable area detection method according to claim 2, characterized in that: After detecting the drivable boundary points corresponding to each three-dimensional ray based on the ray features, the method further includes: Obtaining the marked boundary points corresponding to each of the three-dimensional rays; Calculating the relative distance between the marked boundary point and the drivable boundary point corresponding to each three-dimensional ray; According to the relative distance, the drivable boundary point corresponding to each three-dimensional ray is adjusted to obtain a new drivable boundary point.

4. A vehicle drivable area detection device, characterized in that: The vehicle drivable area detection device comprises: The acquisition module is used to obtain the image to be detected corresponding to the driving scene, and determine the corresponding number of three-dimensional rays emitted by the target vehicle in the driving scene based on the preset resolution, and determine the three-dimensional coordinates of each sampling point on each three-dimensional ray; according to the three-dimensional coordinates, according to each three-dimensional ray from near to far, select a preset number of sampling points from the sampling points on the three-dimensional ray; A projection module is used to extract the multi-scale features of the image to be detected and the position tensor of the preset resolution of the sampling point; project the sampling point to the image to be detected using preset camera parameters, and determine the projection position information of each sampling point in the image to be detected according to the projection result; combine the multi-scale features to the corresponding sampling points according to the projection position information to obtain the content features of each sampling point; fuse the content features of each sampling point with the position tensor to obtain the final feature tensor of each sampling point; generate the ray feature corresponding to each three-dimensional ray based on the feature tensor of each sampling point in each three-dimensional ray; The detection module is used to perform boundary detection on each three-dimensional ray based on the ray characteristics to obtain a detection result, and determine the drivable area of ​​the target vehicle in the driving scene according to the detection result.

5. A vehicle drivable area detection device, characterized in that: The vehicle drivable area detection device comprises: a memory and at least one processor, wherein the memory stores instructions; The at least one processor calls the instructions in the memory to enable the vehicle drivable area detection device to perform the steps of the vehicle drivable area detection method as described in any one of claims 1-3.

6. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the steps of the vehicle drivable area detection method as described in any one of claims 1 to 3 are implemented.