Driving area detection method, system and device, storage medium and vehicle

Abstract

The invention discloses a drivable area detection method, which comprises the following steps of: acquiring all-round fisheye camera images shot by a plurality of vehicle-mounted fisheye cameras, and performing distortion correction processing on each all-round fisheye camera image to obtain a plurality of all-round fisheye camera images after distortion removal; performing feature extraction to obtain multi-scale features of each look-around fisheye camera image; performing feature mapping on the multi-scale features corresponding to the plurality of look-around fisheye camera images by adopting a converter model with a multi-view attention mechanism to obtain BEV map view features; carrying out image semantic segmentation detection and up-sampling to form a mask image with binary classification semantic segmentation; a vehicle drivable area is determined. The invention further provides a corresponding system and device, a storage medium and a vehicle. According to the invention, generalization and robustness of the drivable area detection model can be improved.

Description

Technical Field

[0001] The present invention relates to the field of automatic driving technology, and in particular to a drivable area detection method, system, device, storage medium and vehicle. Background Art

[0002] In automatic parking technology, drivable area detection is very important. The parking process requires the perception of obstacles and drivable areas around the car, which requires high accuracy. In automatic parking, whether it is parking space detection or drivable area detection, 4 fisheye cameras are required to build a 360° real-time visual perception around the car body.

[0003] In the existing autonomous driving or parking system, the original images of four fisheye cameras are used to detect the drivable area. Generally, first, it is assumed that the ground is flat, and then the internal and external parameters of the camera are used to calibrate the parameters, and the four fisheye camera images are projected onto the BEV map view. Then, a geometric model is established for the four BEV map views, and the surrounding images are spliced ​​into a BEV (Bird's Eye View) map.

[0004] In the existing technology, the conventional fisheye camera drivable area detection algorithm needs to assume that the ground of the scene is flat. At the same time, the fisheye camera needs to be explicitly geometrically modeled and surround stitched. However, explicit depth is very challenging to use in downstream tasks, which requires precise calibration and fusion of multiple noise estimates.

[0005] Or in other solutions, a multi-sensor fusion solution is needed to fuse the obstacle distribution information obtained by multiple sensors such as cameras and radars to generate a drivable area around the vehicle.

[0006] In these solutions of the prior art, the methods for detecting the drivable area all have the shortcomings of poor generalization and robustness. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a drivable area detection method, system, device, storage medium and vehicle, which can improve the generalization and robustness of the drivable area detection model.

[0008] In order to solve the above technical problems, as one aspect of the present invention, a method for detecting a drivable area is provided, which at least comprises the following steps:

[0009] Acquire surround-view fisheye camera images taken by multiple vehicle-mounted fisheye cameras, perform distortion correction processing on each surround-view fisheye camera image, and obtain multiple dedistorted surround-view fisheye camera images;

[0010] Performing feature extraction on the plurality of dedistorted surround-view fisheye camera images to obtain multi-scale features of each surround-view fisheye camera image;

[0011] Using a Transformer model with a multi-view attention mechanism to perform feature mapping on multi-scale features corresponding to the multiple surround fisheye camera images to obtain BEV map view features;

[0012] Performing image semantic segmentation detection and upsampling on the BEV map view features to form a binary semantic segmentation mask image;

[0013] The vehicle drivable area is determined according to the binary semantic segmentation mask image.

[0014] The plurality of dedistorted surround fisheye camera images are subjected to feature extraction to obtain multi-scale features of each surround fisheye camera image, specifically:

[0015] The pre-set RegNets network model and BiFPN network model are used to downsample the multiple dedistorted surround fisheye camera images and perform feature extraction to obtain multi-scale features of each surround fisheye camera image; wherein the multi-scale is a different scaling ratio, and the multi-scale features corresponding to each scale at least include image features of each surround fisheye camera image, a rotation matrix, an intrinsic parameter matrix, and a translation vector relative to the center of the vehicle.

[0016] Among them, a converter model with a multi-view attention mechanism is used to perform feature mapping on the multi-scale features corresponding to the multiple surround fisheye camera images to obtain BEV map view features, further comprising:

[0017] Receive multi-scale features of multiple fisheye camera images of at least two scales, where the features of each scale include image features, rotation matrix, intrinsic parameter matrix, and translation vector relative to the center of the vehicle; use the image features of each scale as the value vector of the multi-view attention mechanism;

[0018] Obtain positional encodings of multiple fisheye camera images and combine the image features at each scale as key vectors for the multi-view attention mechanism;

[0019] Get the difference between the position encoding of the BEV map view and the fish camera position encoding as the query vector for the multi-view attention mechanism;

[0020] The similarity at each scale corresponding to each pixel in the BEV map view is obtained, and the similarity at each scale is used as a weight to perform weighted processing with the value vector at the corresponding scale to obtain a view feature representation of each pixel in the BEV map view.

[0021] The obtaining of the similarity at each scale corresponding to each pixel point in the BEV map view is specifically as follows:

[0022] The similarity at each scale corresponding to each pixel in the BEV map view is calculated using the following formula:

[0023]

[0024] Where: k,i Encodes the position of a fisheye camera view; φ k,i Image features of fisheye camera view; c (n) Encode the position on the BEV map view, τ k is the camera position code; k is the camera number, and i is the scale number.

[0025] The step of performing image semantic segmentation detection and upsampling on the BEV map view features to form a binary semantic segmentation mask map further includes:

[0026] Perform bilinear upsampling and convolution processing on the view features of each pixel in the BEV map view to form a BEV map view of a predetermined resolution;

[0027] And perform semantic segmentation to obtain the binary semantic segmentation mask in the BEV map view coordinate system; the loss function of the semantic segmentation used is:

[0028] L loss =w1L cross_entropy +w2L ohem_cross_entropy

[0029] Among them, L cross_entropy is the conventional binary cross entropy loss function, L ohem_cross_entropy It is a binary cross entropy loss function with OHEM added; w1 and w2 are the predetermined weight ratios.

[0030] Wherein, determining the vehicle drivable area according to the binary semantic segmentation mask image includes:

[0031] Output the final BEV map and determine whether the corresponding area is drivable based on the binary semantic segmentation mask value of each pixel;

[0032] Mark the boundaries of the drivable area.

[0033] Among them, further include:

[0034] Using LiDAR to collect information as labels for training data for the drivable area detection model; and

[0035] The RegNets network model, BiFPN network model, and converter model of the multi-view attention mechanism are pre-built and trained.

[0036] Accordingly, another aspect of the present invention further provides a drivable area detection system, which at least comprises:

[0037] A preprocessing unit, used for acquiring surround-view fisheye camera images taken by a plurality of vehicle-mounted fisheye cameras, performing distortion correction processing on each surround-view fisheye camera image, and obtaining a plurality of dedistorted surround-view fisheye camera images;

[0038] A multi-scale feature extraction unit, used to extract features from the plurality of dedistorted surround-view fisheye camera images to obtain multi-scale features of each surround-view fisheye camera image;

[0039] A mapping processing unit, configured to perform feature mapping on multi-scale features corresponding to the plurality of surround fisheye camera images using a converter model with a multi-view attention mechanism to obtain BEV map view features;

[0040] A mask map acquisition unit, used for performing image semantic segmentation detection and upsampling on the BEV map view features to form a binary semantic segmentation mask map;

[0041] A drivable area determination unit is used to determine a drivable area for a vehicle according to the binary semantic segmentation mask image.

[0042] The multi-scale feature extraction unit is specifically used for:

[0043] The preset RegNets network model and BiFPN network model are used to downsample the multiple dedistorted surround-view fisheye camera images and perform feature extraction to obtain multi-scale features of each surround-view fisheye camera image; wherein the multi-scale is a different scaling ratio, and the multi-scale features corresponding to each scale include at least image features of each surround-view fisheye camera image, a rotation matrix, an intrinsic parameter matrix, and a translation vector relative to the center of the vehicle.

[0044] Wherein, the mapping processing unit further comprises:

[0045] A multi-scale feature receiving unit, used to receive multi-scale features of multiple fisheye camera images of at least two scales, wherein the features of each scale include image features, a rotation matrix, an intrinsic parameter matrix, and a translation vector relative to the center of the vehicle;

[0046] A vector setting unit is used to use the image features of each scale as the value vector of the multi-view attention mechanism; obtain the position encodings of multiple fisheye camera images, and combine the image features of each scale as the key vector of the multi-view attention mechanism; obtain the difference between the position encoding of the BEV map view and the fish camera position encoding as the query vector of the multi-view attention mechanism;

[0047] A BEV map view feature acquisition unit, used to acquire the similarity at each scale corresponding to each pixel in the BEV map view, and use the similarity at each scale as a weight to perform weighted processing with the value vector at the corresponding scale to obtain a view feature representation of each pixel in the BEV map view;

[0048] The similarity is calculated using the following formula:

[0049]

[0050] Among them, δ k,i Encodes the position of a fisheye camera view; φ k,i Image features of fisheye camera view; c (n) Encode the position on the BEV map view, τ k is the camera position code; k is the camera number, and i is the scale number.

[0051] Wherein, the mask image acquisition unit further comprises:

[0052] An upsampling processing unit, configured to perform bilinear upsampling and convolution processing on the view feature of each pixel point in the BEV map view to form a BEV map view of a predetermined resolution;

[0053] The semantic segmentation unit is used to perform semantic segmentation and obtain a binary semantic segmentation mask in the BEV map view coordinate system; wherein the semantic segmentation loss function used is:

[0054] L loss =w1L cross_entropy +w2L ohem_cross_entropy

[0055] Among them, L cross_entropy is the conventional binary cross entropy loss function, L ohem_cross_entropy It is a binary cross entropy loss function with OHEM added; w1 and w2 are the predetermined weight ratios.

[0056] Correspondingly, another aspect of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the aforementioned method when executed by a processor.

[0057] Correspondingly, another aspect of the present invention further provides a computing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the aforementioned method when executing the computer program.

[0058] Correspondingly, in another aspect of the present invention, there is also a vehicle, on which is at least provided with an automatic driving planning device, in which the system as described above is deployed.

[0059] The implementation of the embodiments of the present invention has the following beneficial effects:

[0060] The present invention provides a method, system, device, storage medium and vehicle for detecting a drivable area. The surround fisheye camera images taken by multiple on-board fisheye cameras are subjected to distortion correction processing; then RegNets+BiFPN is used to extract features from multiple dedistorted surround fisheye camera images to obtain multi-scale features of each surround fisheye camera image; a trained converter model with a multi-view attention mechanism is used to perform feature mapping on the multi-scale features to obtain BEV map view features; then the BEV map view features are subjected to image semantic segmentation detection and upsampling to form a binary semantic segmentation mask map, and finally the vehicle drivable area is determined. The surround fisheye camera images taken by the on-board fisheye camera can be directly used as input to output a BEV map marked with the vehicle drivable area, which can improve recognition efficiency and accuracy; and through distortion correction and extraction of multi-scale features, the recognition accuracy is further improved.

[0061] In addition, a converter model based on a multi-view attention mechanism is adopted, and a multi-view attention mechanism perceived by multiple (such as 4) surround fisheye cameras is used to implicitly learn the mapping from multi-camera views to BEV map view representation. It uses the camera's internal and external parameter calibration as the position encoding of the converter model, implicitly learns the mapping from multi-camera views to BEV map views, is not limited by the flat ground assumption, and does not require explicit depth estimation of the 4 fisheye cameras, so there is no need to perform geometric modeling and surround stitching of the fisheye cameras, which improves the applicability of the method.

[0062] In this embodiment, a pure vision solution is adopted, which can be implemented using only a fisheye camera sensor. The multi-scale features of the four fisheye camera images can be obtained through the powerful RegNets+BiFPN feature extraction network, which can improve the generalization and robustness of the drivable area detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, without paying creative labor, other drawings obtained based on these drawings still belong to the scope of the present invention.

[0064] Figure 1 A schematic diagram of the main process of an embodiment of a drivable area detection method provided by the present invention;

[0065] Figure 2 for Figure 1 Schematic diagram of the implementation principle;

[0066] Figure 3 A schematic diagram of the RegNets network model involved in the present invention;

[0067] Figure 4 A schematic diagram of a BiFPN network model according to the present invention;

[0068] Figure 5 A schematic diagram of the structure of a drivable area detection system provided by the present invention;

[0069] Figure 6 for Figure 5 A schematic diagram of the structure of the mapping processing unit;

[0070] Figure 7 for Figure 5 Schematic diagram of the structure of the mask image acquisition unit. DETAILED DESCRIPTION

[0071] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings.

[0072] like Figure 1 As shown, a drivable area detection method provided by the present invention is shown. Figures 2 to 4 As shown, in this embodiment, the method at least includes the following steps:

[0073] Step S10, acquiring surround-view fisheye camera images taken by a plurality of vehicle-mounted fisheye cameras, performing distortion correction processing on each surround-view fisheye camera image, and obtaining a plurality of dedistorted surround-view fisheye camera images;

[0074] In one example of the present invention, four fisheye cameras are used to obtain four fisheye camera images of the left, right, front and rear of the vehicle; at the same time, the distortion of the four fisheye camera images is corrected. Specifically, a chessboard correction method can be used to correct the barrel distortion of the fisheye camera image. By printing a chessboard and collecting chessboard images under the fisheye camera, the collected chessboard images are used to calculate the camera's internal parameters, namely, the camera matrix (DIM) and the correction coefficients (K, D).

[0075] Step S11, performing feature extraction on the plurality of dedistorted surround-view fisheye camera images to obtain multi-scale features of each surround-view fisheye camera image;

[0076] In a specific example, the step S11 further includes:

[0077] The pre-set RegNets network model and BiFPN network model are used to downsample the multiple dedistorted surround fisheye camera images and perform feature extraction to obtain multi-scale features of each surround fisheye camera image; wherein the multi-scale is a different scaling ratio, and the multi-scale features corresponding to each scale at least include image features of each surround fisheye camera image, a rotation matrix, an intrinsic parameter matrix, and a translation vector relative to the center of the vehicle.

[0078] In an embodiment of the present invention, an image encoder is used to implement the function of step S11, and the image encoder is used to extract features from the four fisheye camera images and generate a multi-scale feature representation of each input image;

[0079] The image encoder includes a pre-set RegNets network model and a BiFPN network model.

[0080] More specifically, if Figure 3 As shown, in an embodiment of the present invention, the RegNets network is a deep convolutional neural network composed of three components: a stem, a body, and a head.

[0081] in:

[0082] Stem: The stem is the starting part of the RegNets network, which is responsible for some preprocessing and feature extraction operations on the input image. It usually consists of some convolutional layers and pooling layers to initially extract basic features in the image.

[0083] Body: The body is the core part of the RegNets network, which consists of multiple stages. Each stage contains a certain number of identical blocks, which are similar in structure but have different parameter settings. Usually, each stage contains some ordinary blocks and some blocks with downsampling (stride 2) function, which are used to gradually reduce the size of the feature map. The body is mainly responsible for deep calculation and information extraction of input features to obtain richer semantic feature expressions. In an embodiment of the present invention, the body adopts 4 stages.

[0084] Head: The head is the last part of the RegNets network and is used for classification tasks. It usually contains some fully connected layers and pooling layers to transform the feature maps obtained in the main stage into the final classification results. The head is designed according to the specific application scenario and task, and its structure and number of layers can be changed according to the needs.

[0085] Each stage i has four parameters, namely:

[0086] Block number di: indicates the number of blocks in this stage. By increasing the number of blocks, the network can perform more calculations and feature extraction in this stage, thereby improving the network's expressive power.

[0087] Block width wi: represents the width (channels) of each block in this stage. A larger block width can increase the number of parameters in each block, thereby increasing the complexity and feature representation capability of the network.

[0088] Bottleneck ratio bi: represents the ratio between the number of internal channels and the number of input channels in the bottleneck block in this stage. A higher bottleneck ratio can reduce the number of internal channels, thereby reducing the computational complexity of the model.

[0089] Group width gi: represents the number of groups in the group convolution operation within the block. A larger group width can increase the parallelism of the network, improve computational efficiency and feature representation capabilities.

[0090] For different image resolutions, the RegNets network model selects different parameters by following general design principles. Higher resolution images may require larger block widths and group widths to ensure that the network has sufficient feature expression and computing power. By adjusting the parameters, the network can be adapted to input images of different resolutions, thereby improving the generalization performance and adaptability of the model.

[0091] like Figure 3As shown, the BiFPN network structure involved in the present invention is a feature fusion network structure for computer vision tasks such as target detection and semantic segmentation. It mainly achieves better multi-scale feature representation through two paths, top-down and bottom-up, and a weighted feature fusion method.

[0092] exist Figure 3 , the top-down path (the downward solid arrow):

[0093] Its high-level features have strong semantic information but low resolution. The top-down pathway starts from high-level features and passes high-level semantic information downward step by step. At each level, the features are upsampled (for example, using bilinear interpolation) to obtain a feature map with the same resolution as the lower-level features.

[0094] Bottom-up path (upward dashed arrow):

[0095] Low-level features have high resolution and rich detail information, but weak semantic information. The bottom-up pathway starts from low-level features and passes position information upward level by level. At each level, features are downsampled (for example, using maximum pooling) to obtain feature maps with the same resolution as the upper-level features.

[0096] Feature fusion and weight learning (arc solid arrow part of the same layer): a new edge is added between input nodes and input nodes of the same layer. The importance of different input features is learned, and different input features are fused differently.

[0097] At each level, top-down and bottom-up feature maps are combined via feature fusion, a process that carries weights that are learned to distinguish the importance and contribution of different input features.

[0098] In the feature fusion process, the weighted fusion method can better utilize the resolution and semantic information of different input features, thereby obtaining better multi-scale feature representation.

[0099] Through the working principle of the BiFPN network model, a multi-scale feature representation with rich semantic information and high-resolution details can be obtained. This is especially important for target detection tasks, because features of different scales can more accurately capture targets of different sizes and shapes, thereby improving detection performance.

[0100] For example, suppose we need to detect people and vehicles in an image. The top-down pathway can extract high-level semantic information in the image, such as the overall structure and posture of people and vehicles. The bottom-up pathway can provide low-level detail information, such as the edges and textures of these objects. Through the feature fusion and weight learning process of the BiFPN network model, features at different levels will be combined and better represented at different scales to achieve more accurate object detection results.

[0101] According to the description, the BiFPN network model generates multi-scale feature representations after processing the four input fisheye camera images. Scale refers to features of different resolutions (i.e. different scaling ratios).

[0102] It is understandable that traditional feature fusion is often just a simple feature map superposition / addition without distinguishing the feature maps added at the same time. However, different input feature maps have different resolutions, and their contributions to the fused input feature map are also different. Therefore, the BiFPN weighted feature fusion method can obtain better multi-scale feature representation.

[0103] In this way, the BiFPN network model can obtain richer and more comprehensive multi-scale feature representation by integrating features of multiple scales, which helps to improve the detection and recognition performance of targets or scenes of different scales.

[0104] Step S12, using a converter model with a multi-view attention mechanism to perform feature mapping on multi-scale features corresponding to the multiple surround fisheye camera images to obtain BEV map view features;

[0105] It can be understood that, in this embodiment, a trained converter model with a multi-view attention mechanism is used to perform feature mapping on the multi-scale features corresponding to the multiple surround fisheye camera images.

[0106] According to the converter model with multi-view attention mechanism, the multi-view attention mechanism perceived by 4 surround fisheye cameras can be implicitly learned from the mapping of 4 fisheye camera images to BEV map view representation.

[0107] Before describing the contents of step S12 in detail, some principles and parameters of the multi-camera view attention mechanism are briefly introduced.

[0108] In the multi-camera view attention mechanism transformer model, the query vector, key vector, and value vector are three representations obtained by linearly transforming the input sequence. They play different roles in the attention calculation process, among which:

[0109] Query: The query vector is used to measure how well each element of the input is associated with the output. In multi-camera views, the query is usually defined based on the current task or specific information of interest. For example, the image features at the current time step can be used as the query vector to determine the key points or objects associated with the image.

[0110] Key: Key vectors are used to represent features of the input data. They are used to compute a similarity score between the query vector and the input data to determine which elements of the input are most relevant to the query. In multi-camera views, keys are often used to represent features of camera images or specific regions. For example, image features from different cameras can be used as key vectors to correlate query images across multiple viewpoints.

[0111] Value: Value vectors contain information related to the input data. They are used to generate the final output based on the relevance weights of the query and the key. In multi-camera views, values ​​are often used to represent specific information in the input data, such as pixel-level features or semantic information. For example, the feature vector of each pixel in the camera image can be used as the corresponding value vector.

[0112] Let's take an example to illustrate: suppose there are three cameras, each of which takes an image as input. We want to fuse these images through the attention mechanism to generate a synthetic view. In this scenario:

[0113] Query: You can choose one of the images as the query vector. For example, we choose the image from the first camera as the query vector.

[0114] Key: Each camera image can be used as a key vector. Suppose we have camera A, camera B and camera C, then the images of camera A, camera B and camera C are used as key vectors respectively.

[0115] Value: The pixel features in each camera image can be taken as the corresponding value vector. For example, for each pixel in the image of camera A, we can extract its feature vector as the corresponding value vector, and the same for camera B and camera C.

[0116] By computing the similarity score between the query vector and the key vector and weighting the value vector with the corresponding weights, we can generate a synthetic view where the value of each pixel is obtained by weighted fusion of the feature values ​​of the pixels at the corresponding position.

[0117] In a specific example of the present invention, the step S12 uses a trained converter model with a multi-view attention mechanism; in a specific example, the step S12 further includes:

[0118] Step S120, receiving multi-scale features of multiple fisheye camera images of at least two scales, wherein the features of each scale include image features, a rotation matrix, an intrinsic parameter matrix, and a translation vector relative to the center of the vehicle; and using the image features of each scale as a value vector of a multi-view attention mechanism;

[0119] It can be understood that the input of the converter module is the features of four fisheye camera images at two scales, where the two scales refer to two sets of scaled images, for example, one scale corresponds to the original Figure 4 The other scale is 8 times smaller than the original image.

[0120] Among them, the parameters of the image features at each scale are Represent image features, rotation matrix, intrinsic matrix, and translation vector relative to the center of the vehicle, respectively. The goal of the multi-image attention mechanism is to connect the desired BEV map view representation with the view features of the fisheye camera image. For any world coordinate x (W) ∈R 3 , and the image point x (I) The mapping relationship is:

[0121]

[0122] It is understandable that the above formula is only approximately equal, because the actual depth value is unknown and there is uncertainty in scale. Specifically, in computer vision and depth estimation, the depth value usually represents the distance of an object or scene in the real world. However, since a single image cannot provide complete depth information, the estimated depth is usually affected by the uncertainty in scale. In this embodiment, the conventional explicit use of depth information or implicit encoding of depth space distribution is not adopted, but the uncertainty in scale is encoded using position encoding and Transformer network for learning. (W) The x of the image point (I) The similarity relationship is expressed by cosine similarity:

[0123]

[0124] The image encoding module extracts two scale features φ from four fisheye camera images. k,i ,k=1,...4;i=1,...2. ; Where k is the camera number and i is the scale number; These features serve as the query vector (value) of the multi-view attention mechanism.

[0125] It can be understood that in this embodiment, by implicitly encoding the depth information as a position code, the model can learn the similarity relationship between pixel points and 3D space points. Cosine similarity is used to measure this similarity relationship. By encoding and learning scale uncertainty, the model can perform depth estimation and related tasks in the absence of explicit depth information. This approach helps to solve the scale uncertainty problem in single image depth estimation.

[0126] Step S121, obtaining position codes of multiple fisheye camera images, and combining image features of each scale as key vectors of a multi-view attention mechanism;

[0127] It can be understood that the position encoding of the four fisheye camera images depends on the internal and external parameter matrices obtained by calibrating the four cameras respectively; the feature dimensions of the four fisheye camera images are combined with the internal and external parameters of each camera for position encoding, that is, each pixel on the feature map is mapped to the 3D space, that is:

[0128]

[0129] The mapped 3D points are encoded through a linear network to obtain the position encoding of the four fisheye camera views δ k,i ∈R D , combined with the image features in 1) above, that is, δ k,i +φ k,i Serves as the key vector for the multi-view attention mechanism, thereby combining the position embedding with the image.

[0130] Step S122, obtaining the difference between the position encoding of the BEV map view and the position encoding of the fish camera as a query vector for the multi-view attention mechanism;

[0131] More specifically, on a BEV map view, a multilayer perceptron (MLP) is first used to quantize the camera position t k Encoded into τ k ∈R D In our transformer, we construct a map view representation through multiple iterations, generating a learnable position encoding c on the BEV map view. (0) ∈R w×h×D The goal of BEV map view position encoding is to estimate the 3D position of each road element. Through continuous iteration of the network, it can better project the BEV map view coordinates into the 3D environment proxy. Using the BEV map view position encoding c and the camera position encoding τ k The difference between them is used as the query vector (query) of the multi-view attention mechanism.

[0132] Step S123, obtaining the similarity at each scale corresponding to each pixel point in the BEV map view, taking the similarity at each scale as a weight and performing weighted processing with the value vector at the corresponding scale, and obtaining the view feature representation of each pixel point in the BEV map view.

[0133] The obtaining of the similarity at each scale corresponding to each pixel point in the BEV map view is specifically as follows:

[0134] Combining the above formulas (1), (2) and (3), the similarity at each scale corresponding to each pixel in the BEV map view is calculated using the following formula:

[0135]

[0136] Among them, δ k,i Encodes the position of a fisheye camera view; φ k,i Image features of fisheye camera view; c (n) Encode the position on the BEV map view, τ k Encodes the camera position.

[0137] After obtaining the similarity, the corresponding value vector can be weighted according to the similarity score, and the weighted value vectors corresponding to the two scales can be summed to obtain the final output. Thus, by calculating the similarity scores of the query vector and the key vector and weighting the value vectors with the corresponding weights, a synthetic view can be generated, in which the value of each pixel is obtained by weighted fusion of the pixel features at the corresponding position.

[0138] Step S13, performing image semantic segmentation detection and upsampling on the BEV map view features to form a binary semantic segmentation mask image;

[0139] In a specific example, a decoder is used to implement the function of the above step S13, wherein the decoder consists of three layers (bilinear upsampling+convolution).

[0140] In a specific example, the step S13 further includes:

[0141] Step S130, performing bilinear upsampling and convolution processing on the view features of each pixel in the BEV map view to form a BEV map view of a predetermined resolution;

[0142] Specifically, in one example, each upsampling layer increases the resolution by a factor of 2, and the final output resolution is 800×800.

[0143] Step S131, and perform semantic segmentation to obtain a binary semantic segmentation mask in the BEV map view coordinate system;

[0144] It can be understood that the goal of this decoder module is to use the multi-view attention mechanism converter to learn to extract effective information from multiple camera views to predict the binary semantic segmentation mask y∈{0,1} in the BEV map view coordinate system. h×w×C .

[0145] Among them, the loss function of semantic segmentation used is:

[0146] L loss =w1L cross_entropy +w2L ohem_cross_entropy

[0147] Among them, L cross_entropy is the conventional binary cross entropy loss function, L ohem_cross_entropy It is a binary cross entropy loss function with OHEM added; w1 and w2 are the predetermined weight ratios.

[0148] L ohem_cross_entropy In order to solve the problem of imbalance between positive and negative samples, some difficult samples (hard examples) are selected as training samples to improve the network parameter effect. Hard examples refer to samples with diversity and high loss. The two losses are combined with a weight ratio of w1=0.4 and w2=1.

[0149] Step S14, determining the vehicle drivable area according to the binary semantic segmentation mask image.

[0150] In a specific example, the step S14 further includes:

[0151] Output the final BEV map and determine whether the corresponding area is drivable based on the binary semantic segmentation mask value of each pixel;

[0152] Mark the boundaries of the drivable area.

[0153] For example, in a specific example, a BEV map is output, which contains the stitching and fusion information of four fisheye cameras under the BEV perspective. It is a binary semantic mask map, where 0 represents a non-drivable area and 1 represents a drivable area. Finally, the area with a value of 1 is marked as a drivable area for subsequent automatic driving or automatic parking functions.

[0154] It is understood that the method of the present invention further comprises the following steps:

[0155] Using LiDAR to collect information as labels for training data for the drivable area detection model; and

[0156] The RegNets network model, BiFPN network model, and converter model of the multi-view attention mechanism are pre-built and trained.

[0157] It can be understood that in the drivable area detection method provided by the present invention, by adopting a converter model based on a multi-view attention mechanism, using a multi-view attention mechanism perceived by four surround fisheye cameras, implicitly learning the mapping from multi-camera views to BEV map view representation, it is not limited by the flat ground assumption (the ground of outdoor parking scenes is often uneven); there is no need to explicitly estimate the depth of the four fisheye cameras, because explicit depth is very challenging to use in downstream tasks; it depends on the camera and requires accurate calibration and fusion of multiple noise estimates. In the present invention, explicit depth estimation is avoided, and an attention mechanism with position encoding is adopted. The converter module using the multi-view attention mechanism implicitly learns the mapping from multi-camera to BEV map view representation; the internal and external parameter calibration of the camera is used as the position encoding of the converter module of the multi-view attention mechanism to learn the mapping from multi-camera views to BEV map views, without explicit geometric modeling and surround stitching of the fisheye camera.

[0158] In the drivable area detection model, the image features perceived by multiple surround fisheye cameras can be connected and fused with the map view representation by using the converter module with multi-view attention mechanism.

[0159] In the mapping process, due to the lack of completely accurate depth information, there is a certain uncertainty in the mapping relationship between 3D space points and image points. In this embodiment, cosine similarity is used to represent the similarity relationship between 3D space points and image points. Points with higher similarity indicate that they have a more similar mapping relationship in depth.

[0160] The role of using cosine similarity to represent similarity relationships is to encode the uncertainty between 3D space points and image points. By calculating cosine similarity, the degree of directional similarity between two points can be measured, thereby capturing the potential association between them. The more accurate the mapping, the higher the cosine similarity may be.

[0161] like Figure 5 FIG. 1 is a schematic diagram showing a structure of an embodiment of a drivable area detection system provided by the present invention. Figure 6 and Figure 7 As shown, in this embodiment, the drivable area detection system 1 at least includes:

[0162] The preprocessing unit 10 is used to obtain surround-view fisheye camera images taken by multiple vehicle-mounted fisheye cameras, perform distortion correction processing on each surround-view fisheye camera image, and obtain multiple dedistorted surround-view fisheye camera images;

[0163] A multi-scale feature extraction unit 11 is used to extract features from the plurality of dedistorted surround-view fisheye camera images to obtain multi-scale features of each surround-view fisheye camera image;

[0164] A mapping processing unit 12, configured to perform feature mapping on multi-scale features corresponding to the plurality of surround fisheye camera images using a converter model with a multi-view attention mechanism to obtain BEV map view features;

[0165] The mask map acquisition unit 13 is used to perform image semantic segmentation detection and upsampling on the BEV map view features to form a binary classification semantic segmentation mask map;

[0166] The drivable area determination unit 14 is used to determine the drivable area of ​​the vehicle according to the binary semantic segmentation mask image.

[0167] In a specific example, the multi-scale feature extraction unit 11 is specifically used for:

[0168] The preset RegNets network model and BiFPN network model are used to downsample the multiple dedistorted surround-view fisheye camera images and perform feature extraction to obtain multi-scale features of each surround-view fisheye camera image; wherein the multi-scale is a different scaling ratio, and the multi-scale features corresponding to each scale include at least image features of each surround-view fisheye camera image, a rotation matrix, an intrinsic parameter matrix, and a translation vector relative to the center of the vehicle.

[0169] like Figure 6 As shown, in a specific example, the mapping processing unit 12 further includes:

[0170] A multi-scale feature receiving unit 120 is used to receive multi-scale features of multiple fisheye camera images of at least two scales, where the features of each scale include image features, a rotation matrix, an intrinsic parameter matrix, and a translation vector relative to the center of the vehicle;

[0171] A vector setting unit 121 is used to use the image features of each scale as a value vector of the multi-view attention mechanism; obtain the position encodings of multiple fisheye camera images, and combine the image features of each scale as a key vector of the multi-view attention mechanism; obtain the difference between the position encoding of the BEV map view and the fish camera position encoding as a query vector of the multi-view attention mechanism;

[0172] A BEV map view feature acquisition unit 122 is used to acquire the similarity at each scale corresponding to each pixel in the BEV map view, and use the similarity at each scale as a weight to perform weighted processing with the value vector at the corresponding scale to obtain a view feature representation of each pixel in the BEV map view;

[0173] The similarity is calculated using the following formula:

[0174]

[0175] Among them, δ k,i Encodes the position of a fisheye camera view; φ k,i Image features of fisheye camera view; c (n) Encode the position on the BEV map view, τ k is the camera position code; k is the camera number, and i is the scale number.

[0176] like Figure 7 As shown, in a specific example, the mask image acquisition unit 13 further includes:

[0177] An upsampling processing unit 130, configured to perform bilinear upsampling and convolution processing on the view feature of each pixel in the BEV map view to form a BEV map view of a predetermined resolution;

[0178] The semantic segmentation unit 131 is used to perform semantic segmentation and obtain a binary semantic segmentation mask in the BEV map view coordinate system; wherein the semantic segmentation loss function used is:

[0179] L loss =w1L cross_entropy +w2L ohem_cross_entropy

[0180] Among them, L cross_entropy is the conventional binary cross entropy loss function, L ohem_cross_entropy It is a binary cross entropy loss function with OHEM added; w1 and w2 are the predetermined weight ratios.

[0181] It can be understood that, in another example, the system 1 of the present invention further comprises:

[0182] A training data acquisition unit, used to collect information using a laser radar as a label for training data of a drivable area detection model; and

[0183] The training processing unit is used to pre-build the RegNets network model, the BiFPN network model, and the converter model of the multi-view attention mechanism, and perform training.

[0184] Accordingly, in another aspect of the present invention, there is also provided a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the above-mentioned Figures 1 to 4 For more details, please refer to and combine with the above Figures 1 to 4 The description is not repeated here.

[0185] Accordingly, another aspect of the present invention further provides a computing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the aforementioned Figures 1 to 4 For more details, please refer to and combine with the above Figures 1 to 4 The description is not repeated here.

[0186] Accordingly, in another aspect of the present invention, there is also a vehicle, on which at least an automatic driving planning device is provided, in which the aforementioned Figures 5 to 7 For more details, please refer to and combine the above Figures 5 to 7 The description is not repeated here.

[0187] The implementation of the embodiments of the present invention has the following beneficial effects:

[0188] The present invention provides a method, system, device, storage medium and vehicle for detecting a drivable area. The surround fisheye camera images taken by multiple on-board fisheye cameras are subjected to distortion correction processing; then RegNets+BiFPN is used to extract features from multiple dedistorted surround fisheye camera images to obtain multi-scale features of each surround fisheye camera image; a trained converter model with a multi-view attention mechanism is used to perform feature mapping on the multi-scale features to obtain BEV map view features; then the BEV map view features are subjected to image semantic segmentation detection and upsampling to form a binary semantic segmentation mask map, and finally the vehicle drivable area is determined. The surround fisheye camera images taken by the on-board fisheye camera can be directly used as input to output a BEV map marked with the vehicle drivable area, which can improve recognition efficiency and accuracy; and through distortion correction and extraction of multi-scale features, the recognition accuracy is further improved.

[0189] In addition, a converter model based on a multi-view attention mechanism is adopted, and a multi-view attention mechanism perceived by multiple (such as 4) surround fisheye cameras is used to implicitly learn the mapping from multi-camera views to BEV map view representation. It uses the camera's internal and external parameter calibration as the position encoding of the converter model, implicitly learns the mapping from multi-camera views to BEV map views, is not limited by the flat ground assumption, and does not require explicit depth estimation of the 4 fisheye cameras, so there is no need to perform geometric modeling and surround stitching of the fisheye cameras, which improves the applicability of the method.

[0190] In this embodiment, a pure vision solution is adopted, which can be implemented using only a fisheye camera sensor. The multi-scale features of the four fisheye camera images can be obtained through the powerful RegNets+BiFPN feature extraction network, which can improve the generalization and robustness of the drivable area detection model.

[0191] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0192] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0193] The above disclosure is only a preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A method for detecting a drivable area, characterized in that: At least the following steps are included: Acquire surround-view fisheye camera images taken by multiple vehicle-mounted fisheye cameras, perform distortion correction processing on each surround-view fisheye camera image, and obtain multiple dedistorted surround-view fisheye camera images; Performing feature extraction on the plurality of dedistorted surround-view fisheye camera images to obtain multi-scale features of each surround-view fisheye camera image; Using a converter model with a multi-view attention mechanism to perform feature mapping on multi-scale features corresponding to the plurality of surround fisheye camera images to obtain BEV map view features; Performing image semantic segmentation detection and upsampling on the BEV map view features to form a binary semantic segmentation mask image; The vehicle drivable area is determined according to the binary semantic segmentation mask image.

2. The method according to claim 1, characterized in that The plurality of dedistorted surround-view fisheye camera images are subjected to feature extraction to obtain multi-scale features of each surround-view fisheye camera image, specifically: The pre-set RegNets network model and BiFPN network model are used to downsample the multiple dedistorted surround-view fisheye camera images and perform feature extraction to obtain multi-scale features of each surround-view fisheye camera image; wherein the multi-scale is a different scaling ratio, and the multi-scale features corresponding to each scale at least include image features of each surround-view fisheye camera image, a rotation matrix, an intrinsic parameter matrix, and a translation vector relative to the center of the vehicle.

3. The method according to claim 2, characterized in that Using a converter model with a multi-view attention mechanism to perform feature mapping on multi-scale features corresponding to the plurality of surround fisheye camera images to obtain BEV map view features, further comprising: receiving multi-scale features of multiple fisheye camera images of at least two scales, and using the image features of each scale as a value vector of a multi-view attention mechanism; Obtain the position encoding of multiple fisheye camera images according to the fisheye camera's intrinsic parameter matrix and translation vector, and combine the image features of each scale as the key vector of the multi-view attention mechanism; Get the difference between the position encoding of the BEV map view and the fish camera position encoding as the query vector for the multi-view attention mechanism; The similarity at each scale corresponding to each pixel in the BEV map view is obtained, and the similarity at each scale is used as a weight to perform weighted processing with the value vector at the corresponding scale to obtain a view feature representation of each pixel in the BEV map view.

4. The method according to claim 3, characterized in that The method of obtaining the similarity at each scale corresponding to each pixel point in the BEV map view is specifically as follows: The similarity at each scale corresponding to each pixel in the BEV map view is calculated using the following formula: Where: k,i Encodes the position of a fisheye camera view; φ k,i Image features of fisheye camera view; c (n) Encode the position on the BEV map view, τ k is the camera position code; k is the camera number, and i is the scale number.

5. The method according to claim 4, characterized in that The performing image semantic segmentation detection and upsampling on the BEV map view features to form a binary semantic segmentation mask map further includes: Perform bilinear upsampling and convolution processing on the view features of each pixel in the BEV map view to form a BEV map view of a predetermined resolution; Perform semantic segmentation to obtain a binary semantic segmentation mask in the BEV map view coordinate system; the semantic segmentation loss function used is: L loss =w1L cross_entropy +w2L ohem_cross_entropy Among them, L cross_entropy is the conventional binary cross entropy loss function, L ohem_cross_entropy It is a binary cross entropy loss function with OHEM added; w1 and w2 are the predetermined weight ratios.

6. The method according to claim 5, characterized in that Determining a vehicle drivable area according to the binary semantic segmentation mask image includes: Output the final BEV map and determine whether the corresponding area is drivable based on the binary semantic segmentation mask value of each pixel; Mark the boundaries of the drivable area.

7. The method according to any one of claims 1 to 6, further comprising: Use LiDAR to collect information as labels for training data of the drivable area detection model; as well as The RegNets network model, BiFPN network model, and converter model of the multi-view attention mechanism are pre-built and trained.

8. A drivable area detection system, characterized in that: At least: A preprocessing unit, used for acquiring surround-view fisheye camera images taken by a plurality of vehicle-mounted fisheye cameras, performing distortion correction processing on each surround-view fisheye camera image, and obtaining a plurality of dedistorted surround-view fisheye camera images; A multi-scale feature extraction unit, used to extract features from the plurality of dedistorted surround-view fisheye camera images to obtain multi-scale features of each surround-view fisheye camera image; A mapping processing unit, configured to perform feature mapping on multi-scale features corresponding to the plurality of surround fisheye camera images using a converter model with a multi-view attention mechanism to obtain BEV map view features; A mask map acquisition unit, used for performing image semantic segmentation detection and upsampling on the BEV map view features to form a binary semantic segmentation mask map; A drivable area determination unit is used to determine a drivable area for a vehicle according to the binary semantic segmentation mask image.

9. The system according to claim 8, characterized in that The multi-scale feature extraction unit is specifically used for: The preset RegNets network model and BiFPN network model are used to downsample the multiple dedistorted surround fisheye camera images and perform feature extraction to obtain multi-scale features of each surround fisheye camera image; wherein the multi-scale is a different scaling ratio, and the multi-scale features corresponding to each scale include at least image features of each surround fisheye camera image, a rotation matrix, an intrinsic parameter matrix, and a translation vector relative to the center of the vehicle.

10. The system according to claim 9, characterized in that The mapping processing unit further comprises: A multi-scale feature receiving unit, used to receive multi-scale features of a plurality of fisheye camera images of at least two scales; A vector setting unit is used to use the image features of each scale as the value vector of the multi-view attention mechanism; obtain the position encodings of multiple fisheye camera images, and combine the image features of each scale as the key vector of the multi-view attention mechanism; obtain the difference between the position encoding of the BEV map view and the fish camera position encoding as the query vector of the multi-view attention mechanism; A BEV map view feature acquisition unit, used to acquire the similarity at each scale corresponding to each pixel in the BEV map view, and use the similarity at each scale as a weight to perform weighted processing with the value vector at the corresponding scale to obtain a view feature representation of each pixel in the BEV map view; The similarity is calculated using the following formula: Among them, δ k,i Encodes the position of a fisheye camera view; φ k,i Image features of fisheye camera view; c (n) Encode the position on the BEV map view, τ k is the camera position code; k is the camera number, and i is the scale number.

11. The system according to claim 10, characterized in that The mask image acquisition unit further comprises: An upsampling processing unit, configured to perform bilinear upsampling and convolution processing on the view feature of each pixel point in the BEV map view to form a BEV map view of a predetermined resolution; The semantic segmentation unit is used to perform semantic segmentation and obtain a binary semantic segmentation mask in the BEV map view coordinate system; wherein the semantic segmentation loss function used is: L loss =w1L cross_entropy +w2L ohem_cross_entropy Among them, L cross_entropy is the conventional binary cross entropy loss function, L ohem_cross_entropy It is a binary cross entropy loss function with OHEM added; w1 and w2 are the predetermined weight ratios.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

13. A computing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

14. A vehicle, on which at least an automatic driving planning device is provided, wherein the system according to any one of claims 8 to 11 is deployed in the automatic driving planning device.

Images