Travelable area generation method, system and device based on ground contact boundary point detection and vehicle

Abstract

The invention discloses a drivable area generation method based on ground touch boundary point detection. The drivable area generation method comprises the following steps: acquiring a plurality of original fisheye pictures shot by a plurality of vehicle-mounted fisheye cameras; adopting a pre-trained deep learning model to carry out segmentation identification processing on the plurality of original fisheye pictures; distortion correction processing is carried out on the segmented image corresponding to each original fisheye image and the ground touch boundary point information, and the segmented images and the ground touch boundary point information are projected to a BEV view for panoramic stitching; adopting a preset Gaussian probability model to perform weighted fusion on the drivable area detection results at a plurality of historical moments, updating the BEV view, and obtaining a BEV drivable area map; and outputting the BEV drivable area map and marking the ground touch boundary of the drivable area. The invention further discloses a corresponding system, device and vehicle. According to the invention, the ground touch boundary point can be accurately detected and the stable drivable area map can be obtained, so that the accuracy of drivable area detection and the driving safety are improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method, system, device and vehicle for generating a drivable area based on touchdown boundary point detection. Background Art

[0002] In autonomous driving scenarios (such as automatic parking), drivable area detection is very important for downstream decision-making and planning. During parking, it is necessary to perceive the drivable area and obstacle information in the short distance around the vehicle in a timely manner, which requires high perception accuracy. Generally, to realize the automatic parking system, four fisheye cameras are required to build a real-time visual perception function that surrounds the vehicle body 360°.

[0003] In the existing automatic parking system, the drivable area algorithm based on four fisheye cameras is generally implemented by performing binary classification semantic segmentation on the drivable area of ​​the four fisheye camera images, and dividing the images into two categories: drivable and non-drivable. However, due to reasons such as occlusion of the image perspective, many false drivable area boundaries may appear on the drivable boundaries on the fisheye images. For example, the false boundaries of some drivable areas are suspended rather than touching the ground, that is, the false boundaries of the drivable areas are some non-ground touching boundary points.

[0004] However, there are two prerequisites for projecting the drivable area detection images of the four fisheye cameras onto the BEV view: not only is the ground assumed to be flat, but all detected drivable area boundary points must also be assumed to be ground contact boundary points. This is how the camera's internal and external calibration parameters can be used to build a geometric model to stitch together an accurate BEV view.

[0005] However, the BEV view obtained by using existing technologies cannot accurately exclude non-ground contact boundary points. Therefore, obstacles may be stretched and deformed in the drivable area under the BEV view, which is not conducive to the decision-making planning downstream of the automatic parking system, thereby affecting driving safety. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a method, system, device and vehicle for generating a drivable area based on touchdown boundary point detection. The touchdown boundary point can be accurately detected and a stable drivable area map can be obtained, thereby improving the accuracy of drivable area detection and driving safety.

[0007] In order to solve the above technical problems, as one aspect of the present invention, a method for generating a drivable area based on ground contact boundary point detection is provided, which at least comprises the following steps:

[0008] Obtain multiple original fisheye images taken by multiple vehicle-mounted fisheye cameras;

[0009] Using a pre-trained deep learning model to identify ground contact boundary points of the plurality of original fisheye images, and obtaining a segmentation map and ground contact boundary point information corresponding to each original fisheye image;

[0010] Perform distortion correction on the segmentation map and touchdown boundary point information corresponding to each original fisheye image, project the dedistorted segmentation map and touchdown boundary points to the BEV view, and perform panoramic stitching;

[0011] Using a preset Gaussian probability model, weighted fusion is performed on the drivable area detection results at multiple historical moments, the BEV view is updated, and a BEV drivable area map is obtained;

[0012] The BEV drivable area map is outputted, and the ground contact boundary of the drivable area is marked.

[0013] Wherein, using a pre-trained deep learning model to identify the ground contact boundary points of the plurality of original fisheye images to obtain a segmentation map and ground contact boundary point information corresponding to each original fisheye image further includes:

[0014] In the trained deep learning model, a cascaded residual block network is used to extract image features from multiple input original fisheye images;

[0015] According to the extracted image features, semantic segmentation, boundary extraction and ground contact boundary point detection of the drivable area are performed to obtain the segmentation processing result;

[0016] The segmentation processing results are fused to obtain a segmentation map corresponding to each original fisheye image and ground contact boundary point information.

[0017] Among them, according to the extracted image features, semantic segmentation, boundary extraction and ground contact boundary point detection processing of the drivable area are performed to obtain the segmentation processing result, further including:

[0018] The first branch network is used to perform semantic segmentation of the drivable area of ​​each original fisheye image, and the pixels of each fisheye image are divided into two categories: drivable and non-drivable.

[0019] The second branch network is used to extract the boundary of the drivable area of ​​each original fisheye image, and the pixel points of the drivable area are divided into two categories: boundary points and non-boundary points;

[0020] The third branch network is used to extract the boundary points of the drivable area extracted by the second branch network again, and the boundary points are divided into two categories: ground contact boundary points and non-ground contact boundary points.

[0021] Among them: the semantic segmentation loss function used by the first branch network is:

[0022] L loss1 =w 11 L cross_entropy +w 21 L ohem_cross_entropy

[0023] Among them, L cross_entropy is the conventional cross entropy loss function, L ohem_cross_entropy It is the cross entropy loss function with the OHEM parameter added, w 11 and w 21 is the weight ratio of the two;

[0024] The loss function used by the second branch network is:

[0025] L loss2 =L binary_cross_entropy ;

[0026] The loss function used by the third branch network is:

[0027] L loss3 =L binary_cross_entropy ;

[0028] Among them, L binary_cross_entropy is the binary cross entropy loss function; L loss2 and L loss3 Weight of positive sample pos_index And the weight of negative samples neg_index The calculation formula is as follows:

[0029]

[0030]

[0031] Among them, pos_num is the number of positive samples in the entire image pixel, neg_num is the number of negative samples in the entire image pixel; loss2 The positive samples are boundary points, and the negative samples are non-boundary points. loss3 The positive samples are the ground-touching boundary points, and the negative samples are the non-ground-touching boundary points;

[0032] The total loss function of the deep learning model is determined by the following formula:

[0033] loss=w1loss1+w2loss2+w3loss3

[0034] Among them, w1, w2, and w3 are the weights of the three loss functions, which are used to adjust the proportion of each branch, among which w3 has the largest weight value.

[0035] Among them, the segmentation map and the ground contact boundary point information corresponding to each original fisheye image are subjected to distortion correction processing, and the dedistorted segmentation map and the ground contact boundary point are projected to the BEV view for panoramic stitching, further comprising:

[0036] Using a pre-calibrated distortion correction table, multiple fisheye image segmentation maps and ground contact boundary points are distorted to obtain dedistorted segmentation maps and ground contact boundary points;

[0037] Project the dedistorted segmentation map and the ground contact boundary points to the BEV perspective;

[0038] The overlapping areas of the projected images are weighted fused to stitch multiple segmented images into one BEV view.

[0039] The method further comprises: using a preset Gaussian probability model to perform weighted fusion on the drivable area detection results at multiple historical moments, updating the BEV view, and obtaining a BEV drivable area map.

[0040] In a preset Gaussian probability model, the BEV view formed by the splicing is converted into a probability map representing whether it is a touchdown boundary point;

[0041] According to the probability graph, a static Bayesian filter model is used to integrate the drivable area detection results of a fixed number of previous historical moments to generate a comprehensive posterior probability distribution;

[0042] A dynamic Bayesian filter HMM model is used to calculate the probability of evaluating the observation sequence using a forward algorithm, where the observation sequence is the BEV view obtained by detecting the touchdown boundary points of the drivable area at a fixed number of previous historical moments and the current moment;

[0043] The results of the static Bayesian filter model and the dynamic Bayesian filter HMM model are weightedly fused to obtain the current probability of the touchdown boundary point in the drivable area, and the BEV view is updated to obtain a BEV drivable area map.

[0044] Among them, further include:

[0045] Pre-acquire multiple groups of original fisheye images acquired by multiple fisheye cameras, and classify and annotate boundary points of the drivable area into ground contact boundary points and non-ground contact boundary points to obtain a training data set;

[0046] Establish a deep learning model for detecting the touchdown boundary points in the drivable area;

[0047] The deep learning model is trained according to the training data set to obtain a trained deep learning model.

[0048] Accordingly, another aspect of the present invention further provides a drivable area generation system based on touchdown boundary point detection, which at least comprises:

[0049] An original picture acquisition unit, used to acquire a plurality of original fisheye pictures taken by a plurality of vehicle-mounted fisheye cameras;

[0050] A segmentation and recognition unit, configured to use a pre-trained deep learning model to perform ground contact boundary point recognition on the plurality of original fisheye images, and obtain a segmentation map and ground contact boundary point information corresponding to each original fisheye image;

[0051] A projection stitching processing unit is used to perform distortion correction processing on the segmentation map and the ground contact boundary point information corresponding to each original fisheye image, and project the dedistorted segmentation map and the ground contact boundary point to the BEV view for panoramic stitching;

[0052] A probability fusion processing unit, configured to use a preset Gaussian probability model to perform weighted fusion on the drivable area detection results at multiple historical moments, update the BEV view, and obtain a BEV drivable area map;

[0053] The drivable area marking unit is used to output the BEV drivable area map and mark the ground contact boundary of the drivable area.

[0054] Wherein, the segmentation and identification unit further comprises:

[0055] A feature extraction unit is used to extract image features from multiple input original fisheye images using a cascaded residual block network in a trained deep learning model;

[0056] A segmentation detection unit is used to perform semantic segmentation, boundary extraction and ground contact boundary point detection processing of the drivable area according to the extracted image features to obtain a segmentation processing result;

[0057] The first fusion processing unit is used to perform fusion processing on the segmentation processing results to obtain the segmentation map and ground contact boundary point information corresponding to each original fisheye image.

[0058] Wherein, the segmentation detection unit further comprises:

[0059] A first segmentation unit is used to perform semantic segmentation of the drivable area on each original fisheye image using a first branch network, and divide the pixels of each fisheye image into two categories: drivable and non-drivable;

[0060] A first extraction unit is used to extract the boundary of the drivable area of ​​each original fisheye image using a second branch network, and divide the pixel points of the drivable area into two categories: boundary points and non-boundary points;

[0061] The second extraction unit is used to use the third branch network to re-extract the boundary points of the drivable area extracted by the second branch network, and divide the boundary points into two categories: ground contact boundary points and non-ground contact boundary points.

[0062] Among them: the semantic segmentation loss function used by the first branch network is:

[0063] L loss1 =w 11 L cross_entropy +w 21 L ohem_cross_entropy

[0064] Among them, L cross_entropy is the conventional cross entropy loss function, L ohem_cross_entropy It is the cross entropy loss function with the OHEM parameter added, w 11 and w 21 is the weight ratio of the two;

[0065] The loss function used by the second branch network is:

[0066] L loss2 =L binary_cross_entropy ;

[0067] The loss function used by the third branch network is:

[0068] L loss3 =L binary_cross_entropy ;

[0069] Among them, L binary_cross_entropy is the binary cross entropy loss function; L loss2 and L loss3 Weight of positive sample pos_index And the weight of negative samples neg_index The calculation formula is as follows:

[0070]

[0071]

[0072] Among them, pos_num is the number of positive samples in the entire image pixel, neg_num is the number of negative samples in the entire image pixel; loss2 The positive samples are boundary points, and the negative samples are non-boundary points. loss3 The positive samples are the ground-touching boundary points, and the negative samples are the non-ground-touching boundary points;

[0073] The total loss function of the deep learning model is determined by the following formula:

[0074] loss=w1loss1+w2loss2+w3loss3

[0075] Among them, w1, w2, and w3 are the weights of the three loss functions, which are used to adjust the proportion of each branch, among which w3 has the largest weight value.

[0076] Wherein, the projection stitching processing unit further comprises:

[0077] A distortion correction processing unit, used to perform distortion correction on multiple fisheye image segmentation maps and ground contact boundary points using a pre-calibrated distortion correction table to obtain dedistorted segmentation maps and ground contact boundary points;

[0078] A projection unit, used for projecting the dedistorted segmentation map and the ground contact boundary point to the BEV perspective;

[0079] The stitching unit is used to perform weighted fusion on the overlapping areas of the projected images and stitch multiple segmented images into a BEV view.

[0080] Wherein, the probability fusion processing unit further includes:

[0081] A probability map acquisition unit, used for converting the spliced ​​BEV view into a probability map representing whether it is a touchdown boundary point in a preset Gaussian probability model;

[0082] A static filtering processing unit, for integrating the drivable area detection results of a fixed number of previous historical moments using a static Bayesian filtering model according to the probability map to generate a comprehensive posterior probability distribution;

[0083] A dynamic filtering processing unit, for calculating the probability of evaluating an observation sequence using a dynamic Bayesian filtering HMM model using a forward algorithm, wherein the observation sequence is a BEV view obtained by detecting the touchdown boundary points of the drivable area at a fixed number of previous historical moments and the current moment;

[0084] The drivable area map acquisition unit is used to perform weighted fusion on the results of the static Bayesian filter model and the dynamic Bayesian filter HMM model to obtain the current probability of the touchdown boundary point in the drivable area, and to update the BEV view to obtain the BEV drivable area map.

[0085] Among them, further include:

[0086] A data set acquisition unit is used to pre-acquire multiple groups of original fisheye images acquired by multiple fisheye cameras, and classify and mark the boundary points of the drivable area as ground contact boundary points and non-ground contact boundary points to acquire a training data set;

[0087] The model building training unit is used to establish a deep learning model for detecting the touchdown boundary points of the drivable area; and train the deep learning model according to the training data set to obtain a trained deep learning model.

[0088] As another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the aforementioned method are implemented.

[0089] As another aspect of the present invention, a computing device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the aforementioned method when executing the computer program.

[0090] As another vehicle of the present invention, at least an autonomous driving planning device is provided thereon, and the aforementioned system is deployed in the autonomous driving planning device.

[0091] The implementation of the embodiments of the present invention has the following beneficial effects:

[0092] The present invention provides a method, system, device and vehicle for generating a drivable area based on touchdown boundary point detection. By adopting a deep learning model for touchdown boundary point detection, the touchdown boundary points and non-touchdown boundary points are identified by detecting the boundary points of the drivable area of ​​multiple (such as 4) surround fisheye camera images, and then the boundary touchdown points in the drivable area of ​​the multiple surround fisheye images are dedistorted, and the camera's internal and external calibration parameters are used to establish a geometric model and project it under the BEV view. Under the BEV view, a Gaussian probability model is constructed for the drivable area detection of a fixed number of historical moments to generate a drivable area BEV map, in which the drivable area is marked.

[0093] The embodiment of the present invention uses deep learning to detect the touchdown boundary points of the drivable area, directly performs touchdown boundary point detection on multiple input original fisheye images, and includes obstacle contour line detection. It can accurately identify whether the boundary of the drivable area is a touchdown boundary point, avoiding the appearance of false boundaries and obstacle stretching in the drivable area under the projected BEV.

[0094] At the same time, by constructing Bayesian filtering and HMM probability models, the detection results of drivable areas at a fixed number of historical moments are weighted and fused to form a stable and reliable BEV map of drivable areas. This can be used for decision-making planning in automatic parking systems, which can improve the accuracy and safety of decision-making planning.

[0095] By implementing the embodiments of the present invention, the ground contact boundary point can be accurately detected and a stable drivable area map can be obtained, thereby improving the accuracy of drivable area detection and driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, without paying creative labor, other drawings obtained based on these drawings still belong to the scope of the present invention.

[0097] Figure 1 A schematic diagram of the main process of an embodiment of a method for generating a drivable area based on ground contact boundary point detection provided by the present invention;

[0098] Figure 2 It is a schematic diagram of the principle involved in the present invention;

[0099] Figure 3 for Figure 2 Schematic diagram of the working principle of the deep learning model;

[0100] Figure 4 for Figure 3 Schematic diagram of the structure of the pyramid pooling module;

[0101] Figure 5 for Figure 1 Schematic diagram of the process of updating the BEV map;

[0102] Figure 6 for Figure 5 Schematic diagram of the principle of the hidden Markov model HMM;

[0103] Figure 7 A schematic structural diagram of an embodiment of a drivable area generation system based on touchdown boundary point detection provided by the present invention;

[0104] Figure 8 yes Figure 7 Schematic diagram of the structure of the segmentation and recognition unit;

[0105] Fig. 9 for Figure 8 Schematic diagram of the structure of the segmentation detection unit;

[0106] Fig.10 for Figure 7 A schematic diagram of the structure of the projection splicing processing unit;

[0107] Fig.11 for Figure 7 Schematic diagram of the structure of the medium probability fusion processing unit. DETAILED DESCRIPTION

[0108] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings.

[0109] like Figure 1 FIG. 1 is a schematic diagram showing a main process of an embodiment of a method for generating a drivable area based on ground contact boundary point detection provided by the present invention. Figures 2 to 6 As shown, in this embodiment, the method at least includes the following steps:

[0110] Step S10, obtaining a plurality of original fisheye images taken by a plurality of vehicle-mounted fisheye cameras; a plurality of original fisheye images taken by different vehicle-mounted fisheye cameras in the same time series are used as a set of input data;

[0111] Step S11, using a pre-trained deep learning model to identify ground contact boundary points of the plurality of original fisheye images, to obtain a segmentation map and ground contact boundary point information corresponding to each original fisheye image;

[0112] In a specific example, the step S11 further includes:

[0113] Step S110, in the trained deep learning model, using a cascaded residual block network to extract image features from a set of multiple input original fisheye images;

[0114] In this embodiment, a cascaded residual cascade network structure (such as a ResNet residual structure) is used as a shared backbone network for fisheye images to realize image feature extraction; the image features may be feature information such as pixel grayscale values.

[0115] Step S111, performing semantic segmentation, boundary extraction, and ground contact boundary point detection processing on the drivable area according to the extracted image features to obtain a segmentation processing result;

[0116] Combination Figure 3 As shown, in this embodiment, the three segmentation or detection functions are implemented respectively through three branch networks.

[0117] Specifically, the step S111 further includes:

[0118] Step S1110, using the first branch network to perform semantic segmentation of the drivable area on each original fisheye image, and dividing the pixels of each fisheye image into two categories: drivable and non-drivable;

[0119] In a specific example, the first branch network uses three block modules, each of which contains three convolutional layers, and in each block module, the number of channels of the feature map is increased layer by layer, and the resolution of the feature map is changed by downsampling in a tride=2 manner. Such a design helps to extract rich feature representations and increase the receptive field of the network, thereby improving the network's expression ability and classification performance.

[0120] Specifically, each block module can contain the following hierarchy:

[0121] The first convolutional layer: This layer processes the input feature map using a convolution operation and applies a suitable activation function. Usually, this convolutional layer can use a smaller convolution kernel size to capture the local features of the input image.

[0122] The second convolution layer: This layer continues to perform convolution operations on the output of the previous layer and applies an activation function. Typically, this convolution layer can use a larger convolution kernel size to capture a wider range of features. For image classification tasks, this helps extract higher-level semantic features.

[0123] The third convolutional layer: This layer is also a convolutional layer, which is used to further enhance the expressive power of feature representation. Similar to the first two layers, this layer also applies appropriate activation functions.

[0124] In each block module, by adding a pooling layer between convolutional layers or using a convolution operation with stride = 2, the feature map can be downsampled to reduce the resolution of the feature map. While downsampling, the number of channels of the feature map is progressively increased layer by layer to capture more feature information and contextual relationships, improving the modeling ability of the input data.

[0125] More specifically, the semantic segmentation of the drivable area is performed using the first branch network. The pixels of the fisheye image are divided into two categories: drivable and non-drivable. The binary semantic segmentation mask is y∈{0,1} h×w×1 .

[0126] Among them: the semantic segmentation loss function used by the first branch network is:

[0127] L loss1 =w 11 L cross_entropy +w 21 L ohem_cross_entropy

[0128] Among them, L cross_entropy is the conventional cross entropy loss function, Lohem_cross_entropy It is the cross entropy loss function with the OHEM parameter added, w 11 and w 21 is the weight ratio of the two; it can be understood that L ohem_cross_entropy In order to solve the problem of imbalance between positive and negative samples, some hard examples are selected as training samples to improve the effect of network parameters. Hard examples are samples with diversity and high loss.

[0129] Step S1111, using the second branch network to extract the boundary of the drivable area for each original fisheye image, and dividing the pixel points of the drivable area into two categories: boundary points and non-boundary points;

[0130] In a specific example, the second branch network mainly extracts high-resolution feature maps to predict the boundary of the drivable area, using three block modules, each of which also contains three convolutional layers.

[0131] Unlike the first branch network, the second branch network operates with stride = 1 during the sampling process, which does not reduce the resolution of the feature map, but increases the number of channels of the feature map. The purpose of this is to retain high-resolution spatial detail features.

[0132] In addition, the second branch network also adds the feature map after upsampling operation with the corresponding block module of the first branch network. The purpose of this addition operation is to combine low-resolution high-level semantic information and high-resolution detail features to obtain a feature representation with both spatial details and semantic information.

[0133] Through this structure, the second branch network can extract high-resolution feature maps and combine them with the feature maps of the first branch network, which not only has high-resolution spatial detail features but also retains low-resolution high-level semantic information, thereby providing a more accurate and comprehensive feature representation for predicting the boundaries of the drivable area.

[0134] The second branch network is used to extract the boundary of the drivable area, and the pixels of the fisheye image are divided into two categories: boundary points and non-boundary points by binary classification of image segmentation mask. The binary semantic segmentation mask is y∈{0,1} h×w×1 .

[0135] The loss function used by the second branch network is:

[0136] L loss2 =L binary_cross_entropy ;

[0137] Among them, L binary_cross_entropy is the binary cross entropy loss function; Lloss2 Weight of positive sample pos_index And the weight of negative samples neg_index The calculation formula is as follows:

[0138]

[0139]

[0140] Among them, pos_num is the number of positive samples in the entire image pixel, neg_num is the number of negative samples in the entire image pixel; loss2 The positive samples are boundary points, and the negative samples are non-boundary points.

[0141] Step S1112, using the third branch network, re-extracting the boundary points of the drivable area extracted by the second branch network, and dividing the boundary points into two categories: ground contact boundary points and non-ground contact boundary points, wherein the ground contact boundary points refer to the boundary points of the drivable area in contact with the ground, and the non-ground contact boundary points refer to the boundary points of the drivable area that are not in contact with the ground.

[0142] The third branch network is based on the second branch network, and once again extracts the touchdown point boundary (positive sample pos) and the non-touchdown point boundary point (negative sample neg) from the extracted boundary points.

[0143] In a more specific example, the third branch network can share a network layer with the second branch network. While extracting and predicting the boundary, the extracted points on the boundary are further classified into two categories. The loss function of the third branch network is consistent with the loss function of the second branch network. For example, in one example, the loss function used by the third branch network is:

[0144] L loss3 =L binary_cross_entropy ;

[0145] Among them, L binary_cross_entropy is the binary cross entropy loss function; L loss3 Weight of positive sample pos_index And the weight of negative samples neg_index The calculation formula is also as follows:

[0146]

[0147]

[0148] Among them, pos_num is the number of positive samples in the entire image pixel, neg_num is the number of negative samples in the entire image pixel; loss3The positive samples are the ground-touching boundary points, and the negative samples are the non-ground-touching boundary points;

[0149] Step S1113, using the following formula to determine the overall loss function of the deep learning model:

[0150] loss=w1loss1+w2loss2+w3loss3

[0151] Among them, w1, w2, and w3 are the weights of the three loss functions, which are used to adjust the proportion of each branch, among which w3 has the largest weight value.

[0152] It can be understood that in the embodiment of the present invention, three branch networks are used for semantic segmentation of the drivable area, boundary extraction of the drivable area, and detection of the touchdown boundary points of the drivable area. These three branches are progressive, from low-level details to high-level rich semantic information, and extract spatial details through the large receptive field of contextual semantic relevance to extract precise boundary contours. The three are combined and boundary attention is used to guide the fusion of detail and context branches.

[0153] Step S112, performing fusion processing on the segmentation processing results to obtain a segmentation map and ground contact boundary point information corresponding to each original fisheye image.

[0154] In an embodiment of the present invention, by adopting a deep aggregation pyramid pooling module (DAPPM) as a semantic information extraction structure, it is possible to achieve fusion of features of more scales, increase the receptive field, and better integrate image semantic information.

[0155] More details of the DAPPM module can be found in Figure 4 The structural diagram shown in .

[0156] Step S12, performing distortion correction processing on the segmentation map and the ground contact boundary point information corresponding to each original fisheye image, projecting the dedistorted segmentation map and the ground contact boundary point to the BEV view, and performing panoramic stitching;

[0157] In a specific example, the step S12 further includes:

[0158] Step S120, using a pre-calibrated distortion correction table to perform distortion correction on the multiple fisheye image segmentation maps and the ground contact boundary points to obtain dedistorted segmentation maps and ground contact boundary points;

[0159] In a specific example, the camera intrinsic parameter matrix (DIM) and correction coefficients (K, D) are obtained by calibration through the checkerboard calibration method to generate a distortion correction table, and the dedistorted segmentation map and ground contact boundary points are obtained by the table lookup method to eliminate imaging distortion.

[0160] Step S121, projecting the dedistorted segmentation map and the ground contact boundary points to the BEV perspective; specifically, based on the plane assumption, the homography matrix of the image to the ground is calibrated by using a calibration cloth method, and the geometric model is established using the internal and external calibration parameters of the camera, and the dedistorted segmentation map and the ground contact boundary points are projected to the BEV perspective;

[0161] Step S122: weighted fusion is performed on the overlapping areas of the projected images, and the multiple segmented images are spliced ​​into a BEV view.

[0162] Step S13, using a preset Gaussian probability model to perform weighted fusion on the drivable area detection results at multiple historical moments, updating the BEV view, and obtaining a BEV drivable area map;

[0163] It can be understood that in this embodiment, a Gaussian probability model is pre-constructed to perform weighted fusion on the drivable area detection results of the previous n historical moments under the BEV view to generate a BEV drivable area map.

[0164] In this embodiment, a Gaussian probability model is constructed by combining static Bayesian filtering with dynamic Bayesian filtering (i.e., Hidden Markov Model, HMM). It can be understood that the detection of the touchdown boundary point in the drivable area is trained by the model in the form of pixel-level binary classification of image segmentation, and the model predicts whether it is a touchdown boundary point is a binary classification probability problem; therefore, the aforementioned model can output a probability map, which is projected and spliced ​​from the probability maps under the four camera views into a probability map under the BEV view. Then, a Gaussian probability model is constructed for the drivable area detection results at n historical moments for weighted fusion.

[0165] In a specific example, the step S13 further includes:

[0166] Step S130, in a preset Gaussian probability model, converting the spliced ​​BEV view into a probability map representing whether it is a touchdown boundary point;

[0167] Step S131, according to the probability map, using a static Bayesian filter model to integrate the drivable area detection results of a fixed number of previous historical moments to generate a comprehensive posterior probability distribution;

[0168] Specifically, for static Bayesian filtering, the detection of the touchdown boundary point in the drivable area has two states: 0 and 1, representing the touchdown boundary point and the non-touchdown boundary point respectively; the input of the static Bayesian filtering module is the posterior probability of the first observation and the prior probability after initialization, and its output is the posterior probability after n observations. The general calculation formula is as follows:

[0169]

[0170] In the embodiment, the observed posterior probability of the Bayesian filter refers to the probability map of the touchdown boundary point detection in the drivable area under the BEV view, and the initialized prior probability is set to 0.5, that is,

[0171] Where p(x)=0.5.

[0172] Step S132, using a dynamic Bayesian filter HMM model and a forward algorithm to calculate the probability of evaluating an observation sequence, wherein the observation sequence is a BEV view obtained by detecting the touchdown boundary points of the drivable area at a fixed number of previous historical moments and the current moment;

[0173] Specifically, for a dynamic Bayesian filter HMM, i.e. a hidden Markov model, it can evaluate the probability of an observation sequence.

[0174] In this embodiment, first, given the HMM model, its input is λ=(A,B,Π) and the observation sequence O={o1,o2,...,o T}, the output refers to the probability P(O|λ) of the observation sequence O occurring under the model λ.

[0175] For the HMM model of the detection of the touchdown boundary point of the drivable area, the observation sequence O refers to the BEV view of the touchdown boundary point detection of the drivable area at the previous n historical moments and the current moment. Since the detection of the touchdown boundary point of the drivable area has only two states, 0 and 1, the Π in the model λ represents the initial state probability distribution, its dimension is a 2×1 matrix, set to (0.5, 0.5), A represents the state transition probability distribution matrix, its dimension is a 2×2 matrix, set to (0.7, 0.3, 1e-10, 1-1e-10), and B represents the BEV probability map of the touchdown boundary point detection of the drivable area at the previous n historical moments. Figure 6 A schematic diagram showing the principle of the HMM model.

[0176] In this embodiment, the forward algorithm is used to solve the probability of the HMM observation sequence. In the forward algorithm, the local state of the dynamic programming is defined by defining the "forward probability". The hidden state at time t is defined as q i , observation states o1,o2,...,o tThe probability of is the forward probability, which is a t (i), the specific calculation formula is as follows:

[0177]

[0178] Step S133, weighted fusion is performed on the results of the static Bayesian filter model and the dynamic Bayesian filter HMM model to obtain the current probability of the touchdown boundary point in the drivable area, and the BEV view is updated to obtain a BEV drivable area map.

[0179] In a specific example, the probability model for detecting the touchdown boundary point in the drivable area of ​​a BEV is mainly a combination of static Bayesian filtering and hidden Markov model, namely:

[0180] P BEV =w4P bayes +w5P hmm

[0181] Among them, w4 and w5 are the corresponding weighted coefficients.

[0182] It is understandable that, combined with Figure 5 As shown, the process of generating a BEV map for detecting the touchdown boundary points of the drivable area specifically includes the following steps: first, the BEV map is initialized, then the drivable area is received, it is determined whether the boundary points are extracted, the probability map of the touchdown boundary points of the drivable area under the BEV is calculated, and finally the BEV map is updated to obtain the BEV drivable area map.

[0183] Step S14, outputting the BEV drivable area map and marking the ground contact boundary of the drivable area.

[0184] Specifically, in the BEV drivable area map, the touchdown boundary points constitute a non-closed obstacle object contour line, wherein the non-drivable area is mainly an obstacle contour line composed of the touchdown boundary points; and the rest are drivable areas.

[0185] Among them, further include:

[0186] Step S20, pre-acquire multiple groups of original fisheye images acquired by multiple fisheye cameras, and classify and mark the boundary points of the drivable area as ground contact boundary points and non-ground contact boundary points to acquire a training data set;

[0187] Step S21, establishing a deep learning model for detecting the touchdown boundary points of the drivable area;

[0188] Step S22: train the deep learning model according to the training data set to obtain a trained deep learning model.

[0189] It can be understood that in this embodiment, by adopting a deep learning model for touchdown boundary point detection, the touchdown boundary points and non-touchdown boundary points are detected for the drivable area boundary points of the four surround fisheye camera images, and then the boundary touchdown points in the drivable area of ​​the four surround fisheye images are dedistorted, and the geometric model is established using the internal and external calibration parameters of the camera and projected to the BEV view. A Gaussian probability model is constructed for the drivable area detection at multiple historical moments in the BEV view to generate a drivable area BEV map. The drivable area in the BEV view is not closed and is represented by contour lines.

[0190] In the implementation of the embodiment of the present invention, a deep learning method for detecting the touchdown boundary point of the drivable area is adopted to identify whether the boundary of the drivable area in the four fisheye camera images is the touchdown point, thereby avoiding the occurrence of false boundaries and obstacle stretching in the drivable area under the projected BEV.

[0191] In some application scenarios, such as in an automatic parking environment, where the downstream decision-making and planning module does not need to know whether the vehicle is drivable in an instantaneous state, and an obstacle appears at one moment and disappears at the next moment, a stable and reliable BEV drivable area map is very important. In this embodiment, by constructing a Bayesian filter and an HMM probability model, the drivable area detection results at multiple historical moments are weighted and fused to form a stable and reliable BEV drivable area map, thereby improving the safety of decision-making and planning such as automatic parking systems.

[0192] By implementing the embodiments of the present invention, the ground contact boundary point can be accurately detected and a stable drivable area map can be obtained, thereby improving the accuracy of drivable area detection and driving safety.

[0193] like Figure 7 FIG. 1 is a schematic diagram showing a structure of an embodiment of a drivable area generation system based on ground contact boundary point detection provided by the present invention. Figures 8 to 11 As shown, in this embodiment, the system 1 at least includes:

[0194] The original picture acquisition unit 10 is used to acquire a plurality of original fisheye pictures taken by a plurality of vehicle-mounted fisheye cameras;

[0195] The segmentation and recognition unit 11 is used to use a pre-trained deep learning model to identify the ground contact boundary points of the plurality of original fisheye images, and obtain a segmentation map and ground contact boundary point information corresponding to each original fisheye image;

[0196] The projection stitching processing unit 12 is used to perform distortion correction processing on the segmentation map and the ground contact boundary point information corresponding to each original fisheye image, and project the dedistorted segmentation map and the ground contact boundary point to the BEV view for panoramic stitching;

[0197] A probability fusion processing unit 13 is used to use a preset Gaussian probability model to perform weighted fusion on the drivable area detection results at multiple historical moments, update the BEV view, and obtain a BEV drivable area map;

[0198] The drivable area marking unit 14 is used to output the BEV drivable area map and mark the ground contact boundary of the drivable area.

[0199] like Figure 8 As shown, in a specific example, the segmentation and identification unit 11 further includes:

[0200] A feature extraction unit 110 is used to extract image features from multiple input original fisheye images using a cascaded residual block network in a trained deep learning model;

[0201] The segmentation detection unit 111 is used to perform semantic segmentation, boundary extraction and ground contact boundary point detection processing of the drivable area according to the extracted image features to obtain a segmentation processing result;

[0202] The first fusion processing unit 112 performs fusion processing on the segmentation processing results to obtain a segmentation map and ground contact boundary point information corresponding to each original fisheye image.

[0203] like Fig. 9 As shown, in a specific example, the segmentation detection unit 111 further includes:

[0204] The first segmentation unit 113 is used to perform semantic segmentation of the drivable area on each original fisheye image using the first branch network, and divide the pixels of each fisheye image into two categories: drivable and non-drivable;

[0205] The first extraction unit 114 is used to extract the boundary of the drivable area from each original fisheye image using the second branch network, and divide the pixel points of the drivable area into two categories: boundary points and non-boundary points;

[0206] The second extraction unit 115 is used to use the third branch network to re-extract the boundary points of the drivable area extracted by the second branch network, and divide the boundary points into two categories: ground contact boundary points and non-ground contact boundary points.

[0207] Among them: the semantic segmentation loss function used by the first branch network is:

[0208] L loss1 =w 11 L cross_entropy +w 21 L ohem_cross_entropy

[0209] Among them, L cross_entropy is the conventional cross entropy loss function, L ohem_cross_entropy It is the cross entropy loss function with the OHEM parameter added, w 11 and w 21 is the weight ratio of the two;

[0210] The loss function used by the second branch network is:

[0211] L loss2 =L binary_cross_entropy ;

[0212] The loss function used by the third branch network is:

[0213] L loss3 =L binary_cross_entropy ;

[0214] Among them, L binary_cross_entropy is the binary cross entropy loss function; L loss2 and L loss3 Weight of positive sample pos_index And the weight of negative samples neg_index The calculation formula is as follows:

[0215]

[0216]

[0217] Among them, pos_num is the number of positive samples in the entire image pixel, neg_num is the number of negative samples in the entire image pixel; loss2 The positive samples are boundary points, and the negative samples are non-boundary points. loss3 The positive samples are the ground-touching boundary points, and the negative samples are the non-ground-touching boundary points;

[0218] The total loss function of the deep learning model is determined by the following formula:

[0219] loss=w1loss1+w2loss2+w3loss3

[0220] Among them, w1, w2, and w3 are the weights of the three loss functions, which are used to adjust the proportion of each branch, among which w3 has the largest weight value.

[0221] like Fig.10 As shown, in a specific example, the projection stitching processing unit 12 further includes:

[0222] The distortion correction processing unit 120 is used to perform distortion correction on the multiple fisheye image segmentation maps and the ground contact boundary points using a pre-calibrated distortion correction table to obtain dedistorted segmentation maps and ground contact boundary points;

[0223] A projection unit 121, configured to project the dedistorted segmentation map and the ground contact boundary point to a BEV viewing angle;

[0224] The stitching unit 122 is used to perform weighted fusion on the overlapping areas of the projected images and stitch the multiple segmented images into a BEV view.

[0225] like Fig.11 As shown, in a specific example, the probability fusion processing unit 13 further includes:

[0226] A probability map acquisition unit 130 is used to convert the BEV view formed by splicing into a probability map representing whether it is a touchdown boundary point in a preset Gaussian probability model;

[0227] A static filtering processing unit 131 is used to integrate the drivable area detection results of a fixed number of previous historical moments according to the probability map using a static Bayesian filtering model to generate a comprehensive posterior probability distribution;

[0228] A dynamic filtering processing unit 132, configured to use a dynamic Bayesian filtering HMM model to calculate the probability of evaluating an observation sequence using a forward algorithm, wherein the observation sequence is a BEV view obtained by detecting the touchdown boundary points of the drivable area at a fixed number of previous historical moments and the current moment;

[0229] The drivable area map acquisition unit 133 is used to perform weighted fusion on the results of the static Bayesian filter model and the dynamic Bayesian filter HMM model to obtain the current probability of the touchdown boundary point in the drivable area, and to update the BEV view to obtain the BEV drivable area map.

[0230] More specifically, the system 1 further comprises:

[0231] A data set acquisition unit is used to pre-acquire multiple groups of original fisheye images acquired by multiple fisheye cameras, and classify and mark the boundary points of the drivable area as ground contact boundary points and non-ground contact boundary points to acquire a training data set;

[0232] The model building training unit is used to establish a deep learning model for detecting the touchdown boundary points of the drivable area; and train the deep learning model according to the training data set to obtain a trained deep learning model.

[0233] As another aspect of the present invention, a computer-readable storage medium is further provided, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned Figures 1 to 6 For more details, please refer to and combine with the above Figures 1 to 6 The description is not repeated here.

[0234] As another aspect of the present invention, a computing device is also provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the above-mentioned Figures 1 to 6 For more details, please refer to and combine with the above Figures 1 to 6 The description is not repeated here.

[0235] As another aspect of the present invention, a vehicle is provided, on which at least an automatic driving planning device is provided, in which the automatic driving planning device is deployed Figures 7 to 11 For more details, please refer to and combine the above Figures 7 to 11 The description is not repeated here.

[0236] The implementation of the embodiments of the present invention has the following beneficial effects:

[0237] The present invention provides a method, system, device and vehicle for generating a drivable area based on touchdown boundary point detection. By adopting a deep learning model for touchdown boundary point detection, the touchdown boundary points and non-touchdown boundary points are identified by detecting the boundary points of the drivable area of ​​multiple (such as 4) surround fisheye camera images, and then the boundary touchdown points in the drivable area of ​​the multiple surround fisheye images are dedistorted, and the camera's internal and external calibration parameters are used to establish a geometric model and project it under the BEV view. Under the BEV view, a Gaussian probability model is constructed for the drivable area detection of a fixed number of historical moments to generate a drivable area BEV map, in which the drivable area is marked.

[0238] The embodiment of the present invention uses deep learning to detect the touchdown boundary points of the drivable area, directly performs touchdown boundary point detection on multiple input original fisheye images, and includes obstacle contour line detection. It can accurately identify whether the boundary of the drivable area is a touchdown boundary point, avoiding the appearance of false boundaries and obstacle stretching in the drivable area under the projected BEV.

[0239] At the same time, by constructing Bayesian filtering and HMM probability models, the detection results of drivable areas at a fixed number of historical moments are weighted and fused to form a stable and reliable BEV map of drivable areas. This can be used for decision-making planning in automatic parking systems, which can improve the accuracy and safety of decision-making planning.

[0240] By implementing the embodiments of the present invention, the ground contact boundary point can be accurately detected and a stable drivable area map can be obtained, thereby improving the accuracy of drivable area detection and driving safety.

[0241] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0242] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0243] The above disclosure is only a preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A method for generating a drivable area based on ground contact boundary point detection, characterized in that: At least the following steps are included: Obtain multiple original fisheye images taken by multiple vehicle-mounted fisheye cameras; Using a pre-trained deep learning model to recognize the multiple original fisheye images, and obtaining a segmentation map and ground contact boundary point information corresponding to each original fisheye image; Perform distortion correction on the segmentation map and touchdown boundary point information corresponding to each original fisheye image, project the dedistorted segmentation map and touchdown boundary point into the BEV view and splice them; Using a preset Gaussian probability model, weighted fusion is performed on the drivable area detection results at multiple historical moments, the BEV view is updated, and a BEV drivable area map is obtained; The BEV drivable area map is outputted, and the ground contact boundary of the drivable area is marked.

2. The method according to claim 1, characterized in that Using a pre-trained deep learning model to identify the multiple original fisheye images to obtain a segmentation map and ground contact boundary point information corresponding to each original fisheye image further includes: In the trained deep learning model, a cascaded residual block network is used to extract image features from multiple input original fisheye images; According to the extracted image features, semantic segmentation, boundary extraction and ground contact boundary point detection of the drivable area are performed to obtain the segmentation processing result; The segmentation processing results are fused to obtain a segmentation map corresponding to each original fisheye image and ground contact boundary point information.

3. The method according to claim 2, characterized in that According to the extracted image features, semantic segmentation, boundary extraction and ground contact boundary point detection of the drivable area are performed to obtain the segmentation processing result, further including: The first branch network is used to perform semantic segmentation of the drivable area of ​​each original fisheye image, and the pixels of each fisheye image are divided into two categories: drivable and non-drivable. The second branch network is used to extract the boundary of the drivable area of ​​each original fisheye image, and the pixel points of the drivable area are divided into two categories: boundary points and non-boundary points; The third branch network is used to extract the boundary points of the drivable area extracted by the second branch network again, and the boundary points are divided into two categories: ground contact boundary points and non-ground contact boundary points.

4. The method according to claim 3, characterized in that: in: The semantic segmentation loss function used by the first branch network is: L loss1 =w 11 L cross_entropy +w 21 L ohem_cross_entropy Among them, L cross_entropy is the conventional cross entropy loss function, L ohem_cross_entropy It is the cross entropy loss function with the OHEM parameter added, w 11 and w 21 is the weight ratio of the two; The loss function used by the second branch network is: L loss2 =L binary_cross_entropy ; The loss function used by the third branch network is: L loss3 =L binary_cross_entropy ; Among them, L binary_cross_entropy is the binary cross entropy loss function; L loss2 and L loss3 Weight of positive sample pos_index And the weight of negative samples neg_index The calculation formula is as follows: Among them, pos_num is the number of positive samples in the entire image pixel, neg_num is the number of negative samples in the entire image pixel; loss2 The positive samples are boundary points, and the negative samples are non-boundary points. loss3 The positive samples are the ground-touching boundary points, and the negative samples are the non-ground-touching boundary points; The total loss function of the deep learning model is determined by the following formula: loss=w1loss1+w2loss2+w3loss3 Among them, w1, w2, and w3 are the weights of the three loss functions, which are used to adjust the proportion of each branch, among which w3 has the largest weight value.

5. The method according to claim 4, characterized in that The segmentation map and the ground contact boundary point information corresponding to each original fisheye image are subjected to distortion correction processing, and the dedistorted segmentation map and the ground contact boundary point are projected to the BEV view and spliced, further comprising: Using a pre-calibrated distortion correction table, multiple fisheye image segmentation maps and ground contact boundary points are distorted to obtain dedistorted segmentation maps and ground contact boundary points; Project the dedistorted segmentation map and the ground contact boundary points to the BEV perspective; The overlapping areas of the projected images are weighted fused to stitch multiple segmented images into one BEV view.

6. The method according to claim 5, characterized in that Using a preset Gaussian probability model, weighted fusion of the drivable area detection results at multiple historical moments is performed, the BEV view is updated, and a BEV drivable area map is obtained, further comprising: In the preset Gaussian probability model, the BEV view formed by the splicing is converted into a probability map representing whether it is a touchdown boundary point; According to the probability graph, a static Bayesian filter model is used to integrate the drivable area detection results of a fixed number of historical moments before to generate a comprehensive posterior probability distribution; A dynamic Bayesian filter HMM model is used to calculate the probability of evaluating the observation sequence using a forward algorithm, where the observation sequence is the BEV view obtained by detecting the touchdown boundary points of the drivable area at a fixed number of previous historical moments and the current moment; The results of the static Bayesian filter model and the dynamic Bayesian filter HMM model are weightedly fused to obtain the current probability of the touchdown boundary point in the drivable area, and the BEV view is updated to obtain a BEV drivable area map.

7. The method according to any one of claims 1 to 6, characterized in that: Further including: A plurality of groups of original fisheye images obtained by using a plurality of fisheye cameras are obtained in advance, and the boundary points of the drivable area are classified and labeled as ground contact boundary points and non-ground contact boundary points to obtain a training data set; Establish a deep learning model for detecting the touchdown boundary points in the drivable area; The deep learning model is trained according to the training data set to obtain a trained deep learning model.

8. A drivable area generation system based on touchdown boundary point detection, characterized in that: At least: An original picture acquisition unit, used to acquire a plurality of original fisheye pictures taken by a plurality of vehicle-mounted fisheye cameras; A segmentation and recognition unit, configured to recognize the plurality of original fisheye images using a pre-trained deep learning model, and obtain a segmentation map and ground contact boundary point information corresponding to each original fisheye image; A projection stitching processing unit is used to perform distortion correction processing on the segmentation map and the ground contact boundary point information corresponding to each original fisheye image, and project the dedistorted segmentation map and the ground contact boundary point into the BEV view and stitch them together; A probability fusion processing unit, configured to use a preset Gaussian probability model to perform weighted fusion on the drivable area detection results at multiple historical moments, update the BEV view, and obtain a BEV drivable area map; The drivable area marking unit is used to output the BEV drivable area map and mark the ground contact boundary of the drivable area.

9. The system according to claim 8, characterized in that The segmentation and identification unit further comprises: A feature extraction unit, used to extract image features from multiple input original fisheye images using a cascaded residual block network in a trained deep learning model; A segmentation detection unit is used to perform semantic segmentation, boundary extraction and ground contact boundary point detection processing of the drivable area according to the extracted image features to obtain a segmentation processing result; The first fusion processing unit is used to perform fusion processing on the segmentation processing results to obtain the segmentation map and ground contact boundary point information corresponding to each original fisheye image.

10. The system according to claim 9, characterized in that The segmentation detection unit further comprises: A first segmentation unit is used to perform semantic segmentation of the drivable area on each original fisheye image using a first branch network, and divide the pixels of each fisheye image into two categories: drivable and non-drivable; The first extraction unit is used to extract the boundary of the drivable area of ​​each original fisheye image by using the second branch network, and divide the pixel points of the drivable area into two categories: boundary points and non-boundary points; The second extraction unit is used to use the third branch network to re-extract the boundary points of the drivable area extracted by the second branch network, and divide the boundary points into two categories: ground contact boundary points and non-ground contact boundary points.

11. The system according to claim 10, characterized in that: in: The semantic segmentation loss function used by the first branch network is: L loss1 =w 11 L cross_entropy +w 21 L ohem_cross_entropy Among them, L cross_entropy is the conventional cross entropy loss function, L ohem_cross_entropy It is the cross entropy loss function with the OHEM parameter added, w 11 and w 21 is the weight ratio of the two; The loss function used by the second branch network is: L loss2 =L binary_cross_entropy ; The loss function used by the third branch network is: L loss3 =L binary_cross_entropy ; Among them, L binary_cross_entropy is the binary cross entropy loss function; L loss2 and L loss3 Weight of positive sample pos_index And the weight of negative samples neg_index The calculation formula is as follows: Among them, pos_num is the number of positive samples in the entire image pixel, neg_num is the number of negative samples in the entire image pixel; loss2 The positive samples are boundary points, and the negative samples are non-boundary points. loss3 The positive samples are the ground-touching boundary points, and the negative samples are the non-ground-touching boundary points; The total loss function of the deep learning model is determined by the following formula: loss=w1loss1+w2loss2+w3loss3 Among them, w1, w2, and w3 are the weights of the three loss functions, which are used to adjust the proportion of each branch, among which w3 has the largest weight value.

12. The system according to claim 11, characterized in that The projection stitching processing unit further comprises: A distortion correction processing unit, used to perform distortion correction on multiple fisheye image segmentation maps and ground contact boundary points using a pre-calibrated distortion correction table to obtain dedistorted segmentation maps and ground contact boundary points; A projection unit, used for projecting the dedistorted segmentation map and the ground contact boundary point to the BEV perspective; The stitching unit is used to perform weighted fusion on the overlapping areas of the projected images and stitch multiple segmented images into a BEV view.

13. The system of claim 12, wherein: The probability fusion processing unit further comprises: A probability map acquisition unit, used for converting the spliced ​​BEV view into a probability map representing whether it is a touchdown boundary point in a preset Gaussian probability model; A static filtering processing unit, for integrating the drivable area detection results of a fixed number of previous historical moments using a static Bayesian filtering model according to the probability map to generate a comprehensive posterior probability distribution; A dynamic filtering processing unit, for calculating the probability of evaluating an observation sequence using a dynamic Bayesian filtering HMM model using a forward algorithm, wherein the observation sequence is a BEV view obtained by detecting the touchdown boundary points of the drivable area at a fixed number of previous historical moments and the current moment; The drivable area map acquisition unit is used to perform weighted fusion on the results of the static Bayesian filter model and the dynamic Bayesian filter HMM model to obtain the current probability of the touchdown boundary point in the drivable area, and to update the BEV view to obtain the BEV drivable area map.

14. The system according to any one of claims 8 to 13, characterized in that Further including: A data set acquisition unit is used to pre-acquire multiple groups of original fisheye images acquired by multiple fisheye cameras, and classify and mark the boundary points of the drivable area as ground contact boundary points and non-ground contact boundary points to acquire a training data set; A model building training unit, used to build a deep learning model for detecting the touchdown boundary points of the drivable area; And according to the training data set, the deep learning model is trained to obtain a trained deep learning model.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

16. A computing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

17. A vehicle, on which at least an autonomous driving planning device is provided, wherein the system according to any one of claims 8 to 14 is deployed in the autonomous driving planning device.

Images