Training methods and devices for obstacle detection models; obstacle detection methods and devices

Abstract

This disclosure provides a training method and apparatus for an obstacle detection model, as well as an obstacle detection method and apparatus. The training method includes: acquiring sample data, including sample images of a target area captured by a fisheye camera and sample labels; inputting the sample images into an obstacle detection model, which sequentially passes through a first backbone network and a first neck network to obtain a first feature map; a visual converter, based on the first feature map, projects features representing different heights of the target area onto a two-dimensional BEV grid to obtain a BEV feature map; sequentially passing the BEV feature map through a second backbone network and a second neck network for feature extraction and fusion, respectively, to output a second feature map; inputting the second feature map into a prediction head to obtain a prediction heatmap, prediction occupancy information of the grid, and the predicted obstacle height corresponding to each grid; and adjusting the parameters of the obstacle detection model based on the prediction heatmap, the predicted obstacle height corresponding to each grid, and the sample labels.

Description

Technical Field

[0001] This disclosure relates to the field of driver assistance technology, specifically to a training method and apparatus for an obstacle detection model, and an obstacle detection method and apparatus. Background Technology

[0002] Automated parking technology, as an important component of autonomous driving technology, has become one of the key indicators of modern automotive intelligence. During automated parking, the vehicle's controller must accurately identify obstacles around the vehicle to ensure the safety and efficiency of the parking process.

[0003] In complex environments such as underground parking lots, automated parking technology faces multiple challenges. For example, underground parking lots contain many difficult-to-identify obstacles, which are often irregular obstacles with small ground contact areas and large suspended areas, or completely suspended obstacles, such as barriers, suspended pipes, and ground locks. These can all affect the automated parking system's ability to identify and handle obstacles. Summary of the Invention

[0004] In view of this, the present disclosure provides a training method and apparatus for an obstacle detection model, as well as an obstacle detection method and apparatus.

[0005] Firstly, a training method for an obstacle detection model is provided. The obstacle detection model includes: a first backbone network, a first neck network, a visual transducer, a second backbone network, a second neck network, and a prediction head. The training method includes: acquiring sample data, including sample images of the target area captured by a fisheye camera and sample labels; inputting the sample images into the obstacle detection model, which are then processed sequentially through the first backbone network and the first neck network to obtain a first feature map, which includes depth information of the target area; the visual transducer, based on the first feature map, projects features representing different heights of the target area onto a two-dimensional BEV grid to obtain a BEV feature map; the BEV feature map is then processed sequentially through the second backbone network and the second neck network for feature extraction and fusion, respectively, to output a second feature map; the second feature map is input into the prediction head to obtain a prediction heatmap, prediction occupancy information of the grid, and the predicted obstacle height corresponding to each grid; and adjusting the parameters of the obstacle detection model based on the prediction heatmap, the predicted obstacle height corresponding to each grid, and the sample labels.

[0006] In some embodiments, the sample images include multiple sample images captured by multiple fisheye cameras at the same time, and the fields of view of at least two fisheye cameras overlap.

[0007] In some embodiments, a sample image is input into an obstacle detection model and sequentially passed through a first backbone network and a first neck network to obtain a first feature map. This includes: the first backbone network extracting features from multiple sample images, with each sample image outputting multiple sizes of first intermediate feature maps; and the first neck network fusing the multiple sizes of first intermediate feature maps corresponding to each sample image to obtain multiple first feature maps.

[0008] In some embodiments, the vision converter projects features representing different heights of the target region onto a two-dimensional BEV grid based on a first feature map to obtain a BEV feature map, including: obtaining a three-dimensional target region based on the first feature map, camera intrinsics, and depth information of the target region; projecting features of the three-dimensional target region at preset height intervals onto the two-dimensional BEV grid to obtain multiple intermediate BEV feature maps; and fusing the multiple intermediate BEV feature maps to obtain the BEV feature map.

[0009] In some embodiments, the BEV feature map is sequentially passed through a second backbone network and a second neck network for feature extraction and fusion, respectively, to output a second feature map. This includes: the second backbone network extracting features from the BEV feature map and outputting second intermediate feature maps of multiple sizes; and the second neck network fusing the second intermediate feature maps of multiple sizes to obtain the second feature map.

[0010] In some embodiments, the second feature map is input into the prediction head to obtain a prediction heatmap, prediction occupancy information of the grid, and the predicted obstacle height corresponding to each grid, including: the prediction head determines the prediction heatmap through convolution; based on the prediction heatmap, it determines whether each grid is occupied; for occupied grids, it performs layer-by-layer convolution at preset height intervals to find whether different height layers have features; the height of the highest layer with features is taken as the upper edge height of the obstacle in the grid; the height of the lowest layer with features is taken as the lower edge height of the obstacle in the grid; based on the upper edge height and the lower edge height, it determines the obstacle height corresponding to the grid.

[0011] In some embodiments, the sample label includes three-dimensional information of the obstacle. The parameters of the obstacle detection model are adjusted based on the predicted heatmap, the predicted obstacle height for each grid, and the sample label, including: obtaining a sample heatmap and the obstacle sample height for each grid based on the three-dimensional information of the obstacle; and adjusting the parameters of the obstacle detection model based on the difference between the sample heatmap and the predicted heatmap, the predicted obstacle height for each grid, and the obstacle sample height.

[0012] Secondly, an obstacle detection method is provided, comprising: acquiring a fisheye image; inputting the fisheye image into a trained obstacle detection model to obtain grid occupancy information and the obstacle height corresponding to each grid; wherein the obstacle detection model is trained based on the method described in the first aspect above.

[0013] Thirdly, a training device for an obstacle detection model is provided, the obstacle detection model including: a first backbone network, a first neck network, a visual converter, a second backbone network, a second neck network, and a prediction head. The training device includes: a first acquisition module configured to acquire sample data, including sample images of the target area captured by a fisheye camera and sample labels; a first extraction module configured to input the sample images into an obstacle detection model, sequentially passing them through a first backbone network and a first neck network to obtain a first feature map, the first feature map including depth information of the target area; a conversion module configured to input the first feature map into a visual converter, the visual converter projecting features representing different heights of the target area onto a two-dimensional BEV grid based on the first feature map to obtain a BEV feature map; a second extraction module configured to sequentially pass the BEV feature map through a second backbone network and a second neck network for feature extraction and fusion, respectively, and output a second feature map; a processing module configured to input the second feature map into a prediction head to obtain a prediction heatmap, prediction occupancy information of the grid, and the predicted obstacle height corresponding to each grid; and a training module configured to adjust the parameters of the obstacle detection model based on the prediction heatmap, the predicted obstacle height corresponding to each grid, and the sample labels.

[0014] Fourthly, an obstacle detection device is provided, comprising: a second acquisition module configured to acquire a fisheye image; and a prediction module configured to input the fisheye image into a trained obstacle detection model to obtain grid occupancy information and the obstacle height corresponding to each grid; wherein the obstacle detection model is trained based on the method described in the first aspect.

[0015] The obstacle detection model training method disclosed herein extracts first feature information, including obstacle depth information, from sample data. Features of different heights from this first feature information are projected into the BEV space to obtain multiple intermediate BEV feature maps corresponding to each height. These intermediate BEV feature maps are then concatenated to obtain a final BEV feature map. This method compresses the height information in the first feature information, reducing the computational load in subsequent processes and making it suitable for deployment on low-computing-power platforms. After predicting the occupancy information of each grid, the height information of the occupied grids can be restored. Based on the predicted obstacle height, the predicted obstacle height in the occupied grids can be further predicted, avoiding the problem of missing obstacle heights. Therefore, it can improve the detection accuracy for overhanging or irregular obstacles. Attached Figure Description

[0016] Figure 1 The diagram shown is a flowchart illustrating a training method for an obstacle detection model provided in an embodiment of this disclosure.

[0017] Figure 2 The diagram shows a flowchart of an embodiment of this disclosure, illustrating the process of inputting a sample image into an obstacle detection model and sequentially passing it through a first backbone network and a first neck network to obtain a first feature map.

[0018] Figure 3 The diagram shown is a flowchart illustrating the steps of a visual converter according to an embodiment of the present disclosure, which projects features representing different heights of a target region onto a two-dimensional BEV grid based on a first feature map to obtain a BEV feature map.

[0019] Figure 4 The diagram shows a front view of a three-dimensional target region and a schematic diagram of multiple BEV intermediate feature maps corresponding to the three-dimensional target region, according to an embodiment of this disclosure.

[0020] Figure 5 The diagram shown is a flowchart illustrating the steps of extracting and fusing BEV feature maps sequentially through a second backbone network and a second neck network, and outputting a second feature map, according to an embodiment of this disclosure.

[0021] Figure 6 The diagram shown is a flowchart illustrating the steps of inputting a second feature map into a prediction head to obtain a prediction heatmap, prediction occupancy information of the grid, and prediction height of obstacles corresponding to each grid, according to an embodiment of this disclosure.

[0022] Figure 7 The diagram shows a flowchart illustrating the steps of adjusting the parameters of an obstacle detection model based on a predicted heatmap, the predicted obstacle height for each grid, and sample labels, according to an embodiment of this disclosure.

[0023] Figure 8 The diagram shown is a flowchart of an obstacle detection method provided in an embodiment of this disclosure.

[0024] Figure 9 The diagram shown is a structural schematic of a training device for an obstacle detection model provided in an embodiment of this disclosure.

[0025] Figure 10 The diagram shown is a structural schematic of an obstacle detection device provided in an embodiment of this disclosure.

[0026] Figure 11 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0027] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0028] Common obstacles in underground parking lots include barriers, charging stations, and fire hydrants. Barriers typically consist of a main unit and a gate arm, with the arm mounted on one side of the main unit, making it a suspended obstacle. Fire hydrants are usually fixed to walls or pillars, with water pipes connected around them. Charging stations have a small ground contact area on their pillars, and charging cables are often haphazardly suspended from the pillars. Current obstacle detection methods cannot effectively identify these irregular obstacles with small ground contact areas or completely suspended obstacles, posing safety risks to the automated parking process.

[0029] Among related technologies, the Freespace detection method is a common approach for detecting drivable areas. This method uses a deep learning network to predict the boundary contour of the drivable area from the surrounding environmental image. The boundary contour is composed of pixels representing the nearest obstacle grounding points in each column of the environmental image. However, this method has some drawbacks. First, the boundary contour generated by this method lacks 3D positional information. Second, because it predicts the drivable area based on obstacle grounding points, it cannot effectively identify irregular obstacles with small grounding areas but large, ungrounded portions, or overhanging obstacles.

[0030] In addition, some vehicles use multiple fisheye cameras to acquire fisheye environmental images of the vehicle's surroundings from different angles. In related technologies, the vehicle controller uses an Around View Monitor (AVM) system to stitch together multiple fisheye images captured simultaneously by the cameras, obtaining an AVM bird's-eye view of the vehicle's surroundings. This AVM bird's-eye view is then used to predict obstacles in the surrounding environment, thus obtaining the boundary contours of the drivable area. However, this method results in a lack of height information for obstacles, making it even more difficult to identify irregular or overhanging obstacles.

[0031] If the environment is based on two-dimensional images, three-dimensional obstacle data is created in three-dimensional space before obstacle detection is performed. Although three-dimensional data is richer, it requires processing and prediction using three-dimensional convolution, which greatly increases the computational load and demands higher computing power.

[0032] To address the aforementioned technical problems, this disclosure provides a training method and apparatus for an obstacle detection model, as well as an obstacle detection method and apparatus, to improve the accuracy of identifying irregular or suspended obstacles, and is applicable to low-computing-power platforms. To facilitate understanding of the technical solutions provided in the embodiments of this disclosure, a detailed description will be given below in conjunction with the accompanying drawings.

[0033] Figure 1 The diagram shown is a flowchart illustrating a training method for an obstacle detection model according to an embodiment of this disclosure. The obstacle detection model includes: a first backbone network, a first neck network, a visual converter, a second backbone network, a second neck network, and a prediction head.

[0034] like Figure 1 As shown, the training method for the obstacle detection model provided in this embodiment includes the following steps.

[0035] S110, Obtain sample data.

[0036] The sample data includes sample images of the target area taken by a fisheye camera, as well as sample labels.

[0037] Fisheye cameras, with their extremely wide field of view, can comprehensively perceive the entire surrounding environment of a vehicle with a smaller number of cameras, bringing significant advantages to autonomous driving applications. The target region is the real-world environment corresponding to the sample image. The target region can include various types of obstacles, as well as background objects other than obstacles, such as the sky and ground. Sample labels include 3D information or sample heatmaps of obstacles in the target region and the obstacle sample height corresponding to each grid.

[0038] In this embodiment of the disclosure, the sample data can be collected and labeled by ourselves, or an open-source dataset, such as the Occ3D-nuSenses dataset, can be used.

[0039] S120, the sample image is input into the obstacle detection model, and the first feature map is obtained by passing through the first backbone network and the first neck network in sequence.

[0040] The sample image is input into the obstacle detection model. First, the sample image passes through a first backbone network, which is used to extract features from the sample image at multiple levels and scales. Next, the features extracted by the first backbone network are input into a first neck network, which is used to fuse the features extracted by the first backbone network to obtain a first feature map.

[0041] The first feature map includes depth information of the target region. This depth information represents the distance between each feature in the target region and the fisheye camera that captured the sample image. The target region includes obstacle features and background object features outside the obstacles; therefore, the first feature map includes depth information of both obstacles and background objects.

[0042] S130, the visual converter projects the features representing different heights of the target region onto a two-dimensional BEV grid based on the first feature map to obtain the BEV feature map.

[0043] The visual converter is used to convert features at different heights of the target region corresponding to the first feature map into a two-dimensional bird's-eye view (BEV).

[0044] The View Transformer uses Inverse Perspective Mapping (IPM) to project features at different heights of the target region onto a two-dimensional BEV mesh.

[0045] Based on the depth information of the target region, the height information of obstacle features and background object features in the first feature map can be determined. Based on the height information of obstacle features and background object features in the first feature map, the first feature map can be restored to a three-dimensional feature map corresponding to the target region.

[0046] In the 3D features corresponding to the target region, features representing different heights of the target region are projected onto a 2D BEV mesh according to their height coordinates, resulting in multiple intermediate BEV feature maps corresponding to each height. Then, these multiple intermediate BEV feature maps are fused to obtain the final BEV feature map. For example, the above projection process can employ the Inverse Perspective Mapping (IPM) method.

[0047] During this process, the height information in the three-dimensional features is compressed to obtain a two-dimensional BEV feature map.

[0048] S140, the BEV feature map is sequentially passed through the second backbone network and the second neck network for feature extraction and fusion, and the second feature map is output.

[0049] The second backbone network is used to extract features from the BEV feature map, and the second neck network is used to fuse the features extracted by the second backbone network to obtain the second feature map.

[0050] S150, input the second feature map into the prediction head to obtain the prediction heatmap, the predicted occupancy information of the grid, and the predicted obstacle height corresponding to each grid.

[0051] A predictive heatmap is essentially a probability distribution map. The value at each location in the heatmap represents the probability that that location is occupied. This probability of occupancy is used to predict grid occupancy information.

[0052] The grid prediction occupancy information includes whether each grid in the feature map formed by convolving the prediction head with the second feature map is occupied. For each grid, if a grid is occupied, it indicates that there is an obstacle in that grid; if a grid is not occupied, it indicates that there is no obstacle in that grid.

[0053] For occupied grids, the prediction head can further predict the height of obstacles in the grid, obtaining the predicted obstacle height for each grid.

[0054] Since the second feature map is two-dimensional, the prediction head uses lightweight two-dimensional convolution, which greatly reduces the amount of computation in the prediction process and is suitable for low-computing-power platforms.

[0055] Before inputting the second feature map into the prediction head, the second feature map can be upsampled first, and then the upsampled second feature map can be input into the prediction head to improve the resolution of the second feature map.

[0056] S160 adjusts the parameters of the obstacle detection model based on the predicted heatmap, the predicted obstacle height for each grid, and the sample labels.

[0057] There are two forms of sample labels. One form is the 3D information of obstacles in the target area, which can be used to calculate a sample heatmap and the obstacle sample height corresponding to each grid. The other form is to directly obtain the sample heatmap and the obstacle sample height corresponding to each grid from an open-source database. Regardless of which form of sample label is used, the parameters of the obstacle detection model are adjusted based on the difference between the sample heatmap and the predicted heatmap, the predicted obstacle height for each grid, and the obstacle sample height. The obstacle detection model is trained using the above training method until the training termination conditions are met. The training termination conditions include the heatmap loss, the obstacle height loss for each grid being less than a preset value, or the number of training iterations reaching a preset number, at which point the training process stops.

[0058] In this embodiment, a first feature map including depth information of obstacles is extracted from a sample image. Features of different heights in the target region corresponding to the first feature map are converted to a two-dimensional BEV mesh, resulting in multiple intermediate BEV feature maps corresponding to each height. These intermediate BEV feature maps are then fused to obtain a final BEV feature map, thus compressing the height information in the first feature map. When the prediction head uses the processed BEV feature map (second feature map) for prediction, compared to the prediction process using three-dimensional convolution to process three-dimensional spatial data, this embodiment significantly reduces the computational load of subsequent processes by compressing the three-dimensional space to a two-dimensional BEV mesh, making it suitable for deployment on low-computing-power platforms (such as low-computing-power chips).

[0059] Since the BEV feature map is obtained by fusing multiple intermediate BEV feature maps corresponding to different heights, after predicting the occupancy information of each grid, the height information of the occupied grid can be restored. Based on the obstacle height initially predicted by the first feature map, the predicted obstacle height in the occupied grid can be further predicted, avoiding the problem of missing obstacle heights. Therefore, it can improve the detection accuracy of overhanging or irregular obstacles.

[0060] In some embodiments, the sample images include multiple sample images acquired simultaneously by multiple fisheye cameras, and the fields of view of at least two fisheye cameras overlap. Because the fields of view of the fisheye cameras overlap, the target areas corresponding to the fisheye images captured by the fisheye cameras also overlap.

[0061] Since vehicles are typically equipped with multiple fisheye cameras to cover a 360-degree area around the vehicle, multiple sample images can be used to simulate the multiple fisheye images captured by these cameras during parking. Training an obstacle detection model based on these sample images can improve the model's ability to process multiple fisheye images.

[0062] The following section details the specific implementation of the obstacle detection model for processing multi-sample images.

[0063] Figure 2 The diagram shows a flowchart illustrating the process of inputting a sample image into an obstacle detection model, sequentially passing it through a first backbone network and a first neck network to obtain a first feature map, according to an embodiment of this disclosure. Figure 2 As shown, the process of inputting a sample image into an obstacle detection model and sequentially passing it through a first backbone network and a first neck network to obtain a first feature map includes the following steps.

[0064] S121, the first backbone network extracts features from multiple sample images, and each sample image outputs a first intermediate feature map of multiple sizes.

[0065] Multiple sample images are input into the first backbone network. For example, there can be four sample images, which correspond to the fisheye images captured by the fisheye cameras located on the front, rear, left, and right sides of the vehicle, respectively.

[0066] The first backbone network comprises a series of convolutional layers, pooling layers, and activation functions. The first backbone network extracts features from each of the multiple sample images, and each sample image outputs first intermediate feature maps of multiple sizes. For example, each sample image can output three first intermediate feature maps of different sizes, and the first backbone network can employ a DLANet34 network.

[0067] Since the multi-channel sample images were acquired by different fisheye cameras, preprocessing can be performed on the multi-channel sample images before feature extraction. Specific preprocessing steps may include: converting the multi-channel sample images to floating-point type, then resizing them to the same size, and normalizing them according to the number of channels.

[0068] S122, the first neck network fuses multiple first intermediate feature maps of multiple sizes corresponding to each sample image to obtain multiple first feature maps.

[0069] Next, the first intermediate feature maps of multiple sizes for each sample image are input into the first neck network. The first neck network fuses the first intermediate feature maps of multiple sizes corresponding to each sample image to obtain the first feature map of each sample image. For example, the first neck network can be an FPN_LSS network.

[0070] In this embodiment, feature extraction and fusion operations are performed on each sample image to obtain a first feature map corresponding to each image. First intermediate feature maps of different sizes contain semantic and detail information at different levels. Low-level features extracted at shallow layers typically include edges and textures of the sample image, while high-level features extracted at deeper layers typically include the shape and category of obstacles in the sample image. Therefore, obtaining multiple first feature maps from the first backbone network and the first neck network provides rich information for subsequent prediction processes.

[0071] Figure 3 The diagram illustrates a flowchart of a visual converter according to an embodiment of this disclosure, which projects features representing different heights of a target region onto a two-dimensional BEV mesh based on a first feature map to obtain a BEV feature map. Figure 3As shown, the visual converter projects features representing different heights of the target region onto a two-dimensional BEV grid based on the first feature map to obtain the BEV feature map. The steps include the following.

[0072] S131, based on the first feature map, camera intrinsic parameters and depth information of the target region, the three-dimensional target region is obtained.

[0073] The depth information of the target region includes the depth information of each feature in the target region. For example, it includes the depth information of each obstacle.

[0074] Based on camera intrinsic parameters and depth information of the target region, the target region corresponding to the two-dimensional first feature map can be transformed into a three-dimensional target region. The three-dimensional target region includes the three-dimensional features of obstacles.

[0075] For example, if the target area includes streetlights, then the three-dimensional target area includes the three-dimensional features of the streetlights.

[0076] S132, the features of the three-dimensional target region at preset height intervals are projected onto the two-dimensional BEV mesh to obtain multiple intermediate BEV feature maps.

[0077] The 3D target region is segmented along the height direction at preset height intervals to obtain multiple layers of 3D target sub-regions. Each 3D target sub-region corresponds to a portion of the 3D target region within a certain height range. During the segmentation process, features within that height range in the 3D target region are also segmented, and each 3D target sub-region includes slices of features belonging to that height range.

[0078] The upper or lower section of each three-dimensional target sub-region is projected onto a two-dimensional BEV mesh to obtain the BEV intermediate feature map corresponding to each three-dimensional target sub-region.

[0079] Figure 4 The diagram shows a front view of a three-dimensional target region provided in an embodiment of this disclosure, and a schematic diagram of multiple intermediate BEV feature maps corresponding to the three-dimensional target region. Figure 4 As shown in (a), continuing with the street lamp as an example, the three-dimensional target region includes the features of the street lamp 40. The three-dimensional target region is divided along the dividing lines (shown by dashed lines), with the distance between adjacent dividing lines being a preset height interval. In this way, the features of the street lamp 40 are also divided. The street lamp 40 includes lamp 41 and lamp post 42. The street lamp 40 is divided into multiple parts by dividing lines with heights H1, H2, ..., Hn. Next, the cross-sections of the three-dimensional target region at the heights H1, H2, ..., Hn are projected onto the two-dimensional BEV mesh to obtain the intermediate BEV feature maps corresponding to heights H1, H2, ..., Hn.

[0080] Take the intermediate feature maps of BEV corresponding to heights H1, H2, H3, and H4 as an example. Figure 4 (b) shows the intermediate feature map of BEV corresponding to height H1, as follows: Figure 4 As shown in (b), the features of lamp 41 occupy two grids, with coordinates (1,2) and (2,2) respectively; the features of lamp post 42 occupy one grid, with coordinates (3,2). Figure 4 (c) shows the intermediate feature map of BEV corresponding to height H2, such as Figure 4 As shown in (c), the features of lamp 41 occupy two grids, with coordinates (1,2) and (2,2) respectively; the features of lamp post 42 occupy one grid, with coordinates (3,2). Figure 4 (d) shows the intermediate feature map of BEV corresponding to height H3, as follows: Figure 4 As shown in (d), the feature of lamp 41 occupies one grid with coordinates (1,2); the feature of lamp post 42 occupies one grid with coordinates (3,2). Figure 4 (e) shows the intermediate feature map of the BEV corresponding to height H4, as follows: Figure 4 As shown in (e), the feature of lamp post 42 occupies a grid with coordinates (3,2).

[0081] In this way, the three-dimensional target region is segmented along the height direction, and the upper or lower section of each layer is projected onto the two-dimensional BEV mesh to obtain multiple intermediate BEV feature maps.

[0082] Understandably, the preset height interval can be set manually. A smaller preset height interval will result in more accurate height regression for each grid's corresponding obstacle, but it will also increase the computational load. This disclosure does not impose specific limitations on this; it can be set according to actual needs. Figure 4 The number of grids and grid size shown are merely illustrative and should not be construed as limiting this disclosure.

[0083] The above process can be implemented based on the inverse perspective mapping method, and the specific calculation method will not be elaborated here. The inverse perspective mapping method requires less computing power, thus reserving more computing power for subsequent prediction steps, making the obstacle detection model trained by the training method of this disclosure suitable for deployment on low-computing-power platforms.

[0084] S133, after fusing multiple intermediate BEV feature maps, a BEV feature map is obtained.

[0085] For multiple intermediate BEV feature maps, the multiple intermediate BEV feature maps are fused to obtain the BEV feature map.

[0086] The above describes the process of obtaining a three-dimensional target region from a single sample image by using the first feature map, camera intrinsics, and depth information of the target region. The features of the three-dimensional target region are then projected onto the two-dimensional BEV grid at preset height intervals to obtain multiple intermediate BEV feature maps. These multiple intermediate BEV feature maps are then fused to obtain the final BEV feature map.

[0087] For multiple sample images, a BEV feature map corresponding to each sample image can be obtained, i.e., multiple BEV feature maps can be obtained. Then, the multiple BEV feature maps can be fused, and the fused BEV feature map is input into the second backbone network for subsequent steps.

[0088] Using the above method, the first feature map is restored to a three-dimensional target region, and all features of the cross sections at different heights in the three-dimensional target region are projected onto a two-dimensional BEV mesh. This achieves compression of the three-dimensional target region in the height dimension, while retaining the features at different heights, making the subsequent prediction process more efficient and reducing the computational load of the prediction process. At the same time, the height prediction of obstacles is also more accurate.

[0089] Figure 5 The diagram illustrates a step in an embodiment of this disclosure where a BEV feature map is sequentially passed through a second backbone network and a second neck network for feature extraction and fusion, resulting in the output of a second feature map. Figure 5 As shown, the steps for extracting and fusing features from the BEV feature map by sequentially passing it through the second backbone network and the second neck network to output the second feature map include the following steps.

[0090] S141, the second backbone network extracts features from the BEV feature map and outputs second intermediate feature maps of multiple sizes.

[0091] The BEV feature map is input into the second backbone network. The second backbone network includes a series of convolutional layers, pooling layers, and activation functions. The second backbone network extracts features from the BEV feature map and outputs second intermediate feature maps of multiple sizes. For example, two second intermediate feature maps of different sizes can be output, and the second backbone network can be a ResNet network.

[0092] If the sample images include multiple sample images captured simultaneously by multiple fisheye cameras, then the BEV feature maps corresponding to each of the multiple sample images need to be fused into a unified BEV feature map, and then the unified BEV feature map is input into the second backbone network.

[0093] S142, the second neck network fuses the second intermediate feature maps of multiple sizes to obtain the second feature map.

[0094] Next, the second intermediate feature maps of multiple sizes are input into the second neck network. The second neck network fuses the second intermediate feature maps of multiple sizes to obtain the second feature map. For example, the second neck network can adopt an FPN_LSS network. The design features and implementation simplicity of the FPN_LSS network give it a significant advantage in terms of computational resource requirements, making it suitable for deployment on low-computing-power platforms.

[0095] In this embodiment of the disclosure, the BEV feature map is extracted by the second backbone network to obtain a second intermediate feature map of multiple sizes containing semantic and detailed information at different levels, which can provide rich information for the subsequent prediction process.

[0096] Figure 6 The diagram shown is a flowchart illustrating the steps of inputting a second feature map into a prediction head to obtain a prediction heatmap, prediction occupancy information of the grid, and the predicted obstacle height for each grid, according to an embodiment of this disclosure. Figure 6 As shown, the steps to input the second feature map into the prediction head to obtain the prediction heatmap, the predicted occupancy information of the grid, and the predicted obstacle height for each grid include the following steps.

[0097] S151, the prediction head determines the predicted heatmap through convolution.

[0098] The second feature map is input into the prediction head, which includes convolutional layers. These layers perform convolution operations on the second feature map to capture local correlations and hierarchical structures, and predict the probability of each location being occupied, ultimately generating a prediction heatmap. The values ​​in the prediction heatmap represent the probability of that location being occupied.

[0099] S152, based on the predicted heatmap, determines whether each grid is occupied.

[0100] In the predicted heatmap, if the probability of a location being occupied is greater than a preset probability threshold, then that location is determined to be occupied; if the probability of a location being occupied is less than the preset probability threshold, then that location is determined to be unoccupied. For each grid cell, if the proportion of occupied locations within the grid cell's area to the total grid area is greater than a preset occupancy threshold, then that grid cell is determined to be occupied; if the proportion of occupied locations within the grid cell's area to the total grid area is less than the preset occupancy threshold, then that grid cell is determined to be unoccupied.

[0101] This determines whether each grid cell is occupied. If a grid cell is occupied, it means there are obstacles in the grid cell; if a grid cell is not occupied, it means there are no obstacles in the grid cell, or the proportion of obstacles in the grid cell is small.

[0102] S153, for the occupied grid, perform layer-by-layer convolution at preset height intervals to find whether different height layers have features.

[0103] For each occupied grid cell, convolution is performed layer by layer at preset height intervals to find whether the corresponding position of the occupied grid cell has features at different height layers.

[0104] For example, first, the corresponding coordinates of the occupied grid are determined. Then, it is checked whether different height layers have features at that coordinate location. If features are present, it indicates that there is an obstacle at that coordinate location within that height layer; if no features are present, it indicates that there is no obstacle at that coordinate location within that height layer. In this way, the height of the obstacle located at that grid location can be determined.

[0105] S154, the height of the highest layer with features is taken as the upper edge height of the obstacle in the grid.

[0106] S155, the height of the lowest layer with characteristics is taken as the lower edge height of the obstacle in the grid.

[0107] S156, based on the top edge height and bottom edge height, determine the obstacle height corresponding to this grid.

[0108] The top and bottom heights of obstacles within an occupied grid can be determined. The height of the obstacle is then determined using these top and bottom heights. Specifically, during the process of identifying whether different height layers possess features, the highest height layer with features is identified, and its height is used as the top height of the obstacle within that grid. Similarly, the lowest height layer with features is identified, and its height is used as the bottom height of the obstacle within that grid.

[0109] Continue to refer to Figure 4 In the grid at coordinates (1,2), convolution is performed on layers of different heights from H1 to Hn, and each layer is checked to see if it has features. Through convolution, it can be seen that the highest layer with features is at height H1, and the lowest layer with features is at height H3. Therefore, in the grid at (1,2), the corresponding obstacle height is H1 minus H3. In the grid at coordinates (2,2), the highest layer with features is at height H1, and the lowest layer with features is at height H2. Therefore, in the grid at (2,2), the corresponding obstacle height is H1 minus H2. In the grid at coordinates (3,2), the highest layer with features is at height H1, and the lowest layer with features is at height Hn. Therefore, in the grid at (3,2), the corresponding obstacle height is H1 minus Hn.

[0110] In this embodiment, by predicting the upper and lower edge heights of obstacles in the occupied grid, the complex shape of the obstacle is decomposed into a simple numerical representation, reducing the computational load of the prediction process. Simultaneously, based on the lower edge height of the obstacle, it is possible to determine whether the obstacle is in contact with the ground and its actual height above the ground, improving the ability to identify suspended obstacles.

[0111] Figure 7 The diagram shows a flowchart illustrating the steps of adjusting the parameters of an obstacle detection model based on a predicted heatmap, the predicted obstacle height for each grid, and sample labels, according to an embodiment of this disclosure. Figure 7 As shown, the steps for adjusting the parameters of the obstacle detection model based on the predicted heatmap, the predicted obstacle height for each grid, and the sample labels include the following steps.

[0112] S161, based on the three-dimensional information of the obstacles, obtains the sample heat map and the obstacle sample height corresponding to each grid.

[0113] In one implementation, the sample label includes three-dimensional information of obstacles within the target area, and a sample heatmap is calculated based on the information.

[0114] Next, the sample heatmap is divided into multiple grids in the form of a two-dimensional BEV grid, and the height of the obstacle sample corresponding to each grid is determined based on the height of the obstacle.

[0115] For example, the three-dimensional information of the obstacle is determined based on radar point cloud data of the target area, and sample data is determined by combining the radar point cloud data with sample images.

[0116] In another implementation, the sample labels can also be directly based on the sample heatmap obtained from an open-source database and the obstacle sample height corresponding to each grid.

[0117] S162, based on the difference between the sample heatmap and the predicted heatmap, the predicted obstacle height and the obstacle sample height corresponding to each grid, adjust the parameters of the obstacle detection model.

[0118] Next, the heatmap loss is determined based on the difference between the sample heatmap and the predicted heatmap output by the prediction head. For example, the difference between the sample heatmap and the predicted heatmap can be calculated based on a loss function. The loss function can be the CrossEntropyLoss function.

[0119] The height loss is determined based on the obstacle sample height for each grid cell and the predicted obstacle height for each grid cell output by the prediction head. For example, the difference between the predicted obstacle height and the obstacle sample height for each grid cell can be calculated using a height loss function. The height loss function can be the SmoothL1Loss function.

[0120] Heatmap loss and height loss measure the difference between the obstacle detection model's predictions and the actual obstacle situation, respectively, from the perspectives of obstacle location and obstacle height. Smaller heatmap and height losses indicate that the obstacle detection model's predictions are closer to reality, and therefore, better performance.

[0121] If the heatmap loss and height loss do not meet the training termination condition, the parameters of the obstacle detection model are adjusted based on the heatmap loss and height loss.

[0122] The training process of the obstacle detection model repeats the above steps continuously. As the number of iterations increases, the parameters of the obstacle detection model are gradually optimized, the values ​​of heatmap loss and height loss are gradually reduced, and the prediction performance of the obstacle detection model is gradually improved.

[0123] The above text combined Figures 1 to 7 The following describes in detail an embodiment of the training method for the obstacle detection model of this disclosure, in conjunction with... Figure 8 The obstacles detection method embodiments of this disclosure are described in detail below. It should be understood that the descriptions of the training method embodiments correspond to the descriptions of the obstacles detection method embodiments; therefore, any parts not described in detail can be referred to the preceding training method embodiments.

[0124] Figure 8 The diagram shown is a schematic flowchart of an obstacle detection method provided in an embodiment of this disclosure. Figure 8 As shown, the obstacle removal method provided in this embodiment includes the following steps.

[0125] S810, acquire fisheye image.

[0126] The fisheye image was taken using a fisheye camera mounted on the vehicle.

[0127] There can be multiple fisheye cameras, which are set up around the vehicle body and capture images of the surrounding environment from different perspectives.

[0128] S820 inputs the fisheye image into the trained obstacle detection model to obtain grid occupancy information and the obstacle height corresponding to each grid.

[0129] The obstacle detection model is trained using the training method described in the above embodiments.

[0130] The obstacle detection model processes fisheye images to obtain grid occupancy information and the obstacle height corresponding to each grid. Using this grid occupancy information and the obstacle height for each grid, the position and height of obstacles within the corresponding region of the fisheye image can be determined, thus yielding the obstacle detection result.

[0131] The following is combined with Figure 9 This section describes in detail an embodiment of the training apparatus for the obstacle detection model of this disclosure. It should be understood that the description of the training method embodiment corresponds to the description of the training apparatus embodiment; therefore, any parts not described in detail can be referred to the preceding training method embodiment.

[0132] Figure 9 The diagram shown is a structural schematic of a training device for an obstacle detection model provided in an embodiment of this disclosure. Figure 9 As shown, the obstacle detection model training device 900 of this embodiment includes: a first acquisition module 910, a first extraction module 920, a conversion module 930, a second extraction module 940, a processing module 950, and a training module 960.

[0133] Specifically, the obstacle detection model includes: a first backbone network, a first neck network, a visual converter, a second backbone network, a second neck network, and a prediction head.

[0134] The first acquisition module 910 is configured to acquire sample data, including sample images of the target area captured by a fisheye camera and sample labels. The first extraction module 920 is configured to input the sample images into the obstacle detection model, sequentially passing them through a first backbone network and a first neck network to obtain a first feature map, which includes depth information of the target area. The conversion module 930 is configured to input the first feature map into a visual converter, which, based on the first feature map, projects features representing different heights of the target area onto a two-dimensional BEV grid to obtain a BEV feature map. The second extraction module 940 is configured to sequentially pass the BEV feature map through a second backbone network and a second neck network for feature extraction and fusion, respectively, outputting a second feature map. The processing module 950 is configured to input the second feature map into a prediction head to obtain a prediction heatmap, predicted grid occupancy information, and the predicted obstacle height corresponding to each grid. The training module 960 is configured to adjust the parameters of the obstacle detection model based on the prediction heatmap, the predicted obstacle height corresponding to each grid, and the sample labels.

[0135] The following is combined with Figure 10The present disclosure provides a detailed description of embodiments of the obstacle detection device. It should be understood that the descriptions of the obstacle detection model training method embodiments and the obstacle detection method embodiments correspond to the descriptions of the obstacle detection device embodiments. Therefore, any parts not described in detail can be found in the preceding obstacle detection model training method embodiments and obstacle detection method embodiments.

[0136] like Figure 10 As shown, the obstacle detection device 1000 of this embodiment includes: a second acquisition module 1010 and a prediction module 1020.

[0137] The second acquisition module 1010 is configured to acquire fisheye images. The prediction module 1020 is configured to input the fisheye images into a trained obstacle detection model to obtain grid occupancy information and the obstacle height corresponding to each grid. The obstacle detection model is trained based on the obstacle detection model training method described in the above embodiments.

[0138] Below, for reference Figure 11 To describe an electronic device according to embodiments of the present disclosure. Figure 11 The diagram shown is a structural schematic of an electronic device provided according to an embodiment of this disclosure. Figure 11 As shown, the electronic device 1100 includes one or more processors 1110 and memory 1120.

[0139] The processor 1110 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1100 to perform desired functions.

[0140] The memory 1120 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1110 may execute the program instructions to implement the obstacle detection model training method, obstacle detection method, and / or other desired functions of the various embodiments of this disclosure described above.

[0141] In some embodiments, the electronic device 1100 may further include an input device 1130 and an output device 1140, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0142] The input device 1130 may include, for example, a touch screen, microphone, keyboard, mouse, etc. The output device 1140 may include, for example, a display, speaker, and communication network and its connected remote output devices, etc.

[0143] Of course, for the sake of simplicity, Figure 11 Only some of the components of the electronic device 1100 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 1100 may include any other suitable components depending on the specific application.

[0144] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the training method or obstacle detection method of the obstacle detection model according to various embodiments of this disclosure described above.

[0145] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0146] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the training method or obstacle detection method of the obstacle detection model according to various embodiments of this disclosure described above.

[0147] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0148] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0149] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0150] It should also be noted that in the systems, apparatus, and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.

[0151] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0152] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method of training an obstacle detection model, the obstacle detection model comprising: The first backbone network, the first neck network, the visual converter, the second backbone network, the second neck network and the prediction head are characterized in that they comprise: Obtain sample data, the sample data comprising sample images of a target region captured by a fisheye camera, and sample labels; Input the sample images into an obstacle detection model, sequentially pass through the first backbone network and the first neck network to obtain a first feature map, the first feature map comprising depth information of the target region; The visual converter projects features representing different heights of the target region onto a two-dimensional BEV grid based on the first feature map to obtain a BEV feature map; Sequentially pass the BEV feature map through the second backbone network and the second neck network for feature extraction and fusion respectively to output a second feature map; Input the second feature map into the prediction head to obtain a predicted heat map, predicted occupancy information of a grid and an obstacle prediction height corresponding to each grid; Adjust parameters of the obstacle detection model based on the predicted heat map, the obstacle prediction height corresponding to each grid and the sample labels; Wherein, the inputting of the second feature map into the prediction head to obtain a predicted heat map, predicted occupancy information of a grid and an obstacle prediction height corresponding to each grid comprises: The prediction head determines the predicted heat map through convolution; Determine whether each grid is occupied based on the predicted heat map; For the occupied grid, perform layer-by-layer convolution search on different height layers every preset height interval to determine whether the layers have features; The height of the highest layer with features is taken as the upper edge height of the obstacle in the grid; The height of the lowest layer with features is taken as the lower edge height of the obstacle in the grid; Determine the obstacle height corresponding to the grid based on the upper edge height and the lower edge height.

2. The method of claim 1, wherein, The sample images comprise multiple sample images collected by multiple fisheye cameras at the same time, and there is an overlapping area in the field of view of at least two fisheye cameras.

3. The method of claim 2, wherein, The inputting of the sample images into the obstacle detection model, sequentially passing through the first backbone network and the first neck network to obtain a first feature map comprises: The first backbone network extracts features from the multiple sample images, and outputs multiple first intermediate feature maps of different sizes for each sample image; The first neck network fuses the multiple first intermediate feature maps corresponding to each sample image to obtain multiple first feature maps.

4. The method of claim 1, wherein, The visual converter projects features representing different heights of the target region onto a two-dimensional BEV grid based on the first feature map to obtain a BEV feature map, comprising: Based on the first feature map, camera intrinsic parameters and depth information of the target region, a three-dimensional target region is obtained; Project features of the three-dimensional target region every preset height interval onto the two-dimensional BEV grid to obtain multiple BEV intermediate feature maps; Fuse the multiple BEV intermediate feature maps to obtain a BEV feature map.

5. The method of claim 1, wherein, The sequentially passing of the BEV feature map through the second backbone network and the second neck network for feature extraction and fusion respectively to output a second feature map comprises: The second backbone network extracts features of the BEV feature map, and outputs a plurality of sizes of second intermediate feature maps; The second neck network fuses the plurality of sizes of second intermediate feature maps to obtain a second feature map.

6. The method of claim 1, wherein, The sample label includes three-dimensional information of the obstacle, and the parameters of the obstacle detection model are adjusted based on the predicted heat map, the obstacle prediction height corresponding to each grid, and the sample label, including: Based on the three-dimensional information of the obstacle, a sample heat map and an obstacle sample height corresponding to each grid are obtained; Based on the difference between the sample heat map and the predicted heat map, the obstacle prediction height corresponding to each grid, and the obstacle sample height, the parameters of the obstacle detection model are adjusted.

7. An obstacle detection method characterized by, It includes: Obtain a fisheye image; Input the fisheye image into the trained obstacle detection model to obtain the occupancy information of the grid and the obstacle height corresponding to each grid; The obstacle detection model is trained based on the method of any one of claims 1 to 6.

8. An apparatus for training an obstacle detection model, the obstacle detection model comprising: The training device comprises: The first acquisition module is configured to acquire sample data, and the sample data includes a sample image of a target region captured by a fisheye camera and a sample label; The first extraction module is configured to input the sample image into an obstacle detection model to obtain a first feature map through the first backbone network and the first neck network in sequence, and the first feature map includes depth information of the target region; The conversion module is configured to input the first feature map into the visual converter, and the visual converter projects features representing different heights of the target region to a two-dimensional BEV grid based on the first feature map to obtain a BEV feature map; The second extraction module is configured to extract and fuse the BEV feature map through the second backbone network and the second neck network in sequence to output a second feature map; The processing module is configured to input the second feature map into the prediction head to obtain a predicted heat map, predicted occupancy information of the grid, and an obstacle prediction height corresponding to each grid; The training module is configured to adjust the parameters of the obstacle detection model based on the predicted heat map, the obstacle prediction height corresponding to each grid, and the sample label. The processing module is configured to input the second feature map into the prediction head to obtain a predicted heat map, predicted occupancy information of the grid, and an obstacle prediction height corresponding to each grid, including: The prediction head determines the predicted heat map through convolution; Based on the predicted heat map, determine whether each grid is occupied; For the occupied grid, perform layer-by-layer convolution search on different height layers according to every preset height interval to determine whether there is a feature; The height of the highest layer with a feature is taken as the upper edge height of the obstacle in the grid; The height of the lowest layer with a feature is taken as the lower edge height of the obstacle in the grid; Based on the upper edge height and the lower edge height, determine the obstacle height corresponding to the grid.

9. An obstacle detection device characterized by comprising: It includes: The second acquisition module is configured to acquire a fisheye image; A prediction module is configured to input the fisheye image into a trained obstacle detection model to obtain occupancy information of the grid and an obstacle height corresponding to each grid. The obstacle detection model is trained based on the method in any one of claims 1 to 6.

Images