Method for generating three-dimensional (3D) occupancy grid, vehicle controller and vehicle

Abstract

The invention relates to the field of intelligent driving, in particular to a three-dimensional (3D) occupancy grid generation method, a vehicle controller and a vehicle. The method comprises the following steps: acquiring a first aerial view feature; the first aerial view feature comprises feature data of each position on a two-dimensional plane corresponding to the target area. And performing convolution down-sampling on the first aerial view feature to obtain a second aerial view feature. And obtaining a 3D occupation grid corresponding to the target area according to the second aerial view feature.

Description

Technical Field

[0001] The present disclosure relates to the field of intelligent driving, and in particular to a method for generating a three-dimensional 3D occupancy grid, a vehicle controller, and a vehicle. Background Art

[0002] At present, in order to make vehicles with intelligent driving functions drive more safely, accurately identifying objects around the vehicle is a very critical task.

[0003] In the prior art, a three-dimensional (3D) occupancy grid around a vehicle is often used to identify objects around the vehicle. The 3D occupancy grid includes feature information corresponding to each voxel in the 3D space, and the feature information corresponding to each voxel can be used to represent features such as the occupancy rate of the voxel and the semantic category of the voxel. In this way, objects around the vehicle can be identified based on the feature information of each voxel in the 3D occupancy grid.

[0004] However, in the prior art, a large amount of computing power is often consumed to generate a 3D occupancy network. Therefore, how to reduce the computing power cost in the process of generating a 3D occupancy network is a problem that needs to be solved at present. Summary of the invention

[0005] In order to solve the above technical problems, the present disclosure provides a method for generating a three-dimensional 3D occupancy grid, a vehicle controller and a vehicle.

[0006] In a first aspect, a method for generating a three-dimensional 3D occupancy grid is provided, the method comprising: obtaining a first bird's-eye view feature; the first bird's-eye view feature includes feature data of each position on a two-dimensional plane corresponding to a target area. Convolution downsampling the first bird's-eye view feature is performed to obtain a second bird's-eye view feature. According to the second bird's-eye view feature, a 3D occupancy grid corresponding to the target area is obtained.

[0007] In the above method, it is taken into consideration that: in the process of generating a 3D occupied grid, the related technology often requires the use of a large number of 3D convolution calculations, and compared with 2D convolution, 3D convolution requires a lot of computing power. Therefore, in the process of generating a 3D occupied grid, if a 2D convolution operation can be used to replace the 3D convolution operation performed on the 3D voxel feature, the amount of calculation can be reduced and the computing power overhead can be reduced. Based on the above considerations, in the above method, a second bird's-eye view feature is obtained by performing a 2D convolution (i.e., convolution downsampling) on ​​the first 2D bird's-eye view feature, and then a 3D occupied grid is obtained based on the second bird's-eye view feature. In this way, a 3D occupied grid can be obtained without performing a 3D convolution, thereby reducing the amount of calculation and computing power overhead.

[0008] In one implementation, performing convolution downsampling on the first bird's-eye view feature to obtain the second bird's-eye view feature includes: performing convolution downsampling on the first bird's-eye view feature to obtain n feature maps, wherein each feature map in the n feature maps corresponds to a different receptive field, and fusing the n feature maps to obtain the second bird's-eye view feature.

[0009] In the above implementation, the first bird's-eye view feature is convoluted and downsampled to obtain n feature maps corresponding to different receptive fields, and then the n feature maps are fused to obtain the second bird's-eye view feature. In this way, the second bird's-eye view feature can include feature information of different receptive fields, so that it can include both the overall features of the entire target area and the features of finer local areas in the target area.

[0010] In one implementation, obtaining a 3D occupancy grid corresponding to the target area according to the second bird's-eye view feature includes: obtaining a target resolution corresponding to the 3D occupancy grid, converting the second bird's-eye view feature into a third bird's-eye view feature that meets the target resolution, and obtaining a 3D occupancy grid corresponding to the target area that meets the target resolution according to the third bird's-eye view feature.

[0011] In the above implementation, in the process of generating the 3D occupation grid, the corresponding 3D occupation grid can be generated according to the target resolution corresponding to the 3D occupation grid. In this way, the method can generate a 3D occupation grid of the target resolution matching the scene according to different scenes during application.

[0012] In one implementation, a 3D occupancy grid that meets the target resolution and corresponds to the target area is obtained based on the third bird's-eye view feature, including: performing a dimensionality transformation operation on the third bird's-eye view feature to obtain a 3D occupancy grid that meets the target resolution and corresponds to the target area.

[0013] In the above implementation, the dimensionality transformation operation can be performed on the third bird's-eye view feature to quickly and efficiently obtain a 3D occupancy grid.

[0014] In one implementation, the method further includes: obtaining a 3D spatial feature corresponding to the target area; the 3D spatial feature includes feature data of each position in the three-dimensional space corresponding to the target area. According to the 3D spatial feature, a first bird's-eye view feature is obtained.

[0015] In the above implementation, the 3D spatial features corresponding to the target area are first obtained, and then the first bird's-eye view features are obtained according to the 3D spatial features. In this way, during the implementation of the method, a 3D occupancy grid can be output after the 3D spatial features are input. Therefore, the external module does not need to perceive that the 2D convolution is used to generate the 3D occupancy grid in this method, so that the method can be adapted to more application scenarios.

[0016] In one implementation, obtaining the 3D spatial features corresponding to the target area includes: obtaining k captured images corresponding to the target area, extracting image features of the k captured images using a 2D residual network, and obtaining k 2D feature maps; the k 2D feature maps are respectively used to reflect the image features of a captured image. According to the k 2D feature maps, feature data of each voxel in the 3D space is determined to obtain the 3D spatial features corresponding to the target area.

[0017] Through the above implementation, the 3D spatial features corresponding to the target area can be generated according to the k captured images, so as to obtain the 3D occupancy grid by utilizing the 3D spatial features and combining with other steps of the method.

[0018] In one implementation, the method further includes: obtaining a 3D target detection result corresponding to the target area according to the second bird's-eye view feature; the 3D target detection result is used to indicate the size and position of the object in the target area.

[0019] Through the above implementation, after obtaining the second bird's-eye view feature, the method can also use the second bird's-eye view feature to perform 3D target detection, thereby reducing the inference time of the model.

[0020] In a second aspect, a vehicle controller is provided, comprising: an acquisition unit, configured to acquire a first bird's-eye view feature; the first bird's-eye view feature includes feature data of each position on a two-dimensional plane corresponding to a target area; a processing unit, configured to perform convolution downsampling on the first bird's-eye view feature to obtain a second bird's-eye view feature; and the processing unit, further configured to obtain a three-dimensional 3D occupancy grid corresponding to the target area based on the second bird's-eye view feature.

[0021] In one implementation, a processing unit is used to perform convolution downsampling on the first bird's-eye view feature to obtain a second bird's-eye view feature, including: a processing unit is used to perform convolution downsampling on the first bird's-eye view feature to obtain n feature maps; wherein each feature map in the n feature maps corresponds to a different receptive field. The processing unit is also used to fuse the n feature maps to obtain the second bird's-eye view feature.

[0022] In one implementation, the processing unit is further used to obtain a 3D occupancy grid corresponding to the target area according to the second bird's-eye view feature, including: the processing unit is further used to obtain a target resolution corresponding to the 3D occupancy grid. The processing unit is further used to convert the second bird's-eye view feature into a third bird's-eye view feature that meets the target resolution. The processing unit is further used to obtain a 3D occupancy grid corresponding to the target area that meets the target resolution according to the third bird's-eye view feature.

[0023] In one implementation, the processing unit is further used to obtain a 3D occupied grid that meets the target resolution and corresponds to the target area based on the third bird's-eye view feature, including: the processing unit is also used to perform a dimensionality transformation operation on the third bird's-eye view feature to obtain a 3D occupied grid that meets the target resolution and corresponds to the target area.

[0024] In one implementation, the acquisition unit is further used to acquire 3D spatial features corresponding to the target area; the 3D spatial features include feature data of each position in the three-dimensional space corresponding to the target area. The processing unit is further used to obtain the first bird's-eye view features according to the 3D spatial features.

[0025] In one implementation, the acquisition unit is further used to acquire the 3D spatial features corresponding to the target area, including: the acquisition unit is further used to acquire k captured images corresponding to the target area. The acquisition unit is further used to extract the image features of the k captured images using a 2D residual network to obtain k 2D feature maps; the k 2D feature maps are respectively used to reflect the image features of a captured image. The acquisition unit is further used to determine the feature data of each voxel in the 3D space according to the k 2D feature maps to obtain the 3D spatial features corresponding to the target area.

[0026] In one implementation, the processing unit is further used to obtain a 3D target detection result corresponding to the target area according to the second bird's-eye view feature; the 3D target detection result is used to indicate the size and position of the object in the target area.

[0027] In a third aspect, a vehicle controller is provided, comprising a processor and an interface, wherein the processor receives or sends data through the interface, and the processor is used to implement a method as described in the first aspect or any one of the implementations of the first aspect.

[0028] In a fourth aspect, a vehicle is provided, comprising: an on-board memory and an on-board processor, wherein a computer program is stored in the on-board memory, and the on-board processor is configured to run the computer program to execute the method of the above-mentioned first aspect or any implementation method of the first aspect.

[0029] In a fifth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium. When the instructions are executed on a processor, the method of the above-mentioned first aspect or any implementation method of the first aspect is implemented.

[0030] In a sixth aspect, a computer program product is provided, the computer program product comprising instructions, which, when executed on a processor, implement the method of the first aspect or any one of the implementation modes of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0033] Figure 1 A schematic diagram of the structure of a vehicle provided in an embodiment of the present disclosure;

[0034] Figure 2 One of the flowcharts of a method for generating a 3D occupancy grid provided in an embodiment of the present disclosure;

[0035] Figure 3 A second flowchart of a method for generating a 3D occupancy grid provided in an embodiment of the present disclosure;

[0036] Figure 4 A third flowchart of a method for generating a 3D occupancy grid provided in an embodiment of the present disclosure;

[0037] Figure 5 A fourth flowchart of a method for generating a 3D occupancy grid provided in an embodiment of the present disclosure;

[0038] Figure 6 A fifth flowchart of a method for generating a 3D occupancy grid provided in an embodiment of the present disclosure;

[0039] Figure 7 One of the structural schematic diagrams of a vehicle controller provided by an embodiment of the present disclosure;

[0040] Figure 8 The second structural schematic diagram of a vehicle controller provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0041] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0042] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0043] First, the technical terms involved in this disclosure are introduced:

[0044] 1. The three-dimensional (3D) occupancy grid can be understood as a collection of feature information corresponding to each voxel in the 3D space after the 3D space is divided into multiple voxels. The feature information corresponding to each voxel can be used to represent the occupancy rate of the voxel and the semantic category of the voxel. The 3D occupancy grid can predict the geometric shape of the object by predicting the semantic category of each voxel in the 3D space.

[0045] 2. Bird's Eye View (BEV) is a horizontal view used to describe and represent the environment around an object, which is especially common in autonomous driving and intelligent transportation systems. In autonomous driving and intelligent transportation systems, BEV converts the three-dimensional space around the vehicle into a two-dimensional plane (specifically the horizontal plane), and all roads, buildings, pedestrians, other vehicles, etc. are projected onto a virtual horizontal plane and displayed from a bird's-eye view. BEV helps provide clear distance perception, making it easier for the algorithm to identify and analyze the relative position and size of surrounding objects and their impact on the driving path. It is often used in route planning, obstacle detection, obstacle avoidance decisions, etc.

[0046] The following is an introduction to the technical solutions provided by the present disclosure:

[0047] Currently, complex calculations are usually required to generate 3D occupancy grids, including 3D convolution and attention mechanisms. The use of these operators increases computing power overhead, thus increasing the difficulty of board-side deployment.

[0048] In order to reduce the computing power cost in the process of generating 3D occupancy network, some related technologies propose to use sparse 3D convolution and three-view projection to calculate grid occupancy to generate 3D occupancy grid. Among them, on the one hand, since sparse 3D convolution does not need to calculate all 3D voxel grids, but only performs convolution operations on valid 3D voxel grids, the computing power cost of 3D convolution can be reduced. On the other hand, in the three-view projection to calculate grid occupancy, the 3D features can be projected to three planes in the space using the attention mechanism network, and then the cross-attention mechanism is used to obtain the feature expression of each voxel in the 3D voxel in the three planes, which can also reduce the computing power cost.

[0049] The above-mentioned related technologies of using sparse 3D convolution and three-view projection to calculate grid occupancy to generate 3D occupied grids still have high requirements on computing power and have harsh deployment conditions.

[0050] In view of the above technical problems, the present disclosure takes into account that in the process of generating a 3D occupancy grid, the related technology often requires the use of a large number of 3D convolution calculations, and compared with 2D convolution, 3D convolution requires a large amount of computing power. Therefore, in the process of generating a 3D occupancy grid, if the 2D convolution operation can be used to replace the 3D convolution operation performed on the 3D voxel feature, the amount of calculation can be reduced and the computing power overhead can be reduced.

[0051] Based on the above considerations, the present disclosure provides a method for generating a 3D occupancy grid, in which a 2D convolution (including convolution downsampling, etc.) is performed on a 2D bird's-eye view feature instead of a 3D convolution on a 3D voxel feature, thereby reducing the amount of calculation and computing power overhead.

[0052] The implementation process of the method for generating a 3D occupancy grid provided by the present disclosure is described in detail below with reference to a specific example:

[0053] First, the application scenarios of the present disclosure are introduced by examples:

[0054] like Figure 1 Shown is a schematic structural diagram of a vehicle provided by the present disclosure.

[0055] The vehicle 10 may include: an intelligent driving system 11 for realizing assisted driving and intelligent driving, a steering system 12 for controlling the steering of the vehicle, and a power braking system 13 for controlling the power output and braking of the vehicle.

[0056] The intelligent driving system 11 may include: a camera 111 and a vehicle controller 112 , etc.

[0057] The camera 111 is used to collect images around the vehicle.

[0058] The vehicle controller 112 is used to generate a 3D occupancy grid according to the method provided in the embodiment of the present disclosure. In addition, the vehicle controller 112 can also be used to control components such as the vehicle's steering system and power brake system according to the generated 3D occupancy grid.

[0059] It should be noted that in Figure 1 In the example shown, the vehicle controller 112 in the intelligent driving system 11 generates a 3D occupancy grid according to the method provided in the embodiment of the present disclosure as an example for introduction. In actual application, other software and hardware devices in the vehicle independent of the intelligent driving system 11 can also generate a 3D occupancy grid according to the method provided in the embodiment of the present disclosure. There is no limitation on the execution subject of the method provided in the embodiment of the present disclosure in the embodiment of the present disclosure.

[0060] The following takes the operation process of the vehicle controller 112 as an example to introduce the method provided by the embodiment of the present disclosure. Figure 2 As shown, the method may include:

[0061] S201. The vehicle controller 112 obtains a first bird's-eye view feature.

[0062] For example, after acquiring the captured image within the target area captured by the camera 111, the vehicle controller 112 can obtain the first bird's-eye view feature by processing the captured image.

[0063] The specific implementation process of the vehicle controller 112 acquiring the first bird's-eye view feature will be described in detail below and will not be expanded here.

[0064] The first bird's-eye view feature includes feature data of each position on the two-dimensional plane corresponding to the target area.

[0065] For example, the size of the first bird's-eye view feature P1 is expressed as (C, H, W), where H represents the image length corresponding to the first bird's-eye view feature, W represents the image width corresponding to the first bird's-eye view feature, and C represents the number of feature data at each position in the image.

[0066] S202 . The vehicle controller 112 performs convolution downsampling (conv-downsample) on the first bird's-eye view feature to obtain a second bird's-eye view feature.

[0067] By performing convolution down-sampling on the first bird's-eye view feature, feature extraction can be performed on the first bird's-eye view feature, so that the second bird's-eye view feature after feature extraction can be used to generate a 3D occupancy grid.

[0068] Among them, since the first bird's-eye view feature is a 2D image, 2D convolution can be used to perform convolution down-sampling on the first bird's-eye view, so as to realize feature extraction of the first bird's-eye view feature, thereby obtaining the second bird's-eye view feature.

[0069] In one implementation, in order to make the second bird's-eye view feature include both the overall features of the entire target area and the features of a finer local area in the target area, the present disclosure considers that: the first bird's-eye view feature can be convolutionally downsampled to obtain n feature maps corresponding to different receptive fields, and then the n feature maps are fused to obtain the second bird's-eye view feature. In this way, the second bird's-eye view feature can include feature information of different receptive fields, so that it can include both the overall features of the entire target area and the features of a finer local area in the target area. Therefore, S202 can specifically include:

[0070] S2021. The vehicle controller 112 performs convolution down-sampling on the first bird's-eye view features to obtain n feature maps.

[0071] Among them, each feature map in the n feature maps corresponds to a different receptive field.

[0072] For example, the vehicle controller 112 performs the first convolution downsampling on the first bird's-eye view feature P1 (C, H, W) to obtain a feature map P2. Exemplarily, taking the case where the image length and image width are reduced by 1 times each time the convolution downsampling is performed, and the number of feature data at each position in the image is increased by 1 times, the size of the feature map P2 is (2C, H / 2, W / 2). Afterwards, the vehicle controller 112 performs the second convolution downsampling on the feature map P2 to obtain a feature map P3. The size of the feature map P3 is (4C, H / 4, W / 4). Then, the n feature maps may include the above-mentioned feature maps P2 and feature maps P3.

[0073] In addition, it can be understood that the first bird's-eye view feature itself can also be fused as one of the n feature maps.

[0074] S2022. The vehicle controller 112 fuses n feature maps to obtain a second bird's-eye view feature.

[0075] Take the vehicle controller 112 fusing the above-mentioned feature graphs P1, P2 and P3 as an example. Figure 3 As shown, by fusing the feature maps P1, P2 and P3, the second bird's-eye view feature is obtained. The specific process may include:

[0076] First, the feature map P3 (4C, H / 4, W / 4) can be upsampled to obtain the feature map P3', with a size of (4C, H / 2, W / 2). Then, the feature map P3' is fused with the feature map P2 to obtain the feature map P4, with a size of (4C, H / 2, W / 2). Then, the feature map P4 is upsampled to obtain the feature map P4', with a size of (4C, H, W). Then, the feature map P4' is fused with the feature map P1, and the obtained feature map P5 can be used as the second bird's-eye view feature, and the size of P5 can be expressed as (4C, H, W).

[0077] S203 . The vehicle controller 112 obtains a 3D occupancy grid corresponding to the target area according to the second bird's-eye view feature.

[0078] In one implementation, considering that in actual application, the resolution of the 3D occupied grid required in different scenes may also be different. Therefore, in the process of generating the 3D occupied grid, the corresponding 3D occupied grid can be generated according to the target resolution corresponding to the 3D occupied grid. In this way, the method can generate a 3D occupied grid with a target resolution matching the scene according to different scenes during application. Based on the above considerations, S203 may specifically include:

[0079] S2031. The vehicle controller 112 obtains a target resolution corresponding to the 3D occupied grid.

[0080] For example, the vehicle controller 112 may read relevant parameters of the 3D occupancy grid model to obtain a target resolution corresponding to the 3D occupancy grid.

[0081] S2032. The vehicle controller 112 converts the second bird's-eye view feature into a third bird's-eye view feature that meets the target resolution.

[0082] For example Figure 3 As shown, the third bird's-eye view feature P6 (C', H, W) that meets the target resolution can be obtained by performing 2D convolution on the second bird's-eye view feature P5 (4C, H, W).

[0083] S2033. The vehicle controller 112 obtains a 3D occupation grid that meets the target resolution and corresponds to the target area according to the third bird's-eye view feature.

[0084] In one implementation, S2033 may include:

[0085] The vehicle controller 112 performs a dimensional transformation operation (reshape) on the third bird's-eye view feature to obtain a 3D occupancy grid that meets the target resolution and corresponds to the target area.

[0086] For example, through the dimensionality transformation operation, the third bird's-eye view feature P6 (C', H, W) can be converted into a 3D occupancy grid of size (C", Z, H, W). Among them, Z, H, and W respectively represent the three dimensions of the 3D occupancy grid, and C" represents the number of classification categories of the 3D occupancy grid.

[0087] The following is a detailed description of the implementation process of the vehicle controller 112 acquiring the first bird's-eye view feature. Specifically, in one implementation, Figure 4 As shown, the vehicle controller 112 may obtain the first bird's-eye view feature (ie, S201) by specifically including:

[0088] S2011. The vehicle controller 112 obtains the 3D spatial features corresponding to the target area.

[0089] The 3D spatial features include feature data of each position in the three-dimensional space corresponding to the target area.

[0090] In one possible design, S2011 may specifically include:

[0091] S2011a. The vehicle controller 112 obtains k captured images corresponding to the target area.

[0092] For example, the k captured images may be the k captured images collected by the camera 111 on the vehicle in k directions around the vehicle. The vehicle controller 112 may receive the k captured images sent by the camera 111, thereby acquiring the k captured images.

[0093] S2011b, the vehicle controller 112 uses a 2D residual network (2D ResNet) to extract image features of k captured images and obtain k 2D feature maps.

[0094] Among them, k 2D feature maps are respectively used to reflect the image features of a captured image.

[0095] For example, Figure 5 As shown, the vehicle controller 112 uses a 2D residual network (2DResNet) to extract image features of the captured image and obtain a 2D feature map.

[0096] In this design, the image features of k captured images are extracted by using a 2D residual network, so that the image features can be represented as richly as possible in the 2D feature map.

[0097] S2011c. The vehicle controller 112 determines the feature data of each voxel in the 3D space based on the k 2D feature maps, and obtains the 3D space features corresponding to the target area.

[0098] Specifically, a mapping table can be established in the vehicle controller 112, and the mapping table is used to determine the relationship between each voxel in the 3D space and the 2D image feature. Then, through coordinate conversion and index query, the 2D image features in the k 2D feature maps can be mapped to the corresponding voxels, thereby determining the feature data of each voxel in the D space. After filling the 2D image features with each voxel in the 3D space, the 3D space features corresponding to the target area can be obtained.

[0099] S2012. The vehicle controller 112 obtains a first bird's-eye view feature based on the 3D space feature.

[0100] For example, the 3D spatial feature can be transformed into the bird's-eye view feature through a dimensional transformation operation to obtain the first bird's-eye view feature. Figure 4As shown, the vehicle controller 112 can use a dimensional transformation operation to fuse the height direction of the above-mentioned 3D spatial features (i.e., S1), and then determine whether the height vector of the 3D spatial feature is 1 (i.e., S2). If the height vector of the 3D spatial feature is not 1, S1 is executed; if the height vector of the 3D spatial feature is 1, it is determined that the first bird's-eye view feature is obtained.

[0101] For example, Figure 5 As shown, the vehicle controller 112 may transform the 3D space feature into a bird's-eye view feature through a dimensional transformation operation to obtain a first bird's-eye view feature.

[0102] In another implementation, Figure 6 As shown, the method may also include:

[0103] S204. The vehicle controller 112 obtains a 3D target detection result corresponding to the target area according to the second bird's-eye view feature.

[0104] Among them, the 3D target detection results are used to indicate the size and position of objects in the target area.

[0105] For example, Figure 5 As shown, after obtaining the second bird's-eye view feature P5, on the one hand, the third bird's-eye view feature P6 can be obtained by performing 2D convolution on the second bird's-eye view feature P5, and then the 3D occupancy grid can be obtained according to the third bird's-eye view feature P6; on the other hand, the second bird's-eye view feature can also be input into the 3D target detection head to obtain the 3D target detection result.

[0106] In the method provided by the present disclosure, after obtaining the second bird's-eye view feature, the second bird's-eye view feature can also be used to perform 3D target detection, thereby reducing the inference time of the model.

[0107] Based on the above method embodiment, the device provided by the embodiment of the present disclosure is described below. Figure 7 FIG. 3 is a schematic diagram of a vehicle controller provided by an embodiment of the present disclosure. Specifically, the vehicle controller 30 may be a chip or a system on a chip. The vehicle controller 30 may be used to implement Figure 2-Figure 3 Specifically, the vehicle controller 30 may include:

[0108] The acquisition unit 301 is used to acquire a first bird's-eye view feature; the first bird's-eye view feature includes feature data of each position on a two-dimensional plane corresponding to a target area.

[0109] The processing unit 302 is configured to perform convolution down-sampling on the first bird's-eye view feature to obtain a second bird's-eye view feature.

[0110] The processing unit 302 is further configured to obtain a three-dimensional 3D occupancy grid corresponding to the target area according to the second bird's-eye view feature.

[0111] In one implementation, the processing unit 302 is configured to perform convolution down-sampling on the first bird's-eye view feature to obtain the second bird's-eye view feature, including:

[0112] The processing unit 302 is used to perform convolution down-sampling on the first bird's-eye view feature to obtain n feature maps; wherein each feature map in the n feature maps corresponds to a different receptive field.

[0113] The processing unit 302 is used to fuse n feature maps to obtain a second bird's-eye view feature.

[0114] In one implementation, the processing unit 302 is further configured to obtain a 3D occupancy grid corresponding to the target area according to the second bird's-eye view feature, including:

[0115] The processing unit 302 is further configured to obtain a target resolution corresponding to the 3D occupied grid.

[0116] The processing unit 302 is further configured to convert the second bird's-eye view feature into a third bird's-eye view feature that meets the target resolution.

[0117] The processing unit 302 is further configured to obtain a 3D occupancy grid that meets a target resolution and corresponds to the target area according to the third bird's-eye view feature.

[0118] In one implementation, the processing unit 302 is further configured to obtain a 3D occupancy grid that satisfies a target resolution and corresponds to the target area according to the third bird's-eye view feature, including:

[0119] The processing unit 302 is further configured to perform a dimensionality transformation operation on the third bird's-eye view feature to obtain a 3D occupancy grid corresponding to the target area and meeting the target resolution.

[0120] In one implementation, the acquisition unit 301 is further used to acquire 3D spatial features corresponding to the target area; the 3D spatial features include feature data of each position in the three-dimensional space corresponding to the target area.

[0121] The processing unit 302 is further configured to obtain a first bird's-eye view feature according to the 3D space feature.

[0122] In one implementation, the acquisition unit 301 is further configured to acquire the 3D spatial features corresponding to the target area, including:

[0123] The acquisition unit 301 is further configured to acquire k captured images corresponding to the target area.

[0124] The acquisition unit 301 is further used to extract image features of k captured images using a 2D residual network to obtain k 2D feature maps; the k 2D feature maps are respectively used to reflect the image features of one captured image.

[0125] The acquisition unit 301 is further used to determine the feature data of each voxel in the 3D space according to the k 2D feature maps, and obtain the 3D space features corresponding to the target area.

[0126] In one implementation, the processing unit 302 is further used to obtain a 3D object detection result corresponding to the target area according to the second bird's-eye view feature; the 3D object detection result is used to indicate the size and position of the object in the target area.

[0127] like Figure 8 FIG. 4 is a schematic diagram of another vehicle controller provided by an embodiment of the present disclosure. Specifically, the vehicle controller 40 may be a chip or a system on a chip. The vehicle controller 40 may be used to implement Figure 2-Figure 3 Specifically, the vehicle controller 40 may include: a processor 401, a communication line 402, a memory 403, and some or all of at least one communication interface 404.

[0128] The processor 401 is used to execute all or part of the steps in the method provided in this embodiment.

[0129] Specifically, the processor 401 may include a general-purpose central processing unit (CPU), and the processor 401 may also include a microprocessor, a field programmable gate array (Field Programmable Gate Array, FPGA), a digital signal processor (digital signal processing, DSP) or an application-specific integrated circuit (application-specific integrated circuit, ASIC), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0130] In a specific implementation, as an embodiment, the processor 401 may include one or more CPUs. In a specific implementation, as an embodiment, the vehicle controller 40 may include multiple processors, each of which may be a single-CPU processor or a multi-CPU processor. The processor here may refer to one or more devices, circuits, and / or processing cores for processing, for example, computer data (computer program instructions).

[0131] In addition, the memory 403 may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. Among them, the nonvolatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct rambus RAM (DR RAM). Memory 403 can be independent and connected to processor 401 via communication line 402. Memory 403 can also be integrated with processor 401.

[0132] The memory 403 stores computer instructions. The processor 401 can execute the computer instructions stored in the memory 403 to perform all or part of the steps in the method provided in this embodiment.

[0133] Optionally, the computer-executable instructions in this embodiment may also be referred to as application program codes, which is not specifically limited in this embodiment.

[0134] In addition, the communication interface 404 uses any transceiver or other device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0135] In addition, the communication line 402 is used to connect the various components in the vehicle controller 40. Specifically, the communication line 402 may include a data bus, a power bus, a control bus, and a status signal bus, etc. However, for the sake of clarity, various buses are labeled as communication lines 402 in the figure.

[0136] In addition, the vehicle controller 40 may also include a storage medium. The storage medium is used to store computer instructions and various data for implementing the technical solution of this embodiment. When the vehicle controller 40 executes the above method of this embodiment, the computer instructions and various data stored in the storage medium are loaded into the memory 403, so that the processor 401 can execute the computer instructions stored in the memory 403 to execute the method provided by this embodiment.

[0137] The embodiments of the present disclosure may divide the functional modules of the device according to the above method examples. For example, each functional module may be divided according to each function, or two or more functions may be integrated into one processing module. The above integrated modules may be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiments of the present disclosure is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0138] The method steps in this embodiment can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in RAM, flash memory, ROM, PROM, EPROM, EEPROM, registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0139] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in this embodiment is executed in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a communication device, a user device or other programmable device. The computer program or instruction can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program or instruction can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, a hard disk, or a tape; it can also be an optical medium, such as a digital video disc (DVD); it can also be a semiconductor medium, such as an SSD.

[0140] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0141] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating a three-dimensional 3D occupancy grid, characterized in that: The method comprises: Acquire a first bird's-eye view feature; the first bird's-eye view feature includes feature data of each position on a two-dimensional plane corresponding to the target area; Performing convolution down-sampling on the first bird's-eye view feature to obtain a second bird's-eye view feature; A three-dimensional 3D occupancy grid corresponding to the target area is obtained according to the second bird's-eye view feature.

2. The method according to claim 1, characterized in that The performing convolution down-sampling on the first bird's-eye view feature to obtain a second bird's-eye view feature includes: Perform convolution downsampling on the first bird's-eye view feature to obtain n feature maps; wherein each feature map in the n feature maps corresponds to a different receptive field; The n feature maps are fused to obtain the second bird's-eye view feature.

3. The method according to claim 1, characterized in that The obtaining, according to the second bird's-eye view feature, a 3D occupancy grid corresponding to the target area includes: Get the target resolution corresponding to the 3D occupied grid; Converting the second bird's-eye view feature into a third bird's-eye view feature that meets the target resolution; A 3D occupancy grid corresponding to the target area and satisfying the target resolution is obtained according to the third bird's-eye view feature.

4. The method according to claim 3, characterized in that The step of obtaining a 3D occupancy grid that satisfies a target resolution and corresponds to the target area according to the third bird's-eye view feature includes: A dimension transformation operation is performed on the third bird's-eye view feature to obtain a 3D occupancy grid corresponding to the target area that meets the target resolution.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Acquire a 3D spatial feature corresponding to the target area; the 3D spatial feature includes feature data of each position in the three-dimensional space corresponding to the target area; The first bird's-eye view feature is obtained according to the 3D space feature.

6. The method according to claim 5, characterized in that The acquiring of the 3D spatial features corresponding to the target area includes: Acquire k captured images corresponding to the target area; Using a 2D residual network, extracting image features of the k captured images to obtain k 2D feature maps; the k 2D feature maps are respectively used to reflect the image features of one of the captured images; According to the k 2D feature maps, feature data of each voxel in the 3D space is determined to obtain the 3D space features corresponding to the target area.

7. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: A 3D target detection result corresponding to the target area is obtained according to the second bird's-eye view feature; the 3D target detection result is used to indicate the size and position of the object in the target area.

8. A vehicle controller, characterized in that: include: An acquisition unit, configured to acquire a first bird's-eye view feature; the first bird's-eye view feature includes feature data of each position on a two-dimensional plane corresponding to a target area; a processing unit, configured to perform convolution down-sampling on the first bird's-eye view feature to obtain a second bird's-eye view feature; The processing unit is further used to obtain a three-dimensional 3D occupancy grid corresponding to the target area according to the second bird's-eye view feature.

9. A vehicle controller, characterized in that: The method comprises a processor and a communication interface, wherein the processor receives or sends data through the communication interface, and the processor is used to implement the method according to any one of claims 1 to 7.

10. A vehicle, characterized in that: include: An on-board memory and an on-board processor, wherein the on-board memory stores a computer program, and the on-board processor is configured to run the computer program to execute the vehicle charging method described in any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a processor, the method according to any one of claims 1 to 7 is implemented.

12. A computer program product, characterized in that The computer program product comprises instructions, and when the instructions are executed on a processor, the method according to any one of claims 1 to 7 is implemented.

Images