3D occupancy grid prediction method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a 3D occupancy grid prediction method and device, electronic equipment and a storage medium, and relates to the technical field of automatic driving. The method comprises the following steps: acquiring a plurality of multi-view images of different views acquired at the same moment, respectively performing illumination enhancement on each multi-view image to obtain a corresponding enhanced multi-view image, respectively performing feature extraction on each enhanced multi-view image to respectively obtain each 2D feature map, and projecting each 2D feature map to the same 3D space. The method comprises the following steps: acquiring 2D feature maps, inputting the 2D feature maps into a pre-trained query generation network to obtain an initial voxel query, clustering the 2D feature maps, fusing the clustered 2D feature maps into the initial voxel query to obtain an enhanced voxel query, performing up-sampling on the enhanced voxel query, generating a high-resolution voxel query, and inputting the high-resolution voxel query into a classification network. And obtaining a prediction result of the 3D occupied grid of each multi-view image. Therefore, the prediction of the 3D occupied grid in the low-light scene is more accurate.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and more specifically, to a 3D occupancy grid prediction method, apparatus, electronic device, and storage medium. Background Technology

[0002] As a key method for environmental perception in autonomous driving, 3D occupancy grid prediction technology can accurately identify the spatial position, geometric shape and motion state of vehicles, pedestrians and obstacles by constructing a refined semantic representation of the distribution of objects in three-dimensional space in real time. It can also effectively detect important environmental elements that are easily missed by traditional two-dimensional perception systems, such as low curbs, high-altitude traffic signs and complex terrain, thus providing a complete environmental cognition basis for autonomous driving decision-making systems.

[0003] In specific applications, this technology needs to address the challenges of environmental perception under varying lighting conditions. Notably, in low-light environments, the quality of data acquired by the sensor deteriorates significantly, manifesting as a decrease in signal-to-noise ratio, loss of detailed features, and distortion of depth information. This reduction in data quality directly impacts the detection accuracy and reliability of the 3D occupancy grid prediction model under low-light conditions. Summary of the Invention

[0004] In view of this, the object of the present invention is to provide a 3D occupancy grid prediction method, apparatus, electronic device and storage medium to at least partially improve the above-mentioned problems.

[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of the present invention are as follows: In a first aspect, embodiments of the present invention provide a 3D occupancy grid prediction method, comprising: Acquire multiple multi-view images taken at the same time from different perspectives; Each of the multi-view images is enhanced with illumination to obtain the corresponding enhanced multi-view image; Feature extraction is performed on each of the enhanced multi-view images to obtain a 2D feature map; The 2D feature maps are projected onto the same 3D space to obtain a 3D feature volume, and the 3D feature volume is input into a pre-trained query generation network to obtain an initial voxel query. The 2D feature maps are clustered and then fused into the initial voxel query to obtain the enhanced voxel query. The enhanced voxel query is upsampled to generate a high-resolution voxel query, and the high-resolution voxel query is input into a classification network to obtain the prediction results of the 3D occupancy grid of each multi-view image.

[0006] Optionally, the step of performing illumination enhancement on each of the multi-view images to obtain a corresponding enhanced multi-view image includes: For each of the multi-view images, the multi-view images are input into a global illumination network to generate global guidance features; The multi-view image is segmented into multiple local image blocks; Each of the local image patches and the global guidance features are input into a local contrast enhancement network to obtain each enhanced local image patch with enhanced illumination. The enhanced local image blocks are stitched together and input into the optimization module to obtain the final enhanced multi-view image.

[0007] Optionally, the step of inputting each of the local image patches and the global guidance features into a local contrast enhancement network to obtain illuminated enhanced local image patches includes: For each of the local image blocks, determine whether the local image block has reached the preset illumination level; If not, the contrast of the local image block is enhanced by the contrast enhancement module to obtain the first image block; Based on the modulation formula, the first image block is modulated according to the global guidance features to obtain the second image block; The second image patch is processed by CDC convolution to obtain an enhanced local image patch with improved illumination.

[0008] Optionally, the modulation formula is:

[0009] in, and These are global pooling and global mean pooling, respectively. For the global guidance feature, This refers to the first image block.

[0010] Optionally, the step of clustering the 2D feature maps and fusing them into the initial voxel query to obtain the enhanced voxel query includes: Clustering the feature points in all the 2D feature maps yields multiple 2D prototypes; wherein each 2D prototype represents a semantically similar region. Multiple 3D prototypes are obtained by projecting each of the 2D prototypes into 3D space using a multilayer perceptron. The 3D prototypes are weighted and fused into the initial voxel query to obtain the enhanced voxel query.

[0011] Optionally, the step of weightedly fusing each of the 3D prototypes into the initial voxel query to obtain an enhanced voxel query includes: Calculate the cosine similarity between each voxel of the initial voxel query and each of the 3D prototypes; The cosine similarity values ​​are normalized to obtain the affinity matrix between the initial voxel and each of the 3D prototypes; each element in the affinity matrix represents the matching probability between a voxel and a 3D prototype. Based on the affinity matrix, the prototype perception features of each voxel are calculated by weighted summation of each 3D prototype and each matching probability. Each voxel in the initial voxel query is concatenated or added to its corresponding prototype-aware feature to obtain the enhanced voxel query.

[0012] Optionally, the upsampling of the enhanced voxel query to generate a high-resolution voxel query, and the input of the high-resolution voxel query into the classification network to obtain the prediction results of the 3D occupancy grid for each of the multi-view images, includes: The enhanced voxel query is perturbed at least once to obtain at least one perturbed voxel query; The enhanced voxel query and each of the perturbation voxel queries are upsampled into multiple high-resolution voxel queries by using a transposed 3D convolution with shared weights. After performing consistency regularization on each of the high-resolution voxel queries, the data is input into a classification network to obtain the prediction results of the 3D occupancy grid for each of the multi-view images.

[0013] Secondly, embodiments of the present invention provide a 3D occupancy grid prediction device, comprising: The image acquisition unit is used to acquire multiple multi-view images from different perspectives captured at the same time. The illumination enhancement unit is used to perform illumination enhancement on each of the multi-view images to obtain the corresponding enhanced multi-view images; The feature extraction unit is used to extract features from each of the enhanced multi-view images to obtain each 2D feature map. The initial voxel query construction unit is used to project each of the 2D feature maps into the same 3D space to obtain a 3D feature map, and multiply the 3D feature map by the pre-trained weights to obtain the initial voxel query; The voxel query enhancement unit is used to cluster the 2D feature maps and then fuse them into the initial voxel query to obtain an enhanced voxel query. The occupancy grid prediction unit is used to upsample the enhanced voxel query to generate a high-resolution voxel query, and input the high-resolution voxel query into the classification network to obtain the prediction results of the 3D occupancy grid for each of the multi-view images.

[0014] Thirdly, embodiments of the present invention provide an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the method described in any of the above-mentioned embodiments.

[0015] Fourthly, embodiments of the present invention provide a storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in any of the preceding claims.

[0016] The present invention provides a 3D occupancy grid prediction method, apparatus, electronic device and storage medium, which improves the image quality acquired in low light scenes by performing illumination enhancement processing on the acquired multi-view images, and makes 3D occupancy grid prediction in low light scenes more accurate by using the illumination-enhanced multi-view images.

[0017] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A schematic structural block diagram of an electronic device provided in an embodiment of the present invention; Figure 2 A flowchart illustrating a 3D occupancy grid prediction method provided in an embodiment of the present invention; Figure 3 This is another flowchart illustrating a 3D occupancy grid prediction method provided in an embodiment of the present invention; Figure 4 A flowchart illustrating step S223 provided in an embodiment of the present invention; Figure 5 This is another flowchart illustrating a 3D occupancy grid prediction method provided in an embodiment of the present invention; Figure 6 A flowchart illustrating step S253 provided in an embodiment of the present invention; Figure 7 This is another flowchart illustrating a 3D occupancy grid prediction method provided in an embodiment of the present invention; Figure 8This is a schematic structural block diagram of a 3D occupancy grid prediction device provided in an embodiment of the present invention.

[0020] Icons: 100 - Electronic device; 101 - Memory; 102 - Communication interface; 103 - Processor; 104 - Communication bus; 300 - 3D occupancy grid prediction device; 310 - Image acquisition unit; 320 - Illumination enhancement unit; 330 - Feature extraction unit; 340 - Initial voxel query construction unit; 350 - Voxel query enhancement unit; 360 - Occupancy grid prediction unit. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0022] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0023] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0024] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0025] As described in the background section, 3D occupancy grid prediction technology accurately identifies vehicles, pedestrians, and obstacles by constructing a refined semantic representation of object distribution in three-dimensional space in real time. It also detects important environmental elements that traditional two-dimensional perception systems easily miss, providing a complete environmental cognitive foundation for autonomous driving decision-making systems. However, in low-light environments, the data quality acquired by sensors deteriorates, directly affecting the detection accuracy and reliability of the 3D occupancy grid prediction model under low-light conditions.

[0026] Based on the above, embodiments of the present invention provide a 3D occupancy grid prediction method, apparatus, electronic device, and storage medium. By performing illumination enhancement processing on the acquired multi-view images and then performing 3D occupancy grid prediction, the 3D occupancy grid prediction in low-light scenes can be made more accurate.

[0027] To implement the process steps and functions of the various examples of this invention, please refer to [link / reference]. Figure 1 , Figure 1 This is a schematic structural block diagram of an electronic device provided in an embodiment of the present invention. The electronic device 100 includes a memory 101 and a processor 103, which are electrically connected directly or indirectly to each other to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses 104 or signal lines. The memory 101 can be used to store software programs and modules, and the processor 103 executes the software programs and modules stored in the memory 101, thereby performing various functional applications and data processing.

[0028] Electronic device 100 can be, but is not limited to, a personal computer (PC), a server, a distributed computer, etc. It is understood that electronic device 100 is not limited to a physical server, but can also be a virtual machine on a physical server, a virtual machine built on a cloud platform, or any other computer that can provide the same functionality as the server or virtual machine. The operating system of electronic device 100 can be, but is not limited to, Windows, Linux, etc.

[0029] The memory 101 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0030] The communication connection between the electronic device 100 and external devices is achieved through at least one communication interface 102 (which can be wired or wireless).

[0031] Processor 103 may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of this embodiment can be completed by integrated logic circuits in the hardware of processor 103 or by instructions in software form. Processor 103 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0032] Understandable. Figure 1 The structure shown is for illustrative purposes only; the electronic device 100 may also include components that are more advanced than those shown. Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown. Figure 1 The components shown can be implemented using hardware, software, or a combination thereof.

[0033] The 3D occupancy grid prediction method provided by this invention will be described below by way of example. See also Figure 2 The subject executing this method can be one of the above. Figure 1 The electronic device 100 shown, the method includes as follows Figure 2 The following steps are described: S210: Acquire multiple multi-view images from different perspectives captured at the same time.

[0034] S220: Perform illumination enhancement on each multi-view image to obtain the corresponding enhanced multi-view image.

[0035] S230: Extract features from each enhanced multi-view image to obtain each 2D feature map.

[0036] S240: Project each 2D feature map onto the same 3D space to obtain a 3D feature volume, and input the 3D feature volume into a pre-trained query generation network to obtain an initial voxel query.

[0037] S250: Cluster each 2D feature map and then fuse it into the initial voxel query to obtain the enhanced voxel query.

[0038] S260: Upsample the enhanced voxel query to generate a high-resolution voxel query, and input the high-resolution voxel query into the classification network to obtain the prediction results of the 3D occupancy grid for each multi-view image.

[0039] First, acquire multiple multi-view images taken at the same time from different perspectives. For example, in a scenario where there are 6 cameras around a vehicle, 6 multi-view images will be acquired at the same time. These 6 multi-view images are taken from different perspectives.

[0040] After acquiring multi-view images, each image undergoes illumination enhancement, which can be processed in parallel for faster completion, resulting in enhanced multi-view images. For example, images may appear dark and lack detail due to insufficient lighting at night. Illumination enhancement improves image brightness and contrast, sharpens edges, and provides higher-quality data for subsequent feature extraction.

[0041] Feature extraction is performed on each multi-view image after illumination enhancement. The feature extraction network can use ResNet50 + FPN. After feature extraction, each multi-view image corresponds to a 2D feature map.

[0042] After obtaining each 2D feature map, each 2D feature map will be processed in two parts.

[0043] First, the 2D feature maps are back-projected into 3D space using camera parameters (intrinsic and extrinsic parameters). Features from all viewpoints are fused in a unified coordinate system to form a preliminary 3D feature volume. This 3D feature volume is then input into a query generation network to obtain initial voxel queries. These initial voxel queries form the basis for subsequent predictions and can be understood as the initial semantic representation of each voxel in 3D space. For example, feature maps from six 2D images are back-projected into a unified 3D space around a vehicle to form a preliminary 3D feature volume. The query generation network then compresses this 3D feature volume into a learnable set of voxel queries.

[0044] Secondly, the feature points of each 2D feature map are clustered, and the clustered features are fused to perform an initial voxel query, so that the fused enhanced voxel query has richer semantic information.

[0045] Finally, the enhanced voxel query is upsampled and input into the classification network to perform semantic classification on each voxel, obtaining the occupancy status and category label of each voxel, and finally obtaining the prediction result of the 3D occupancy grid.

[0046] This method improves the image quality acquired in low-light scenes by performing illumination enhancement processing on the acquired multi-view images, and uses the illumination-enhanced multi-view images to perform 3D occupancy grid prediction, thereby making the 3D occupancy grid prediction in low-light scenes more accurate.

[0047] There are various ways to enhance the illumination of multi-view images. For example, one can directly enhance the colors of the original multi-view image, or enhance the features extracted from the multi-view image. Exemplarily, this embodiment of the invention uses feature enhancement to enhance the illumination of multi-view images, referring to... Figure 3 The above step S220 may include the following steps: S221: For each multi-view image, input the multi-view image into the global illumination network to generate global guidance features.

[0048] S222: Divide a multi-view image into multiple local image blocks.

[0049] S223: Input each local image patch and the global guidance features into the local contrast enhancement network to obtain each enhanced local image patch with enhanced illumination.

[0050] S224: Stitch together the enhanced local image blocks and input them into the optimization module to obtain the final enhanced multi-view image.

[0051] The same processing is applied to each multi-view image; specifically, the multi-view image is processed in two parts.

[0052] First, multi-view images are input into a global illumination network to generate global guidance features. This global illumination network can be a Swin Transformer. The Swin Transformer extracts features from the multi-view images to obtain global guidance features, providing prior knowledge of the global illumination distribution for the subsequent local contrast enhancement network, ensuring that the enhanced local image patches are consistent in overall illumination.

[0053] Secondly, the multi-view image is segmented into multiple local image blocks. The segmented local image blocks and global guidance features are input into a local contrast enhancement network to enhance the illumination of each local image block, resulting in enhanced local image blocks with improved illumination.

[0054] Finally, the enhanced local image patches are stitched together and input into the optimization module to obtain the final enhanced multi-view image. This optimization module can be a lightweight network containing two stacked 1×1 and 3×3 convolutional layers, each followed by a PReLU activated network. It can fuse features from different enhanced local image patches, eliminate mesh artifacts caused by stitching, improve the naturalness of details, and output the final enhanced local image patches.

[0055] To ensure that the enhanced local image patches are consistent with the overall illumination, prior knowledge of the global illumination distribution can be provided when enhancing local image patches. For example, global guidance features can be used to modulate each local image patch. See also [example description]. Figure 4 Step S223 may include: S2231: For each local image block, determine whether the local image block has reached the preset illumination level.

[0056] S2232: If not, perform contrast enhancement on the local image block using the contrast enhancement module to obtain the first image block.

[0057] S2233: Based on the modulation formula, the first image block is modulated according to the global guidance features to obtain the second image block.

[0058] S2234: The second image block is processed by CDC convolution to obtain an enhanced local image block with enhanced illumination.

[0059] For each local image patch, it is first determined whether the local image patch has reached the preset illumination level. In this embodiment of the invention, a cascaded 3×3 and 5×5 convolutional layer is used to reach a fully connected layer and then a Sigmoid activation. The binary classification result is output to determine whether the current local image patch has reached a sufficient illumination level.

[0060] If the current local image patch already has sufficient illumination, no processing is performed.

[0061] If the current local image patch does not reach a sufficient level of illumination, the contrast of the local image patch is enhanced by a contrast enhancement module to obtain the first image patch. This contrast enhancement module is a module containing 3×3 convolution and PReLU activation to enhance the contrast.

[0062] Furthermore, based on the modulation formula, the first image block is modulated according to the global guidance features to obtain the second image block. The modulation formula is:

[0063] in, and These are global pooling and global mean pooling, respectively. As a global guidance feature, This is the first image block.

[0064] Finally, the second image patch is processed by CDC convolution to enhance fine-grained texture details, resulting in an enhanced local image patch with improved illumination.

[0065] To obtain more feature information from the voxel query, feature points in each 2D feature map can be clustered, and then each category can be fused with the voxel query. See [link / reference] Figure 5 Step S250 may include the following steps: S251: Cluster the feature points in all 2D feature maps to obtain multiple 2D prototypes; where each 2D prototype represents a semantically similar region.

[0066] S252: Project each 2D prototype onto 3D space using a multilayer perceptron to obtain multiple 3D prototypes.

[0067] S253: The 3D prototypes are weighted and fused into the initial voxel query to obtain the enhanced voxel query.

[0068] While generating initial voxel queries from all 2D feature maps, feature points in all 2D feature maps are clustered to obtain multiple 2D prototypes. For example, k-means analysis is performed on these 2D features from different images, locations, and scales. The aim is to group together features from visually or semantically similar image regions (e.g., all belonging to "vehicle boundaries," "sidewalk textures," or "tree regions").

[0069] For example, the 2D feature map is iteratively grouped and clustered to generate Q prototypes E = {E_1, E_2, ..., E_Q}. Each prototype represents a semantic pattern.

[0070] A multilayer perceptron (MLP) is used to project the 2D prototype E into 3D space, resulting in the 3D prototype E' = {E'_1, E'_2, ..., E'_Q}. This step is to adapt the 2D semantic information to the representation in 3D space.

[0071] Then, each 3D prototype E' is weighted and fused into the previously generated initial voxel query to obtain the enhanced voxel query.

[0072] The method of weighted fusion of 3D prototypes into the initial voxel query can be based on the correlation between each 3D prototype and each voxel in the initial voxel query. See [link to relevant documentation]. Figure 6 Step S253 may include: S2531: Calculate the cosine similarity between each voxel of the initial voxel query and each 3D prototype.

[0073] S2532: Normalize each cosine similarity to obtain the affinity matrix between the initial voxel and each 3D prototype; where an element in the affinity matrix represents the matching probability between a voxel and a 3D prototype.

[0074] S2533: Based on the affinity matrix, the prototype perception features of each voxel are calculated by weighted summation of each 3D prototype and each matching probability.

[0075] S2534: Concatenate or add each voxel of the initial voxel query with its corresponding prototype-aware feature to obtain the enhanced voxel query.

[0076] First, the affinity matrix between the initial voxels and each 3D prototype is calculated using cosine similarity. This affinity matrix represents the correlation or matching degree between each voxel Q_i and each 3D prototype E'_j. Cosine similarity is used to measure the similarity between the feature vectors of Q_i and E'_j. The higher the similarity, the better the feature at the query position of the voxel matches the semantic pattern represented by the prototype. The Sigmoid function (or other normalization methods, such as Softmax row / column normalization) is applied to the calculated original affinity matrix to convert the similarity into a probability distribution, representing the likelihood (probability) that voxel Q_i belongs to the semantic pattern represented by prototype E'_j, thus obtaining the final affinity matrix.

[0077] Using the affinity matrix A as weights, the features of all prototypes E' are weighted and combined to generate a new feature vector for each voxel Q_i that incorporates information from all prototypes. This can be understood as: This new prototype-aware feature, VoxelFeature_i, represents the aggregation of semantic information of all prototypes associated with the voxel Q_i location.

[0078] Finally, the original voxel queries Q generated above are fused together, for example, by concatenating Concat(Q_i, VoxelFeature_i) or adding Q_i+ VoxelFeature_i. The result of the fusion is the enhanced voxel query.

[0079] In one alternative implementation, see [link to implementation details]. Figure 7 The above step S260 may include: S261: Perturb the enhanced voxel query at least once to obtain at least one perturbed voxel query.

[0080] S262: Upsample the enhanced voxel query and each perturbation voxel query into multiple high-resolution voxel queries by using a transposed 3D convolution with shared weights.

[0081] S263: After performing consistency regularization on each high-resolution voxel query, input it into the classification network to obtain the prediction results of the 3D occupancy grid of each multi-view image.

[0082] Before performing classification prediction, the enhanced voxel queries can be enhanced first, then upsampled, and finally all the upsampled high-resolution voxel queries are fed into the classification network for prediction and classification to obtain the prediction results of the 3D occupied grid.

[0083] For example, firstly, enhanced voxel queries are input and various enhancements are applied, including feature-level enhancements such as random discarding and adding Gaussian noise to disrupt local features and improve robustness. Spatial-level enhancements include voxel transposition and flipping to change the spatial perspective and generate multiple perturbed voxel queries. Secondly, the low-resolution enhanced and perturbed voxel queries are upsampled into multiple high-resolution voxel queries through a shared-weighted transposed 3D convolution. To ensure consistency in the decoding process across different perspectives and avoid overfitting, consistent regularization is applied to the multiple high-resolution voxel queries to prevent inconsistencies in predictions caused by enhancements and improve model robustness. Finally, a classification network performs semantic classification on each voxel in the multiple high-resolution voxel queries to generate high-resolution 3D occupancy predictions, including the occupancy status and category label of each voxel.

[0084] Furthermore, embodiments of the present invention also provide a 3D occupancy grid prediction device, see [link to related document]. Figure 8 The 3D occupancy grid prediction device 300 includes: The image acquisition unit 310 is used to acquire multiple multi-view images from different perspectives collected at the same time.

[0085] The illumination enhancement unit 320 is used to enhance the illumination of each multi-view image to obtain the corresponding enhanced multi-view image.

[0086] The feature extraction unit 330 is used to extract features from each enhanced multi-view image to obtain each 2D feature map.

[0087] The initial voxel query construction unit 340 is used to project each 2D feature map onto the same 3D space to obtain a 3D feature map, and multiply the 3D feature map by the pre-trained weights to obtain the initial voxel query.

[0088] The voxel query enhancement unit 350 is used to cluster each 2D feature map and then fuse it into the initial voxel query to obtain the enhanced voxel query.

[0089] Occupied raster prediction unit 360 is used to upsample the enhanced voxel query, generate a high-resolution voxel query, and input the high-resolution voxel query into the classification network to obtain the prediction results of the 3D occupied raster for each multi-view image.

[0090] In summary, the 3D occupancy grid prediction method, apparatus, electronic device, and storage medium provided by this invention improve the image quality acquired in low-light scenes by performing illumination enhancement processing on the acquired multi-view images. Utilizing the enhanced multi-view images for 3D occupancy grid prediction makes the prediction more accurate in low-light scenes. Modulating each local image patch using global guidance features ensures that the enhanced local image patches are consistent in overall illumination. Clustering feature points in each 2D feature map and fusing each category with a voxel query allows the voxel query to obtain more feature information.

[0091] In the embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0092] In addition, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0093] If the functionality is implemented as a software module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a computer-readable storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0094] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0095] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A 3D occupancy grid prediction method, characterized in that, include: Acquire multiple multi-view images taken at the same time from different perspectives; Each of the multi-view images is enhanced with illumination to obtain the corresponding enhanced multi-view image; Feature extraction is performed on each of the enhanced multi-view images to obtain a 2D feature map; The 2D feature maps are projected onto the same 3D space to obtain a 3D feature volume, and the 3D feature volume is input into a pre-trained query generation network to obtain an initial voxel query. The 2D feature maps are clustered and then fused into the initial voxel query to obtain the enhanced voxel query. The enhanced voxel query is upsampled to generate a high-resolution voxel query, and the high-resolution voxel query is input into a classification network to obtain the prediction results of the 3D occupancy grid of each multi-view image.

2. The method according to claim 1, characterized in that, The step of performing illumination enhancement on each of the multi-view images to obtain corresponding enhanced multi-view images includes: For each of the multi-view images, the multi-view images are input into a global illumination network to generate global guidance features; The multi-view image is segmented into multiple local image blocks; Each of the local image patches and the global guidance features are input into a local contrast enhancement network to obtain each enhanced local image patch with enhanced illumination. The enhanced local image blocks are stitched together and input into the optimization module to obtain the final enhanced multi-view image.

3. The method according to claim 2, characterized in that, The step of inputting each of the local image patches and the global guidance features into a local contrast enhancement network to obtain enhanced local image patches with enhanced illumination includes: For each of the local image blocks, determine whether the local image block has reached the preset illumination level; If not, the contrast of the local image block is enhanced by the contrast enhancement module to obtain the first image block; Based on the modulation formula, the first image block is modulated according to the global guidance features to obtain the second image block; The second image patch is processed by CDC convolution to obtain an enhanced local image patch with improved illumination.

4. The method according to claim 3, characterized in that, The modulation formula is: in, and These are global pooling and global mean pooling, respectively. For the global guidance feature, This refers to the first image block.

5. The method according to claim 1, characterized in that, The step of clustering the 2D feature maps and fusing them into the initial voxel query to obtain the enhanced voxel query includes: Clustering the feature points in all the 2D feature maps yields multiple 2D prototypes; wherein each 2D prototype represents a semantically similar region. Multiple 3D prototypes are obtained by projecting each of the 2D prototypes into 3D space using a multilayer perceptron. The 3D prototypes are weighted and fused into the initial voxel query to obtain the enhanced voxel query.

6. The method according to claim 5, characterized in that, The step of weightedly fusing each of the 3D prototypes into the initial voxel query to obtain the enhanced voxel query includes: Calculate the cosine similarity between each voxel of the initial voxel query and each of the 3D prototypes; The cosine similarity values ​​are normalized to obtain the affinity matrix between the initial voxel and each of the 3D prototypes; each element in the affinity matrix represents the matching probability between a voxel and a 3D prototype. Based on the affinity matrix, the prototype perception features of each voxel are calculated by weighted summation of each 3D prototype and each matching probability. Each voxel in the initial voxel query is concatenated or added to its corresponding prototype-aware feature to obtain the enhanced voxel query.

7. The method according to claim 1, characterized in that, The upsampling of the enhanced voxel query to generate a high-resolution voxel query, and inputting the high-resolution voxel query into the classification network to obtain the prediction results of the 3D occupancy grid for each of the multi-view images, includes: The enhanced voxel query is perturbed at least once to obtain at least one perturbed voxel query; The enhanced voxel query and each of the perturbation voxel queries are upsampled into multiple high-resolution voxel queries by using a transposed 3D convolution with shared weights. After performing consistency regularization on each of the high-resolution voxel queries, the data is input into a classification network to obtain the prediction results of the 3D occupancy grid for each of the multi-view images.

8. A 3D occupancy grid prediction device, characterized in that, include: The image acquisition unit is used to acquire multiple multi-view images from different perspectives captured at the same time. The illumination enhancement unit is used to perform illumination enhancement on each of the multi-view images to obtain the corresponding enhanced multi-view images; The feature extraction unit is used to extract features from each of the enhanced multi-view images to obtain each 2D feature map. The initial voxel query construction unit is used to project each of the 2D feature maps into the same 3D space to obtain a 3D feature map, and multiply the 3D feature map by the pre-trained weights to obtain the initial voxel query; The voxel query enhancement unit is used to cluster the 2D feature maps and then fuse them into the initial voxel query to obtain an enhanced voxel query. The occupancy grid prediction unit is used to upsample the enhanced voxel query to generate a high-resolution voxel query, and input the high-resolution voxel query into the classification network to obtain the prediction results of the 3D occupancy grid for each of the multi-view images.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 7.

Images