A three-dimensional grid occupancy prediction method, device and equipment

Abstract

The present disclosure belongs to the technical field of computer vision, and particularly relates to a three-dimensional grid occupancy prediction method, device and equipment. The method comprises: obtaining three-dimensional voxel features of a target scene; inputting the three-dimensional voxel features into a layered Mamba module to generate three-dimensional voxel features with enhanced spatial features, wherein the layered Mamba module comprises a plurality of Mamba blocks for processing different scale features, and is used for considering global information and local information under multi-scale to enhance spatial features, and the Mamba blocks take a one-dimensional sequence formed by self-adaptive expansion and rearrangement of the three-dimensional voxel features based on the spatial structure of the target scene as input; and performing three-dimensional grid occupancy prediction based on the three-dimensional voxel features with enhanced spatial features. The present disclosure improves the prediction effect of three-dimensional grid occupancy prediction of a complex scene.

Description

TECHNICAL FIELD

[0001] The present disclosure belongs to the technical field of computer vision, and particularly relates to a three-dimensional grid occupancy prediction method, device and equipment. BACKGROUND

[0002] The target of three-dimensional grid occupancy prediction is to estimate the occupancy and semantic labels of a three-dimensional space using various input signals.

[0003] Most existing three-dimensional grid occupancy prediction networks are built based on convolutional neural networks. The network built in this way can better utilize local scene information and has certain prediction capability. The Mamba network has become a hot spot in recent research work because it integrates a selection mechanism on the basis of a state space model, thereby effectively capturing scene information while greatly reducing computational complexity.

[0004] However, on the one hand, the Mamba network is mainly designed for language models, and the input data is a one-dimensional sequence, while the feature representation of three-dimensional grid occupancy prediction is usually three-dimensional voxels. When directly converting three-dimensional voxels to one-dimensional, the inherent spatial relationship between adjacent voxels in the three-dimensional space may be lost, resulting in being separated far apart in the one-dimensional sequence. The loss of this spatial relationship weakens the ability of the Mamba network to effectively understand the global scene, thereby affecting the prediction performance. Therefore, when rearranging three-dimensional voxels into a one-dimensional sequence, it is necessary to consider how to effectively express the correlation of the inherent spatial relationship between adjacent voxels. However, the existing methods are mainly designed based on the spatial structure of the target scene, and if the actual scene type is different from the target scene type, the efficiency of expressing the correlation of the inherent spatial relationship between adjacent voxels will decrease. In particular, a complex scene contains multiple scene types, and a single unfolding method cannot efficiently express the correlation of the inherent spatial relationship between adjacent voxels for a complex scene containing different scene types. On the other hand, directly using the Mamba network to process complex scenes involving large-scale voxels, such as autonomous driving environments, because the global information of these scenes is rich and the details are complex, the Mamba network is difficult to fully understand such a large amount of voxels, resulting in not accurate enough grasp of the overall scene, thereby affecting the reliability and accuracy of the prediction result. In summary, the existing three-dimensional grid occupancy prediction scheme based on the Mamba network has poor processing effect on complex scenes. SUMMARY

[0005] The embodiment of the present disclosure proposes a three-dimensional grid occupancy prediction scheme based on a hierarchical Mamba module, which solves the problem of poor prediction effect of the existing three-dimensional grid occupancy prediction scheme based on the Mamba network on complex scenes.

[0006] The first aspect of the embodiment of the present disclosure provides a three-dimensional grid occupancy prediction method, comprising:

[0007] acquiring three-dimensional voxel features of a target scene;

[0008] inputting the three-dimensional voxel features into a hierarchical Mamba module to generate three-dimensional voxel features with enhanced spatial features, wherein the hierarchical Mamba module comprises a plurality of Mamba blocks for processing different scale features, and is configured to balance global information and local information at multiple scales to enhance spatial features, and each Mamba block takes as input a one-dimensional sequence formed by adaptive unfolding and rearrangement of the three-dimensional voxel features based on a spatial structure of the target scene;

[0009] performing three-dimensional grid occupancy prediction based on the three-dimensional voxel features with enhanced spatial features.

[0010] In some embodiments, the hierarchical Mamba module is composed of an encoder and a decoder, wherein the encoder and the decoder each comprise a plurality of Mamba blocks for processing different scale features, each Mamba block contains at least two Mamba networks, the different Mamba blocks of the encoder comprise down-sampling operations for multi-scale voxel feature extraction, the different Mamba blocks of the decoder comprise up-sampling operations for multi-scale voxel feature integration, the Mamba blocks of the same scale of the encoder and the decoder are connected by skip connections, the input of the Mamba block is a one-dimensional sequence generated by rearrangement of three-dimensional voxel features, and the output is three-dimensional voxel features integrated from the one-dimensional sequence.

[0011] In some embodiments, the acquiring three-dimensional voxel features of a target scene comprises:

[0012] acquiring a two-dimensional image of the target scene, and generating image voxel features of the target scene based on the two-dimensional image;

[0013] acquiring point cloud data of the target scene, and generating point cloud voxel features of the target scene based on the point cloud data;

[0014] fusing the image voxel features and the point cloud voxel features of the target scene to generate three-dimensional voxel features of the target scene.

[0015] In some embodiments, the adaptive unfolding and rearrangement of the three-dimensional voxel features based on the spatial structure of the target scene to form a one-dimensional sequence comprises:

[0016] determining a scene type of the target scene, wherein the scene type comprises a horizontal moving scene, an omnidirectional moving scene, and a low activity scene;

[0017] unfolding and rearranging the three-dimensional voxel features to form a one-dimensional sequence based on the scene type of the target scene.

[0018] In some embodiments, the determining the scene type of the target scene comprises:

[0019] obtaining a prediction range of the target scene and a pose change of a prediction subject in the target scene, and determining the scene type of the target scene based on the prediction range and the pose change.

[0020] In some embodiments, the rearranging the three-dimensional voxel features into a one-dimensional sequence based on the scene type of the target scene comprises:

[0021] when the scene type is a horizontal movement scene, representing the target scene with a (X, Y, Z) three-dimensional coordinate system, and generating a vertical voxel sequence for an xy plane according to the three-dimensional voxel features of the target scene having the same (X, Y) coordinate values in a low-to-high order of Z coordinate values;

[0022] all the vertical voxel sequences generated by all the three-dimensional voxel features of the target scene form a vertical voxel sequence set, and the vertical voxel sequences are sorted and connected according to a two-dimensional Hilbert curve on the xy plane based on the (X, Y) coordinate values of each of the vertical voxel sequences in the vertical voxel sequence set, to form the one-dimensional sequence.

[0023] In some embodiments, the rearranging the three-dimensional voxel features into a one-dimensional sequence based on the scene type of the target scene comprises:

[0024] when the scene type is an omnidirectional movement scene, rearranging the three-dimensional voxel features of the target scene based on a three-dimensional Hilbert curve to form the one-dimensional sequence.

[0025] In some embodiments, the rearranging the three-dimensional voxel features into a one-dimensional sequence based on the scene type of the target scene comprises:

[0026] when the scene type is a low-activity scene, representing the target scene with a (X, Y, Z) three-dimensional coordinate system, and generating a vertical voxel sequence for an xy plane according to the three-dimensional voxel features of the target scene having the same (X, Y) coordinate values in a low-to-high order of Z coordinate values;

[0027] all the vertical voxel sequences generated by all the three-dimensional voxel features of the target scene form a vertical voxel sequence set, and the vertical voxel sequences are sorted and connected according to a two-dimensional Hilbert curve on the xy plane based on the (X, Y) coordinate values of each of the vertical voxel sequences in the vertical voxel sequence set, to form the one-dimensional sequence.

[0028] A second aspect of the embodiments of the present disclosure provides a three-dimensional grid occupancy prediction device, comprising:

[0029] a feature acquisition module, configured to acquire three-dimensional voxel features of a target scene;

[0030] a feature enhancement module, configured to input the three-dimensional voxel features into a hierarchical Mamba module to generate three-dimensional voxel features with enhanced spatial features, wherein the hierarchical Mamba module comprises a plurality of Mamba blocks for processing features of different scales, and is configured to consider global information and local information under multiple scales to enhance spatial features, and the Mamba blocks take as input a one-dimensional sequence formed by adaptive unfolding and rearrangement of the three-dimensional voxel features based on a spatial structure of the target scene;

[0031] a prediction module, configured to perform three-dimensional grid occupancy prediction based on the three-dimensional voxel features with enhanced spatial features.

[0032] A third aspect of the embodiments of the present disclosure provides a three-dimensional grid occupancy prediction device, comprising a memory and a processor:

[0033] the memory, configured to store a computer program;

[0034] the processor, configured to implement the method of any one of the first aspect of the present disclosure when the computer program is executed.

[0035] In summary, the three-dimensional grid occupancy prediction method, device and equipment provided by the embodiments of the present disclosure can improve the performance of three-dimensional grid occupancy prediction of complex scenes by unfolding and rearranging three-dimensional voxel features based on the spatial structure of complex scenes into a one-dimensional sequence as input of the Mamba network to retain as much as possible the inherent spatial relationship between adjacent voxels in three-dimensional space in the rearrangement process. Meanwhile, the hierarchical Mamba module extracts spatial features from multiple scales and integrates them, which solves the problem that the Mamba network is difficult to fully understand the global features of large-scale three-dimensional voxels, thereby enhancing the ability to accurately grasp complex scenes involving large-scale three-dimensional voxels and extract spatial features, and improving the prediction effect of complex scenes. BRIEF DESCRIPTION OF DRAWINGS

[0036] The features and advantages of the present disclosure will be more clearly understood through reference to the following drawings, which are illustrative rather than limiting of the present disclosure. In the drawings:

[0037] Figure 1 is a schematic diagram of a computer system to which the present disclosure is applicable;

[0038] Figure 2 is a system framework of a three-dimensional grid occupancy prediction method based on a Mamba network according to some embodiments of the present disclosure;

[0039] Figure 3 This is a flowchart illustrating a three-dimensional grid occupancy prediction method according to some embodiments of the present disclosure;

[0040] Figure 4 This is a schematic diagram of the Hilbert unfolding of the adaptive scene shown in this disclosure;

[0041] Figure 5 This is a schematic diagram of a layered Mamba module according to some embodiments of the present disclosure;

[0042] Figure 6 This is a schematic diagram of the Mamba block framework according to some embodiments of the present disclosure;

[0043] Figure 7 This is a performance comparison of different 3D grid occupancy prediction methods;

[0044] Figure 8 It is the RGB image data of the target scene in OpenOccupancy;

[0045] Figure 9 It is the point cloud data of the target scene in OpenOccupancy;

[0046] Figure 10 Based on Figure 3 The method described in S310-S330 predicts the occupancy of the three-dimensional raster in the target scene;

[0047] Figure 11 yes Figures 8-10 The truth value of the target scene;

[0048] Figure 12 This is a schematic diagram of a three-dimensional grid occupancy prediction device according to some embodiments of the present disclosure;

[0049] Figure 13 This is a schematic diagram of a three-dimensional grid occupancy prediction device according to an embodiment of the present disclosure. Detailed Implementation

[0050] In the following detailed description, numerous specific details of this disclosure are illustrated by way of example in order to provide a thorough understanding of the relevant disclosure. However, it will be apparent to those skilled in the art that this disclosure can be practiced without these details. It should be understood that the terms “system,” “apparatus,” “unit,” and / or “module” used in this disclosure are a method of distinguishing different parts, elements, sections, or components at different levels in a sequential arrangement. However, these terms may be replaced with other expressions if they can achieve the same purpose.

[0051] It should be understood that when a device, unit, or module is referred to as being "on", "connected to", or "coupled to" another device, unit, or module, it can be directly on, connected, or coupled to the other device, unit, or module, or intervening devices, units, or modules can be present, unless the context clearly dictates otherwise. For example, the term "and / or" as used herein refers to and encompasses any and all combinations of one or more of the associated listed items.

[0052] The terminology used in the disclosure is only for the purpose of describing particular embodiments and is not intended to limit the scope of the disclosure. As used in the specification and claims of the disclosure, the words "a", "an", "one", and / or "the" are not intended to refer to only a singular entity, but rather can include one or more entities, unless the context clearly indicates otherwise. In general, the term "includes" or "including" means the inclusion of an explicitly identified item, but not the exclusion of other items, and such a term does not constitute an exhaustive or exclusive list of the items.

[0053] These and other features and characteristics of the disclosure, the methods of operation, the functions of the related elements of structure and the combination of parts and economies of manufacture will become more apparent upon consideration of the following description and the appended claims with reference to the accompanying drawings, all of which form a part of this specification. It is to be expressly understood, however, that the drawings are for the purpose of illustration and description only and are not intended as a definition of the limits of the disclosure. It should be understood that the drawings are not drawn to scale.

[0054] Various structural diagrams are used in the disclosure to illustrate various modifications according to embodiments of the disclosure. It should be understood that the foregoing or the following structures are not intended to limit the disclosure. The scope of protection of the disclosure is subject to the claims.

[0055] Figure 1 is a schematic diagram of a computer system to which the disclosure is applicable. Figure 1 The system shown includes a three-dimensional grid occupancy prediction server connected to an image sensor and a point cloud distance sensor, the three-dimensional grid occupancy prediction server being configured to generate a three-dimensional grid occupancy prediction of a target scene based on three-dimensional voxel features of the target scene generated by the image sensor and the point cloud distance sensor.

[0056] The image sensor is configured to acquire image or video data of the target scene, including various monocular or multi-view cameras, monocular or multi-view video cameras; the point cloud distance sensor is configured to acquire point cloud data of the target scene, including various laser radars or millimeter wave radars.

[0057] The three-dimensional grid occupancy prediction server generates three-dimensional voxel features of the target scene based on image data and point cloud data of the target scene, and then performs three-dimensional grid occupancy prediction on the target scene based on the three-dimensional voxel features of the target scene. The three-dimensional grid occupancy prediction server can be any one of a single machine, a cluster or a distributed server.

[0058] The three-dimensional grid occupancy prediction server is deployed with multiple Mamba blocks for processing different scale features, which are used to balance global information and local information at multiple scales to enhance spatial features, and the Mamba blocks take as input a one-dimensional sequence formed by adaptive unfolding and rearrangement of the three-dimensional voxel features based on the spatial structure of the target scene.

[0059] The target of three-dimensional grid occupancy prediction is to estimate the occupancy and semantic labels of three-dimensional space using various input signals. Most existing three-dimensional grid occupancy prediction networks are built based on convolutional neural networks. Networks built in this way can better utilize local scene information and have certain prediction capabilities. However, due to the inherent defects of convolutional neural networks, it is difficult to capture global scene information, and there is still room for improvement. Inspired by the breakthrough progress of transformer networks in the field of computer vision, some recent work has enhanced the processing capability of global scene information by combining transformer networks, thereby achieving certain performance improvement. However, when they process large voxel grids, the quadratic computational complexity makes the computational burden particularly heavy. Especially when deployed in some outdoor scenes, they have to rely on complex feature compression or key point algorithms, which inevitably suffer from information loss. In order to solve the problem of high computational complexity of transformer networks, people have proposed more efficient operators. Among them, state space models are a popular choice. Inspired by control theory, state space models map system inputs to system outputs through hidden states, thereby effectively processing information sequences. As an important variant of state space models, Mamba networks are known for their integrated selection mechanism. Mamba networks can effectively capture global information and process large-scale data in linear time, and have become a strong candidate for the next generation of structures to replace transformer networks. However, Mamba networks are mainly designed for language models, and the input data is a one-dimensional sequence, while the input data of three-dimensional grid occupancy prediction is three-dimensional voxel. When directly converting three-dimensional voxel to one-dimensional, the inherent spatial relationship between adjacent voxels in three-dimensional space is likely to be lost, resulting in being separated far apart in the one-dimensional sequence.

[0060] This problem can be alleviated by embedding the expression of inherent spatial relationship between adjacent voxels when rearranging the three-dimensional voxels into a one-dimensional sequence. However, existing methods are mainly designed based on the spatial structure of the target scene, and if the actual scene type is different from the target scene type, the expression efficiency of the inherent spatial relationship between adjacent voxels will decrease sharply. In particular, a complex scene contains multiple scene types, and a single unfolding method cannot efficiently express the correlation of the inherent spatial relationship between adjacent voxels for a complex scene containing different scene types; at the same time, when directly using the Mamba network to process a complex scene involving a large number of voxels, such as an autonomous driving environment, because the global information of these scenes is rich and the details are complex, the Mamba network is difficult to fully understand such a large amount of voxels, resulting in that the overall scene is not grasped accurately, and thus affecting the reliability and accuracy of the prediction result. In summary, the existing three-dimensional grid occupancy prediction scheme based on the Mamba network has poor processing effect on complex scenes.

[0061] To solve the above problems, the present disclosure proposes a new method for three-dimensional grid occupancy prediction based on the Mamba network, and the system framework is as shown in Figure 2 Specifically, first, the input data features from the RGB camera and the laser radar are extracted and converted into three-dimensional voxel form, then fused and input to the layered Mamba module for processing, and finally output the three-dimensional grid occupancy by the prediction head.

[0062] Figure 3 is a flowchart of a three-dimensional grid occupancy prediction method according to some embodiments of the present disclosure. In some embodiments, the three-dimensional grid occupancy prediction function is performed by a three-dimensional grid occupancy prediction server as shown in Figure 1 The three-dimensional grid occupancy prediction method comprises the following steps:

[0063] S310, obtaining three-dimensional voxel features of a target scene.

[0064] Specifically, the positions of the laser radar and the RGB camera are adjusted to ensure that their respective receptive fields can cover as much scene information as possible. For the laser radar branch, we first perform voxelization processing on the input point cloud P, and then use a laser radar encoder E L to generate laser radar voxel features Similarly, for the RGB camera branch, we input the image data I into an image encoder, which extracts image features using a residual network, aggregates multi-scale features through a feature pyramid network, and then generates image voxel features using a two-dimensional to three-dimensional view transformer wherein, and is the spatial dimension of the voxel feature, and is the channel dimension of these voxel features. We ensure V L and V C are consistent in spatial dimension and concatenate these voxel features in channel dimension to get the multi-modal encoded features V:

[0065] V=concat(V L ,V C )。

[0066] where V is the multi-modal three-dimensional voxel feature, V L is the lidar voxel feature, V C is the image voxel feature, and concat() is the concatenation operation.

[0067] S320, input the three-dimensional voxel feature into a hierarchical Mamba module to generate a three-dimensional voxel feature with enhanced spatial features, wherein the hierarchical Mamba module includes a plurality of Mamba blocks for processing different scale features, and is used to balance global information and local information at multiple scales to enhance spatial features, and the Mamba block takes as input a one-dimensional sequence formed by self-adaptive unfolding and rearrangement of the three-dimensional voxel feature based on the spatial structure of the target scene.

[0068] Before inputting the three-dimensional voxel feature V into the hierarchical Mamba module, it is necessary to rearrange it into a one-dimensional sequence. Inspired by the Hilbert curve, the present disclosure proposes a self-adaptive scene Hilbert unfolding, as shown in Figure 4 Figure 4 The method shown can dynamically and adaptively rearrange the three-dimensional voxel feature into a one-dimensional sequence based on different spatial structures in the three-dimensional grid occupancy prediction task under each scene. Some embodiments of the present disclosure predict the trend of the change of the subject pose by tracking the change of the input point cloud, and adjust the scene scheme by referring to the prediction range, to realize the adaptive function. The scene is mainly divided into three categories: horizontal movement scene, omnidirectional movement scene and low activity scene, which correspond to the main application scenes of outdoor autonomous driving, outdoor unmanned aerial vehicle and indoor augmented reality, respectively.

[0069] ​Specifically, for the horizontal movement scenario, some embodiments of the present disclosure divide the three-dimensional coordinates xyz into the xy plane and the z dimension. Starting from the xy plane of z = 0, each xy coordinate value extends along the z axis direction to form a one-dimensional vertical sequence corresponding to the xy point, and finally all voxels are sorted into a vertical voxel sequence set. Then the vertical voxel sequences in the set are sorted and connected to each other according to the two-dimensional Hilbert curve on the xy plane. This method prioritizes the spatial information of the z axis while maintaining strong spatial proximity in the xy plane; for the omnidirectional movement scenario, considering the uncertainty of the direction, some embodiments of the present disclosure directly use the three-dimensional Hilbert curve for rearrangement; for the low activity scenario, the main concern is the things around the predicted subject. Therefore, some embodiments of the present disclosure are processed in a similar manner to the horizontal movement scenario: first, generate a vertical sequence along the z axis direction for each xy coordinate value, then sort the sequences in layers in the order from the center to the periphery on the xy plane, and finally connect the sequences in order to obtain the rearrangement result.

[0070] The present disclosure proposes a hierarchical Mamba module for balancing global information and local information at multiple scales to enhance spatial features, as shown in Figure 5 . The hierarchical Mamba module includes a hierarchical Mamba encoder E Figure 5 , as shown in M , and a decoder D M . E M is composed of multiple Mamba blocks as shown in Figure 6 , each block containing two Mamba networks, with down-sampling operations between blocks for multi-scale voxel feature extraction. Correspondingly, D M mirrors this structure, each block containing two Mamba networks, with up-sampling operations between blocks for integrating multi-scale voxel features. Figure 6 is a schematic diagram of a Mamba block framework according to some embodiments of the present disclosure. The up-sampling operation in D M introduces a skip connection from the same height block of E M , and maintains the consistency of feature dimensions. The rearrangement strategy of the present disclosure is applied before and after each layer of Mamba blocks, so that the scene features maintain three-dimensional voxel feature representation during the down-sampling and up-sampling stages. Overall, the multi-modal encoding features V are input into the hierarchical Mamba module to generate Mamba features V', which can be described as:

[0071] V' = D M (E M (V))

[0072] where V is the multi-modal three-dimensional voxel feature, V' is the three-dimensional voxel feature of the enhanced spatial feature output by the hierarchical Mamba module, V' maintains the same space and feature as V, E Mis an encoding operation, D M is a decoding operation.

[0073] S330, performing three-dimensional grid occupancy prediction based on the three-dimensional voxel features with enhanced spatial features.

[0074] Specifically, we interpolate the Mamba feature V' space dimension size to match the true value, and then use the MLP to predict the category of each three-dimensional grid occupancy to obtain the prediction result O occ :

[0075] O occ = MLP (Interpolation (V'))

[0076] One embodiment of the present disclosure quantitatively analyzes various three-dimensional grid occupancy prediction methods on the OpenOccupancy dataset. The experimental method includes training using the training set and testing on the validation set. The indicators of the experiment are IoU and mIoU, i.e. intersection over union and mean intersection over union. Among them, mIoU measures the accuracy and completeness of the prediction method in each class, which is the most important indicator of this task. As shown in the experimental results and comparisons such as Figure 7 , it can be seen that the method described in the present disclosure (the method described in S310-S330) outperforms previous methods in each modality, achieving a decisive advantage. Figure 3

[0077] In qualitative analysis, Figure 8 and Figure 9 are the RGB image and point cloud input of a certain scene in the validation set, respectively. We take them as input, and the output obtains the three-dimensional grid occupancy prediction result as Figure 10 As a comparison, Figure 11 is the labeled true value, and we can see that the present disclosure can accurately and completely predict the semantic details in the scene, and has certain prediction ability for the occluded part of the scene.

[0078] ​The present disclosure has strong global modeling capability compared with previous methods based on convolutional neural networks by using a brand-new Mamba network; compared with methods based on transformers, its linear computational complexity makes it not need to rely on complex feature compression or key point algorithms, avoiding the information loss caused thereby. Therefore, compared with previous methods, the present disclosure has unique advantages in the structure used. When adapting the Mamba network to three-dimensional grid occupancy prediction, we not only designed a hierarchical Mamba module, but also designed a simple but effective three-dimensional to one-dimensional adaptive rearrangement strategy for this task. This hierarchical module design and adaptive rearrangement strategy enable the present disclosure to efficiently process multi-scale global information and make accurate and complete three-dimensional grid occupancy prediction in various complex large scenes.

[0079] Figure 12 is a schematic diagram of a three-dimensional grid occupancy prediction device according to some embodiments of the present disclosure. As Figure 12 indicated, the three-dimensional grid occupancy prediction device 1200 includes a feature acquisition module 1210, a feature enhancement module 1220, and a prediction module 1230. In some embodiments of the present disclosure, the three-dimensional grid occupancy prediction function is performed by a three-dimensional grid occupancy prediction server as Figure 1 indicated. Wherein:

[0080] The feature acquisition module 1210 is configured to acquire three-dimensional voxel features of a target scene.

[0081] The feature enhancement module 1220 is configured to input the three-dimensional voxel features into a hierarchical Mamba module to generate three-dimensional voxel features with enhanced spatial features, wherein the hierarchical Mamba module includes a plurality of Mamba blocks for processing different scale features, and is configured to balance global information and local information at multiple scales to enhance spatial features, and the Mamba block takes as input a one-dimensional sequence formed by adaptive unfolding and rearrangement of the three-dimensional voxel features based on the spatial structure of the target scene.

[0082] The prediction module 1230 is configured to perform three-dimensional grid occupancy prediction based on the three-dimensional voxel features with enhanced spatial features.

[0083] One embodiment of the present disclosure provides a three-dimensional grid occupancy prediction device. As Figure 13 indicated, the three-dimensional grid occupancy prediction device 1300 includes a memory 1320 and a processor 1310, the memory 1320 is configured to store a computer program; the processor 1310 is configured to implement the method in S310-S330 Figure 3 when executing the computer program.

[0084] In summary, the three-dimensional grid occupancy prediction method, device and equipment provided by each embodiment of the present disclosure can preserve as much as possible the inherent spatial relationship between adjacent voxels in three-dimensional space in the rearrangement process by rearranging the three-dimensional voxel features into a one-dimensional sequence based on the spatial structure of the complex scene to serve as the input of the Mamba network, thereby improving the performance of three-dimensional grid occupancy prediction of the complex scene. At the same time, by using the hierarchical Mamba module to extract spatial features from multiple scales and integrate them, the problem that the Mamba network is difficult to fully understand the global features of large-scale three-dimensional voxels is solved, thereby enhancing the ability to accurately grasp the complex scene involving large-scale three-dimensional voxels and extract spatial features, and improving the prediction effect of the complex scene.

[0085] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device and the module described above can refer to the corresponding description in the foregoing device embodiment, and will not be repeated here.

[0086] Although the subject matter described herein is provided in the general context of computer-executable instructions executed by a computer system on a computer system, those skilled in the art will recognize that other implementations can be performed in combination with other types of program modules. Generally, program modules include routines, programs, components, data structures, and other types of structures that perform particular tasks or implement particular abstract data types. Those skilled in the art will appreciate that the subject matter described herein can be practiced with other computer system configurations, including hand-held devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like. The subject matter can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote memory storage devices.

[0087] Those of ordinary skill in the art can realize that the units and method steps of the examples described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present disclosure.

[0088] It should be understood that the foregoing detailed description of the disclosure, rather than limiting the disclosure, is intended to exemplify and elicit the principles of the disclosure. Accordingly, any modification, equivalent replacement, improvement, etc. made without departing from the spirit and scope of the disclosure should be included in the scope of protection of the disclosure. In addition, the claims of the disclosure are intended to cover all changes and modifications falling within the scope and boundary of the appended claims, or the equivalent form of such scope and boundary.

Claims

1. A three-dimensional grid occupancy prediction method, characterized in that, include: Obtain the 3D voxel features of the target scene; The three-dimensional voxel features are input into a hierarchical Mamba module to generate three-dimensional voxel features with enhanced spatial features. The hierarchical Mamba module includes multiple Mamba blocks for processing features at different scales, which are used to enhance spatial features by taking into account both global and local information at multiple scales. The Mamba blocks take a one-dimensional sequence formed by adaptively unfolding and rearranging the three-dimensional voxel features based on the spatial structure of the target scene as input. Three-dimensional raster occupancy prediction is performed based on the three-dimensional voxel features with enhanced spatial characteristics. The acquisition of the three-dimensional voxel features of the target scene includes: Acquire a two-dimensional image of the target scene, and generate image voxel features of the target scene based on the two-dimensional image; Obtain point cloud data of the target scene, and generate point cloud voxel features of the target scene based on the point cloud data; The image voxel features and point cloud voxel features of the target scene are fused to generate the three-dimensional voxel features of the target scene; The step of adaptively unfolding and rearranging the three-dimensional voxel features based on the spatial structure of the target scene to form a one-dimensional sequence includes: The scene type of the target scene is determined, wherein the scene type includes horizontal movement scene, omnidirectional movement scene and low activity scene; Based on the scene type of the target scene, the three-dimensional voxel features are unfolded and rearranged to form a one-dimensional sequence; The scenario type for determining the target scenario includes: The prediction range of the target scene and the pose change of the predicted subject in the target scene are obtained, and the scene type of the target scene is determined based on the prediction range and the pose change. The process of unfolding and rearranging three-dimensional voxel features to form a one-dimensional sequence based on the scene type of the target scene includes: When the scene type is a horizontal movement scene, the target scene is represented by a three-dimensional coordinate system of (X, Y, Z). For three-dimensional voxel features with the same (X, Y) coordinate values ​​in the target scene, a sequence of voxels perpendicular to the xy plane is generated according to the order of Z coordinate values ​​from low to high. All the vertical voxel sequences generated from all the three-dimensional voxel features of the target scene form a vertical voxel sequence set. Based on the (X, Y) coordinate values ​​of each vertical voxel sequence in the vertical voxel sequence set, the vertical voxel sequences are sorted according to the two-dimensional Hilbert curve on the xy plane and connected to each other to form the one-dimensional sequence. The process of unfolding and rearranging three-dimensional voxel features to form a one-dimensional sequence based on the scene type of the target scene includes: When the scene type is an omnidirectional moving scene, the three-dimensional voxel features of the target scene are rearranged based on the three-dimensional Hilbert curve to form the one-dimensional sequence; The process of unfolding and rearranging three-dimensional voxel features to form a one-dimensional sequence based on the scene type of the target scene includes: When the scene type is a low-activity scene, the target scene is represented by a three-dimensional coordinate system (X, Y, Z). For three-dimensional voxel features with the same (X, Y) coordinate values ​​in the target scene, a sequence of voxels perpendicular to the xy plane is generated according to the order of Z coordinate values ​​from low to high. All the vertical voxel sequences generated from the three-dimensional voxel features of the target scene form a vertical voxel sequence set. Based on the (X, Y) coordinate values ​​of each vertical voxel sequence in the vertical voxel sequence set, the vertical voxel sequences are unfolded in layers on the xy plane in order from the center of the predicted subject outwards to form the one-dimensional sequence.

2. The method according to claim 1, characterized in that: The hierarchical Mamba module consists of an encoder and a decoder. The encoder and the decoder each include multiple Mamba blocks for processing features at different scales. Each Mamba block contains at least two Mamba networks. The different Mamba blocks of the encoder include downsampling operations for multi-scale voxel feature extraction, and the different Mamba blocks of the decoder include upsampling operations for multi-scale voxel feature integration. The Mamba blocks of the encoder and the Mamba blocks of the decoder at the same scale are skip connections. The input of the Mamba block is a one-dimensional sequence generated by rearranging three-dimensional voxel features, and the output is a three-dimensional voxel feature integrated from the one-dimensional sequence.

3. A three-dimensional grid occupancy prediction device, characterized in that, include: The feature acquisition module is used to acquire the three-dimensional voxel features of the target scene; The feature enhancement module is used to input the three-dimensional voxel features into the hierarchical Mamba module to generate three-dimensional voxel features with enhanced spatial features. The hierarchical Mamba module includes multiple Mamba blocks for processing features at different scales, which are used to enhance spatial features by taking into account both global and local information at multiple scales. The Mamba blocks take a one-dimensional sequence formed by adaptively unfolding and rearranging the three-dimensional voxel features based on the spatial structure of the target scene as input. The prediction module is used to perform three-dimensional raster occupancy prediction based on the three-dimensional voxel features with enhanced spatial characteristics. The acquisition of the three-dimensional voxel features of the target scene includes: Acquire a two-dimensional image of the target scene, and generate image voxel features of the target scene based on the two-dimensional image; Obtain point cloud data of the target scene, and generate point cloud voxel features of the target scene based on the point cloud data; The image voxel features and point cloud voxel features of the target scene are fused to generate the three-dimensional voxel features of the target scene; The step of adaptively unfolding and rearranging the three-dimensional voxel features based on the spatial structure of the target scene to form a one-dimensional sequence includes: The scene type of the target scene is determined, wherein the scene type includes horizontal movement scene, omnidirectional movement scene and low activity scene; Based on the scene type of the target scene, the three-dimensional voxel features are unfolded and rearranged to form a one-dimensional sequence; The scenario type for determining the target scenario includes: The prediction range of the target scene and the pose change of the predicted subject in the target scene are obtained, and the scene type of the target scene is determined based on the prediction range and the pose change. The process of unfolding and rearranging three-dimensional voxel features to form a one-dimensional sequence based on the scene type of the target scene includes: When the scene type is a horizontal movement scene, the target scene is represented by a three-dimensional coordinate system of (X, Y, Z). For three-dimensional voxel features with the same (X, Y) coordinate values ​​in the target scene, a sequence of voxels perpendicular to the xy plane is generated according to the order of Z coordinate values ​​from low to high. All the vertical voxel sequences generated from all the three-dimensional voxel features of the target scene form a vertical voxel sequence set. Based on the (X, Y) coordinate values ​​of each vertical voxel sequence in the vertical voxel sequence set, the vertical voxel sequences are sorted according to the two-dimensional Hilbert curve on the xy plane and connected to each other to form the one-dimensional sequence. The process of unfolding and rearranging three-dimensional voxel features to form a one-dimensional sequence based on the scene type of the target scene includes: When the scene type is an omnidirectional moving scene, the three-dimensional voxel features of the target scene are rearranged based on the three-dimensional Hilbert curve to form the one-dimensional sequence; The process of unfolding and rearranging three-dimensional voxel features to form a one-dimensional sequence based on the scene type of the target scene includes: When the scene type is a low-activity scene, the target scene is represented by a three-dimensional coordinate system (X, Y, Z). For three-dimensional voxel features with the same (X, Y) coordinate values ​​in the target scene, a sequence of voxels perpendicular to the xy plane is generated according to the order of Z coordinate values ​​from low to high. All the vertical voxel sequences generated from the three-dimensional voxel features of the target scene form a vertical voxel sequence set. Based on the (X, Y) coordinate values ​​of each vertical voxel sequence in the vertical voxel sequence set, the vertical voxel sequences are unfolded in layers on the xy plane in order from the center of the predicted subject outwards to form the one-dimensional sequence.

4. A three-dimensional grid occupancy prediction device, characterized in that, Including memory and processor: The memory is used to store computer programs; The processor is configured to, when executing the computer program, implement the method according to any one of claims 1-2.

Images