Three-dimensional grid occupation prediction method, device and equipment

Abstract

The invention belongs to the technical field of computer vision, and particularly relates to a three-dimensional grid occupation prediction method, device and equipment. The method comprises the following steps: acquiring three-dimensional voxel features of a target scene; the three-dimensional voxel features are input into a layered Mama module, three-dimensional voxel features with enhanced spatial features are generated, the layered Mama module comprises a plurality of Mama blocks used for processing features of different scales and is used for giving consideration to global information and local information under multiple scales to enhance the spatial features, and the spatial features of the three-dimensional voxel features are generated; the Mama block takes a one-dimensional sequence formed by adaptively expanding and rearranging three-dimensional voxel features based on the space structure of the target scene as input; and performing three-dimensional grid occupation prediction based on the three-dimensional voxel features with the enhanced spatial features. According to the invention, the prediction effect of the three-dimensional grid occupation prediction of the complex scene is improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a three-dimensional grid occupancy prediction method, device and equipment. Background Art

[0002] The goal of 3D grid occupancy prediction is to estimate the occupancy and semantic labels of a 3D space using various input signals.

[0003] Most of the existing 3D grid occupancy prediction networks are built based on convolutional neural networks. This kind of network can make good use of local scene information and has certain prediction capabilities. Mamba network has become a hot topic in recent research because it integrates a selective mechanism based on the state space model, which can effectively capture scene information while greatly reducing computational complexity.

[0004] However, on the one hand, the Mamba network is mainly designed for language models, and the input data is a one-dimensional sequence, while the feature representation of the 3D grid occupancy prediction is usually a 3D voxel. When directly converting 3D voxels to one dimension, the inherent spatial relationship between adjacent voxels in the 3D space may be lost, resulting in a large separation in the one-dimensional sequence. This loss of spatial relationship weakens the ability of the Mamba network to effectively understand the global scene, which in turn affects the prediction performance. Therefore, when rearranging 3D voxels into a one-dimensional sequence, it is necessary to consider how to effectively express the correlation of the inherent spatial relationship between adjacent voxels. However, existing methods are mainly designed based on the spatial structure of the target scene. If the actual scene type is different from the target scene type, the expression efficiency of the correlation of the inherent spatial relationship between adjacent voxels is reduced. In particular, complex scenes contain multiple scene types. A single expansion method cannot efficiently express the correlation of the inherent spatial relationship between adjacent voxels for complex scenes containing different scene types. On the other hand, when directly using the Mamba network to process complex scenes involving large-scale voxels, such as autonomous driving environments, because these scenes have rich global information and complex details, the Mamba network is difficult to fully understand such a large amount of voxels, resulting in an inaccurate grasp of the overall scene, which in turn affects the reliability and accuracy of the prediction results. In summary, the existing three-dimensional grid occupancy prediction scheme based on the Mamba network does not work well for complex scenes. Summary of the invention

[0005] The disclosed embodiment proposes a three-dimensional grid occupancy prediction scheme based on a hierarchical Mamba module, which solves the problem that the existing three-dimensional grid occupancy prediction scheme based on the Mamba network has poor prediction effect on complex scenes.

[0006] A first aspect of an embodiment of the present disclosure provides a three-dimensional grid occupancy prediction method, comprising:

[0007] Obtaining three-dimensional voxel features of the target scene;

[0008] Inputting the three-dimensional voxel features into a hierarchical Mamba module to generate three-dimensional voxel features with enhanced spatial features, wherein the hierarchical Mamba module includes a plurality of Mamba blocks for processing features of different scales, and is used to take into account global information and local information at multiple scales to enhance spatial features, and the Mamba block takes a one-dimensional sequence formed by adaptively unfolding and rearranging the three-dimensional voxel features based on the spatial structure of the target scene as input;

[0009] Three-dimensional grid occupancy prediction is performed based on the three-dimensional voxel features with enhanced spatial features.

[0010] In some embodiments, the hierarchical Mamba module is composed of an encoder and a decoder, wherein the encoder and the decoder respectively include multiple Mamba blocks for processing features of different scales, each of the Mamba blocks contains at least two Mamba networks, different Mamba blocks of the encoder include downsampling operations for multi-scale voxel feature extraction, different Mamba blocks of the decoder include upsampling operations for multi-scale voxel feature integration, and the Mamba blocks of the encoder and the Mamba blocks of the decoder of the same scale are jump-connected, the input of the Mamba block is a one-dimensional sequence generated by rearranging three-dimensional voxel features, and the output is a three-dimensional voxel feature integrated from the one-dimensional sequence.

[0011] In some embodiments, acquiring the three-dimensional voxel features of the target scene includes:

[0012] Acquire a two-dimensional image of the target scene, and generate image voxel features of the target scene based on the two-dimensional image;

[0013] Acquire point cloud data of the target scene, and generate point cloud voxel features of the target scene based on the point cloud data;

[0014] The image voxel features and the point cloud voxel features of the target scene are fused to generate three-dimensional voxel features of the target scene.

[0015] In some embodiments, the step of adaptively expanding and rearranging the three-dimensional voxel features based on the spatial structure of the target scene to form a one-dimensional sequence comprises:

[0016] Determining a scene type of the target scene, wherein the scene type includes a horizontal movement scene, an omnidirectional movement scene, and a low activity scene;

[0017] The three-dimensional voxel features are expanded and rearranged to form a one-dimensional sequence based on the scene type of the target scene.

[0018] In some embodiments, determining the scene type of the target scene includes:

[0019] The prediction range of the target scene and the predicted posture change of the subject in the target scene are obtained, and the scene type of the target scene is determined based on the prediction range and the posture change.

[0020] In some embodiments, the expanding and rearranging the three-dimensional voxel features to form a one-dimensional sequence based on the scene type of the target scene comprises:

[0021] When the scene type is a horizontal moving scene, the target scene is represented by a (X, Y, Z) three-dimensional coordinate system, and for three-dimensional voxel features of the target scene having the same (X, Y) coordinate value, a vertical voxel sequence on the xy plane is generated according to the order of the Z coordinate value from low to high;

[0022] All the vertical voxel sequences generated by all the three-dimensional voxel features of the target scene form a vertical voxel sequence set, and based on the (X, Y) coordinate values ​​of each of the vertical voxel sequences in the vertical voxel sequence set, the vertical voxel sequences are sorted according to the two-dimensional Hilbert curve on the xy plane and connected to each other to form the one-dimensional sequence.

[0023] In some embodiments, the expanding and rearranging the three-dimensional voxel features to form a one-dimensional sequence based on the scene type of the target scene comprises:

[0024] When the scene type is an omnidirectional moving scene, the three-dimensional voxel features of the target scene are rearranged based on a three-dimensional Hilbert curve to form the one-dimensional sequence.

[0025] In some embodiments, the expanding and rearranging the three-dimensional voxel features to form a one-dimensional sequence based on the scene type of the target scene comprises:

[0026] When the scene type is a low activity scene, the target scene is represented by a (X, Y, Z) three-dimensional coordinate system, and for three-dimensional voxel features of the target scene having the same (X, Y) coordinate value, a vertical voxel sequence on the xy plane is generated according to the order of the Z coordinate value from low to high;

[0027] All the vertical voxel sequences generated by all the three-dimensional voxel features of the target scene form a vertical voxel sequence set. Based on the (X, Y) coordinate values ​​of each of the vertical voxel sequences in the vertical voxel sequence set, the vertical voxel sequences are layered and unfolded in the xy plane in the order from the center position of the predicted subject to the surrounding areas to form the one-dimensional sequence.

[0028] A second aspect of an embodiment of the present disclosure provides a three-dimensional grid occupancy prediction device, comprising:

[0029] A feature acquisition module, used to acquire three-dimensional voxel features of the target scene;

[0030] A feature enhancement module, used for inputting the three-dimensional voxel features into a hierarchical Mamba module to generate three-dimensional voxel features with enhanced spatial features, wherein the hierarchical Mamba module includes a plurality of Mamba blocks for processing features of different scales, and is used for taking into account global information and local information at multiple scales to enhance spatial features, and the Mamba block takes as input a one-dimensional sequence formed by adaptively expanding and rearranging the three-dimensional voxel features based on the spatial structure of the target scene;

[0031] A prediction module is used to perform three-dimensional grid occupancy prediction based on the three-dimensional voxel features with enhanced spatial features.

[0032] A third aspect of the embodiments of the present disclosure provides a three-dimensional grid occupancy prediction device, including a memory and a processor:

[0033] The memory is used to store computer programs;

[0034] The processor is used to implement the method described in any one of the first aspects of the present disclosure when executing the computer program.

[0035] In summary, the three-dimensional grid occupancy prediction method, apparatus and device provided by the embodiments of the present disclosure improves the performance of three-dimensional grid occupancy prediction for complex scenes by adaptively expanding and rearranging the three-dimensional voxel features into a one-dimensional sequence based on the spatial structure of the complex scene as the input of the Mamba network, thereby retaining as much inherent spatial relationship between adjacent voxels in the three-dimensional space as possible during the rearrangement process. At the same time, the hierarchical Mamba module extracts and integrates spatial features from multiple scales, thereby solving the problem that the Mamba network is difficult to fully understand the global features of large-scale three-dimensional voxels, thereby enhancing the ability to accurately grasp complex scenes involving large-scale three-dimensional voxels and extract spatial features, and improving the prediction effect for complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The features and advantages of the present disclosure will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present disclosure in any way. In the accompanying drawings:

[0037] Figure 1 is a schematic diagram of a computer system to which the present disclosure is applicable;

[0038] Figure 2 It is a system framework of a three-dimensional grid occupancy prediction method based on Mamba network according to some embodiments of the present disclosure;

[0039] Figure 3 is a flowchart of a three-dimensional grid occupancy prediction method according to some embodiments of the present disclosure;

[0040] Figure 4 is a schematic diagram of the Hilbert expansion of the adaptive scenario shown in the present disclosure;

[0041] Figure 5 is a schematic diagram of a hierarchical Mamba module according to some embodiments of the present disclosure;

[0042] Figure 6 is a schematic diagram of a Mamba block framework according to some embodiments of the present disclosure;

[0043] Figure 7 It is a performance comparison of different three-dimensional grid occupancy prediction methods;

[0044] Figure 8 It is the RGB image data of the target scene in OpenOccupancy;

[0045] Fig. 9 It is the point cloud data of the target scene in OpenOccupancy;

[0046] Fig.10 is based on Figure 3 The method described in S310-S330 predicts the three-dimensional grid occupancy of the target scene;

[0047] Fig.11 yes Figure 8-Figure 10 The true value of the target scene in

[0048] Fig.12 is a schematic diagram of a three-dimensional grid occupancy prediction device according to some embodiments of the present disclosure;

[0049] Fig.13 It is a schematic diagram of a three-dimensional grid occupancy prediction device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0050] In the detailed description below, many specific details of the present disclosure are set forth by way of example in order to provide a thorough understanding of the relevant disclosure. However, it will be apparent to one of ordinary skill in the art that the present disclosure can be implemented without these details. It should be understood that the use of the terms "system," "device," "unit," and / or "module" in the present disclosure is a method for distinguishing different parts, elements, parts, or assemblies at different levels in a sequential arrangement. However, these terms may be replaced by other expressions if other expressions can achieve the same purpose.

[0051] It should be understood that when a device, unit or module is referred to as being "on," "connected to," or "coupled to" another device, unit or module, it may be directly on, connected to, coupled to, or communicating with other devices, units or modules, or there may be intermediate devices, units or modules, unless the context clearly indicates an exception. For example, the term "and / or" as used in this disclosure includes any and all combinations of one or more of the relevant listed items.

[0052] The terms used in this disclosure are only for describing specific embodiments and are not intended to limit the scope of this disclosure. As shown in the specification and claims of this disclosure, unless the context clearly indicates an exception, the words "one", "a kind" and / or "the" do not specifically refer to the singular, but may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of clearly identified features, wholes, steps, operations, elements and / or components, and such expressions do not constitute an exclusive list, and other features, wholes, steps, operations, elements and / or components may also be included.

[0053] These and other features and characteristics, methods of operation, functions of related elements of the structure, combinations of parts, and economies of manufacture of the present disclosure may be better understood with reference to the following description and drawings, which form a part of the specification. However, it is to be clearly understood that the drawings are for illustration and description purposes only and are not intended to limit the scope of protection of the present disclosure. It is to be understood that the drawings are not drawn to scale.

[0054] Various structural diagrams are used in this disclosure to illustrate various variations of the embodiments of the present disclosure. It should be understood that the above or below structures are not intended to limit the present disclosure. The scope of protection of the present disclosure shall be subject to the claims.

[0055] Figure 1 is a schematic diagram of a computer system to which the present disclosure is applicable. Figure 1 The system shown includes a three-dimensional grid occupancy prediction server connected to image sensor and point cloud distance sensor data, and the three-dimensional grid occupancy prediction server is used to perform three-dimensional grid occupancy prediction on the target scene based on the three-dimensional voxel features of the target scene generated by the image sensor and the point cloud distance sensor.

[0056] The image sensor is used to obtain image or video data of the target scene, including various monocular or multi-eye cameras, monocular or multi-eye video cameras; the point cloud distance sensor is used to obtain point cloud data of the target scene, including various laser radars or millimeter wave radars.

[0057] The 3D grid occupancy prediction server generates 3D voxel features of the target scene based on the image data and point cloud data of the target scene, and then performs 3D grid occupancy prediction on the target scene based on the 3D voxel features of the target scene. The 3D grid occupancy prediction server can be any of a single machine, a cluster or a distributed server.

[0058] The three-dimensional grid occupancy prediction server is deployed with multiple Mamba blocks for processing features of different scales, and is used to take into account global information and local information at multiple scales to enhance spatial features. The Mamba block takes as input a one-dimensional sequence formed by adaptively expanding and rearranging three-dimensional voxel features based on the spatial structure of the target scene.

[0059] The goal of 3D grid occupancy prediction is to estimate the occupancy and semantic labels of 3D space using various input signals. Most of the existing 3D grid occupancy prediction networks are built based on convolutional neural networks. The network built in this way can make good use of local scene information and has certain prediction capabilities. However, due to the inherent defects of convolutional neural networks themselves, it is difficult to capture global scene information, and there is still a lot of room for improvement. Inspired by the breakthrough progress of transformer networks in the field of computer vision, some recent works have enhanced the processing capabilities of global scene information by combining transformer networks, thereby achieving certain performance improvements. However, when they process large voxel grids, their quadratic computational complexity makes the computational burden particularly heavy. Especially when deployed in some outdoor scenes, they have to rely on complex feature compression or key point algorithms, which inevitably suffers from information loss. In order to solve the problem of high computational complexity of transformer networks, people are vying to propose more efficient operators. Among them, state-space models are a type of choice that has attracted much attention. Inspired by control theory, it maps system inputs to system outputs through hidden states, thereby effectively processing information sequences. As an important variant of the state-space model, the Mamba network is well-known for its integrated selective mechanism. The Mamba network can effectively capture global information and process large-scale data in linear time, and has become a strong candidate for the next-generation structure to replace the transformer network. However, the Mamba network is mainly designed for language models, and the input data is a one-dimensional sequence, while the input data for three-dimensional grid occupancy prediction is three-dimensional voxels. When directly converting three-dimensional voxels to one dimension, the inherent spatial relationship between adjacent voxels in the three-dimensional space is likely to be lost, resulting in far separation in the one-dimensional sequence.

[0060] This problem can be alleviated by embedding the expression of the inherent spatial relationship between adjacent voxels when rearranging the three-dimensional voxels into a one-dimensional sequence. However, existing methods are mainly designed based on the spatial structure of the target scene. If the actual scene type is different from the target scene type, the efficiency of expressing the inherent spatial relationship between adjacent voxels will drop sharply. In particular, complex scenes contain multiple scene types. A single unfolding method cannot efficiently express the correlation of the inherent spatial relationship between adjacent voxels for complex scenes containing different scene types. At the same time, when directly using the Mamba network to process complex scenes involving large-scale voxels, such as autonomous driving environments, because these scenes have rich global information and complex details, it is difficult for the Mamba network to fully understand such a large amount of voxels, resulting in an inaccurate grasp of the overall scene, which in turn affects the reliability and accuracy of the prediction results. In summary, the existing three-dimensional grid occupancy prediction scheme based on the Mamba network is not effective in processing complex scenes.

[0061] In order to solve the above problems, this paper proposes a new method for three-dimensional grid occupancy prediction based on Mamba network. Figure 2 Specifically, the input data features from the RGB camera and LiDAR are first extracted and converted into 3D voxel form, then fused and input into the hierarchical Mamba module for processing, and finally handed over to the prediction head to output the 3D grid occupancy.

[0062] Figure 3 is a flowchart of a three-dimensional grid occupancy prediction method according to some embodiments of the present disclosure. In some embodiments, the three-dimensional grid occupancy prediction function is Figure 1 The three-dimensional grid occupancy prediction server is executed, and the three-dimensional grid occupancy prediction method includes the following steps:

[0063] S310, obtaining three-dimensional voxel features of the target scene.

[0064] Specifically, adjust the positions of the LiDAR and RGB camera to ensure that their receptive fields can cover as much scene information as possible. For the LiDAR branch, we first voxelize the input point cloud P, and then use the LiDAR encoder E based on sparse convolution L Generate LiDAR voxel features Similarly, for the RGB camera branch, we input the image data I into the image encoder, which extracts image features using a residual network, aggregates multi-scale features through a feature pyramid network, and then generates image voxel features using a 2D to 3D view transformer in, and is the spatial dimension of the voxel feature, and is the channel dimension of these voxel features. We ensure that V L and V C Keep consistent in the spatial dimension and connect these voxel features in the channel dimension to obtain the multimodal encoding feature V:

[0065] V = concat(V L , V C ).

[0066] Among them, V is the multimodal 3D voxel feature, V L is the lidar voxel feature, V C It is the image voxel feature and concat() is the connection operation.

[0067] S320, inputting the three-dimensional voxel features into a hierarchical Mamba module to generate three-dimensional voxel features with enhanced spatial features, wherein the hierarchical Mamba module includes a plurality of Mamba blocks for processing features of different scales, and is used to take into account both global information and local information at multiple scales to enhance spatial features, and the Mamba block takes as input a one-dimensional sequence formed by adaptively expanding and rearranging the three-dimensional voxel features based on the spatial structure of the target scene.

[0068] Before inputting the 3D voxel feature V into the hierarchical Mamba module, it is necessary to rearrange it into a one-dimensional sequence. Inspired by the Hilbert curve, the present disclosure proposes a Hilbert expansion of an adaptive scene, such as Figure 4 shown. Figure 4 The method shown can dynamically and adaptively rearrange the three-dimensional voxel features into a one-dimensional sequence based on the different spatial structures in the three-dimensional grid occupancy prediction task in each scene. Some embodiments of the present disclosure predict the changing trend of the subject posture by tracking the changes in the input point cloud, and adjust the scene scheme with reference to the prediction range to achieve the adaptive function. The scenes are mainly divided into three categories: horizontal movement scenes, omnidirectional movement scenes and low activity scenes, and the corresponding main application scenarios are outdoor autonomous driving, outdoor drones and indoor augmented reality.

[0069] Specifically, for horizontal movement scenes, some embodiments of the present disclosure divide the three-dimensional coordinate xyz into an xy plane and a z dimension. Starting from the xy plane at z=0, each xy coordinate value extends along the z-axis direction to form a one-dimensional vertical sequence corresponding to the xy point, and finally all voxels are sorted into a set of vertical voxel sequences. The vertical voxel sequences in the set are then sorted and connected to each other according to the two-dimensional Hilbert curve on the xy plane. This method gives priority to the spatial information of the z-axis while maintaining strong spatial proximity on the xy plane; for omnidirectional movement scenes, considering the uncertainty of the direction, some embodiments of the present disclosure directly use the three-dimensional Hilbert curve for rearrangement; for low-activity scenes, the main concern is to predict things around the subject. Therefore, some embodiments of the present disclosure are processed in a manner similar to horizontal movement scenes: first, a vertical sequence along the z-axis direction is generated for each xy coordinate value, and then hierarchically sorted in the xy plane from the center to the surroundings, and finally these sequences are connected in order to obtain the rearrangement result.

[0070] The present disclosure proposes a hierarchical Mamba module for taking into account global information and local information at multiple scales to enhance spatial features, such as Figure 5 The hierarchical Mamba modules include Figure 5 The layered Mamba encoder E shown M and decoder D M . E M By multiple Figure 6 The Mamba blocks shown in the figure are composed of two Mamba networks each, and there is a downsampling operation between the blocks for multi-scale voxel feature extraction. M This structure is mirrored, with each block containing two Mamba networks, and upsampling operations between blocks to integrate multi-scale voxel features. Figure 6 FIG. 1 is a schematic diagram of a Mamba block framework according to some embodiments of the present disclosure. M The upsampling operation in E M The skip connection is introduced into the same height block of the Mamba block, and the consistency of the feature dimension is maintained. The rearrangement strategy disclosed in the present invention is applied before and after the Mamba block of each layer, so that the scene features maintain the three-dimensional voxel feature representation during the downsampling and upsampling stages. In general, the multimodal encoded feature V is input into the hierarchical Mamba module to generate the Mamba feature V′, which can be described as:

[0071] V′=D M (E M (V)

[0072] Among them, V is the multimodal 3D voxel feature, V′ is the 3D voxel feature of the enhanced spatial feature output by the hierarchical Mamba module, V′ maintains the same space and features as V, E M() is the encoding operation, D M () is the decoding operation.

[0073] S330: Perform three-dimensional grid occupancy prediction based on the three-dimensional voxel features with enhanced spatial features.

[0074] Specifically, we interpolate the spatial dimension of the Mamba feature V′ to match the true value, and then use MLP to predict the category occupied by each three-dimensional grid to obtain the prediction result O occ :

[0075] O occ =MLP(Interpolation(V′))

[0076] An embodiment of the present disclosure quantitatively analyzes various 3D grid occupancy prediction methods on the OpenOccupancy dataset. The experimental method includes training with a training set and testing with a validation set. The experimental indicators are IoU and mIoU, i.e., intersection over union and mean intersection over union. mIoU measures the accuracy and completeness of the prediction method in each category and is the most important indicator for this task. The experimental results are comparable to those of the comparison Figure 7 As shown, it can be seen that the method described in the present disclosure ( Figure 3 The method described in S310-S330 in FIG. 4 surpasses previous methods in all modes and achieves an overwhelming advantage.

[0077] In qualitative analysis, Figure 8 and Fig. 9 They are the RGB image and point cloud input of a scene in the validation set, respectively. We use them as input and output as follows Fig.10 The three-dimensional grid occupancy prediction result of . For comparison, Fig.11 To mark the true value, we can see that the present disclosure can predict the semantic details in the scene more accurately and completely, and has a certain prediction ability for the occluded part of the scene.

[0078] By using the new Mamba network, the present disclosure has a powerful global modeling capability compared to previous methods based on convolutional neural networks; compared to transformer-based methods, its linear computational complexity means that it does not need to rely on complex feature compression or key point algorithms, thus avoiding the resulting information loss. Therefore, compared to previous methods, the present disclosure has unique advantages in the structure used. When adapting the Mamba network to three-dimensional grid occupancy prediction, we not only designed a hierarchical Mamba module, but also designed a simple but effective three-dimensional to one-dimensional adaptive rearrangement strategy for this task. This hierarchical module design and adaptive rearrangement strategy enable the present disclosure to still efficiently perform multi-scale global information processing in various complex large scenes, and perform accurate and complete three-dimensional grid occupancy prediction.

[0079] Fig.12 FIG. 1 is a schematic diagram of a three-dimensional grid occupancy prediction device according to some embodiments of the present disclosure. Fig.12 As shown, the three-dimensional grid occupancy prediction device 1200 includes a feature acquisition module 1210, a feature enhancement module 1220, and a prediction module 1230. In some embodiments of the present disclosure, the three-dimensional grid occupancy prediction function is Figure 1 The three-dimensional grid shown occupies the prediction server execution. Where:

[0080] A feature acquisition module 1210 is used to acquire three-dimensional voxel features of a target scene;

[0081] A feature enhancement module 1220 is used to input the three-dimensional voxel feature into a hierarchical Mamba module to generate a three-dimensional voxel feature with enhanced spatial features, wherein the hierarchical Mamba module includes a plurality of Mamba blocks for processing features of different scales, and is used to enhance spatial features by taking into account global information and local information at multiple scales, and the Mamba block takes a one-dimensional sequence formed by adaptively expanding and rearranging the three-dimensional voxel feature based on the spatial structure of the target scene as input;

[0082] The prediction module 1230 is used to perform three-dimensional grid occupancy prediction based on the three-dimensional voxel features with enhanced spatial features.

[0083] One embodiment of the present disclosure provides a three-dimensional grid occupancy prediction device. Fig.13 As shown, the three-dimensional grid occupancy prediction device 1300 includes a memory 1320 and a processor 1310, wherein the memory 1320 is used to store a computer program; and the processor 1310 is used to implement when executing the computer program. Figure 3 The method described in S310-S330.

[0084] In summary, the three-dimensional grid occupancy prediction method, apparatus and device provided by the embodiments of the present disclosure improves the performance of three-dimensional grid occupancy prediction for complex scenes by adaptively expanding and rearranging the three-dimensional voxel features into a one-dimensional sequence based on the spatial structure of the complex scene as the input of the Mamba network, thereby retaining as much inherent spatial relationship between adjacent voxels in the three-dimensional space as possible during the rearrangement process. At the same time, the hierarchical Mamba module extracts and integrates spatial features from multiple scales, thereby solving the problem that the Mamba network is difficult to fully understand the global features of large-scale three-dimensional voxels, thereby enhancing the ability to accurately grasp complex scenes involving large-scale three-dimensional voxels and extract spatial features, and improving the prediction effect for complex scenes.

[0085] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding descriptions in the aforementioned device embodiments and will not be repeated here.

[0086] Although the subject matter described herein is provided in the general context of being executed in conjunction with the execution of operating systems and application programs on computer systems, it will be appreciated by those skilled in the art that other implementations may also be performed in conjunction with other types of program modules. In general, program modules include routines, programs, components, data structures, and other types of structures that perform specific tasks or implement specific abstract data types. It will be appreciated by those skilled in the art that the subject matter described herein may be practiced using other computer system configurations, including handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc., and may also be used in a distributed computing environment in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0087] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0088] It should be understood that the above specific embodiments of the present disclosure are only used to illustrate or explain the principles of the present disclosure, and do not constitute a limitation of the present disclosure. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present disclosure should be included in the protection scope of the present disclosure. In addition, the claims attached to the present disclosure are intended to cover all changes and modifications that fall within the scope and boundaries of the attached claims, or the equivalent forms of such scope and boundaries.

Claims

1. A three-dimensional grid occupancy prediction method, characterized in that: include: Obtaining three-dimensional voxel features of the target scene; Inputting the three-dimensional voxel features into a hierarchical Mamba module to generate three-dimensional voxel features with enhanced spatial features, wherein the hierarchical Mamba module includes a plurality of Mamba blocks for processing features of different scales, and is used to take into account global information and local information at multiple scales to enhance spatial features, and the Mamba block takes a one-dimensional sequence formed by adaptively unfolding and rearranging the three-dimensional voxel features based on the spatial structure of the target scene as input; Three-dimensional grid occupancy prediction is performed based on the three-dimensional voxel features with enhanced spatial features.

2. The method according to claim 1, characterized in that: The hierarchical Mamba module is composed of an encoder and a decoder, wherein the encoder and the decoder respectively include multiple Mamba blocks for processing features of different scales, each of the Mamba blocks contains at least two Mamba networks, different Mamba blocks of the encoder include downsampling operations for multi-scale voxel feature extraction, different Mamba blocks of the decoder include upsampling operations for multi-scale voxel feature integration, the Mamba blocks of the encoder and the Mamba blocks of the decoder of the same scale are jump-connected, the input of the Mamba block is a one-dimensional sequence generated by rearranging three-dimensional voxel features, and the output is a three-dimensional voxel feature integrated from the one-dimensional sequence.

3. The method according to claim 1, characterized in that: The step of acquiring the three-dimensional voxel features of the target scene comprises: Acquire a two-dimensional image of the target scene, and generate image voxel features of the target scene based on the two-dimensional image; Acquire point cloud data of the target scene, and generate point cloud voxel features of the target scene based on the point cloud data; The image voxel features and the point cloud voxel features of the target scene are fused to generate three-dimensional voxel features of the target scene.

4. The method according to claim 1, characterized in that: The step of adaptively expanding and rearranging the three-dimensional voxel features based on the spatial structure of the target scene to form a one-dimensional sequence comprises: Determining a scene type of the target scene, wherein the scene type includes a horizontal movement scene, an omnidirectional movement scene, and a low activity scene; The three-dimensional voxel features are expanded and rearranged to form a one-dimensional sequence based on the scene type of the target scene.

5. The method according to claim 4, characterized in that The determining the scene type of the target scene includes: The prediction range of the target scene and the predicted posture change of the subject in the target scene are obtained, and the scene type of the target scene is determined based on the prediction range and the posture change.

6. The method according to claim 4, characterized in that: The step of expanding and rearranging the three-dimensional voxel features to form a one-dimensional sequence based on the scene type of the target scene includes: When the scene type is a horizontal moving scene, the target scene is represented by a (X, Y, Z) three-dimensional coordinate system, and for three-dimensional voxel features of the target scene having the same (X, Y) coordinate value, a vertical voxel sequence on the xy plane is generated according to the order of the Z coordinate value from low to high; All the vertical voxel sequences generated by all the three-dimensional voxel features of the target scene form a vertical voxel sequence set, and based on the (X, Y) coordinate values ​​of each of the vertical voxel sequences in the vertical voxel sequence set, the vertical voxel sequences are sorted according to the two-dimensional Hilbert curve on the xy plane and connected to each other to form the one-dimensional sequence.

7. The method according to claim 4, characterized in that: The step of expanding and rearranging the three-dimensional voxel features to form a one-dimensional sequence based on the scene type of the target scene includes: When the scene type is an omnidirectional moving scene, the three-dimensional voxel features of the target scene are rearranged based on a three-dimensional Hilbert curve to form the one-dimensional sequence.

8. The method according to claim 4, characterized in that: The step of expanding and rearranging the three-dimensional voxel features to form a one-dimensional sequence based on the scene type of the target scene includes: When the scene type is a low activity scene, the target scene is represented by a (X, Y, Z) three-dimensional coordinate system, and for three-dimensional voxel features of the target scene having the same (X, Y) coordinate value, a vertical voxel sequence on the xy plane is generated according to the order of the Z coordinate value from low to high; All the vertical voxel sequences generated by all the three-dimensional voxel features of the target scene form a vertical voxel sequence set. Based on the (X, Y) coordinate values ​​of each of the vertical voxel sequences in the vertical voxel sequence set, the vertical voxel sequences are layered and unfolded in the xy plane in the order from the center position of the predicted subject to the surrounding areas to form the one-dimensional sequence.

9. A three-dimensional grid occupancy prediction device, characterized in that: include: A feature acquisition module, used to acquire three-dimensional voxel features of the target scene; A feature enhancement module, used for inputting the three-dimensional voxel features into a hierarchical Mamba module to generate three-dimensional voxel features with enhanced spatial features, wherein the hierarchical Mamba module includes a plurality of Mamba blocks for processing features of different scales, and is used for taking into account global information and local information at multiple scales to enhance spatial features, and the Mamba block takes as input a one-dimensional sequence formed by adaptively expanding and rearranging the three-dimensional voxel features based on the spatial structure of the target scene; A prediction module is used to perform three-dimensional grid occupancy prediction based on the three-dimensional voxel features with enhanced spatial features.

10. A three-dimensional grid occupancy prediction device, characterized in that: Including memory and processor: The memory is used to store computer programs; The processor is configured to implement the method according to any one of claims 1-8 when executing the computer program.

Images