Low-altitude three-dimensional occupation perception prediction method, system and computer device

Abstract

The application relates to a low-altitude three-dimensional occupancy perception prediction method, system and computer device. The method comprises the following steps: obtaining a target feature sequence by rearranging occupancy mask values corresponding to each voxel in a feature mask sequence; inputting the target feature sequence into a target prediction model of air perception to extract target feature information corresponding to a plurality of voxels at a current time of the target feature sequence, and performing spatial occupancy prediction on each voxel according to the target feature information and outputting a target prediction result; in this way, based on the multi-modal data fusion and feature rearrangement mechanism, the problems of lack of three-dimensional semantic occupancy data in the low-altitude scene of the unmanned aerial vehicle, the migration defects of the vehicle-mounted perception method and the efficiency problems caused by the spatial sparsity are solved. Through compact pre-processing of key entity features, the reasoning efficiency and feature aggregation capability of long sequence context modeling are significantly improved.

Description

Technical Field

[0001] This application relates to the field of three-dimensional vision technology, and in particular to a method, system and computer device for low-altitude three-dimensional occupancy perception and prediction. Background Technology

[0002] With the development of drone technology, there are technical bottlenecks in drone low-altitude 3D semantic occupancy perception technology. In aerial scenarios such as urban inspection and navigation in high-risk environments, drones need to analyze unconstrained 3D spatial structures, but the industry lacks dense 3D semantic occupancy datasets adapted to aerial scenarios.

[0003] Existing datasets are mostly designed for ground-based autonomous driving and are applied to vehicle-mounted applications. Their fixed altitude and limited vertical field of view cannot cover the dynamic characteristics of drones flying in aerial scenarios. Furthermore, obtaining semantic annotations from the drone's perspective relies on high manual costs, resulting in a lack of a solid foundation in 3D spatial semantic understanding. Meanwhile, mainstream 3D occupancy prediction methods are based on the prior knowledge of a flat ground surface. When transferred to drone scenarios with drastic altitude fluctuations and variable perspectives, the reliability of depth estimation drops sharply. Therefore, it is difficult to construct accurate semantic annotation data for aerial scenarios, further resulting in low accuracy and efficiency in predictive inference for aerial scenarios, failing to meet the actual flight requirements of drones. Summary of the Invention

[0004] Therefore, it is necessary to provide a low-altitude 3D occupancy perception prediction method, system, and computer equipment that can construct semantically labeled data for aerial scenes and improve the accuracy and efficiency of prediction and reasoning in aerial scenes, in order to address the above-mentioned technical problems.

[0005] In a first aspect, this application provides a low-altitude three-dimensional occupancy perception prediction method, including:

[0006] Image and radar data collected in an aerial scene are acquired, and the radar data is voxelized to obtain a 3D mesh; the 3D mesh consists of multiple voxels divided from the aerial scene.

[0007] Extract two-dimensional visual features from image data and three-dimensional geometric features from radar data;

[0008] Based on the semantic information in two-dimensional visual features and the depth information in three-dimensional geometric features, feature fusion is performed on each voxel to obtain the corresponding feature fusion result.

[0009] Based on the feature fusion results, the occupancy information corresponding to multiple voxels is identified, and the feature mask sequence is obtained based on the occupancy information; wherein, the feature mask sequence is a sequence composed of the occupancy mask values ​​corresponding to multiple voxels;

[0010] The occupancy mask values ​​corresponding to each voxel in the feature mask sequence are rearranged to obtain the target feature sequence.

[0011] The target feature sequence is input into the target prediction model for aerial perception to extract target feature information corresponding to multiple voxels at the current time of the target feature sequence. Based on the target feature information, the space occupancy of each voxel is predicted, and the target prediction result is output. The target prediction result is used to represent the occupancy information and semantic information of each voxel in the real scene.

[0012] In one embodiment, after the steps of extracting two-dimensional visual features from image data and three-dimensional geometric features from radar data, the method further includes:

[0013] Extracting high-dimensional semantic features from two-dimensional visual features; whereby high-dimensional semantic features are used to represent semantic information in image data;

[0014] Projecting three-dimensional geometric features onto a two-dimensional plane yields two-dimensional geometric depth features;

[0015] Convolutional operations are performed on the two-dimensional geometric depth features to generate depth cue information corresponding to the three-dimensional mesh, and the depth cue information is encoded to obtain geometric cue features;

[0016] By combining two-dimensional visual features and geometric cue features, the depth probability distribution of each voxel in the three-dimensional mesh is calculated; whereby the depth probability distribution is used to represent the depth information corresponding to multiple voxels in radar data.

[0017] In one embodiment, the step of fusing features of each voxel based on semantic information in two-dimensional visual features and depth information in three-dimensional geometric features to obtain the corresponding feature fusion result includes:

[0018] Based on the high-dimensional semantic features corresponding to the two-dimensional visual features and the depth probability distribution corresponding to the three-dimensional geometric features, the semantic features corresponding to the semantic information are extended along the depth axis corresponding to the depth information through the outer product operation to generate three-dimensional view frustum features.

[0019] The 3D view frustum features are mapped onto a 3D mesh to obtain the 3D visual features;

[0020] By concatenating 3D visual features and 3D geometric features along the channel dimension, a 3D combined feature is obtained.

[0021] A 3D convolution operation is performed on the 3D combined features. By sliding the convolution kernel, the 3D visual features and 3D geometric features contained in the 3D combined features are fused, the number of channels is compressed, and the context information is aggregated to obtain the feature fusion result.

[0022] In one embodiment, the steps of identifying occupancy information corresponding to multiple voxels based on the feature fusion result, and obtaining a feature mask sequence based on the occupancy information, include:

[0023] The feature fusion results corresponding to multiple voxels are serialized and expanded to obtain an initial feature sequence; wherein, the initial feature sequence is a one-dimensional feature vector after the feature fusion results of each voxel are compressed into a one-dimensional space.

[0024] The feature fusion results corresponding to multiple voxels are projected as scalars, and the scalars are used as occupancy probability scores. Thresholding is performed based on the occupancy probability scores to identify the occupancy information of multiple voxels and obtain the occupancy grid mask. The occupancy grid mask includes the occupancy mask value corresponding to the thresholding identification of each voxel.

[0025] Align the initial feature sequence with the occupied grid mask to obtain the feature mask sequence.

[0026] In one embodiment, the occupancy information in the feature mask sequence includes physical occupancy and air occupancy;

[0027] The steps for rearranging the occupancy mask values ​​corresponding to each voxel in the feature mask sequence to obtain the target feature sequence include:

[0028] The first voxel representing the entity's occupation information in the feature mask sequence is placed at the beginning, and the second voxel representing the air's occupation information in the feature mask sequence is placed at the end, to obtain the target feature sequence.

[0029] In one embodiment, prior to the step of inputting the target feature sequence into the air-aware target prediction model, the method further includes:

[0030] Construct a target simulation sample dataset; wherein, the target simulation sample dataset includes multiple sample data pairs, the sample data pairs include sample acquisition data of aerial scenes, and corresponding label values, the sample acquisition data includes sample image data and sample radar data, the label values ​​include occupancy information corresponding to the sample acquisition data, the occupancy information is used to represent the real occupancy information of each voxel in the three-dimensional mesh corresponding to the aerial scene where the sample acquisition data is located;

[0031] Based on the sample image data and sample radar data, determine the sample feature sequence;

[0032] The sample feature sequence is input into a preset initial perception prediction network to predict the occupancy information of each voxel in the three-dimensional grid corresponding to the sample acquisition data, and the predicted occupancy information is obtained.

[0033] The occupancy loss of each voxel in the 3D mesh is calculated based on the predicted occupancy information and the actual occupancy information. The initial perception prediction network is backpropagated based on the occupancy loss, and the network parameters are updated to obtain the target prediction model for aerial perception.

[0034] In one embodiment, the steps of constructing the target simulation sample dataset include:

[0035] Obtain the simulation occupancy dataset; the simulation occupancy dataset includes multiple simulation objects in the aerial scene;

[0036] Based on the distribution information of simulation objects in the simulation occupancy dataset, the simulation occupancy dataset is classified into regions.

[0037] Based on the region type of different areas in the simulation occupancy dataset, the simulation occupancy dataset is sampled according to the corresponding strategy to obtain sample image data and sample radar data covering multiple simulation objects;

[0038] The initial point cloud data is voxelized to obtain an initial 3D mesh containing multiple corresponding voxels. The initial 3D mesh is traversed to obtain the center coordinates and average normal vectors of multiple voxels in the initial 3D mesh.

[0039] Based on the sample semantic features in the sample image data and the dot product of the center coordinates and average normal vectors of multiple voxels, it is determined whether the voxels effectively hit the entity, so as to determine the occupancy information corresponding to multiple voxels.

[0040] The target simulation sample dataset is obtained by using sample image data and sample radar data as sample acquisition data, and using the occupancy information corresponding to multiple voxels as label values.

[0041] In one embodiment, after the step of sampling the simulation occupancy dataset according to the region type of different regions in the simulation occupancy dataset using a corresponding strategy to obtain sample image data and sample radar data covering multiple simulation objects, the method further includes:

[0042] Transform the coordinate system of the sample radar data into the local coordinate system of the acquisition device;

[0043] Based on the local coordinate system, the observable range of the acquisition equipment is determined in the sample radar data, and the target radar data within the observable range is identified.

[0044] Secondly, this application also provides an aerial perception prediction system, comprising:

[0045] The acquisition module is used to acquire image data and radar data collected in the aerial scene, and to perform voxelization processing on the radar data to obtain a three-dimensional mesh; wherein, the three-dimensional mesh includes multiple voxels divided by the aerial scene;

[0046] The feature extraction module is used to extract two-dimensional visual features from image data and three-dimensional geometric features from radar data;

[0047] The feature fusion module is used to fuse features of each voxel based on semantic information in two-dimensional visual features and depth information in three-dimensional geometric features to obtain the corresponding feature fusion result.

[0048] The perception module is used to identify the occupancy information corresponding to multiple voxels based on the feature fusion result, and to obtain the feature mask sequence based on the occupancy information; wherein, the feature mask sequence is a sequence composed of the occupancy mask values ​​corresponding to multiple voxels;

[0049] The feature rearrangement module is used to rearrange the occupancy mask values ​​corresponding to each voxel in the feature mask sequence to obtain the target feature sequence;

[0050] The prediction module is used to input the target feature sequence into the target prediction model of the air perception, so as to extract the target feature information corresponding to multiple voxels at the current time of the target feature sequence, and perform spatial occupancy prediction for each voxel based on the target feature information, and output the target prediction result; wherein, the target prediction result is used to represent the occupancy information and semantic information of each voxel in the real scene.

[0051] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the control perception prediction method provided in the first aspect.

[0052] The aforementioned low-altitude 3D occupancy perception and prediction method, system, and computer equipment acquire image data and radar data collected in an aerial scene, and perform voxelization processing on the radar data to obtain a 3D mesh. The 3D mesh comprises multiple voxels divided by the aerial scene. Two-dimensional visual features are extracted from the image data, and three-dimensional geometric features from the radar data are extracted. Based on the semantic information in the two-dimensional visual features and the depth information in the three-dimensional geometric features, feature fusion is performed on each voxel to obtain the corresponding feature fusion result. Based on the feature fusion result, occupancy information corresponding to multiple voxels is identified, and a feature mask sequence is obtained based on the occupancy information. The feature mask sequence is a sequence composed of occupancy mask values ​​corresponding to multiple voxels. The occupancy mask values ​​corresponding to each voxel in the feature mask sequence are rearranged to obtain a target feature sequence. The target feature sequence is input into an aerial perception target prediction model to extract target feature information corresponding to multiple voxels at the current time of the target feature sequence, and spatial occupancy prediction is performed on each voxel based on the target feature information, outputting the target prediction result. The target prediction result represents the occupancy information and semantic information of each voxel in the real scene. Therefore, based on multimodal data fusion and feature rearrangement mechanisms, this approach effectively addresses the lack of 3D semantic data in low-altitude UAV scenarios, the transfer defects of vehicle-mounted perception methods, and the efficiency issues caused by spatial sparsity. Through compact preprocessing of key entity features, it significantly improves the inference efficiency and feature aggregation capabilities of long-sequence context modeling, providing a precise 3D spatial cognitive foundation for intelligent obstacle avoidance and autonomous navigation of UAVs in complex aerial scenarios. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 A diagram illustrating the application environment of the low-altitude three-dimensional occupancy perception and prediction method;

[0055] Figure 2 This is a flowchart illustrating the first embodiment of the low-altitude three-dimensional occupancy perception and prediction method.

[0056] Figure 3 This is a flowchart illustrating the second embodiment of the low-altitude three-dimensional occupancy perception and prediction method.

[0057] Figure 4 This is a schematic diagram of a scene in the second embodiment of the low-altitude three-dimensional occupancy perception and prediction method;

[0058] Figure 5 This is a flowchart illustrating the third embodiment of the low-altitude three-dimensional occupancy perception and prediction method.

[0059] Figure 6 This is a schematic diagram of the first scene in the third embodiment of the low-altitude three-dimensional occupancy perception and prediction method.

[0060] Figure 7 This is a schematic diagram of the second scene in the third embodiment of the low-altitude three-dimensional occupancy perception and prediction method;

[0061] Figure 8 This is a flowchart illustrating the fourth embodiment of the low-altitude three-dimensional occupancy perception and prediction method.

[0062] Figure 9 This is a schematic diagram of the first scene in the fourth embodiment of the low-altitude three-dimensional occupancy perception and prediction method.

[0063] Figure 10 This is a schematic diagram of the second scene in the fourth embodiment of the low-altitude three-dimensional occupancy perception and prediction method;

[0064] Figure 11 This is a schematic diagram of the third scene in the fourth embodiment of the low-altitude three-dimensional occupancy perception and prediction method;

[0065] Figure 12 This is a schematic diagram of the structure of a prediction system for aerial perception in one embodiment;

[0066] Figure 13 This is a schematic diagram of the internal structure of a computer device in one embodiment. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0068] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0069] The low-altitude three-dimensional occupancy perception and prediction method provided in this application can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Terminal 102 can acquire image data and point cloud data in an aerial scene. After acquiring the image data and point cloud data, it determines the feature fusion result of each voxel in the corresponding 3D mesh of the aerial scene through feature extraction and feature fusion. Based on the feature fusion result, it identifies the occupancy information corresponding to multiple voxels and obtains a feature mask sequence based on the occupancy information; wherein, the feature mask sequence is a sequence composed of occupancy mask values ​​corresponding to multiple voxels; the occupancy mask values ​​corresponding to each voxel in the feature mask sequence are rearranged to obtain a target feature sequence; the target feature sequence is input into the aerial perception target prediction model to extract target feature information corresponding to multiple voxels at the current time where the target feature sequence is located, and spatial occupancy prediction is performed on each voxel based on the target feature information, outputting the target prediction result; wherein, the target prediction result is used to represent the occupancy information and semantic information of each voxel in the real scene.

[0070] Terminal 102 may be, but is not limited to, various drones, low-altitude aircraft, low-altitude drones, flying cars, robots, etc.

[0071] Server 104 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0072] Traditional 3D semantic occupancy perception technologies are primarily designed for ground-based autonomous driving. In low-altitude drone scenarios, they suffer from a lack of dense 3D semantic occupancy data and automated construction methods. Furthermore, existing vehicle-mounted 3D occupancy perception methods rely on the prior knowledge of a flat ground surface. Directly transferring these methods to drone scenarios, which exhibit high volatility and extreme spatial sparsity, leads to severe performance bottlenecks and wasted computational resources due to scale ambiguity, unreliable depth estimation, and redundant and sparse feature sequences. In addition, the drone perception grid presents a vast high-altitude volume, with a very high proportion of open free space. Traditional prediction methods need to traverse the entire space within the perception range, resulting in extremely redundant feature sequences. Key geometric entity features are fragmented by massive air regions, severely wasting computational resources and weakening the ability to model long-sequence contexts, thus impacting prediction accuracy and inference efficiency.

[0073] Based on this, a low-altitude three-dimensional occupancy perception and prediction method is proposed, such as... Figure 2 As shown, in an exemplary embodiment, the method is applied to Figure 1 The terminal 102 in the example is a drone, and the process includes the following steps 21 to 26, wherein:

[0074] Step 21: Acquire image data and radar data collected in the aerial scene, and perform voxelization processing on the radar data to obtain a three-dimensional mesh.

[0075] The 3D mesh consists of multiple voxels divided from the aerial scene.

[0076] Here, the aerial scene refers to the environment in which the drone flies at low altitude, characterized by fluctuating altitude, varied perspectives, and sparse space, with most areas consisting of open, free space.

[0077] Image data can refer to two-dimensional image information collected by visual sensors such as cameras mounted on drones. Image information can include visual cues such as texture and color in the environment in which the aerial scene is located.

[0078] Radar data can refer to a set of three-dimensional spatial points collected by depth sensors such as lidar. Each point in radar data usually contains three-dimensional coordinate information, which can be used to represent the geometry of the scene in the air where the acquisition device is located.

[0079] Specifically, image data can be acquired in real time by a visible light camera mounted on the drone, while radar data can be obtained by a lidar sensor. Alternatively, image data can be extracted from frames in a pre-stored video stream, and radar data can be obtained from a depth camera or a structured light sensor.

[0080] It should be noted that after obtaining point cloud data from radar data based on pose calculation, voxelization of the point cloud data can be a process of converting continuous or discrete point cloud data into a discrete 3D mesh structure. This is achieved by dividing the 3D space into multiple regular small cubes, and using these small cubes as voxels to form the 3D mesh. Here, the 3D mesh refers to a discrete 3D spatial representation composed of multiple voxels obtained through voxelization, used for structured description of aerial scenes. A voxel in the 3D mesh can be the smallest unit within the 3D mesh; each voxel represents a specific region in 3D space and can be used to store the feature or state information of that region.

[0081] One implementation approach is to mesh the point cloud data using a fixed resolution, directly mapping the point cloud data to a predefined 3D voxel space, and then counting the number or average position of points within each voxel until multiple small cubes in the point cloud data are mapped to their corresponding 3D voxel spaces. Alternatively, adaptive voxel structures such as octrees can be used to dynamically adjust the voxel size based on the point cloud density, thereby reducing storage and computational overhead.

[0082] Step 22: Extract the two-dimensional visual features from the image data and the three-dimensional geometric features from the radar data.

[0083] Two-dimensional visual features can refer to feature representations extracted from image data to describe the content and semantics of an image, such as edge, corner, and texture features.

[0084] Here, three-dimensional geometric features can refer to the feature representations extracted from each voxel in the three-dimensional mesh corresponding to the point cloud data, used to describe the spatial location, shape, or structural information of the voxels, such as the coordinates, normals, or density of the voxels.

[0085] In practical applications, pre-trained deep learning models can be used to extract multi-scale semantic features to obtain two-dimensional visual features in image data. In addition, information such as the normal vector, surface area, or distance between each voxel in the three-dimensional network and the sensor can be calculated to obtain three-dimensional geometric features.

[0086] Step 23: Based on the semantic information in the two-dimensional visual features and the depth information in the three-dimensional geometric features, perform feature fusion on each voxel to obtain the corresponding feature fusion result.

[0087] Semantic information can refer to information that semantically describes the meaning and category of objects or areas in an aerial scene based on image data. For example, if an object or area is identified as a building, vegetation, or vehicle, the corresponding semantic information can be used to describe it.

[0088] Here, depth information can refer to the distance information of an object or voxel relative to the observation device, used to represent the positional relationship in three-dimensional space.

[0089] One implementation method is to directly connect the semantic information of two-dimensional visual features with the depth information of three-dimensional geometric features along the feature dimension. For example, the image feature vector can be concatenated with the depth values ​​of voxels to form a longer feature vector, which serves as the feature fusion result.

[0090] As another implementation method, element-wise multiplication or addition operations can be used to interact the semantic information of two-dimensional visual features with the depth information of three-dimensional geometric features. For example, after upsampling and aligning the image feature map with the depth map, pixel-by-pixel multiplication operations can be performed to weight the semantic features with depth, thereby obtaining the feature fusion result.

[0091] Step 24: Based on the feature fusion results, identify the occupancy information corresponding to multiple voxels, and obtain the feature mask sequence based on the occupancy information.

[0092] The feature mask sequence is a sequence of occupancy mask values ​​corresponding to multiple voxels, and each occupancy mask value is used to represent the occupancy status of the corresponding voxel.

[0093] Here, occupancy information can refer to the state description of whether each voxel in the 3D mesh is occupied by a physical entity (entity occupancy) or is empty free space (air occupancy).

[0094] In practical applications, a simple classifier can be used to perform threshold classification on the feature fusion results of each voxel to determine whether it is occupied, thereby obtaining the occupancy information and the occupancy feature value corresponding to each voxel. After determining the occupancy feature value corresponding to each voxel, the occupancy feature values ​​corresponding to each voxel can be arranged according to the fixed scanning order of each voxel in the 3D mesh (e.g., Z-axis first, Y-axis second, X-axis third) to obtain the feature mask sequence.

[0095] As another implementation method, a neural network can be used to perform multi-classification on the feature fusion results corresponding to each voxel. This not only determines whether a voxel is occupied by an entity, but also distinguishes the different occupancy categories of each voxel. After determining the occupancy information and occupancy category of each voxel in the 3D mesh, they can be arranged according to the linear index order of each voxel to obtain a feature mask sequence.

[0096] Step 25: Rearrange the occupancy mask values ​​corresponding to each voxel in the feature mask sequence to obtain the target feature sequence.

[0097] The target feature sequence can refer to the sequence obtained by rearranging the occupancy mask values ​​in the feature mask sequence. Compared with the feature mask sequence, the rearranged target feature sequence can optimize the arrangement order of each voxel with different occupancy information to improve processing efficiency.

[0098] It should be noted that rearranging the occupancy mask values ​​corresponding to each voxel in the feature mask sequence can be based on a preset sorting rule. The aim is to arrange the different occupancy information represented by each voxel in the feature mask sequence in an orderly manner. In this way, the entity occupancy data, which is valid computational data, can be arranged continuously, and the data types can be distinguished from the air occupancy data, which is invalid computational data, thereby improving the efficiency of feature sequence processing during calculation.

[0099] In one embodiment, the occupancy mask value corresponding to the voxel whose occupancy information is represented as "entity occupancy" can be moved to the front of the feature mask sequence, while the occupancy mask value corresponding to the voxel whose occupancy information is represented as "air occupancy" can be moved to the back of the feature mask sequence, thereby forming the rearranged target feature sequence.

[0100] As another implementation, the occupancy mask values ​​in the feature mask sequence can be rearranged based on the spatial location information of the voxels. For example, the occupancy mask values ​​corresponding to voxels closer to the sensor or the center of the scene can be prioritized to ensure that the features of important areas are processed first.

[0101] Step 26: Input the target feature sequence into the target prediction model of the airborne perception to extract the target feature information corresponding to multiple voxels at the current time where the target feature sequence is located, and perform space occupancy prediction for each voxel based on the target feature information, and output the target prediction result.

[0102] The target prediction results are used to represent the occupancy and semantic information of each voxel in the real scene.

[0103] Here, the target prediction model for aerial perception can refer to a trained neural network model specifically designed to process aerial scene data, predict the spatial occupancy of each voxel in the corresponding 3D grid of the aerial scene at future times based on the input feature sequence, as well as semantic information used to describe the spatial occupancy.

[0104] Specifically, the target prediction model for aerial perception can be a model based on the Transformer architecture, which processes the target feature sequence through a self-attention mechanism, captures the long-distance dependencies between voxels in the sequence, and thus performs space occupancy prediction to determine the target prediction result.

[0105] In the aforementioned low-altitude 3D occupancy perception and prediction method, image data and radar data collected in the aerial scene are acquired, and the radar data is voxelized to obtain a 3D mesh. The 3D mesh comprises multiple voxels divided by the aerial scene. Two-dimensional visual features are extracted from the image data, and three-dimensional geometric features from the radar data. Based on the semantic information in the two-dimensional visual features and the depth information in the three-dimensional geometric features, feature fusion is performed on each voxel to obtain the corresponding feature fusion result. Based on the feature fusion result, occupancy information corresponding to multiple voxels is identified, and a feature mask sequence is obtained based on the occupancy information. The feature mask sequence is a sequence composed of occupancy mask values ​​corresponding to multiple voxels. The occupancy mask values ​​corresponding to each voxel in the feature mask sequence are rearranged to obtain a target feature sequence. The target feature sequence is input into the aerial perception target prediction model to extract target feature information corresponding to multiple voxels at the current time of the target feature sequence, and spatial occupancy prediction is performed on each voxel based on the target feature information, outputting the target prediction result. The target prediction result represents the occupancy information and semantic information of each voxel in the real scene. Therefore, this application, based on multimodal data fusion and feature rearrangement mechanisms, effectively solves the problems of insufficient 3D semantic data, transfer defects of vehicle-mounted perception methods, and efficiency issues caused by spatial sparsity in low-altitude UAV scenarios. Through compact preprocessing of key entity features, it significantly improves the inference efficiency and feature aggregation capabilities of long sequence context modeling, providing a precise 3D spatial cognitive foundation for UAVs' intelligent obstacle avoidance and autonomous navigation in complex aerial scenarios.

[0106] In an exemplary embodiment, after extracting the two-dimensional visual features from the image data and the three-dimensional geometric features from the radar data in step 22, further feature processing can be performed on the two-dimensional visual features and the three-dimensional geometric features to extract more feature information. After feature processing of the two-dimensional visual features and the three-dimensional geometric features, feature fusion is performed on each voxel to obtain the feature fusion result. For example... Figure 3 As shown, steps 31 to 34 and steps 35 to 38 are included, wherein:

[0107] Step 31: Extract high-dimensional semantic features from the two-dimensional visual features.

[0108] Among them, high-dimensional semantic features are used to represent semantic information in image data.

[0109] When extracting high-dimensional semantic features from two-dimensional visual features, deep learning models, such as Convolutional Neural Networks (CNNs) or Transformer networks, can be used to further extract and abstract the original two-dimensional visual features.

[0110] It's important to note that deep learning models can learn deeper semantic information from images, such as identifying object categories (e.g., vehicles, pedestrians, buildings, ground), scene context, and the relationships between them. Through operations like multi-layer convolution, pooling, and activation functions, low-level visual features (e.g., edges, textures, colors) are transformed into high-dimensional, more discriminative semantic representations, thus providing richer semantic contextual information for subsequent feature fusion.

[0111] Step 32: Project the three-dimensional geometric features onto a two-dimensional plane to obtain two-dimensional geometric depth features.

[0112] Specifically, using known camera intrinsic and extrinsic parameter matrices, the center coordinates or vertex coordinates of each voxel in the 3D mesh can be projected from 3D space onto a 2D image plane. For each voxel, its projection region on the image plane can be calculated, and the geometric attributes within that region can be extracted to obtain the corresponding 2D geometric features.

[0113] For example, each pixel can be assigned a depth value, normal vector, or surface area information corresponding to its voxel, thereby forming a depth map, normal map, or other geometric property map aligned with the image as a two-dimensional geometric depth feature. In this way, three-dimensional geometric information can be aligned and interact with two-dimensional visual information on the same plane.

[0114] Step 33: Perform a convolution operation on the two-dimensional geometric depth features to generate depth cue information corresponding to the three-dimensional mesh, and encode the depth cue information to obtain geometric cue features.

[0115] In practical computation, a series of convolutional layers can be used to process two-dimensional geometric depth features (such as depth maps). By learning the depth information of two-dimensional geometric depth features at different scales through convolutional layers, and capturing local and global structures in the depth map, such as depth boundaries and planar regions, more structured and contextual depth cue information can be generated.

[0116] Furthermore, after determining the depth cue information, it can be encoded. One approach is to use a fully connected layer or another convolutional layer to perform dimensionality reduction or feature transformation, matching the dimension of the depth cue information with the high-dimensional semantic features to obtain geometric cue features. Another approach is to encode the depth cue information into a representation of the depth range and depth uncertainty of each pixel or region, thus obtaining geometric cue features.

[0117] Step 34: Combine two-dimensional visual features and geometric cue features to calculate the depth probability distribution of each voxel in the three-dimensional mesh.

[0118] The depth probability distribution is used to represent the depth information corresponding to multiple voxels in radar data.

[0119] Here, the depth probability distribution can be a discrete histogram to represent the probability of a voxel being occupied in different depth intervals, or a continuous probability density function. The depth probability distribution, based on the probability representation, can better handle the uncertainty of depth estimation, especially in areas with occlusion, missing textures, or sparse point clouds.

[0120] In one implementation, the extracted high-dimensional semantic features can be used as representatives of the two-dimensional visual features, concatenated with the geometric cue features in the channel dimension, and input into a multilayer perceptron (MLP) or convolutional network to output the probability value of each voxel at different depth values. Alternatively, an attention mechanism can be designed to allow the high-dimensional semantic features and geometric cue features to interact and jointly predict the depth probability distribution.

[0121] Step 35: Based on the high-dimensional semantic features corresponding to the two-dimensional visual features and the depth probability distribution corresponding to the three-dimensional geometric features, the semantic features corresponding to the semantic information are extended along the depth axis corresponding to the depth information through the outer product operation to generate three-dimensional view frustum features.

[0122] In this 3D view frustum feature, each point contains original 2D semantic information and is assigned a corresponding probability weight based on its depth position in 3D space, thus achieving preliminary spatialization and depth perception of semantic information from 2D to 3D. The 3D view frustum feature can be used to represent the feature information obtained after effectively upscaling the semantic information in a 2D image to 3D space and performing preliminary fusion with depth information.

[0123] Specifically, the image feature vector obtained by the deep learning model can be used as a high-dimensional semantic feature. This image feature vector encodes semantic information such as object category and texture in the image. By using the high-dimensional semantic features extracted from the two-dimensional visual features and the depth probability distribution calculated from the three-dimensional geometric features, the two-dimensional semantic features can be extended along the depth axis by performing an outer product operation and modulated by the depth probability distribution, thereby generating a three-dimensional feature volume in the camera's view frustum.

[0124] Step 36: Map the 3D view frustum features to a 3D mesh to obtain the 3D visual features.

[0125] After generating the 3D frustum features, the 3D frustum features can be transformed from the camera frustum space and integrated into a unified, pre-divided 3D mesh. This allows the semantic information in the 3D frustum features to be projected onto each voxel of the 3D mesh, resulting in 3D visual features mapped from the 3D frustum features onto each voxel.

[0126] In specific implementations, the 3D frustum features in each voxel can be mapped using coordinate transformation and feature interpolation or aggregation operations to obtain the corresponding 3D visual features. For example, the center coordinates of the voxel can be back-projected onto the camera coordinate system, and then the corresponding semantic information can be sampled from the 3D frustum features. Alternatively, multiple frustum features can be aggregated into the same voxel through summation, averaging, or maximization, thereby generating a feature vector containing its semantic information for each voxel as a 3D visual feature.

[0127] Step 37: The three-dimensional visual features and the three-dimensional geometric features are spliced ​​together in the channel dimension to obtain the three-dimensional combined features.

[0128] After obtaining the 3D visual features containing semantic information for each voxel, they can be spliced ​​together with the 3D geometric features corresponding to that voxel in the channel dimension to obtain the 3D combined features.

[0129] For example, if the 3D visual features have C1 channels and the 3D geometric features have C2 channels, then the stitched 3D combined features will have C1+C2 channels, so that the feature vector of each voxel simultaneously contains semantic information from the image data and geometric information from the point cloud data.

[0130] Step 38: Perform a 3D convolution operation on the 3D combined features. Through the sliding calculation of the convolution kernel, perform feature fusion, channel number compression and context information aggregation on the 3D visual features and 3D geometric features contained in the 3D combined features to obtain the feature fusion result.

[0131] After stitching together the 3D combined features, deep feature fusion can be performed by applying 3D convolution operations. The 3D convolution kernel slides on the 3D combined feature volume, performing weighted summation on the voxel features in the local neighborhood. This allows it to learn and extract the complex interrelationships between 3D visual features and 3D geometric features, achieving deep fusion of the two at the semantic and geometric levels. Furthermore, the design of the convolutional layers can compress the number of channels, reduce feature dimensionality, and improve computational efficiency.

[0132] Meanwhile, the local receptive field characteristics of 3D convolution enable it to effectively aggregate contextual information around voxels, capture local structures and dependencies in 3D space, and generate a more discriminative and robust feature fusion result that can more comprehensively and accurately represent the integrated information of each voxel.

[0133] For example, please refer to Figure 4 , Figure 4This is a schematic diagram of a scenario in this embodiment. The following example is presented in this schematic diagram, using a drone as an example of terminal 102, to illustrate feature extraction and feature fusion of image data and point cloud data collected by the drone in a low-altitude flight scenario.

[0134] Specifically, drones can acquire multi-view image data corresponding to low-altitude flight scenarios and point cloud data (LiDAR data) collected by radar equipment within them.

[0135] For point cloud data, voxelization can be performed first to obtain a 3D mesh containing multiple voxels. This 3D mesh is then input into a 3D coding layer to extract 3D geometric features, denoted as... .

[0136] Furthermore, point cloud data can be projected from 3D to 2D, and sparse depth cue information can be generated through convolution operations, encoded as geometric cue features, denoted as... .

[0137] For multi-view image data, the image data can be input into a 2D coding layer to extract two-dimensional visual features, denoted as... .

[0138] Here, it is understandable that multi-view image data can include image data from different perspectives in multiple directions such as front, back, left, right, up, and down.

[0139] After performing basic feature extraction on image and point cloud data and obtaining the corresponding feature information, including depth branch and context branch.

[0140] In the depth branch, two-dimensional visual features can be combined. and geometric cue features Predict the pixel-level depth probability distribution (depth probability distribution) of each voxel in a discrete depth interval, denoted as In the context branch, high-dimensional semantic features can also be extracted from the image data, denoted as... .

[0141] Furthermore, high-dimensional semantic features can be... With depth probability distribution Perform a pointwise outer product operation to expand the semantic features along the depth axis, generating a 3D view frustum feature, denoted as . The specific calculation formula is as follows: .

[0142] In acquiring 3D view frustum features Then, the three-dimensional view frustum features can be... With three-dimensional geometric features Feature fusion is performed. Specifically, this involves 3D view frustum features. A mapping can be performed first, utilizing the extrinsic parameter matrix from the image and point cloud data collected by the UAV to map the 3D view frustum features. Mapping to a 3D mesh yields 3D visual features. .

[0143] After mapping, the three-dimensional visual features With three-dimensional geometric features Channel stitching and 3D convolution are performed in a unified 3D coordinate system to achieve feature fusion. The output 3D fused voxel features are used as the feature fusion result, denoted as . .

[0144] Here, the feature fusion result The fusion calculation formula can be as follows:

[0145] .

[0146] In this embodiment, using the above technical solution, this application first utilizes the outer product operation to extend the high-dimensional semantic features corresponding to the two-dimensional visual features along the depth probability distribution corresponding to the three-dimensional geometric features, generating three-dimensional frustum features with depth perception and initially aligned with the three-dimensional space, effectively solving the initial alignment problem of mapping two-dimensional semantic information to three-dimensional space. Subsequently, these three-dimensional frustum features are precisely mapped to a unified three-dimensional mesh, ensuring the accurate positional correspondence of semantic information in three-dimensional space. On this basis, by concatenating three-dimensional visual features and three-dimensional geometric features in the channel dimension, comprehensive multimodal input is provided for subsequent deep fusion. Finally, three-dimensional convolution operation is used to perform deep fusion of the three-dimensional combined features, which not only achieves a tight combination of semantic and geometric information, but also extracts more discriminative and robust feature fusion results through channel compression and context information aggregation. This enables the model to more accurately understand the semantics and spatial occupancy status of each voxel in the aerial scene, significantly improving the accuracy and reliability of subsequent spatial occupancy prediction.

[0147] In one exemplary embodiment, please refer to Figure 5 After obtaining the feature fusion result, the occupancy information corresponding to multiple voxels can be identified based on the feature fusion result. A feature mask sequence can be obtained based on the occupancy information, and the order of the feature mask sequence can be rearranged, including steps 241 to 244, wherein:

[0148] Step 241: The feature fusion results corresponding to multiple voxels are serialized and expanded to obtain the initial feature sequence.

[0149] The initial feature sequence is a one-dimensional feature vector compressed into a one-dimensional space from the feature fusion results of each voxel.

[0150] Here, the initial feature sequence can be used to represent the transformation of the feature fusion result of each voxel from its original multidimensional representation into a unified one-dimensional vector form, and to sort the one-dimensional feature vectors corresponding to each voxel according to the arrangement order of each voxel in the three-dimensional network to obtain the initial feature sequence.

[0151] It should be noted that the feature fusion results corresponding to multiple voxels can be flattened, arranging all dimensions of data into a long vector in a specific order. In this way, the rich feature information of each voxel is preserved and standardized into a consistent format.

[0152] Step 242: Project the feature fusion results corresponding to multiple voxels into scalars, use the scalars as occupancy probability scores, and perform thresholding based on the occupancy probability scores to identify the occupancy information of multiple voxels and obtain the occupancy grid mask.

[0153] The occupancy grid mask includes the occupancy mask value corresponding to the thresholding identification of each voxel.

[0154] In one implementation, the feature fusion result of each voxel can be input into a lightweight prediction layer to map multidimensional features into a single scalar value. After processing this scalar value through an activation function, the activated scalar value can be interpreted as the probability score of the voxel being occupied. Subsequently, this occupancy probability score is compared with a preset threshold (e.g., if the score is greater than 0.5, it is considered occupied; otherwise, it is considered unoccupied), thereby converting continuous probability values ​​into discrete occupancy mask values ​​(e.g., 1 represents occupied, and 0 represents unoccupied). In this way, a complete occupancy grid mask is generated, clearly representing the occupancy status of each voxel in the aerial scene.

[0155] Step 243: Align the initial feature sequence and the occupied grid mask to obtain the feature mask sequence.

[0156] Based on this, the initial feature sequence and the occupancy grid mask are aligned to obtain a feature mask sequence, so as to effectively associate the detailed feature information (initial feature sequence) of each voxel with its corresponding discrete occupancy state (occupancy grid mask).

[0157] In one implementation, the occupancy mask value (or its embedded representation) of each voxel can be concatenated with the initial feature sequence of that voxel to form an enhanced feature vector containing dual information. For example, the occupancy mask value can be used as an additional channel or dimension of the initial feature sequence. In this way, each element in the feature mask sequence not only contains the original visual and geometric features of the voxel, but also explicitly indicates the occupancy state of that voxel.

[0158] Step 244: The first voxel representing the entity's occupation information in the feature mask sequence is moved forward, and the second voxel representing the air's occupation information in the feature mask sequence is moved backward, to obtain the target feature sequence.

[0159] In one implementation, the occupancy information in the feature mask sequence includes physical occupancy and air occupancy.

[0160] In this context, entity occupancy refers to the state in which voxels in a 3D mesh are occupied by actual physical objects, such as buildings, trees, other aircraft, or ground obstacles. This entity occupancy information is crucial for path planning, collision detection, and obstacle avoidance for aerial platforms, as it directly relates to flight safety.

[0161] Furthermore, air occupancy refers to the state where voxels in a 3D mesh are in free space and not occupied by any physical objects, representing passable areas. Based on the distinction between entity occupancy and air occupancy information, the structure of an aerial scene can be clearly understood.

[0162] Based on a clear understanding of entity occupancy and air occupancy in a 3D mesh, the occupancy mask values ​​corresponding to each voxel in the feature mask sequence can be rearranged to obtain the target feature sequence. This step aims to perform a structured rearrangement of the feature mask sequence to optimize its information transmission efficiency when input into the target prediction model.

[0163] Specifically, the structured rearrangement of the mask values ​​in the feature mask sequence can be achieved by placing the first voxel representing the entity's occupation information at the front of the feature mask sequence, that is, placing the occupation mask values ​​corresponding to all voxels identified as being occupied by entities at the beginning of the sequence.

[0164] In practice, this can be achieved by traversing the feature mask sequence, identifying all voxels occupied by entities, collecting their corresponding feature vectors or occupancy mask values, and then placing this collected information at the beginning of the target feature sequence.

[0165] At the same time, the second voxel representing air occupancy in the occupancy information in the feature mask sequence is placed at the end, that is, the occupancy mask values ​​corresponding to all voxels identified as air occupancy are placed at the end of the sequence.

[0166] In practice, the feature vectors or occupancy mask values ​​corresponding to the remaining voxels identified as air occupancy can be placed after the entity occupancy voxel information in the target feature sequence.

[0167] Understandably, based on the above rearrangement strategy, it is ensured that the target prediction model can prioritize receiving and processing key information related to physical obstacles when processing sequences, thereby guiding the model to focus more attention on areas that are crucial to flight safety.

[0168] For example, please refer to Figure 6 , Figure 6 This is a schematic diagram of the first scenario in this embodiment. The following example is presented in this schematic diagram, using a drone as an example of terminal 102, to illustrate the occupancy guidance sequence rearrangement for the drone.

[0169] After processing the image and radar data collected by the UAV, the feature fusion results of each voxel in the 3D mesh corresponding to the aerial scene are obtained. Subsequently, the occupancy information of each voxel in the aerial scene can be identified and predicted based on the Mamba module to further determine the flight strategy of the UAV.

[0170] Specifically, to address the problem of wasted computing power caused by a large amount of useless air in aerial scenes, a 3D Hilbert curve mapping mechanism can be used to fuse the feature results of each voxel in the upper 3D mesh. Perform Hilbert sorting to serialize into a one-dimensional initial feature sequence. Specifically, its serialization process is represented as follows: .

[0171] in, This is the initial feature sequence after flattening into one dimension. The mapping rule from three-dimensional voxel coordinates to one-dimensional Hilbert curve indices. This refers to the operation of extracting features based on an index.

[0172] Here, the initial feature sequence Adjacent voxels in space maintain the continuity of their local positions after being flattened.

[0173] Furthermore, the voxel features are projected into scalars through an auxiliary layer (Aux Head), and the scalars are used as occupancy probability scores. An occupancy grid mask is then generated by binarization using a preset threshold. Specifically, the calculation formula (calculation process) for the occupancy grid mask is as follows:

[0174] ;

[0175] .

[0176] in, The predicted occupancy probability score. For activation function, This represents the occupancy mask value of each voxel in the 3D mesh after binarization. To determine the threshold.

[0177] Here, the grid mask value is used. It can be used to distinguish whether each voxel in a 3D mesh is a physical entity or redundant air.

[0178] For example, determining the threshold The occupancy probability fraction of voxels in the 3D mesh can be preferably 0.2. Higher than the judgment threshold The occupancy mask value is =1 indicates that the voxel is a physical entity; occupying probability score Below the judgment threshold The occupancy mask value is =0 indicates that the voxel is occupied by physical air.

[0179] After obtaining the occupied grid mask, the occupied grid mask can be compared with the initial feature sequence. Alignment yields a feature mask sequence. And based on this feature mask sequence A stable sorting process is performed, prioritizing all subsequences identified as physical entities to form an occupied sort, and placing all subsequences of empty voxels at the end to form an air sort. The combination of these two forms a highly compact occupied-air joint sort, i.e., the target feature sequence. Specifically, the rearrangement formula (steps) can be referenced in the following expression:

[0180] ;

[0181] .

[0182] in, The sorted index obtained after performing a stable sort. This is the inverted occupancy mask, used to ensure that entity features appear first. This is the final rearranged sequence. Corresponding to a compact sequence of preceding entity features, Corresponding to the subsequent air feature sequence, This indicates the concatenation and merging of sequences.

[0183] Furthermore, in determining the target feature sequence Then, the target feature sequence can be extracted. The front end includes physical entities. Part of the input is fed into a hierarchical Mamba module. Through a gated dual-branch structure and a state-space model, the high information density sequence after rearrangement is efficiently modeled for long-range geometric dependencies and extracted for multi-scale features. After feature recovery and de-rearrangement mapping by the decoder, the one-dimensional sequence is reversed to restore to a three-dimensional grid space. The final dense UAV three-dimensional semantic occupancy prediction result is output as the target prediction result.

[0184] Please refer to Figure 7 , Figure 7 This is a schematic diagram of the second scenario in this embodiment. Specifically, in the process of serializing and expanding the feature fusion results corresponding to multiple voxels to obtain the initial feature sequence, this embodiment preferably uses a three-dimensional Hilbert curve. In some alternative embodiments, the serialization mapping mechanism can also arrange the voxels in the three-dimensional mesh according to other different arrangements. Specifically, it can include: Hilbert sorting with Y-cycle priority, XYZ raster scan sorting, Z-order curve sorting, or height-priority Hilbert curves.

[0185] The above technical solution clearly distinguishes between entity occupancy and air occupancy in the feature mask sequence, and rearranges the sequence so that voxel information representing entity occupancy is placed first, while voxel information representing air occupancy is placed last. This structured input method enables the target prediction model for aerial perception to prioritize processing and learning key information related to physical obstacles. This helps the model identify and understand potential hazardous areas in the aerial scene more quickly, thereby improving the accuracy and robustness of future space occupancy predictions. The model can more effectively focus on entity occupancy information that is crucial to flight safety, reducing the computational resources scattered on non-critical information, thus optimizing the model's training efficiency and inference performance, and providing a more reliable perception foundation for real-time decision-making and safe operation of the aerial platform.

[0186] In one exemplary embodiment, such as Figure 8 As shown, before the step of inputting the target feature sequence into the target prediction model for airborne perception, the steps for effectively training the model include steps 51 to 54, wherein:

[0187] Step 51: Construct the target simulation sample dataset.

[0188] The target simulation sample dataset includes multiple sample data pairs. Each sample data pair includes sample acquisition data from an aerial scene and corresponding label values. The sample acquisition data includes sample image data and sample radar data. The label values ​​include occupancy information corresponding to the sample acquisition data. The occupancy information is used to represent the actual occupancy information of each voxel in the 3D mesh corresponding to the aerial scene where the sample acquisition data is located.

[0189] Here, the target simulation sample dataset can be a collection of data used to train a target prediction model for airborne perception. It contains a series of sample data pairs, each consisting of sample data (including sample image data and sample radar data) collected in an airborne scene and its corresponding real occupancy information (label value). The purpose of constructing this dataset based on sample data pairs is to provide the model with rich, realistically labeled training instances, enabling it to learn the mapping relationship from perception data to space occupancy prediction.

[0190] In one implementation, existing public datasets are used to generate data through a simulation environment, or real-world collected data is processed through manual or semi-automatic annotation to obtain the required label information. During the construction process, it is necessary to ensure the diversity and representativeness of the sample data to cover various situations that may occur in aerial scenarios.

[0191] It's important to note that most current 3D semantic occupancy perception technologies are tailored for ground-based autonomous driving. In the field of low-altitude drones, acquiring dense voxel-level 3D labeled data is extremely expensive, resulting in a severe lack of datasets that can be directly used for drone 3D occupancy perception training. More importantly, the industry lacks a fully automated pipeline capable of perfectly aligning high-precision geometric models of the physical world with drone multimodal sensor simulation data and automatically generating high-fidelity 3D voxel labels without human intervention.

[0192] Based on this, a method for constructing a target simulation sample dataset is proposed. As a fully automated generation pipeline for dense occupancy labels based on hierarchical flight planning and spatiotemporal alignment, it can generate sample data pairs including sample image data, sample radar data and corresponding label values ​​in batches to construct a target simulation sample dataset.

[0193] In one specific implementation, the step of automatically generating a pipeline to generate sample data pairs, including sample image data and sample radar data and corresponding label values, to construct a target simulation sample dataset may include:

[0194] (1) Obtain the simulation occupancy dataset.

[0195] The simulation occupancy dataset includes multiple simulation objects in an aerial scene.

[0196] Here, the simulation occupancy dataset can be used to represent a set of data created in a virtual environment that simulates a real aerial scenario.

[0197] It should be noted that the simulation occupancy dataset can contain pre-defined simulation objects with precise 3D geometric and semantic information, such as buildings, vehicles, pedestrians, trees, and terrain. These simulation objects are distributed in a virtual aerial scene according to predetermined rules or random patterns to simulate various complex situations that may occur in the real world.

[0198] As an example, the simulation occupancy dataset provided in this embodiment can be an Urban Building Instance Segmentation (Urbanbis) dataset. The Urbanbis dataset is deployed in a simulation environment by laying out the Urbanbis dataset in a preset simulation platform, and image data and point cloud data constituting the target simulation sample dataset are obtained based on a preset sample data acquisition strategy.

[0199] Another example is the sensor layout in a drone, which can be a 5-view camera + 64-line LiDAR configuration. Alternatively, the drone's sensor layout can also be a 4-view, 6-view, or even panoramic fisheye camera. Furthermore, the depth measurement arm can replace the mechanical LiDAR with a solid-state LiDAR, millimeter-wave radar, or a binocular depth camera.

[0200] (2) Based on the distribution information of the simulation objects in the simulation occupancy dataset, the simulation occupancy dataset is classified into regions.

[0201] After obtaining the simulation occupancy dataset, it can be classified into regions based on the distribution information of the simulation objects within the dataset. Region classification aims to divide the entire simulation occupancy dataset into several regions with different characteristics based on the spatial distribution characteristics of the simulation objects in the virtual scene.

[0202] Specifically, the distribution information of simulated objects can include their density, type diversity, size range, motion state, etc. For example, regions can be divided into "sparse regions" and "dense regions" based on object density; regions can be divided into "urban regions" (many buildings, vehicles) and "suburban regions" (most vegetation, few buildings) based on object type.

[0203] For example, clustering can be performed based on the number and type of objects within each grid after grid division; or the division can be based on predefined geographical or semantic boundaries to determine different regions in the simulation dataset.

[0204] (3) Based on the region type of different regions in the simulation occupancy dataset, sample the simulation occupancy dataset according to the corresponding strategy to obtain sample image data and sample radar data covering multiple simulation objects.

[0205] After classifying the simulation occupancy dataset, in order to ensure that the generated sample data can fully cover various complex aerial scenarios and improve the diversity and representativeness of the dataset, different sampling strategies can be selected according to different region types in the dataset.

[0206] For example, in densely populated urban areas, a high-density sampling strategy can be used to increase the frequency of sampling viewpoints and locations to capture more details and occlusions; in sparsely populated suburban areas, a sparse sampling strategy can be used to save computing resources while still covering the main objects.

[0207] It should be noted that sampling strategies can include random sampling, uniform sampling, importance-based sampling (e.g., increasing sampling density near key objects), or sampling based on a preset flight path. By simulating the perspectives and parameters of different sensors (such as cameras and LiDAR), corresponding sample image data (such as RGB images and depth maps) and sample radar data are rendered or generated from the virtual scene.

[0208] Please refer to Figure 9 , Figure 9 This is a schematic diagram of the first scenario of an embodiment of this application. As an example, the following is an example description of the region classification of the simulation occupancy dataset and the sampling of the corresponding strategy.

[0209] Specifically, after deploying the Urbanbis dataset into a pre-defined simulation platform to form a simulation environment, the simulation environment can be abstractly divided based on its vertical occupancy and geometric density. After dividing the simulation environment, a hierarchical optimization strategy is used to generate UAV acquisition trajectories to obtain sample data.

[0210] First, complex areas containing dense high-rise buildings and drastic height changes can be classified as complex areas. For complex areas in the simulation environment, an optimization algorithm based on the Traveling Salesman Problem (TSP) can be used to generate a first sampling path 81 with dense viewpoints as a search path to explore the complex areas, so as to achieve multi-view, all-round coverage of the three-dimensional physical structure in the simulation environment within the complex areas.

[0211] Secondly, sparse regions with small elevation changes and relatively open areas can be divided into boundaries and sparse regions. For the boundaries and sparse regions in the simulation environment, a connectivity-driven greedy search strategy can be adopted to generate a second sampling path 82 including sparse regular viewpoints as a search path to explore the boundaries and sparse regions, so that the UAV can quickly capture background and free space data with the most efficient flight path.

[0212] Here, as an alternative, the path planning algorithm can be replaced by the A-star Search Algorithm (A-star), the Rapidly-exploring Random Tree (RRT) algorithm, or an exploration algorithm based on Deep Reinforcement Learning (DRL).

[0213] Based on the exploration method in the above simulation environment, the UAV can simultaneously record multimodal observation data, which may include multi-view RGB images (front, left, right, bottom, and back), sparse lidar point clouds, and six-degree-of-freedom (6-DoF) pose information, thereby constructing a target simulation sample dataset.

[0214] (4) Voxelize the initial point cloud data to obtain an initial three-dimensional mesh containing multiple corresponding voxels. Traverse the initial three-dimensional mesh to obtain the center coordinates and average normal vectors of multiple voxels in the initial three-dimensional mesh.

[0215] Here, voxelization is the process of converting discrete point data in the initial point cloud data into a regular three-dimensional voxel mesh, where each voxel represents a small cubic region in three-dimensional space.

[0216] After voxelization, the center coordinates of each voxel can be calculated by traversing each voxel in the initial 3D mesh, representing the geometric center position of that voxel in 3D space. Simultaneously, for each voxel occupied by point data, its average normal vector can be calculated based on the normal vectors of all points within it. This average normal vector can be used to characterize the orientation of the voxel's surface in 3D space.

[0217] (5) Based on the sample semantic features in the sample image data and the dot product of the center coordinates and average normal vectors of multiple voxels, determine whether the voxels effectively hit the entity, so as to determine the occupancy information corresponding to multiple voxels.

[0218] Among them, sample semantic features refer to the features extracted from sample image data that are used to describe the semantic information of image content, such as object categories and region attributes identified by image segmentation networks.

[0219] Judgment can be made by combining semantic information and geometric information of voxels in the image.

[0220] For example, for a voxel in a 3D mesh, the ray observed from the perspective of a virtual sensor can be simulated based on its center coordinates and average normal vector. The ray can then be associated with the semantic features in the sample image data to determine whether the voxel corresponds to a region with a specific semantic meaning (such as "building" or "vehicle") in the image.

[0221] Simultaneously, the dot product of the average normal vector of the voxel and the line connecting the sensor to the center of the voxel can be calculated. If the dot product result is close to 1 (i.e., the normal direction is roughly parallel to the viewing direction), it indicates that the surface of the voxel is facing the sensor, which can effectively and accurately filter physical occlusion and artifacts, and further observe and confirm that it belongs to an entity.

[0222] In this way, by taking the dot product of the center coordinates and the average normal vector of each voxel, it can be accurately determined whether each voxel belongs to entity occupancy (occupied by an object) or air occupancy (free space), thereby generating high-precision three-dimensional occupancy information as a label.

[0223] (6) The sample image data and sample radar data are used as sample acquisition data, and the occupancy information corresponding to multiple voxels is used as the label value to obtain the target simulation sample dataset.

[0224] Among them, sample image data and sample radar data constitute the input of the model, namely sample acquisition data; while the occupancy information (entity occupancy or air occupancy) of each voxel obtained through the above-mentioned refined judgment serves as the output target of the model, namely the label value.

[0225] In one specific implementation, after sampling the simulation occupancy dataset according to the region type of different areas in the simulation occupancy dataset using corresponding strategies to obtain sample image data and sample radar data covering multiple simulation objects, the coordinate system of the sample radar data can be converted into the local coordinate system where the acquisition device is located; based on the local coordinate system, the observable range of the acquisition device is determined in the sample radar data, and the target radar data within the observable range is determined.

[0226] It should be noted that simulation environments typically use a global coordinate system to describe the entire scene, while actual aerial data acquisition devices (such as lidar or cameras mounted on drones) establish a local coordinate system with themselves as the origin. Therefore, the global geometric point cloud in the simulation environment can be converted into the local coordinate system of the drone.

[0227] For example, this can be achieved using a coordinate transformation matrix that contains the translation and rotation information of the acquisition device in the global coordinate system. First, the precise pose (including position and orientation) of the acquisition device in the global coordinate system of the simulation scene needs to be obtained. Then, using this pose information, the coordinates of each point in the simulation point cloud data are transformed from the global coordinate system to a local coordinate system with the acquisition device as the origin.

[0228] For example, if the acquisition device is located at (X_dev, Y_dev, Z_dev) in the global coordinate system and has a specific rotational orientation, the coordinate system can be transformed by subtracting the translation vector (X_dev, Y_dev, Z_dev) from the coordinates of each point in the point cloud data and applying the corresponding rotation matrix.

[0229] To more realistically simulate the perception limitations of actual data acquisition devices, this application determines the observable range of the acquisition device within the sample radar data using a local coordinate system, and identifies target radar data within this observable range. Actual acquisition devices, such as lidar or cameras, have a limited field of view (FoV) and maximum detection range. In the transformed local coordinate system, the relative angle between a point and the origin of the acquisition device can be used to determine whether a point falls within the device's field of view. Simultaneously, the Euclidean distance from each point to the device's origin is calculated and compared with the device's preset maximum detection range to filter out points within the effective detection range. Furthermore, a more refined simulation can consider occlusion effects, where points blocked by other objects will not be observed by the sensor; this can be simulated using techniques such as ray tracing. Through these filtering methods, the final target radar data represents the points that the simulated acquisition device can actually perceive in its current pose, thus more accurately reflecting the sensor input in the real world.

[0230] For example, please refer to Figure 10 , Figure 10 This is a schematic diagram of the second scenario in this embodiment. Specifically, this embodiment provides a fully automated pipeline for generating dense 3D semantic occupancy data that can be deployed offline, and can be used to automatically generate sample data for training.

[0231] Specifically, in the urban bimodal occupancy dataset on the left, which serves as the simulation occupancy dataset, sample data can be collected based on the region type of different areas in the simulation occupancy dataset after spatial registration to obtain sample data.

[0232] Here, the sample data may include coefficient radar data, UAV pose data, and RGB images from five cameras from different perspectives. In addition, the sample data may also include dense network labels corresponding to the radar data, pose data, and RGB images.

[0233] It should be noted that dense network labels can be obtained by cropping, aligning coordinates, geometrically consistent ray casting, and voxelizing the point cloud segmentation labels that serve as ground truth in the simulated occupancy dataset.

[0234] Step 52: Determine the sample feature sequence based on the sample image data and sample radar data.

[0235] After constructing the target simulation sample dataset, the sample feature sequence can be determined based on the sample image data and sample radar data in the target simulation sample dataset, so as to convert the original sample image data and sample radar data into a feature representation with rich semantic and geometric information that the model can process.

[0236] One implementation approach involves first voxelizing the sample radar data to obtain a 3D mesh. Then, 2D visual features are extracted from the sample image data, and 3D geometric features are extracted from the 3D mesh. Next, these features are fused to identify voxel occupancy information and generate a feature mask sequence. Finally, the occupancy mask values ​​in the feature mask sequence are rearranged to obtain a sample feature sequence for training. This process ensures that the format and information structure of the training data remain consistent with the input data during model inference.

[0237] Step 53: Input the sample feature sequence into the preset initial perception prediction network to predict the occupancy information of each voxel in the three-dimensional grid corresponding to the sample acquisition data, and obtain the predicted occupancy information.

[0238] The initial perception prediction network can be a neural network model that has not yet been fully trained, and its structure is similar to the final target prediction model for aerial perception.

[0239] Here, occupancy information can be represented in the form of probability values ​​or classification results. For example, for each voxel, the initial perception prediction network may output the probability that it is in a physical occupancy state, or predict whether it belongs to the occupancy category of "physical occupancy" or "air occupancy".

[0240] In order to enable the initial perception prediction network to perform forward propagation of input data based on the parameters of the sample feature sequence, preliminary prediction of the occupancy information of each voxel can be made based on the sample feature sequence to obtain the predicted occupancy information.

[0241] Step 54: Calculate the occupancy loss of each voxel in the 3D mesh based on the predicted occupancy information and the actual occupancy information. Perform backpropagation on the initial perception prediction network based on the occupancy loss and update the network parameters to obtain the target prediction model for aerial perception.

[0242] Occupancy loss is an indicator that measures the difference between the prediction results of the initial perception prediction network and the actual labels.

[0243] Here, the loss can be calculated using loss functions such as cross-entropy loss, Dice loss, or Focal loss. For example, for binary classification problems, a binary cross-entropy loss function can be used, specifically measuring the loss based on the log-likelihood difference between the predicted probability and the true label. The larger the loss value, the greater the prediction error of the model, and the greater the adjustment required.

[0244] After determining the occupancy loss, the initial sensing and prediction network can be backpropagated based on the occupancy loss to update the network parameters and obtain the target prediction model for aerial sensing. Specifically, based on the occupancy loss, the gradient of the occupancy loss with respect to each parameter in the initial sensing and prediction network is calculated using the chain rule, and the network parameters are updated accordingly.

[0245] It should be noted that gradients can be used to instruct the adjustment of each parameter in the initial sensing and prediction network to minimize the occupancy loss (convergence). By using gradients and a preset learning rate, the optimizer can adjust the network's weights and biases, iteratively reducing the occupancy loss. This allows the initial sensing and prediction network to gradually learn the mapping relationship from sample feature sequences to actual occupancy information, ultimately converging into a high-performance aerial sensing target prediction model.

[0246] Through the above technical solution, this application provides a systematic training method for a target prediction model for aerial perception. By constructing a target simulation sample dataset containing sample image data, sample radar data, and labels of actual occupancy information, and using this dataset to iteratively train the initial perception prediction network, the network can effectively learn to extract features from multimodal perception data and accurately predict spatial occupancy information. Specifically, by calculating the occupancy loss between predicted and actual occupancy information, and based on this, performing backpropagation and updating network parameters, the model can continuously optimize its internal parameters, thereby significantly improving its recognition accuracy and generalization ability for the occupancy information of various voxels in the aerial scene. This enables the final aerial perception target prediction model to predict spatial occupancy more accurately and robustly in practical applications, providing a reliable perception foundation for subsequent tasks such as path planning and obstacle avoidance, and effectively solving the problem that model performance depends on human experience or is difficult to obtain high-quality training data.

[0247] Please refer to Figure 11 , Figure 11This is a schematic diagram of the third scenario in this embodiment. This embodiment proposes a system framework for constructing and perceiving prediction of three-dimensional semantic occupancy data for low-altitude scenarios. Overall, this embodiment includes: (1) a data automatic labeling pipeline of hierarchical path planning and geometric filtering on the right to construct a target simulation sample dataset; and (2) a perceiving prediction network of multimodal fusion and sequence rearrangement on the left, which can accurately identify the prediction information of each voxel in the three-dimensional mesh corresponding to the aerial scene.

[0248] Here, the dense grid labels generated by the pipeline on the right (target simulation sample dataset) can be used as real values ​​to supervise the training of the prediction network on the left.

[0249] It is understood that although the above embodiments are all based on "drones" and designed for the aerial scene in which drones are located, the sample dataset and aerial scene recognition and prediction proposed in this application can be applied to a variety of other application scenarios. The following are some examples that are not exhaustive:

[0250] (1) Flying cars and low-altitude economic aircraft, as the core perception system of future urban low-altitude transportation tools, provide them with all-weather 3D obstacle avoidance and drivable space prediction.

[0251] (2) Ground-based high-level autonomous driving system, which can be applied to intelligent connected vehicles equipped with lidar and multiple cameras to solve the 3D semantic occupancy prediction of vehicles in scenarios with high requirements for vertical height vision, such as complex overpasses and long tunnels.

[0252] (3) Indoor and outdoor embodied intelligent robots, such as quadruped robot dogs and logistics delivery robots, utilize the multi-scale prediction capability of the present invention to construct high-precision local 3D semantic maps to assist them in navigation and obstacle avoidance in unknown environments.

[0253] (4) Urban digital twin and high-precision map construction can automatically build pipelines using dense 3D semantic occupancy data from offline terminals, and automatically generate a high-fidelity 3D urban semantic grid base for smart cities and mixed reality (MR) applications.

[0254] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0255] Based on the same inventive concept, this application also provides an airborne perception prediction system for implementing the low-altitude three-dimensional occupancy perception prediction method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations of one or more airborne perception prediction system embodiments provided below can be found in the limitations of the low-altitude three-dimensional occupancy perception prediction method described above, and will not be repeated here.

[0256] In one exemplary embodiment, such as Figure 12 As shown, an aerial perception prediction system is provided, comprising: an acquisition module, a feature extraction module, a feature fusion module, a perception module, a feature rearrangement module, and a prediction module, wherein:

[0257] The acquisition module is used to acquire image data and radar data collected in the aerial scene, and to perform voxelization processing on the radar data to obtain a three-dimensional mesh; wherein, the three-dimensional mesh includes multiple voxels divided by the aerial scene;

[0258] The feature extraction module is used to extract two-dimensional visual features from image data and three-dimensional geometric features from radar data;

[0259] The feature fusion module is used to fuse features of each voxel based on semantic information in two-dimensional visual features and depth information in three-dimensional geometric features to obtain the corresponding feature fusion result.

[0260] The perception module is used to identify the occupancy information corresponding to multiple voxels based on the feature fusion result, and to obtain the feature mask sequence based on the occupancy information; wherein, the feature mask sequence is a sequence composed of the occupancy mask values ​​corresponding to multiple voxels;

[0261] The feature rearrangement module is used to rearrange the occupancy mask values ​​corresponding to each voxel in the feature mask sequence to obtain the target feature sequence;

[0262] The prediction module is used to input the target feature sequence into the target prediction model of the air perception, so as to extract the target feature information corresponding to multiple voxels at the current time of the target feature sequence, and perform spatial occupancy prediction for each voxel based on the target feature information, and output the target prediction result; wherein, the target prediction result is used to represent the occupancy information and semantic information of each voxel in the real scene.

[0263] The modules in the aforementioned airborne perception prediction system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0264] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 13 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores image data, point cloud data, and pose data. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a low-altitude three-dimensional occupancy perception prediction method.

[0265] Those skilled in the art will understand that Figure 13 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0266] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described low-altitude three-dimensional occupancy perception prediction method.

[0267] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0268] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0269] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0270] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A low-altitude three-dimensional occupancy perception and prediction method, characterized in that, The method includes: Image data and radar data collected in an aerial scene are acquired, and the radar data is voxelized to obtain a three-dimensional mesh; wherein, the three-dimensional mesh includes multiple voxels divided by the aerial scene; Extract two-dimensional visual features from the image data and three-dimensional geometric features from the radar data; Based on the semantic information in the two-dimensional visual features and the depth information in the three-dimensional geometric features, feature fusion is performed on each voxel to obtain the corresponding feature fusion result; Based on the feature fusion result, the occupancy information corresponding to the plurality of voxels is identified, and a feature mask sequence is obtained based on the occupancy information; wherein, the feature mask sequence is a sequence composed of the occupancy mask values ​​corresponding to the plurality of voxels; The occupancy mask values ​​corresponding to each voxel in the feature mask sequence are rearranged to obtain the target feature sequence; wherein, the occupancy information in the feature mask sequence includes entity occupancy and air occupancy, and the step of rearranging the occupancy mask values ​​corresponding to each voxel in the feature mask sequence to obtain the target feature sequence includes: placing the first voxel representing entity occupancy in the feature mask sequence at the beginning, and placing the second voxel representing air occupancy in the feature mask sequence at the end, to obtain the target feature sequence; The target feature sequence is input into the target prediction model for aerial perception to extract target feature information corresponding to the multiple voxels at the current time where the target feature sequence is located, and space occupancy prediction is performed on each voxel based on the target feature information to output the target prediction result; wherein, the target prediction result is used to represent the occupancy information and semantic information of each voxel in the real scene.

2. The method according to claim 1, characterized in that, After the steps of extracting the two-dimensional visual features from the image data and the three-dimensional geometric features from the radar data, the method further includes: Extract high-dimensional semantic features from the two-dimensional visual features; wherein, the high-dimensional semantic features are used to represent the semantic information in the image data; The three-dimensional geometric features are projected onto a two-dimensional plane to obtain two-dimensional geometric depth features; The two-dimensional geometric depth features are convolved to generate depth cue information corresponding to the three-dimensional mesh, and the depth cue information is encoded to obtain geometric cue features; Combining the two-dimensional visual features and the geometric cue features, the depth probability distribution of each voxel in the three-dimensional grid is calculated; wherein, the depth probability distribution is used to represent the depth information corresponding to the multiple voxels in the radar data.

3. The method according to claim 1 or 2, characterized in that, The step of fusing features of each voxel based on the semantic information in the two-dimensional visual features and the depth information in the three-dimensional geometric features to obtain the corresponding feature fusion result includes: Based on the high-dimensional semantic features corresponding to the two-dimensional visual features and the depth probability distribution corresponding to the three-dimensional geometric features, the semantic features corresponding to the semantic information are extended along the depth axis corresponding to the depth information through the outer product operation to generate three-dimensional frustum features. The three-dimensional frustum features are mapped onto the three-dimensional mesh to obtain the three-dimensional visual features; The three-dimensional visual features and the three-dimensional geometric features are concatenated along the channel dimension to obtain a three-dimensional combined feature; A three-dimensional convolution operation is performed on the three-dimensional combined features. By sliding the convolution kernel, the three-dimensional visual features and the three-dimensional geometric features contained in the three-dimensional combined features are fused, the number of channels is compressed, and the context information is aggregated to obtain the feature fusion result.

4. The method according to claim 1, characterized in that, The step of identifying the occupancy information corresponding to the plurality of voxels based on the feature fusion result, and obtaining a feature mask sequence based on the occupancy information includes: The feature fusion results corresponding to the multiple voxels are serialized and expanded to obtain an initial feature sequence; wherein, the initial feature sequence is a one-dimensional feature vector of each voxel after the feature fusion result is compressed into a one-dimensional space. The feature fusion results corresponding to the multiple voxels are projected as scalars, and the scalars are used as occupancy probability scores. Thresholding processing is performed based on the occupancy probability scores to identify the occupancy information of the multiple voxels and obtain an occupancy grid mask. The occupancy grid mask includes the occupancy mask value corresponding to the thresholding identification of each voxel. Align the initial feature sequence with the occupied grid mask to obtain the feature mask sequence.

5. The method according to claim 1, characterized in that, Prior to the step of inputting the target feature sequence into the airborne perception target prediction model, the method further includes: Construct a target simulation sample dataset; wherein, the target simulation sample dataset includes multiple sample data pairs, the sample data pairs include sample acquisition data of aerial scenes and corresponding label values, the sample acquisition data includes sample image data and sample radar data, the label values ​​include occupancy information corresponding to the sample acquisition data, the occupancy information is used to represent the actual occupancy information of each voxel in the three-dimensional mesh corresponding to the aerial scene where the sample acquisition data is located; Based on the sample image data and the sample radar data, determine the sample feature sequence; The sample feature sequence is input into a preset initial perception prediction network to predict the occupancy information of each voxel in the three-dimensional grid corresponding to the sample collection data, and to obtain the predicted occupancy information. The occupancy loss of each voxel in the 3D mesh is calculated based on the predicted occupancy information and the actual occupancy information. The initial perception prediction network is backpropagated based on the occupancy loss, and the network parameters are updated to obtain the target prediction model for aerial perception.

6. The method according to claim 5, characterized in that, The steps for constructing the target simulation sample dataset include: Obtain a simulation occupancy dataset; wherein, the simulation occupancy dataset includes multiple simulation objects in an aerial scene; Based on the distribution information of the simulation objects in the simulation occupancy dataset, the simulation occupancy dataset is classified into regions. Based on the region type of different areas in the simulation occupancy dataset, the simulation occupancy dataset is sampled according to the corresponding strategy to obtain sample image data and sample radar data covering multiple simulation objects; The initial point cloud data is voxelized to obtain an initial 3D mesh containing multiple corresponding voxels. The initial 3D mesh is traversed to obtain the center coordinates and average normal vectors corresponding to multiple voxels in the initial 3D mesh. Based on the sample semantic features in the sample image data and the dot product of the center coordinates and average normal vectors of the multiple voxels, it is determined whether the voxels effectively hit the entity, so as to determine the occupancy information corresponding to the multiple voxels. The sample image data and the sample radar data are used as sample acquisition data, and the occupancy information corresponding to the multiple voxels is used as label values ​​to obtain the target simulation sample dataset.

7. The method according to claim 6, characterized in that... The step of sampling the simulation occupancy dataset according to the region type of different regions in the simulation occupancy dataset using a corresponding strategy to obtain sample image data and sample radar data covering multiple simulation objects includes: If the region type is a complex region in the simulation environment, an optimization algorithm based on the traveling salesman problem is used to generate a first sampling path including dense viewpoints, and the first sampling path is used as a search path to explore the complex region and obtain the corresponding sample image data and sample radar data of the simulation object. If the region type is a boundary and sparse region in the simulation environment, a connectivity-driven greedy search strategy is adopted to generate a second sampling path including sparse regular viewpoints, and the second sampling path is used as the search path to explore the boundary and sparse region to obtain the corresponding sample image data and sample radar data of the simulation object.

8. The method according to claim 6, characterized in that, After the step of sampling the simulation occupancy dataset according to the region type of different regions in the simulation occupancy dataset using a corresponding strategy to obtain sample image data and sample radar data covering multiple simulation objects, the method further includes: The coordinate system of the sample radar data is converted to the local coordinate system of the acquisition device. Based on the local coordinate system, the observable range of the acquisition device is determined in the sample radar data, and the target radar data within the observable range is determined.

9. A predictive system for aerial perception, characterized in that, The system includes: An acquisition module is used to acquire image data and radar data collected in an aerial scene, and to perform voxelization processing on the radar data to obtain a three-dimensional mesh; wherein, the three-dimensional mesh includes multiple voxels divided by the aerial scene; The feature extraction module is used to extract two-dimensional visual features from the image data and three-dimensional geometric features from the radar data; The feature fusion module is used to perform feature fusion on each voxel based on the semantic information in the two-dimensional visual features and the depth information in the three-dimensional geometric features to obtain the corresponding feature fusion result; The perception module is used to identify the occupancy information corresponding to the plurality of voxels based on the feature fusion result, and to obtain a feature mask sequence based on the occupancy information; wherein, the feature mask sequence is a sequence composed of the occupancy mask values ​​corresponding to the plurality of voxels; A feature rearrangement module is used to rearrange the occupancy mask values ​​corresponding to each voxel in the feature mask sequence to obtain a target feature sequence; wherein, the occupancy information in the feature mask sequence includes entity occupancy and air occupancy, and the step of rearranging the occupancy mask values ​​corresponding to each voxel in the feature mask sequence to obtain the target feature sequence includes: placing the first voxel representing entity occupancy in the feature mask sequence at the beginning, and placing the second voxel representing air occupancy in the feature mask sequence at the end, to obtain the target feature sequence; The prediction module is used to input the target feature sequence into the target prediction model for aerial perception, to extract the target feature information corresponding to the multiple voxels at the current time where the target feature sequence is located, and to perform space occupancy prediction on each voxel according to the target feature information, and output the target prediction result; wherein, the target prediction result is used to represent the occupancy information and semantic information of each voxel in the real scene.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

Images