A satellite and ground collaborative scenario semantic completion method, device and medium

Abstract

The application discloses a satellite and ground cooperative scene semantic completion method and device and a medium. After a three-dimensional voxel generation module in a scene semantic completion model extracts voxels from a ground image, a feature processing module is combined to perform feature extraction and segmentation on the ground image and a satellite image in parallel, so that voxel features, first satellite bird's-eye view features and a semantic segmentation map are obtained. A LM geometric alignment module with a fixed damping factor is used to iteratively optimize a global positioning system (GPS) reference pose, the semantic segmentation map, ground bird's-eye view features obtained based on voxel feature pooling and the first satellite bird's-eye view features, and a GPS optimal pose is used to perform spatial conversion on the first satellite bird's-eye view features, so that second satellite bird's-eye view features after alignment are obtained. After a fusion module sequentially performs cross-attention fusion and gate fusion on the second satellite bird's-eye view features and the ground bird's-eye view features, a prediction module is combined to predict semantic completion results based on voxel features.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, autonomous driving perception, and multimodal remote sensing image analysis, and relates to, but is not limited to, a method for scene semantic completion based on satellite and ground collaboration. Background Technology

[0002] 3D scene semantic completion is a technique that jointly infers and predicts the complete geometry and semantic category of a 3D scene using sparse or incomplete observation data (such as monocular images). It is primarily applied in fields such as autonomous driving, robot navigation, and human-computer interaction. Its core advantage lies in its ability to overcome the limitations of sensor viewpoint and occlusion, providing the system with a high-precision, high-completeness 3D environmental data foundation. In recent years, camera-based methods have gained widespread attention due to their low cost; however, a single ground viewpoint inherently suffers from field-of-view occlusion problems, making it difficult to reconstruct the complete scene layout.

[0003] In related technologies, a satellite-ground collaborative semantic completion framework has been proposed, which fuses satellite features with ground features through a deformable attention mechanism. However, this method has the following shortcomings: First, the attention mechanism implicitly learns geometric alignment, lacking explicit spatial geometric constraints, which limits its performance on categories with strong geometric structures such as vehicles and parking spaces, and is prone to overfitting in the later stages of training; Second, although pure geometric methods can improve the accuracy of geometric categories, they are difficult to handle fine-grained semantic categories that depend on texture and annotation conventions, such as trucks and traffic signs.

[0004] Therefore, how to efficiently and accurately obtain semantic completion results of the surrounding environment including occluded areas during vehicle operation has become an urgent problem to be solved. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a semantic completion method for scenes based on satellite and ground collaboration, which at least solves the problem that related technologies cannot efficiently and accurately obtain semantic completion results of the surrounding environment containing occluded areas.

[0006] According to a first aspect of the present invention, a method for semantic completion of a scene based on satellite and ground collaboration is provided, comprising:

[0007] Acquire ground images, GPS reference pose, and satellite images of the surrounding environment while the vehicle is currently in motion; The ground image is voxel extracted using a 3D voxel generation module to obtain non-empty voxels; Based on the feature processing module and non-empty voxels, feature extraction and segmentation are performed on ground images and satellite images in parallel to obtain target voxel features and semantic segmentation maps corresponding to ground images, as well as first satellite bird's-eye view features corresponding to satellite images. The LM geometric alignment module is used to iteratively optimize the GPS reference pose, semantic segmentation map, ground bird's-eye view features obtained by target voxel feature pooling, and first satellite bird's-eye view features to obtain the optimal GPS pose. The optimal GPS pose is then used to spatially transform the first satellite bird's-eye view features to obtain the second satellite bird's-eye view features aligned with the ground bird's-eye view features. The third satellite bird's-eye view features are obtained by sequentially performing cross-attention fusion and gating fusion on the features of the second satellite bird's-eye view and the features of the ground bird's-eye view through the fusion module. Based on the prediction module, the features of the third satellite bird's-eye view and the target voxel features are predicted to obtain the semantic completion results of the surrounding environment.

[0008] Optionally, the step of extracting voxels from the ground image using a three-dimensional voxel generation module to obtain non-empty voxels includes: Obtain the disparity map corresponding to the ground image; and calculate the depth value of each pixel point by point based on the disparity map; Back projection is performed based on the depth value to determine the x and y coordinates of each pixel; and the depth point corresponding to each pixel is constructed based on the x and y coordinates and the depth value. Project all depth points into a preset three-dimensional voxel space. If any voxel contains at least one depth point, then that voxel is considered a non-empty voxel.

[0009] Optionally, the feature processing module includes a satellite feature submodule, a ground feature submodule, and a segmentation submodule; the ground feature submodule includes a feature extraction layer, a query encoding layer, a cross-attention layer, a self-attention layer, and a pooling layer; The feature processing module and non-empty voxels perform feature extraction and segmentation on ground and satellite images in parallel to obtain target voxel features and semantic segmentation maps corresponding to the ground images, and first satellite bird's-eye view features corresponding to the satellite images, including: The ground image is used to extract features through a feature extraction layer to obtain two-dimensional ground features; The coordinates of non-empty voxels are matched with the trained voxel query embeddings in the three-dimensional voxel space by the query encoding layer, and the matched voxel query embeddings are used as the first query vector. The first query vector and the two-dimensional ground features are input into the cross-attention layer. The first value vector and the first key vector are determined through the two-dimensional ground features. The initial voxel features are determined based on the first key vector, the first value vector, and the first query vector. The initial voxel features are enhanced using a self-attention layer to obtain the target voxel features; and the target voxel features are pooled along the height dimension to obtain the ground bird's-eye view features. The satellite feature submodule extracts features from the satellite image to obtain the first satellite bird's-eye view features; and the segmentation submodule segments the ground image to obtain a semantic segmentation map.

[0010] Optionally, the LM geometric alignment module is used to iteratively optimize the GPS reference pose, semantic segmentation map, ground bird's-eye view features obtained based on target voxel feature pooling, and first satellite bird's-eye view features to obtain the optimal GPS pose, including: The target weight matrix is ​​calculated using the semantic segmentation map. In the first optimization iteration, the coordinates of each pixel in the GPS reference pose and the first satellite bird's-eye view features are substituted into the spatial transformation formula to obtain the sampling coordinates of the satellite image. Bilinear interpolation sampling is performed on the sampling coordinates to obtain the first satellite bird's-eye view feature temporarily aligned with the ground bird's-eye view feature in the same coordinate system; The ground bird's-eye view features and the temporarily aligned first satellite bird's-eye view features are used to perform point-by-point calculations on the pixels at the same location to obtain the first target residual vector; The pose increment is calculated based on the target weight matrix, a fixed damping factor, and the first target residual vector; the pose increment is summed with the GPS reference pose to obtain the new pose after the first iteration. The above steps are repeated based on the new pose and the features of the first satellite bird's-eye view to obtain the new target residual vector; and the target pose is determined based on the weighted sum of the new target residual vector and the first target residual vector in the new pose and the GPS reference pose. The target pose is used as the GPS reference pose for the second optimization iteration, and the above optimization steps are repeated using the same fixed damping factor until the number of iterations is reached to obtain the optimal GPS pose.

[0011] Optionally, the step of substituting the coordinates of each pixel in the GPS reference pose and the features of the first satellite bird's-eye view into the spatial transformation formula to obtain the sampling coordinates of the satellite image includes: Substitute the GPS reference pose and the coordinates of each pixel into the spatial transformation formula to obtain the sampled coordinates; The space transformation formula is as follows: ; ; In the above formula, For the floor function, The x-coordinate of the sampling coordinates The vertical coordinate of the sampling coordinates. The yaw angle is the GPS reference pose. x The x-coordinate of the GPS reference pose. y The ordinate is the GPS reference pose. uThe x-coordinate of each pixel, v The ordinate of each pixel.

[0012] Optionally, calculating the target weight matrix from the semantic segmentation graph includes: The semantic segmentation map is projected onto the satellite image to obtain the corresponding first real number mask set; based on the non-static categories on the semantic segmentation map, the corresponding real number masks in the first real number mask set are set to 0 to obtain the second real number mask set. Substituting the real number masks from the second set of real number masks into the numerical truncation formula yields the third set of real number masks. The third set of real-valued masks is used as the initial weight matrix, and the initial weight matrix is ​​substituted into the weight adjustment formula to obtain the target weight matrix; The numerical truncation formula is as follows: ; In the above formula, For the second set of real number masks, the first i A real number mask, The third real number mask set i A real number mask; It is a numerical truncation function; The weight adjustment formula is as follows: ; In the above formula, The weight matrix is ​​the first i Each weight value The power exponent, This is the global scaling factor.

[0013] Optionally, the fusion module includes a cross-attention submodule and a gating fusion submodule; The third satellite bird's-eye view features are obtained by sequentially performing cross-attention fusion and gated fusion on the second satellite bird's-eye view features and the ground bird's-eye view features through the fusion module, including: The second satellite bird's-eye view features and the ground bird's-eye view features are input into the cross-attention submodule. The second satellite bird's-eye view features are linearly transformed to obtain the second query vector. The ground bird's-eye view features are linearly transformed to obtain the second value vector and the second key vector. The enhanced satellite bird's-eye view features are obtained based on the second query vector, the second value vector and the second key vector. The enhanced satellite bird's-eye view features and the second satellite bird's-eye view features are gated and fused using the gated fusion formula in the gated fusion submodule to obtain the third satellite bird's-eye view features; the gated fusion formula is as follows:

[0014] In the above formula, Features of the third satellite aerial view, The gating coefficient, Features of the second satellite aerial view, Features of the enhanced satellite bird's-eye view.

[0015] Optionally, the prediction module includes a first feature submodule, a second feature submodule, a fusion submodule, a feature filtering submodule, and a prediction submodule; The prediction module predicts features from the third satellite bird's-eye view to obtain semantic completion results for the surrounding environment, including... The first feature submodule is used to extract features from the target voxel features to obtain the enhanced target voxel features; The second feature submodule is used to extract features from the third satellite bird's-eye view features to obtain the target satellite bird's-eye view features. The target satellite bird's-eye view features are up-dimensional, and the up-dimensional target satellite bird's-eye view features and the enhanced target voxel features are fused using the fusion submodule to obtain fused voxel features; The feature filtering submodule filters the fused voxel features, and the prediction module predicts the filtered voxel features to obtain the scene semantic completion result.

[0016] According to a second aspect of the present invention, an electronic device is provided, the electronic device including a processor and a memory, the memory storing a computer program, which, when executed by the processor, implements the steps of the method described in the first aspect.

[0017] According to a third aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0018] According to the scheme provided in the embodiments of the present invention, the scene semantic completion model includes a feature processing module, a three-dimensional voxel generation module, an LM geometric alignment module with a fixed damping factor, a fusion module, and a prediction module. The model acquires ground images, GPS reference poses, and satellite images of the surrounding environment under the current driving state of the vehicle; it also acquires ground images, GPS reference poses, and satellite images of the surrounding environment under the current driving state of the vehicle; it extracts voxels from the ground images using the three-dimensional voxel generation module to obtain non-empty voxels; and it performs feature extraction and segmentation on the ground images and satellite images in parallel based on the feature processing module and the non-empty voxels to obtain the target voxel features and semantic segmentation map corresponding to the ground images, as well as the satellite image features. The system first obtains the first satellite bird's-eye view features; then, using the LM geometric alignment module, iteratively optimizes the GPS reference pose, semantic segmentation map, ground bird's-eye view features obtained from target voxel feature pooling, and the first satellite bird's-eye view features to obtain the optimal GPS pose. The optimal GPS pose is then used to spatially transform the first satellite bird's-eye view features to obtain the second satellite bird's-eye view features aligned with the ground bird's-eye view features. The fusion module sequentially performs cross-attention fusion and gated fusion on the second satellite bird's-eye view features and the ground bird's-eye view features to obtain the third satellite bird's-eye view features. Finally, the prediction module predicts the semantic completion results of the surrounding environment based on the third satellite bird's-eye view features and the target voxel features. In this process, satellite images are acquired simultaneously with ground images and GPS reference poses. Satellite images, as low-cost and readily available reference information, provide a bird's-eye view of the global scene structure, naturally complementing the ground view. By filtering out non-empty voxels from the ground images, the ineffective calculation of empty voxels can be avoided. Satellite bird's-eye images provide global and unobstructed static scene layout information. Aligning them with ground bird's-eye image features to the same coordinate system can effectively compensate for the global context information that cannot be obtained from the ground view due to limited field of view and occlusion. Cross-attention fusion processing is applied to the second satellite bird's-eye image features and the ground bird's-eye image features, allowing the local details of the ground and the satellite bird's-eye image features to complement each other. Further gating fusion processing allows the accurate spatial location provided by the aligned second satellite bird's-eye image features and the semantic features provided by the cross-attention fusion output satellite bird's-eye image feature map to be effectively combined. Finally, after prediction, the accuracy and completeness of semantic completion can be improved more accurately. In summary, the solution of the present invention can retain the advantages of geometric alignment while supplementing semantic details, and can efficiently and accurately obtain the semantic completion results of the surrounding environment of the vehicle during driving. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein: Figure 1 A flowchart illustrating a satellite-and-ground collaborative scene semantic completion method provided in an embodiment of the present invention; Figure 2 A schematic diagram of the overall framework of a satellite-and-ground collaborative scene semantic completion method provided in this embodiment of the invention; Figure 3 This is a schematic diagram of the internal processing of an LM geometry alignment module with a fixed damping factor, provided in an embodiment of the present invention. Figure 4 This is a schematic diagram of the structure of a semantic refinement flow provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of a learnable weighted fusion structure provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of a dual bird's-eye view supervision structure provided in an embodiment of the present invention; Figure 7 A comparison chart of qualitative results with baseline methods on the SemanticKITTI dataset provided for embodiments of the invention; Figure 8 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. The following embodiments are used to illustrate the present invention, but are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0022] It should be noted that the terms "first, second, and third" used in the embodiments of the present invention are only used to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, and third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present invention described herein can be implemented in an order other than that illustrated or described herein.

[0023] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which these embodiments of the invention pertain. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0024] Figure 1 This is a flowchart illustrating a satellite-and-ground collaborative scene semantic completion method provided in an embodiment of the present invention. The satellite-and-ground collaborative scene semantic completion method provided in this embodiment of the present invention can be executed by an electronic device, such as a computer or server.

[0025] like Figure 1 As shown, the semantic completion method for satellite and ground-based collaborative scenarios includes: S101. Acquire ground images, GPS reference pose, and satellite images of the surrounding environment while the vehicle is currently in motion.

[0026] In embodiments of the present invention, during vehicle operation, ground images, satellite images, and Global Positioning System (GPS) reference poses of the vehicle's current surrounding driving environment are acquired. Ground images can be acquired using cameras installed in the vehicle, with a resolution of 1226×370. Satellite images can be downloaded from a map service provider via the network, with a resolution of 512×512. The GPS reference pose, i.e., the GPS coarse pose, specifically includes lateral displacement, longitudinal displacement, and yaw angle, and can be acquired by receiving satellite signals and parsing NMEA protocol data through the GPS module inside the vehicle.

[0027] S102. Voxels are extracted from the ground image using the three-dimensional voxel generation module to obtain non-empty voxels.

[0028] S103. Based on the feature processing module and non-empty voxels, feature extraction and segmentation are performed on the ground image and satellite image in parallel to obtain the target voxel features and semantic segmentation map corresponding to the ground image, as well as the first satellite bird's-eye view features corresponding to the satellite image.

[0029] In embodiments of the present invention, a non-empty voxel is any voxel containing at least one three-dimensional depth point. The semantic segmentation map contains the classification results of each pixel in the ground image, such as roads, pedestrians, and trees. Satellite bird's-eye view features refer to the top-down feature representations directly extracted from satellite orthophoto or oblique imagery that reflect the land cover type, geometric contours of ground features, and macroscopic scene semantics.

[0030] In an embodiment of the present invention, a scene semantic completion model is proposed. The scene semantic completion model includes a three-dimensional voxel generation module and a feature processing module. The three-dimensional voxel generation module is used to extract non-empty voxels from the input ground image. The feature processing module includes multiple branches, which perform feature extraction on the ground image in parallel to obtain ground bird's-eye view features, segment the ground image to obtain a semantic segmentation map, and extract features from the satellite image to obtain a first satellite bird's-eye view feature.

[0031] S104. The LM geometric alignment module is used to iteratively optimize the GPS reference pose, semantic segmentation map, ground bird's-eye view features obtained by target voxel feature pooling, and first satellite bird's-eye view features to obtain the optimal GPS pose. The optimal GPS pose is then used to spatially transform the first satellite bird's-eye view features to obtain the second satellite bird's-eye view features aligned with the ground bird's-eye view features.

[0032] In embodiments of the present invention, the scene semantic completion model further includes an LM geometric alignment module with a fixed damping factor. This module is obtained by fixing the damping factor in the standard Levenberg-Marquardt algorithm (LM). Ground bird's-eye view features refer to the feature representations reflecting the planar layout and spatial geometric relationships of ground objects extracted after projecting a first-person ground view image onto a top-down view through perspective transformation or deep learning. The target voxel features are max-pooled along the height axis to obtain ground bird's-eye view features. These features, along with the GPS reference pose, semantic segmentation map, and first satellite bird's-eye view features, are input into the LM geometric alignment module. Iterative optimization is performed based on the GPS reference pose until the optimal GPS pose is obtained. Spatial transformation of the first satellite bird's-eye view features using the optimal GPS pose aligns them with the ground bird's-eye view features in the coordinate system of the ground bird's-eye view features.

[0033] When spatially transforming the features of the first satellite bird's-eye view using the optimal GPS pose, the transformation can be performed using subsequent spatial transformation formulas.

[0034] The damping factor can be fixed at 0.01, and the total number of iterations can be set to 10.

[0035] S105. The second satellite bird's-eye view features and the ground bird's-eye view features are sequentially subjected to cross-attention fusion and gating fusion through the fusion module to obtain the third satellite bird's-eye view features.

[0036] S106. Based on the prediction module, predict the features of the third satellite bird's-eye view and the target voxel features to obtain the semantic completion results of the surrounding environment.

[0037] In an embodiment of the present invention, the scene semantic completion model further includes a fusion module. This fusion module involves two fusion processes. First, the second satellite bird's-eye view features and the ground bird's-eye view features are linearly projected to calculate the corresponding query vector, value vector, and key vector. Then, these three vectors are fused through cross-attention. Finally, the enhanced satellite bird's-eye view features and the second satellite bird's-eye view features are gated and fused again. During the gated fusion, the gating coefficients are optimized after the scene semantic completion model has been trained.

[0038] Furthermore, the scene semantic completion model also includes a prediction module. This module is specifically used to perform feature extraction, fusion, feature filtering, and prediction operations on the features of the third satellite bird's-eye view and the target voxel features, ultimately obtaining the semantic completion result corresponding to the surrounding environment. This semantic completion result is presented as a voxel semantic map, which can reflect the objects and object categories in the surrounding environment covering the occluded area during vehicle movement. It can spatially fill in and infer the category of the surrounding environment when the vehicle encounters visually occluded areas, thereby generating a wide-area scene description containing complete information about surrounding objects and their semantic labels. This aims to provide the vehicle with a cognitive spatial reference system during movement, enabling it to maintain reliable contextual awareness and path planning even under certain observation conditions.

[0039] Figure 2 This is a schematic diagram of the overall framework of a satellite-and-ground collaborative scene semantic completion method provided in an embodiment of the present invention. Figure 2 The system initially consists of two branches: a ground branch and a satellite branch. In the ground branch, ground images are processed through a backbone network to obtain two-dimensional ground features. Simultaneously, the ground image undergoes depth estimation processing to obtain voxel candidates, all of which are non-empty voxels. The coordinates of the non-empty voxels are matched with the trained voxel query embeddings in the 3D voxel space to determine the first query vector. After determining the first query vector, it and the 2D ground features are input into the Cross-Attention (CA) layer. The 2D ground features are used to transform the first value vector V and the first key vector K. Combined with the first query vector Q, feature enhancement is performed. The output is then input into the Self-Attention (SA) layer to obtain the initial voxel features. After max pooling, the ground bird's-eye view features are obtained. The process involves parallel input of ground images into the FastCNN model, calculating a first set of real-valued masks (containing multiple static masks) based on the output. Then, based on the non-static categories and numerical truncation formula, static weights corresponding to each mask value are calculated (the initial weight matrix), and the final target weight matrix is ​​further calculated. On the satellite branch, feature extraction from satellite images is performed through the backbone network. The extracted features are then subjected to flat convolutional projection to obtain the first satellite bird's-eye view features. Simultaneously, while acquiring ground and satellite images, GPS reference poses of the vehicle during its movement are collected. Using the LM algorithm with a fixed damping factor, features from the first satellite bird's-eye view and the ground bird's-eye view are aligned based on the GPS reference poses. During alignment, the corresponding initialized GPS reference pose is acquired at each iteration. The corresponding residuals are calculated based on the GPS reference pose, the features of the first satellite bird's-eye view, and the features of the ground bird's-eye view. The pose increment is calculated based on the residual. The pose for the next iteration is calculated using the pose increment. The pose at the last iteration and the pose at the next iteration. That is, the optimal pose Furthermore, alignment is performed using the optimal pose, and the aligned features of the second satellite bird's-eye view are then used. and The input is fed into the cross-attention module, and the output is the enhanced satellite bird's-eye view features. Using the trained gating coefficients Will and Gating fusion is performed to combine the features of the fused third satellite bird's-eye view. Feature extraction was performed using 2D-unet to obtain the target satellite bird's-eye view features. Later dimensional ascension, and simultaneously Feature extraction was performed using 3D-unet to obtain the target voxel features. After the two features are fused by the fusion module, the top K voxel features are selected by feature filtering, and finally the prediction layer is used to predict and obtain the scene semantic completion result.

[0040] It is understood that, in the embodiments of the present invention, the scene semantic completion model includes a feature processing module, a three-dimensional voxel generation module, an LM geometric alignment module with a fixed damping factor, a fusion module, and a prediction module. The model acquires ground images, GPS reference poses, and satellite images of the surrounding environment under the current driving state of the vehicle; it also acquires ground images, GPS reference poses, and satellite images of the surrounding environment under the current driving state of the vehicle; it extracts voxels from the ground images using the three-dimensional voxel generation module to obtain non-empty voxels; and it performs feature extraction and segmentation on the ground images and satellite images in parallel based on the feature processing module and the non-empty voxels to obtain the target voxel features and semantic segmentation map corresponding to the ground images, as well as the satellite images. The first satellite bird's-eye view features are obtained. The LM geometric alignment module iteratively optimizes the GPS reference pose, semantic segmentation map, ground bird's-eye view features obtained by target voxel feature pooling, and the first satellite bird's-eye view features to obtain the optimal GPS pose. The optimal GPS pose is then used to spatially transform the first satellite bird's-eye view features to obtain the second satellite bird's-eye view features aligned with the ground bird's-eye view features. The second satellite bird's-eye view features and the ground bird's-eye view features are then subjected to cross-attention fusion and gating fusion in sequence through the fusion module to obtain the third satellite bird's-eye view features. The third satellite bird's-eye view features and the target voxel features are then predicted based on the prediction module to obtain the semantic completion results of the surrounding environment. In this process, satellite images are acquired simultaneously with ground images and GPS reference poses. Satellite images, as low-cost and readily available reference information, provide a global scene structure from a bird's-eye view, naturally complementing the ground view. By selecting non-empty voxels from the ground images, the ineffective calculation of empty voxels can be avoided. Satellite bird's-eye images provide global and unobstructed static scene layout information. After aligning them with the ground bird's-eye image features to the same coordinate system using the LM algorithm with a fixed damping factor, it can effectively compensate for the global context information that cannot be obtained from the ground view due to limited field of view and occlusion. Furthermore, fixing the damping factor during the alignment process can improve the efficiency of image alignment. Cross-attention fusion processing allows the local details of the ground and the features of the satellite bird's-eye image to complement each other. Further gated fusion processing combines the accurate spatial location provided by the aligned second satellite bird's-eye image features with the semantic features provided by the satellite bird's-eye image feature map output by cross-attention fusion. Finally, after prediction, the accuracy and completeness of semantic completion can be improved more accurately. In summary, the solution of the present invention can retain the advantages of geometric alignment while supplementing semantic details, and can efficiently and accurately obtain the semantic completion results of the surrounding environment of the vehicle during driving.

[0041] In some embodiments of the present invention, S102 can be implemented by the following steps: obtaining a disparity map corresponding to the ground image; calculating the depth value of each pixel point by point based on the disparity map; performing back projection based on the depth value to determine the horizontal and vertical coordinates corresponding to each pixel; constructing a depth point corresponding to each pixel based on the horizontal and vertical coordinates and the depth value; projecting all depth points into a preset three-dimensional voxel space, and if any voxel contains at least one depth point, then the arbitrary voxel is regarded as a non-empty voxel.

[0042] Specifically, a depth estimation model can be set up, and the ground image can be input into the depth estimation model to obtain the corresponding disparity map. The camera focal length, baseline distance, and disparity map at the time the ground image was captured are obtained. Then, using the disparity value of each pixel in the disparity map, combined with existing triangulation formulas, the corresponding depth value is calculated point by point. Combining existing back projection formulas and depth values, the x-coordinate and y-coordinate of each pixel are calculated. Based on the x-coordinate, y-coordinate, and depth value of each pixel, a depth point for each pixel is constructed.

[0043] Furthermore, a three-dimensional voxel space is established, with a voxel range of H×W×Z=128×128×16, and each voxel having a size of 0.4m×0.4m×0.4m, where H is the height dimension, W is the width dimension, and Z is the depth dimension. All depth points are projected into the established three-dimensional voxel space. In the projected three-dimensional voxel space, if any voxel contains at least one three-dimensional depth point, then that voxel is considered a non-empty voxel; otherwise, it is considered an empty voxel.

[0044] The depth estimation model can be a pre-trained MobileStereoNet model.

[0045] In some embodiments of the present invention, S103 can be implemented through the following steps: the feature processing module includes a satellite feature submodule, a ground feature submodule, and a segmentation submodule; the ground feature submodule includes a feature extraction layer, a query encoding layer, a cross-attention layer, and a self-attention layer. The feature extraction layer extracts features from the ground image to obtain two-dimensional ground features; the query encoding layer matches the coordinates of non-empty voxels with the trained voxel query embeddings in the three-dimensional voxel space, and uses the matched voxel query embeddings as the first query vector; the first query vector and the two-dimensional ground features are input to the cross-attention layer, and the first value vector and the first key vector are determined through the two-dimensional ground features, and the initial voxel features are determined based on the first key vector, the first value vector, and the first query vector; the self-attention layer enhances the initial voxel features to obtain target voxel features; the satellite feature submodule extracts features from the satellite image to obtain the first satellite bird's-eye view features; and the segmentation submodule segments the ground image to obtain a semantic segmentation map.

[0046] Specifically, the satellite feature submodule, ground feature submodule, and segmentation submodule are three parallel branches. The ground feature submodule processes the ground image, and its features extraction layer, query encoding layer, cross-attention layer, and self-attention layer are cascaded sequentially. The input of each layer is the output of the previous layer. The satellite feature submodule processes the input satellite image, and the segmentation submodule processes the input ground image. A set of learnable voxel query embeddings is initialized in 3D voxel space. After the scene semantic completion model is trained, the coordinates of non-empty voxels are matched with the trained voxel query embeddings in 3D voxel space to determine the first query vector.

[0047] Specifically, a cascaded architecture consisting of an EfficientNet-B7 backbone network and a Feature Pyramid Network (FPN) can be built into the feature extraction layer. This cascaded architecture is used to extract features from the ground image to obtain multi-scale two-dimensional ground features.

[0048] Specifically, a cascaded architecture consisting of ResNet-50 and a feature pyramid network can be built into the satellite feature submodule. This cascaded architecture is used to extract features from the ground image to obtain the first satellite bird's-eye view features.

[0049] Ground images can be segmented using a pre-trained FastSCNN network.

[0050] In some embodiments of the present invention, S106 can be implemented through the following steps: calculating the target weight matrix through the semantic segmentation map; during the first optimization iteration, substituting the coordinates of each pixel in the GPS reference pose and the first satellite bird's-eye view feature into the spatial transformation formula to obtain the sampling coordinates of the satellite image; performing bilinear interpolation sampling on the sampling coordinates to obtain the first satellite bird's-eye view feature temporarily aligned with the ground bird's-eye view feature in the same coordinate system; performing point-by-point calculation on the pixels at the same position of the ground bird's-eye view feature and the temporarily aligned first satellite bird's-eye view feature to obtain the first target residual vector; based on The pose increment is calculated using the target weight matrix, a fixed damping factor, and the first target residual vector. The pose increment is summed with the GPS reference pose to obtain the new pose after the first iteration. The above steps are repeated based on the new pose and the features of the first satellite bird's-eye view to obtain the new target residual vector. The target pose is determined based on the weighted sum of the new target residual vector and the first target residual vector in the new pose and the GPS reference pose. The target pose is used as the GPS reference pose for the second optimization iteration, and the above optimization steps are repeated using the same fixed damping factor until the number of iterations is reached to obtain the optimal GPS pose.

[0051] Specifically, after projecting the semantic segmentation graph, a corresponding set of real-valued masks is obtained, and the corresponding target weight matrix is ​​calculated based on the set of real-valued masks. i The formula for calculating the residual vector of pixels is as follows:

[0052] In the formula, For the first i The residual vector of pixels, For the first i The coordinates of a pixel. Features of the first satellite bird's-eye view, temporarily aligned. T p This is for subsequent space transformation operations.

[0053] Furthermore, the residual vectors of all pixels are concatenated sequentially to obtain the first target residual vector, which has a dimension of . Using the chain rule, the partial derivative of the residual vector of each pixel with respect to the GPS reference pose is calculated. The pose is decomposed into the product of the image gradient (i.e., the partial derivative of the residual with respect to the sampled coordinates) and the partial derivative of the sampled coordinates with respect to the GPS reference pose, yielding the Jacobian matrix. The Jacobian matrix, target weight matrix, first target residual vector, and damping factor are substituted into the existing weighted normal equation to calculate the pose increment. The pose increment is then summed with the GPS reference pose to obtain the pose after the first iteration, i.e., the new pose. Furthermore, at the start of the second iteration for optimization, the spatial transformation formula of the first bird's-eye view feature space is re-performed using the new pose to calculate a new target residual vector. Then, following the standard LM algorithm, the weighted sums of the new target residual vector and the first target residual vector are compared. If the weighted sum of the new target residual vector is not greater than the weighted sum of the first target residual vector, the new pose is used as the GPS reference pose for the second iteration, and the same operation as in the first iteration is performed for the second iteration until the iteration condition is met and the optimal GPS pose is obtained; otherwise, the GPS reference pose from the first iteration is used as the GPS reference pose for the second iteration.

[0054] In this process, the damping factor remains constant throughout the first to the last optimization process, using the same value. The number of iterations is a hyperparameter that is pre-set based on experience, and can be 10.

[0055] In some embodiments of the present invention, the sampling coordinates of the satellite image are obtained by substituting the GPS reference pose and the coordinates of each pixel in the first satellite bird's-eye view feature into the spatial transformation formula. This can be achieved through the following steps: substituting the GPS reference pose and the coordinates of each pixel into the spatial transformation formula to obtain the sampling coordinates; the spatial transformation formula is as follows: ; ; In the above formula, For the floor function, The x-coordinate of the sampling coordinates The ordinate of the sampling coordinates. The yaw angle is the GPS reference pose. x The x-coordinate of the GPS reference pose. y The ordinate is the GPS reference pose. u The x-coordinate of each pixel, v The ordinate of each pixel.

[0056] Figure 3 This is a schematic diagram of the internal processing structure of an LM geometry alignment module with a fixed damping factor, provided in an embodiment of the present invention. Figure 3 In the LM geometry alignment module, the inputs are the initial satellite bird's-eye view features (i.e., the second satellite bird's-eye view features), the GPS coarse pose (i.e., the GPS reference pose), and the ground bird's-eye view reference features (i.e., the ground bird's-eye view features). The output is the second satellite bird's-eye view features aligned with the bit-aligned ground bird's-eye view features. (i.e. During the optimization process using the LM algorithm, a total of 10 iterations are performed. Spatial transformation is performed using the aforementioned spatial transformation formula. The target weight matrix is ​​calculated using the aforementioned residual calculation formula. The input of FastSCNN is processed to obtain a semantic segmentation map, which is then projected onto a satellite image and subjected to a series of operations to obtain a static mask. The weighted normal equation in the image is a known formula, where J is the Jacobian matrix and W is the target weight. Let I be a fixed damping factor, I be the identity matrix, r be the first objective residual vector, and T be the transpose. Solving the left-hand side of the weighted normal equation using Cholesky's algorithm yields the following result. , Let L be the new pose in each optimization process, and L be the damped Hessian matrix. The Cholesky formula is known and will not be elaborated upon. After solving the Jacobian matrix and the normal equation, the pose increment is obtained. Combined with the current initial pose P, the updated new pose P is calculated. After multiple optimizations, the final optimal pose is obtained. .

[0057] In some embodiments of the present invention, the calculation of the target weight matrix from the semantic segmentation map can be achieved through the following steps: projecting the semantic segmentation map onto a satellite image to obtain a corresponding first real-valued mask set; setting the corresponding real-valued masks in the first real-valued mask set to 0 according to the non-static categories on the semantic segmentation map to obtain a second real-valued mask set; substituting the real-valued masks in the second real-valued mask set into the numerical truncation formula to obtain a third real-valued mask set; using the third real-valued mask set as the initial weight matrix and substituting the initial weight matrix into the weight adjustment formula to obtain the target weight matrix; The numerical truncation formula is as follows: ; In the above formula, For the second set of real number masks, the first i A real number mask, The third real number mask set i A real number mask; It is a numerical truncation function; ; In the above formula, The first weight of the target weight matrix i Each weight value The power exponent, This is the global scaling factor.

[0058] For example, when calculating the numerical truncation formula, y = clamp ( x,min ),when x > min hour y = x ;when x <= min hour y = min .

[0059] The semantic segmentation graph categories include sky, buildings, trees, ground / roads, pedestrians, and cars, while the non-static categories are all categories except ground / roads / driving areas, such as sky, buildings, vegetation, and vehicles.

[0060] In this process, the semantic segmentation map is projected onto the satellite image to obtain a first set of real number masks with values ​​ranging from [0,1]. The mask value represents the confidence that the object corresponding to each pixel in the satellite image belongs to the ground static category.

[0061] Specifically, when the semantic segmentation map determines that a certain pixel position belongs to a non-static category, the corresponding mask value in the first real number mask set is set to 0.

[0062] in, It can be 1.0. It can be 1.0.

[0063] In some embodiments of the present invention, S105 can be implemented through the following steps: The fusion module includes a cross-attention submodule and a gated fusion submodule. The second satellite bird's-eye view features and the ground bird's-eye view features are input to the cross-attention submodule. The second satellite bird's-eye view features are linearly transformed to obtain a second query vector. The ground bird's-eye view features are linearly transformed to obtain a second value vector and a second key vector. The enhanced satellite bird's-eye view features are obtained based on the second query vector, the second value vector, and the second key vector. The enhanced satellite bird's-eye view features and the second satellite bird's-eye view features are gatedly fused using the gated fusion formula in the gated fusion submodule to obtain a third satellite bird's-eye view feature. The gated fusion formula is as follows:

[0064] In the above formula, Features of the third satellite aerial view, The gating coefficient, Features of the second satellite aerial view, Features of the enhanced satellite bird's-eye view.

[0065] Specifically, the ground-based bird's-eye view features and the second satellite bird's-eye view features are flattened. The two feature dimensions [H, W, C] are flattened to [N, C], where N = H × W, representing the number of spatial locations, i.e., the number of tokens. The flattened bird's-eye view features are then input into a cross-attention submodule. The second query vector, second value vector, and second key vector are processed through a multi-head cross-attention mechanism, residual connections and normalization, a feedforward network, and residual connections and normalization to obtain the enhanced satellite bird's-eye view features. These enhanced satellite bird's-eye view features and the second satellite bird's-eye view features have the same spatial dimensions H × W and the same number of channels C. This design allows each geometrically aligned satellite feature point to actively query the texture and detail information of its corresponding location in the ground view, thereby filling the geometric framework with accurate semantic labels. For example, when the geometric base stream identifies a large cubic region, the semantic refinement stream can distinguish whether the object is a truck or another vehicle based on the ground texture.

[0066] The cross-attention module adopts a standard Transformer block structure, including a multi-head cross-attention layer with 8 attention heads, residual connections, layer normalization, and a feedforward neural network. The embedding dimension C=128, and the shapes of the ground bird's-eye view features and the second satellite bird's-eye view features are both 128×128×128.

[0067] Specifically, during the training phase, a learnable fusion parameter is introduced into the cross-attention submodule. a ∈[0,1]. Parameter a Learnable scalar The Sigmoid function is generated as follows: ,and It can be initialized to 5.0 to facilitate the initial training phase. a ≈1.0. This design ensures that the value is automatically reduced via backpropagation only when the enhanced satellite view features provide a better feature representation than the second satellite view features. a The value; any reduction a However, attempts that lead to performance degradation will increase the total loss, forcing the optimizer to be scaled up again. a This allows for the protection of the features of the second satellite bird's-eye view.

[0068] Figure 4 This is a schematic diagram of the structure of a semantic refinement flow provided in an embodiment of the present invention. Figure 4 In this process, the cross-attention module of the standard Transformer block structure is used to process the input ground bird's-eye view features. (i.e., ground-view derived BEV features) and second satellite bird's-eye view features (The aforementioned LM-aligned satellite BEV features) are processed to... As a geometric base, such as a car, parking area, tree trunk, or road, Semantic reference cues can include vegetation, buildings, curbs, and obstacles. Before input, the two features are flattened. Before flattening, they are [H, W, C], and after flattening, they become [N, C], where N = H × W, representing the number of spatial locations, i.e., the number of tokens. After inputting the flattened features, the process is... Determine the second query vector, through After determining the second value vector and the second key vector, a multi-head cross-attention layer with eight attention heads, residual connections, layer normalization, and a feedforward neural network were used to process the data, resulting in the fused semantically refined features. (The fused semantic refinement features are also the enhanced satellite bird's-eye view features mentioned above) The dimensions are H×W×C. After this processing, entities with clear semantic definitions can be identified, such as trucks, traffic signs, other vehicles, and poles.

[0069] Figure 5 This is a schematic diagram of a learnable weighted fusion structure provided in an embodiment of the present invention. Figure 5 In this process, a learnable fusion parameter is introduced. a ∈[0,1]. Parameter aLearnable scalar The Sigmoid function is generated as follows: ,and It can be initialized to 5.0 to facilitate the initial training phase. a ≈1.0. After training, the value obtained from geometric protection will be... and Multiplication, resulting in semantically refined... and Multiply the two results, then sum them to obtain the fused BEV features. (That is, the aforementioned third satellite bird's-eye view features) ).

[0070] In some embodiments of the present invention, S106 can be implemented through the following steps: the prediction module includes a first feature submodule, a second feature submodule, a fusion submodule, a feature filtering submodule, and a prediction submodule. The first feature submodule is used to extract features from the target voxel features to obtain enhanced target voxel features; the second feature submodule is used to extract features from the third satellite bird's-eye view features to obtain target satellite bird's-eye view features; the target satellite bird's-eye view features are upgraded in dimensionality, and the fusion submodule is used to fuse the upgraded target satellite bird's-eye view features and the enhanced target voxel features to obtain fused voxel features; the feature filtering submodule is used to filter the fused voxel features, and the prediction module is used to predict the filtered voxel features to obtain scene semantic completion results.

[0071] The first feature submodule can incorporate a 3D-UNET network for feature extraction, and the second feature submodule can incorporate a 2D-UNET network for feature extraction.

[0072] In this process, the features of the target satellite bird's-eye view are extended to three-dimensional space through feature mapping or projection transformation to obtain an upgraded target satellite bird's-eye view. Then, the upgraded target satellite bird's-eye view and the enhanced target voxel features can be fused in the form of feature stitching to obtain fused voxel features.

[0073] In the feature selection submodule, the fused voxel features are input into a fully connected layer for processing. The output of the fully connected layer is then mapped using a sigmoid activation function to obtain a score for each fused voxel feature. This score represents the probability that the voxel contains a real object. Next, all voxel features are sorted according to their scores, and only the top K voxel features with the highest scores are retained. The value of K is set according to the actual application.

[0074] The prediction layer can be a fully connected layer or a 1D convolutional layer. The first K voxel features are input into the prediction layer to obtain the final scene semantic completion result.

[0075] In this embodiment of the invention, when calculating the loss function during the training phase of the scene semantic completion model, Two convolutional layers are set up to serve as two independent bird's-eye view segmentation maps. The enhanced satellite bird's-eye view features and the second satellite bird's-eye view features are respectively input into the two auxiliary convolutional layers set during training to obtain their respective class probability maps, i.e., class score maps. The value of each pixel position on each channel is the raw score of the pixel belonging to the corresponding class. The 3D voxel ground truth values ​​are obtained, which contain the true class labels (0 represents empty, 1-19 represent different objects / scenes). The 3D voxel ground truth values ​​are max-pooled along the height axis to obtain the bird's-eye view ground truth values. The ground truth of the bird's-eye view is used as the label. A loss function, such as the cross-entropy loss function, is set. The category probability map and label corresponding to the enhanced satellite bird's-eye view features are substituted into the loss function to calculate the first loss. The category probability map corresponding to the second satellite bird's-eye view features is substituted into the set loss function to calculate the second loss.

[0076] When calculating the scene class affinity loss, such as when calculating the scene class affinity loss Lgeo_scal corresponding to the features of the second satellite bird's-eye view, the loss function is as follows: ; In the above formula, , , ; in, Let c be the number of voxels whose true class is c and whose predicted class is c. Let c be the number of voxels whose true class is not c but whose predicted class is c. Let c be the number of voxels whose true class is c but whose predicted class is not c. Let c be the number of voxels whose true class is not c but whose predicted class is also not c. For each class c, find the locations of all voxels with true class c in the 3D voxel ground truth map, and find the locations of all voxels with true class c in the predicted class probability map, and then count them. , , and Then, by combining the above formulas, the final Lgeo_scal is calculated.

[0077] Meanwhile, the scene affinity loss Lsem_scal corresponding to the enhanced satellite bird's-eye view features is calculated in the same way as Lgeo_scal.

[0078] Furthermore, a loss function, such as a weighted cross-entropy loss function, is set. The semantic category prediction and the corresponding true semantic label for each voxel in the semantic completion result are substituted into the set loss function to calculate the weighted cross-entropy loss for the 3D voxel. .

[0079] Furthermore, all the aforementioned losses, such as the first loss, the second loss, the scene class affinity loss, and the weighted cross-entropy loss, are substituted into the total loss calculation formula. The calculated total loss is then used to train the scene semantic completion model, resulting in the trained scene semantic completion model. The total loss calculation formula is as follows: ; In the above formula, =1.0.

[0080] The optimizer used AdamW with an initial learning rate of 4e-4 and a training period of 30 epochs.

[0081] The true value of the three-dimensional voxel can be obtained by combining lidar point cloud fusion with fine manual annotation.

[0082] Figure 6 This is a schematic diagram of a dual bird's-eye view supervision structure provided in an embodiment of the present invention. Figure 6 In the process, the enhanced satellite bird's-eye view features and the second satellite bird's-eye view features are respectively input into the two segmentation heads, BEV heads. geo and BEV head sem In the diagram, the two segmentation heads are two auxiliary convolutional layers set during training, each producing its corresponding classification score and BEV prediction. geo and BEV forecast sem (i.e., category score map), the obtained 3D voxel truth values Max pooling is performed along the height axis to obtain the shared BEV truth value BEV. gt (i.e., the ground truth of the bird's-eye view), using the ground truth as the label. A loss function is set, and the category probability map and labels corresponding to the enhanced satellite bird's-eye view features are substituted into the loss function to calculate... Substitute the category probability map corresponding to the features of the second satellite bird's-eye view into the set loss function to calculate the... Weighted cross-entropy loss corresponding to 3D voxels As mentioned above, Lgeo_scal and Lsem_scal eventually sum all the losses to obtain the total loss.

[0083] Figure 7 A comparison chart of qualitative results on the SemanticKITTI dataset with baseline methods, provided for embodiments of the invention. Figure 7The selected images are: Seq08 / 000002 (frame 2 of video sequence 08 in the SemanticKITTI dataset), Seq08 / 000078 (frame 78 of video sequence 08 in the SemanticKITTI dataset), Seq08 / 000110 (frame 110 of video sequence 08 in the SemanticKITTI dataset), and Seq08 / 000156 (frame 156 of video sequence 08 in the SemanticKITTI dataset). A baseline model was set, using the core method proposed in the paper "SGFormer: Satellite-Ground Fusion for 3D Semantic Scene Completion" as the baseline model. The results of manually annotated Ground Truth, the baseline model output, and the proposed method Our Model were compared on the selected images, and the changes in the proposed method's output compared to the baseline model output were presented. Blue represents cars in the semantic completion results, purple represents sidewalks, dark green represents vegetation, dark purple represents roads, yellow represents buildings, light green represents terrain, and red represents traffic signs. In Figure 7, bright green (Our Model corrects) indicates the baseline model is incorrect, but the proposed model predicts correctly; dark red (Our Model loses) indicates the baseline model predicts correctly, but the proposed model predicts incorrectly; very light gray (both correct) indicates both models predict correctly; and dark gray (both wrong) indicates both models predict incorrectly.

[0084] Furthermore, the experimental results from several frames show that, by comparing the prediction results of the baseline model and the model proposed in this invention, and combining this with the Change difference map on the far right, it can be seen that the baseline model often exhibits ambiguity or misjudgment at the intersections of road edges and building outlines, specifically in the red area of ​​the Change map. In this area, the model proposed in this invention has fewer misjudgments, can more accurately fit the object boundaries, and corrects a large number of incorrect predictions from the baseline model, which is represented by a large bright green area in the Change map, i.e., Our Model corrects. Especially when dealing with small targets such as distant vehicles and pole-shaped objects, the model proposed in this invention has fewer missed detections and stronger overall semantic consistency, making the segmentation results closer to the ground truth. The specific improvements of the Delta+0.1269, Delta+0.0275, Delta+0.0382, and Delta+0.0358 models compared to the baseline model in the Intersection over Union (IoU) evaluation metric are shown.

[0085] In an embodiment of the present invention, Table 1 shows the training results on the SemanticKITTI dataset, as follows: Table 1 Training results on the SemanticKITTI dataset

[0086] In Table 1, the first four rows represent the average crossover ratio (CWR), CWR, precision, and recall for the entire training set. The fifth row and subsequent rows represent each specific semantic category defined in the training dataset, with each row corresponding to the CWR of that specific category. The average CWR is 0.1584, indicating that the proposed method possesses reliable scene understanding capabilities. Furthermore, the scores for each category after the fifth row exhibit a highly reasonable gradient distribution, with high scores for large background areas, medium-sized objects in the center, and low scores for extremely small targets. This aligns with the objective physical laws of semantic segmentation, fully demonstrating that the proposed method is logically rigorous in feature extraction, without overfitting or serious logical errors, exhibiting robust overall performance and high practical value.

[0087] The present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements a method for semantic completion of a scene based on satellite and ground collaboration.

[0088] Reference Figure 8 The diagram shows a structural schematic of an electronic device according to an embodiment of the present invention. The specific embodiments of the present invention do not limit the specific implementation of the electronic device.

[0089] like Figure 8 As shown, the electronic device may include: a processor 502, a communications interface 504, a memory 506, and a communications bus 508.

[0090] in: The processor 502, communication interface 504, and memory 506 communicate with each other via communication bus 508.

[0091] Communication interface 504 is used to communicate with other electronic devices or servers.

[0092] The processor 502 is used to execute program 510, specifically the relevant steps in the above method embodiments.

[0093] Specifically, program 510 may include program code that includes computer operation instructions.

[0094] Processor 502 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The smart device may include one or more processors of the same type, such as one or more CPUs; or it may include processors of different types, such as one or more CPUs and one or more ASICs.

[0095] Memory 506 is used to store program 510. Memory 506 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0096] Specifically, program 510 can be used to cause processor 502 to perform the operations corresponding to the methods described in the above method embodiments.

[0097] The specific implementation of each step in program 510 can be found in the corresponding descriptions of the steps and units in the above method embodiments, and will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0098] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of the present invention can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present invention.

[0099] The methods described above according to embodiments of the present invention can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code originally stored on a remote recording medium or a non-transitory machine-readable medium and subsequently stored on a local recording medium, downloaded via a network. Thus, the methods described herein can be processed by software stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0100] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of the embodiments of the present invention.

[0101] The above embodiments are only used to illustrate the embodiments of the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of the present invention. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of the present invention, and the patent protection scope of the embodiments of the present invention should be defined by the claims.

Claims

1. A method for semantic completion of scenes based on satellite and ground collaboration, characterized in that, The method is implemented through a scene semantic completion model, which includes a feature processing module, a 3D voxel generation module, an LM geometric alignment module with a fixed damping factor, a fusion module, and a prediction module. The method includes: Acquire ground images, GPS reference pose, and satellite images of the surrounding environment while the vehicle is currently in motion; The ground image is voxel extracted using a 3D voxel generation module to obtain non-empty voxels; Based on the feature processing module and non-empty voxels, feature extraction and segmentation are performed on ground images and satellite images in parallel to obtain target voxel features and semantic segmentation maps corresponding to ground images, as well as first satellite bird's-eye view features corresponding to satellite images. The LM geometric alignment module is used to iteratively optimize the GPS reference pose, semantic segmentation map, ground bird's-eye view features obtained by target voxel feature pooling, and first satellite bird's-eye view features to obtain the optimal GPS pose. The optimal GPS pose is then used to spatially transform the first satellite bird's-eye view features to obtain the second satellite bird's-eye view features aligned with the ground bird's-eye view features. The third satellite bird's-eye view features are obtained by sequentially performing cross-attention fusion and gating fusion on the features of the second satellite bird's-eye view and the features of the ground bird's-eye view through the fusion module. Based on the prediction module, the features of the third satellite bird's-eye view and the target voxel features are predicted to obtain the semantic completion results of the surrounding environment.

2. The method according to claim 1, characterized in that, The step of extracting voxels from the ground image using a three-dimensional voxel generation module to obtain non-empty voxels includes: Obtain the disparity map corresponding to the ground image; and calculate the depth value of each pixel point by point based on the disparity map; Back projection is performed based on the depth value to determine the x and y coordinates of each pixel; and the depth point corresponding to each pixel is constructed based on the x and y coordinates and the depth value. Project all depth points into a preset three-dimensional voxel space. If any voxel contains at least one depth point, then that voxel is considered a non-empty voxel.

3. The method according to claim 1, characterized in that, The feature processing module includes a satellite feature submodule, a ground feature submodule, and a segmentation submodule; the ground feature submodule includes a feature extraction layer, a query encoding layer, a cross-attention layer, and a self-attention layer; The feature processing module and non-empty voxels perform feature extraction and segmentation on ground and satellite images in parallel to obtain target voxel features and semantic segmentation maps corresponding to the ground images, and first satellite bird's-eye view features corresponding to the satellite images, including: The ground image is used to extract features through a feature extraction layer to obtain two-dimensional ground features; The coordinates of non-empty voxels are matched with the trained voxel query embeddings in the three-dimensional voxel space by the query encoding layer, and the matched voxel query embeddings are used as the first query vector. The first query vector and the two-dimensional ground features are input into the cross-attention layer. The first value vector and the first key vector are determined through the two-dimensional ground features. The initial voxel features are determined based on the first key vector, the first value vector, and the first query vector. The initial voxel features are enhanced using a self-attention layer to obtain the target voxel features; The satellite feature submodule extracts features from the satellite image to obtain the first satellite bird's-eye view features; and the segmentation submodule segments the ground image to obtain a semantic segmentation map.

4. The method according to claim 1, characterized in that, The process of using the LM geometric alignment module to iteratively optimize the GPS reference pose, semantic segmentation map, ground bird's-eye view features obtained based on target voxel feature pooling, and first satellite bird's-eye view features to obtain the optimal GPS pose includes: The target weight matrix is ​​calculated using the semantic segmentation map. In the first optimization iteration, the coordinates of each pixel in the GPS reference pose and the first satellite bird's-eye view features are substituted into the spatial transformation formula to obtain the sampling coordinates of the satellite image. Bilinear interpolation sampling is performed on the sampling coordinates to obtain the first satellite bird's-eye view feature temporarily aligned with the ground bird's-eye view feature in the same coordinate system; The ground bird's-eye view features and the temporarily aligned first satellite bird's-eye view features are used to perform point-by-point calculations on the pixels at the same location to obtain the first target residual vector; The pose increment is calculated based on the target weight matrix, a fixed damping factor, and the first target residual vector; the pose increment is summed with the GPS reference pose to obtain the new pose after the first iteration. The above steps are repeated based on the new pose and the features of the first satellite bird's-eye view to obtain the new target residual vector; and the target pose is determined based on the weighted sum of the new target residual vector and the first target residual vector in the new pose and the GPS reference pose. The target pose is used as the GPS reference pose for the second optimization iteration, and the above optimization steps are repeated using the same fixed damping factor until the number of iterations is reached to obtain the optimal GPS pose.

5. The method according to claim 4, characterized in that, The step of substituting the coordinates of each pixel in the GPS reference pose and the features of the first satellite bird's-eye view into the spatial transformation formula to obtain the sampling coordinates of the satellite image includes: Substitute the GPS reference pose and the coordinates of each pixel into the spatial transformation formula to obtain the sampled coordinates; The space transformation formula is as follows: ； ； In the above formula, For the floor function, The x-coordinate of the sampling coordinates The vertical coordinate of the sampling coordinates. The yaw angle is the GPS reference pose. x The x-coordinate of the GPS reference pose. y The ordinate is the GPS reference pose. u The x-coordinate of each pixel, v The ordinate of each pixel.

6. The method according to claim 4, characterized in that, The step of calculating the target weight matrix through the semantic segmentation graph includes: The semantic segmentation map is projected onto the satellite image to obtain the corresponding first real number mask set; based on the non-static categories on the semantic segmentation map, the corresponding real number masks in the first real number mask set are set to 0 to obtain the second real number mask set. Substituting the real number masks from the second set of real number masks into the numerical truncation formula yields the third set of real number masks. The third set of real-valued masks is used as the initial weight matrix, and the initial weight matrix is ​​substituted into the weight adjustment formula to obtain the target weight matrix; The numerical truncation formula is as follows: ； In the above formula, For the second set of real number masks, the first i A real number mask, The third real number mask set i A real number mask; It is a numerical truncation function; The weight adjustment formula is as follows: ； In the above formula, The first weight of the target weight matrix i Each weight value The power exponent, This is the global scaling factor.

7. The method according to claim 1, characterized in that, The fusion module includes a cross-attention submodule and a gated fusion submodule; The third satellite bird's-eye view features are obtained by sequentially performing cross-attention fusion and gated fusion on the features of the second satellite bird's-eye view and the ground bird's-eye view through the fusion module, including: The second satellite bird's-eye view features and the ground bird's-eye view features are input into the cross-attention submodule. The second satellite bird's-eye view features are linearly transformed to obtain the second query vector. The ground bird's-eye view features are linearly transformed to obtain the second value vector and the second key vector. The enhanced satellite bird's-eye view features are obtained based on the second query vector, the second value vector and the second key vector. The enhanced satellite bird's-eye view features and the second satellite bird's-eye view features are gated and fused using the gated fusion formula in the gated fusion submodule to obtain the third satellite bird's-eye view features; the gated fusion formula is as follows: In the above formula, Features of the third satellite aerial view, The gating coefficient, Features of the second satellite aerial view, Features of the enhanced satellite bird's-eye view.

8. The method according to claim 3, characterized in that, The prediction module includes a first feature submodule, a second feature submodule, a fusion submodule, a feature filtering submodule, and a prediction submodule; The prediction module predicts the features of the third satellite bird's-eye view and the target voxel features to obtain the semantic completion results of the surrounding environment, including... The first feature submodule is used to extract features from the target voxel features to obtain the enhanced target voxel features; The second feature submodule is used to extract features from the third satellite bird's-eye view features to obtain the target satellite bird's-eye view features. The target satellite bird's-eye view features are up-dimensional, and the up-dimensional target satellite bird's-eye view features and the enhanced target voxel features are fused using the fusion submodule to obtain fused voxel features; The feature filtering submodule filters the fused voxel features, and the prediction module predicts the filtered voxel features to obtain the scene semantic completion result.

9. An electronic device, characterized in that, The electronic device includes a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; The memory is used to store at least one executable instruction that causes the processor to perform the operation corresponding to the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method according to any one of claims 1-8.

Images