Scene and sensor data generation method based on remote sensing image and electronic equipment

Abstract

The invention discloses a scene and sensor data generation method based on a remote sensing image and electronic equipment, and relates to the technical field of data generation, and the data generation method comprises the steps: taking remote sensing data collected by an unmanned aerial vehicle or a satellite as input to obtain multi-source modal data, and based on the derivation, fusion, reconstruction and generation of the multi-source modal data, obtaining the multi-source modal data; therefore, the consistent time sequence track image and laser point cloud data can be obtained, and the three-dimensional space information and visual features of the surrounding environment of the vehicle can be obtained. According to the scene and sensor data generation method based on the remote sensing image, a small amount of unmanned aerial vehicle aerial photography data or short-period high-resolution optical remote sensing satellite data can be utilized to generate a time sequence track image and laser point cloud data, the data acquisition difficulty in a complex or extreme traffic scene can be reduced, and the acquisition cost can be effectively reduced; the comprehensive coverage of different terrains, road structures and environmental conditions is realized, and the problem that real vehicle acquisition is limited by routes, traffic flows and safety factors is solved.

Description

Scene and sensor data generation methods and electronic devices based on remote sensing imagery Technical Field

[0001] This invention relates to the field of data generation technology, and more specifically, to a method and electronic device for generating scene and sensor data based on remote sensing imagery. Background Technology

[0002] With the rapid development of autonomous driving technology, vehicles' perception and decision-making regarding the environment are highly dependent on the support of multimodal data. The realization of high-precision autonomous driving systems requires the collection of core sensor data, including LiDAR point clouds and camera images, to obtain three-dimensional spatial information and visual features of the vehicle's surrounding environment.

[0003] Existing data collection methods mainly rely on real-vehicle data collection, which usually requires a large number of vehicles, sensor devices and professional personnel. This is not only costly, but also limited by driving routes and traffic conditions, making it difficult to cover complex or extreme traffic scenarios. As a result, the scope of data acquisition is limited, which restricts the performance improvement of autonomous driving systems in diverse environments. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a method and electronic device for generating scene and sensor data based on remote sensing imagery, so as to increase the coverage and diversity of the collected data while reducing the data acquisition cost.

[0005] To achieve the above objectives, this invention provides a method for generating scene and sensor data based on remote sensing imagery, comprising: Step S1: acquiring several single high-resolution images using UAV aerial photography data or short-period high-resolution optical remote sensing satellite data; Step S2: acquiring multi-source modal data based on the single high-resolution images, wherein the multi-source modal data includes embedded features of multi-view images, virtual vehicle trajectories within the coverage area of ​​the remote sensing data, and scene text descriptions; Step S3: obtaining multi-layer feature self-attention for the corresponding modality by encoding different modal data, and performing different... The mutual attention features of the modal data are fused to obtain the updated feature representation; Step S4: The new feature representation under this pose is processed by the DPT model to obtain a dense depth map and a high-precision 3D point cloud with GS flow embedding features; Step S5: Based on the dense depth map and high-precision 3D point cloud in Step S4, a diffusion model is used to generate multi-view images and high-quality 3D assets observed by the same sensor configuration under the next pose; Step S6: The multi-view images and high-quality 3D assets in Step 5 are processed by a recurrent neural network to obtain consistent temporal trajectory images and laser point cloud data.

[0006] Furthermore, in step S2, the acquisition of multi-source modal data based on a single high-resolution image includes: step S21: generating multi-view images based on a single high-resolution image; step S22: generating embedded features of scene text descriptions for each view based on the multi-view images; step S23: generating virtual driving trajectories within the coverage area for each view based on the multi-view images.

[0007] Further, in step S21, the generation of multi-view images based on a single high-resolution image includes: step S211: using a deep learning model to predict the depth information of each pixel and generate a depth map; step S212: reconstructing the sparse three-dimensional geometry of the image scene; step S213: using the reconstructed three-dimensional geometry to simulate a camera and re-render the scene from any new perspective, simulating a new view of the scene from different perspectives.

[0008] Further, in step S22, the embedding features for generating scene text descriptions for each view based on multi-view images include: Step S221: confirming the backbone network for feature extraction and the scene description model used; Step S222: inputting a single high-resolution remote sensing image and its new view into the scene description model used, so that the backbone network outputs a high-dimensional feature tensor rich in semantic information; Step S223: using a self-attention mechanism to integrate the visual features extracted in step 222 into a structured representation rich in contextual semantic information; Step S224: using a decoder to gradually and autoregressively convert the contextually rich, fixed-size visual representation in step S223 into a word sequence, forming the final image scene text description.

[0009] Further, in step S23, generating a virtual driving trajectory within the coverage area of ​​each view based on the multi-view image includes: Step S231: Creating a realistic and accurate digital environment using remote sensing data; Step S232: Converting the processed remote sensing data into a vector road network model that can be understood and simulated by a computer; Step S233: Adding simulation rules to the generated vector road network, the simulation rules including road grade, number of lanes, speed limit, traffic rules, and gradient; Step S234: Configuring simulation parameters for the preset simulation model using a genetic algorithm and parameter calibration tool, and performing path planning based on the shortest time and shortest path, the simulation parameters including traffic demand, vehicle characteristics, and driving behavior patterns; Step S235: Outputting trajectory data and loading the trajectory file using a professional visualization analysis tool to overlay it onto the remote sensing image to obtain the virtual driving trajectory.

[0010] Further, in step S3, the step of obtaining multi-layer feature self-attention of the corresponding modality by encoding different modal data, and performing mutual attention feature fusion of different modal data based on multi-layer feature self-attention to obtain the updated feature representation includes: Step S31: Encoding multi-source modal data obtained based on a single high-resolution image, wherein the multi-view image adopts frame encoding, the virtual driving trajectory within the remote sensing data coverage area adopts pose encoding, and the embedded features of the scene text description adopt CLIP encoding; Step S32: Performing information fusion of multi-layer feature self-attention of different modal data based on a preset mutual attention model to obtain the updated feature representation.

[0011] Further, in step S4, the step of obtaining a dense depth map and a high-precision 3D point cloud with GS flow embedding features by processing the new feature representation under the pose using the DPT model includes: Step S41: Using the DPT model to read the new feature representation in step S3, and obtaining a dense depth map through encoding fusion; Step S42: Converting each pixel and its depth value in the optimized depth map into a 3D point in world coordinates through back projection to obtain an initial dense point cloud; Step S43: Using a point cloud optimization algorithm to denoise and simplify the initial dense point cloud to obtain the final dense point cloud; Step S44: Inferring the complete geometric and semantic properties of the 3D space based on the Transformer decoder, thereby converting the 3D points into data with optimizable features and a high-dimensional, learnable feature vector; this feature vector is the GS flow embedding feature.

[0012] Further, in step S5, the step of generating multi-view images and high-quality 3D assets observed by the same sensor configuration in the next pose based on the dense depth map and high-precision 3D point cloud in step S4 through a diffusion model includes: Step S51: Generating a 3D Gaussian model based on the dense depth map and the high-precision 3D point cloud with GS flow embedding features in step S4; Step S52: Performing differentiable rendering using the 3D Gaussian model in the target pose to obtain the RGB image and depth information rendered by the 3D Gaussian model in that viewpoint; Step S53: Preparing the diffusion model. The input conditions are processed and control conditions are applied to obtain a noisy latent representation; Step S54: The noisy latent representation is refined and upsampled to obtain a refined image; Step S55: Steps S52, S53 and S54 are repeated multiple times to obtain refined images from different perspectives; Step S56: The 3D Gaussian model is optimized again using the newly refined images from multiple different perspectives until the 3D Gaussian model and the diffusion model converge to an optimal state, resulting in highly consistent multi-view images and detailed, high-quality 3D assets.

[0013] Further, in step S6, the process of using a recurrent neural network to process the multi-view images and 3D assets in step 5 to obtain consistent temporal trajectory images and laser point cloud data includes: Step S61: Inputting images from different perspectives at a certain time point into a pre-trained CNN, obtaining the output of the last convolutional layer or global average pooling layer of the CNN, and obtaining a fixed-length feature vector; Step S62: Using an attention mechanism to weightedly fuse the features from different perspectives at a certain time point, obtaining a single, highly representative joint feature vector that can represent that time point, repeating the above operation to process all time points, and obtaining a time series; Step S63: Inputting the time series obtained in step S62 into a preset RNN unit, so that the RNN updates the hidden state A at each time point in sequence, and obtaining a hidden state A sequence; Step S64: Abstracting the features in the hidden space of the RNN through neural network layers. The trajectory prediction sequence is obtained by mapping to the real output space, and a time-series trajectory image based on the trajectory prediction sequence is output through a standard scientific plotting library; Step S65: The current sensor pose and hidden state in each time step are processed by a preset RNN unit, and a new hidden state B and the predicted pose increment are output to realize the beam pose update iteration, and a continuously changing laser beam pose sequence is generated during the iteration process; Step S66: A laser ray is constructed based on the pose of the lidar in the current time step, and the sampling range and the initial settings of the laser ray are set; Step S67: Voxel rendering is performed along the ray to obtain a set of discrete sampling points and corresponding weight values; Step S68: Based on the weight distribution function obtained in Step S67, an optimal depth estimate is determined, and the world coordinates and reflection intensity of the point are calculated; Step S69: The original point sequence is converted into high-fidelity, directly usable virtual laser point cloud data using a preset noise model.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the scene and sensor data generation method based on remote sensing imagery as described in any one of claims 1 to 9.

[0015] Compared with the prior art, the present invention has the following advantages and effects: 1. The scene and sensor data generation method based on remote sensing imagery in the present invention can generate time-series trajectory images and laser point cloud data using a small amount of UAV aerial photography data or short-cycle high-resolution optical remote sensing satellite data. This not only reduces the difficulty of data acquisition in complex or extreme traffic scenarios, but also effectively reduces the acquisition cost, achieves comprehensive coverage of different terrains, road structures and environmental conditions, and solves the problem of real vehicle data acquisition being limited by routes, traffic flow and safety factors.

[0016] 2. The scene and sensor data generation method based on remote sensing images in this invention can generate road scenes under extreme weather conditions such as rain, snow, and night, as well as high-density traffic or accident scenes through remote sensing images, making up for the rare situations that are difficult to reach by real vehicle data collection and improving the robustness of autonomous driving systems.

[0017] 3. Compared with the scene and sensor data generation method based on remote sensing imagery in this invention, the large-scale deployment of real vehicle data collection, using drones or satellites to collect remote sensing images and combining them with algorithms to generate other sensor data such as lidar point clouds and camera images, eliminates the need to deploy real vehicles for every route, saving vehicle, sensor and manpower costs, while also enabling the rapid generation of large-scale training datasets.

[0018] 4. The scene and sensor data generation method based on remote sensing images in this invention utilizes remote sensing images to generate data for the same road segment under different weather (rain, snow, fog, etc.) or different time (day, night, dawn and dusk) conditions, without waiting for the right opportunity to repeatedly collect data on-site. Attached Figure Description

[0019] Figure 1 is a flowchart illustrating the method for generating scene and sensor data based on remote sensing imagery in an embodiment of the present invention; Figure 2 is a flowchart illustrating step S2 in Figure 1; Figure 3 is a flowchart illustrating step S3 in Figure 1; Figure 4 is a flowchart illustrating step S4 in Figure 1; Figure 5 is a flowchart illustrating step S5 in Figure 1; Figure 6 is a flowchart illustrating step S6 in Figure 1. Detailed Implementation

[0020] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can also refer to the internal connection of two components; and they can refer to a wireless connection or a wired connection. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0022] Please refer to Figures 1-6. This embodiment of the invention provides a method and electronic device for generating scene and sensor data based on remote sensing images, including: Step S1: acquiring several single high-resolution images through UAV aerial photography data or short-period high-resolution optical remote sensing satellite data.

[0023] Step S2: Acquire multi-source modal data based on a single high-resolution image. The multi-source modal data includes embedded features of multi-view images, virtual vehicle trajectories within the coverage area of ​​remote sensing data, and scene text descriptions. Step S3: Obtain multi-layer feature self-attention for the corresponding modality by encoding different modal data, and perform mutual attention feature fusion of different modal data based on multi-layer feature self-attention to obtain updated feature representations. Step S4: Process the new feature representations under this pose using a DPT model to obtain dense depth maps and high-precision 3D point clouds with GS flow embedding features. Step S5: Based on the predicted image information in Step S4, generate multi-view images and high-quality 3D assets observed by the same sensor configuration under the next pose using a diffusion model. Step S6: Process the multi-view images and high-quality 3D assets in Step 5 using a recurrent neural network to obtain consistent temporal trajectory images and laser point cloud data.

[0024] Please refer to Figure 1-2. In step S2, multi-source modal data is acquired based on a single high-resolution image, including: step S21: generating multi-view images based on a single high-resolution image; step S22: generating embedded features of scene text descriptions for each view based on the multi-view images; step S23: generating virtual driving trajectories within the coverage area for each view based on the multi-view images.

[0025] Specifically, in step S21, generating a multi-view image based on a single high-resolution image includes: step S211: using a deep learning model to predict the depth information of each pixel and generating a depth map.

[0026] In this step, the deep learning model is a pre-trained monocular depth estimation model, such as the MiDaS model, the DPT (Dense Prediction Transformer) model, the AdaBins model, etc. Monocular depth estimation models are suitable for inferring depth from cues such as texture, shadow, perspective, and target size. The pre-processed single high-resolution image (RGB image) is input into the monocular depth estimation model, and the model outputs a depth map registered with the input image.

[0027] Step S212: Reconstruct the sparse 3D geometry of the image scene.

[0028] In this step, based on the camera's intrinsic and extrinsic parameters, each image pixel (u,v) and its depth value d are transformed into a point (X,Y,Z) in the three-dimensional world coordinate system through back projection; the transformation formula is as follows:

[0029]

[0030]

[0031] Where f is the focal length in the camera parameters. The x-coordinate of the principal point in the camera parameters. The ordinate of the principal point in the camera parameters; , The principal point is the location of the principal point in the camera parameters. The principal point refers to the intersection of the camera optical axis and the image sensor plane (image plane) in the camera intrinsic parameter matrix. Based on the above formula, the position information of each point in the image is calculated to generate a three-dimensional point cloud.

[0032] Step S213: Use the reconstructed 3D geometry to create a virtual camera and re-render the scene from any new perspective to simulate a new view of the scene from different angles.

[0033] In this step, the camera parameters for the new image to be generated must first be confirmed, including the coordinates of the new camera center in the world coordinate system (X_c', Y_c', Z_c'), the rotation angle of the new camera (yaw, pitch, roll), the focal length of the new image, and the sensor size. Each 3D point on the 3D point cloud or mesh is projected onto the new 2D image plane according to the parameters of the new camera. Using the standard rasterization or ray tracing workflow of computer graphics, the textured mesh is rendered into a new 2D image to obtain the new view.

[0034] Specifically, in step S22, the embedding features for generating scene text descriptions for each view based on the multi-view image include: Step S221: confirming the backbone network for feature extraction and the scene description model used.

[0035] In this step, the backbone network refers to a convolutional neural network (CNN) or a Vision Transformer (ViT) pre-trained on a large natural image dataset (such as ImageNet), and the scene description model used is one of ResNet, VGG, DenseNet, and EfficientNet.

[0036] Step S222: Input a single high-resolution remote sensing image and its new view into the scene description model used, so that the backbone network outputs a high-dimensional feature tensor rich in semantic information.

[0037] In this step, the high-dimensional feature tensor can be a global feature vector or a feature map.

[0038] Step S223: Use a self-attention mechanism to integrate the visual features extracted in step 222 into a structured representation rich in contextual semantic information.

[0039] In this step, the spatial feature map extracted in step 222 is first flattened and regarded as a sequence. Then, this sequence is input into a network composed of multiple Transformer Encoder modules. Inside the Encoder, each image region token interacts with all other tokens in the sequence (calculating attention weights) so that each token absorbs global information and updates its own representation. Finally, a context-aware visual token sequence is output, and each vector in the sequence already contains information about its relationship with all other regions. A token is a numerical vector representing a specific part or region in the image.

[0040] Step S224: The decoder is used to progressively and autoregressively convert the context-rich, fixed-size visual representation from step S223 into a word sequence, forming the final image scene text description.

[0041] In this step, the decoder architecture is Transformer Decoder, which uses masked self-attention to focus on the generated text sequence and cross-attention to focus on the features extracted from the image. The specific process is as follows: input the visual context vector and the currently generated sequence into the decoder; the decoder outputs a probability score on all words in the vocabulary; use beam search to generate the corresponding words; append the newly generated words to the sequence as input for the next step; repeat the above steps until a complete word sequence is generated or the maximum generation length is reached, and finally generate a natural language description.

[0042] Specifically, in step S23, generating a virtual driving trajectory within the coverage area for each of the multiple view images includes: step S231: creating a realistic and accurate digital environment using remote sensing data.

[0043] In this step, the geographical boundaries of the virtual trajectory generation (i.e., the "remote sensing data coverage area") need to be clearly defined first. Then, multi-source remote sensing data is acquired based on image information. Specifically, this includes acquiring true-color or false-color surface images from optical satellite / aerial imagery, acquiring high-precision digital surface models (DSM) and digital elevation models (DEM) from lidar data, and acquiring supplementary optical data from radar data. The digital surface models (DSM) and digital elevation models (DEM) can provide three-dimensional elevation information including buildings and trees, which is key to calculating slope, curvature, and field of view. After that, data preprocessing is required to improve spectral accuracy, geometric precision, and information richness.

[0044] Specifically, data preprocessing methods include radiometric calibration and atmospheric correction, geometric correction and orthorectification, and data registration and fusion. Radiometric calibration refers to using calibration coefficients (gain and offset) provided by the sensor manufacturer to perform linear transformation, converting the raw digital quantization values ​​(DN values) recorded by the sensor into physically meaningful apparent radiance or apparent reflectance. The conversion formula is as follows:

[0045] in, Represents radiance, This represents the conversion factor, used to scale DN values ​​to physical quantities such as radiance or reflectivity. These are the values ​​originally recorded by the sensor, representing the grayscale values ​​of the pixels (raw data without calibration). It is the offset, used to adjust the reference.

[0046] Atmospheric correction refers to the processing of remote sensing images using the dark pixel method to eliminate the influence of atmospheric scattering, absorption, and reflection on the reflectance signals of ground objects, thereby obtaining the true surface reflectance of ground objects to accurately reflect their spectral characteristics.

[0047] Geometric correction refers to using ground control points (GCPs) and polynomial models to correct systematic distortions caused by sensor attitude (tilt), Earth curvature, etc. Orthorectification refers to using digital elevation models (DEMs) to correct projection differences caused by terrain undulations. An image that has undergone orthorectification is called an orthophoto, and each pixel has accurate geographic coordinates, just like a map with a uniform scale.

[0048] Image registration refers to the process of selecting a large number of corresponding feature points on an image (reference image) and an image to be registered, and achieving precise alignment through polynomial transformation or triangular mesh distortion correction, so as to accurately align images from different sources, times, and angles to the same coordinate system in space. Image fusion refers to the process of using algorithms such as Brovy transform, PCA (principal component analysis) transform, and Gram-Schmidt transform to combine image data with different spatial and spectral resolutions to generate a new image with both high spatial and high spectral resolution, so as to improve the image's clarity and interpretation capabilities, and make details such as road edges and lane lines more clearly distinguishable.

[0049] Step S232: Convert the processed remote sensing data into a vector road network model that can be understood and simulated by a computer.

[0050] In this step, the first step is to extract a high-precision network and use a deep learning semantic segmentation model to perform pixel-level classification of the remote sensing image, automatically identify road areas, and then convert them into vector lines (road center lines) through post-processing. In this application, the deep learning semantic segmentation model is U-Net as an example. Before use, the deep learning semantic segmentation model needs to be trained with a large amount of labeled "image-road label" data using a deep convolutional neural network (CNN) so that the model can learn the deep abstract features of the road and enable the model to have strong recognition and generalization capabilities.

[0051] Step S233: Add simulation rules to the generated vector road network. The simulation rules include road grade, number of lanes, speed limit, traffic rules, and gradient. As a further description of the above steps, the simulation rules are stored in the GIS attribute table and are the "traffic rules" and "physical laws" of the simulation.

[0052] Step S234: Configure simulation parameters for the preset simulation model using a genetic algorithm and parameter calibration tool, and perform path planning based on the shortest time and shortest path. The simulation parameters include traffic demand, vehicle characteristics, and driving behavior patterns. In this step, it is necessary to use a genetic algorithm to generate paths for tens of thousands of points in batches, set road attributes in batches, and process the OD matrix. Specifically, this includes using a script to transmit data estimated through traffic surveys, mobile phone signaling data, GPS big data, etc., to the simulation software to configure the simulation parameters. The OD matrix defines the number of vehicles departing from each traffic zone to other traffic zones within a specific time interval. During this process, the calibration tool can automatically perform multiple iterative simulations to find the optimal parameter combination, thereby automatically adjusting the model parameters (such as maximum acceleration, expected vehicle distance, etc.) to plan the shortest time and shortest path while making the simulation output results as close as possible to the real data. The calibration tool in this application is SUMO's calibrator tool as an example.

[0053] Step S235: Output the trajectory data and load the trajectory file using a professional visualization analysis tool to overlay it onto the remote sensing image to obtain a virtual driving trajectory.

[0054] In this step, the processed road network files, vehicle files, and configuration files are imported into microscopic traffic simulation software (such as SUMO, VISSIM, Aimsun). The simulation engine then proceeds at a fixed time step (e.g., 0.1 seconds) so that the engine calculates the acceleration, velocity, and new position (x, y) of each vehicle within each time step based on all configured models and rules, until the simulation ends. After the simulation is complete, detailed trajectory data for each vehicle can be output, including vehicle ID, timestamp, planar coordinates, elevation, velocity, acceleration, and lane ID.

[0055] Please refer to Figures 1-3. In step S3, the step of obtaining multi-layer feature self-attention of the corresponding modality by encoding different modal data, and performing mutual attention feature fusion of different modal data based on multi-layer feature self-attention to obtain the updated feature representation includes: Step S31: Encoding multi-source modal data obtained based on a single high-resolution image, wherein the multi-view image adopts frame encoding, the virtual driving trajectory within the remote sensing data coverage area adopts pose encoding, and the embedded features of the scene text description adopt CLIP encoding.

[0056] In this step, frame encoding of the multi-view image is suitable for converting the original image into a smaller data format through a compression algorithm. The process is as follows: ① Divide the image to be frame encoded into 16×16 pixel macroblocks. Then, based on the richness of image detail, each macroblock can be further divided into smaller subblocks. The more detailed and complex the motion, the smaller and more refined the subblocks should be for more accurate processing; ② Based on the pixel values ​​of the adjacent encoded blocks above and to the left of the current frame image, predict the value of the current block using specific directional patterns (such as horizontal, vertical, diagonal, etc.) to eliminate spatial redundancy in the image; ③ Search between the current frame block and the reference frame to find the best matching region; the encoder calculates a motion vector to describe where the block moved from the reference frame to the current position, in order to... ④ Eliminate temporal redundancy in the video sequence; ⑤ Subtract the predicted block from the original block to obtain the residual block; the more accurate the prediction, the smaller the residual value, the less data redundancy, and the more conducive to subsequent compression; ⑥ Use Discrete Cosine Transform (DCT) to concentrate image energy into the low-frequency region. After the transformation, the upper left corner of the matrix contains low-frequency coefficients (high energy, most important), and the lower right corner contains high-frequency coefficients (low energy, usually details and noise, not important), thereby achieving the effect of transforming the residual block from the spatial domain to the frequency domain, rearranging the information, and making it easier to compress; ⑦ Use algorithms like CAVLC or CABAC to convert the quantized coefficients (as well as motion vectors, prediction modes, and all other information) into the final binary bitstream to eliminate statistical redundancy and further compress the data; so that the compressed data (bitstream) can be output or transmitted.

[0057] In this step, the pose encoding of the virtual vehicle trajectory within the remote sensing data coverage area is suitable for converting high-precision, continuous pose sequences into a compact, discrete data representation to save storage space, improve transmission efficiency, and facilitate processing. The process is as follows: ① Use filtering algorithms (such as Kalman filtering, low-pass filtering, and particle filtering) to remove noise and outliers from the output data in step 235; ensure the physical rationality of the trajectory, avoid sudden changes in acceleration and curvature, and make the motion smoother; ② Determine a unified global reference coordinate system and transform all pose data to this unified coordinate system to ensure data consistency; ③ While maintaining the trajectory shape and kinematic characteristics as much as possible, use keyframe extraction and uniform resampling to reduce the number of data points to be processed. The keyframe extraction method retains more points in areas of drastic trajectory changes (such as curves) and fewer points in areas of gentle changes (such as straight lines) based on indicators such as curvature and velocity changes, to significantly reduce the amount of data without significantly losing accuracy; the uniform resampling method refers to discarding data points in simple trajectory shapes such as straight road segments. Discarding a large number of intermediate points and retaining only a few points at fixed time intervals to define the path makes the data more regular; ④ Using a parametric mathematical model to fit the whole or segmented trajectory. This application adopts a segmented parameterization approach, dividing the whole trajectory into multiple segments, fitting each segment with a low-order polynomial, and ensuring that these segments are smoothly connected at the connection points. For example, using cubic splines or B-splines to generate a smooth curve with a series of control points and polynomial parameters; ⑤ Using precision quantization and compression to package the encoded digital data into a storable or transmittable binary stream or file; The quantization process is as follows: First, determine the possible range of each variable to be quantized and determine the required precision according to the application requirements. Then, convert each floating-point number into an integer. After that, add a text header and metadata to each encoded file. Finally, select the sequence number format and package it. The above sequence number format is one of the following: custom binary format, Protocol Buffers (Protobuf), JSON / XML, and a specific standard format; Compression refers to applying a general compression algorithm (such as Delta Encoding + ZigZag + entropy encoding) to perform secondary compression on the quantized integer sequence.

[0058] In this step, CLIP encoding of scene text description embedding features is used to convert a text describing a scene into a high-dimensional numerical vector that can represent its semantic information. The specific process is as follows: ① Configure the Python environment and install dependent libraries (such as openai-clip or transformers library) and deep learning frameworks (such as PyTorch or TensorFlow); ② Load the CLIP model and its corresponding preprocessing functions; ③ Import the text file into the CLIP model, so that the CLIP model converts the text into a format called "tokens"; ④ Input the tokenized text into the model's text encoder to obtain the normalized feature vector.

[0059] Step S32: Based on the preset mutual attention model, perform multi-layer feature self-attention information fusion of different modal data and obtain the updated feature representation; in this step, the features of different modalities are concatenated at the input level to form a unified, hybrid representation, and then handed over to the model (using Transformer as an example in this application) to automatically learn all the complex relationships within and between all modalities.

[0060] ① The three modal information representations are converted into feature sequences, and the feature sequences of the three modalities are simply concatenated end to end to form a long, mixed multimodal sequence; ② Modality type embedding is added to enable the model to distinguish different modalities, and positional encoding is added to add standard positional encoding to each position of the constant sequence, injecting order information; this facilitates the generation of the final input representation; ③ The prepared mixed sequence is input into a standard Transformer encoder consisting of multiple layers to achieve completely arbitrary and dense cross-modal interaction; ④ After processing through L layers of Transformer, the final output sequence is obtained, where each vector is a "multimodal vector"; it no longer purely represents the original modal information, but contains relevant information converged from all other modalities.

[0061] Please refer to Figures 1-4. In step S4, the process of obtaining a dense depth map and a high-precision 3D point cloud with GS flow embedding features by processing the new feature representation under the pose using the DPT model includes: Step S41: Using the DPT model to read the new feature representation from step S3, and obtaining a dense depth map through encoding fusion; In this step, the feature representation sequence obtained in step S3 is input into the ViT encoder of the DPT model to read all multi-scale features. The ViT encoder is composed of multiple stacked Transformer blocks. In this process, not all Transformer blocks are used. Usually, 4 to 5 representative layers are selected, and these layers are evenly distributed or strategically distributed in the encoder; The corresponding starting feature vector is extracted from the output of each selected target layer, and a small fully connected layer (Linear) is used. The Layer projects each initial feature vector to a uniform number of channels (D), treating each vector with a uniform number of channels as a pseudo-feature map. Then, it uses deconvolution or bilinear upsampling to upsample each pseudo-feature map to a specific spatial size, thereby forming multiple feature maps with different spatial resolutions but rich in information. This set of feature maps is then used to construct the multi-scale feature pyramid required by the decoder. Finally, a simple convolutional layer (usually a 1x1 convolution) is used as the prediction head, which is applied to the feature vector of each pixel to map the high-dimensional features to a scalar value, namely the predicted depth value of the pixel, and outputs a dense depth map.

[0062] Step S42: Convert each pixel and its depth value in the optimized depth map into a 3D point in world coordinates through back projection to obtain the initial dense point cloud; In this step, the essence of generating the point cloud is to calculate the 3D coordinates P in the world coordinate system for each pixel (u,v) in the image in order to achieve the alignment of the point cloud from multiple perspectives.

[0063] In the depth map, the depth value of each pixel is d=D(u,v), where D is a hyperparameter representing the number of special channels; the formula for calculating the 3D coordinates of a pixel in the world coordinate system is as follows:

[0064] in, It is the inverse of the camera intrinsic parameter matrix. These are the homogeneous coordinates of the pixels. This is the depth value of that pixel. The superscript of T indicates the transpose sign. All of these are part of the camera's external parameters. Represents the translation vector. Represents a rotation matrix.

[0065] Step S43: Use a point cloud optimization algorithm to denoise and simplify the initial dense point cloud to obtain the final dense point cloud.

[0066] In this step, the point cloud optimization algorithm of this application includes a denoising algorithm and a simplification algorithm. The denoising algorithm is one or a combination of statistical filtering, radius filtering, Gaussian filtering and bilateral filtering algorithms, while the simplification algorithm is one of voxel grid filtering, curvature sampling and feature preservation sampling algorithms. Step S44: Based on the Transformer decoder, the complete geometric and semantic attributes of the 3D space are inferred, so that the 3D points are converted into data with optimizable features and a high-dimensional, learnable feature vector, which is the GS stream embedding feature. In this step, after obtaining the 3D coordinates in the world coordinate system, the Transformer decoder calculates the correlation between the 3D points and each element in the input sequence (these elements are associated with image positions, text words, and trajectory points), so as to obtain the GS stream embedding features corresponding to the 3D points, so as to enrich the point cloud.

[0067] Please refer to Figures 1-5. In step S5, the generation of multi-view images and high-quality 3D assets observed by the same sensor configuration in the next pose based on the dense depth map and high-precision 3D point cloud in step S4 through the diffusion model includes: Step S51: Generate a 3D Gaussian model based on the dense depth map and the high-precision 3D point cloud with GS flow embedding features in step S4.

[0068] In this step, a viewpoint is first selected from the initial remote sensing image and the multi-view images it produces, or a new viewpoint is randomly generated within a reasonable range. A differentiable renderer is used to render a 2D image from this viewpoint based on the current 3D representation and appearance parameters. The rendered image is compared with the pre-selected image, and the rendering loss and regularization loss are calculated. The gradient of the total loss with respect to all optimizable parameters is calculated. Since the renderer is differentiable, this gradient can propagate backward from the pixel space of the 2D image to the geometric and appearance parameters in the 3D space. An optimizer (such as Adam) can be used to update the parameters (position, shape, color, transparency) based on the gradient, so that when the loss function converges, the final optimized high-precision 3D asset, i.e., a high-precision 3D Gaussian model, can be output.

[0069] As a preferred embodiment of the above scheme, the rendering loss includes RGB L1 / L2 Loss (a loss function used to measure the difference between the predicted and actual values ​​of an RGB image), perceptual loss (used to compare whether the high-level semantic features of two images are similar), and adversarial loss (referring to the introduction of a discriminator to determine whether the rendered image is "real" or "generated" in order to improve the realism of the rendered image). The regularization loss includes geometric smoothness loss (preventing unnatural bumps on the surface by encouraging smooth changes in adjacent vertices or SDF values) and sparsity loss (making the surface simpler by encouraging unnecessary geometry to be pruned).

[0070] Step S52: Perform differentiable rendering using a 3D Gaussian model under the target pose to obtain the RGB image and depth information rendered by the 3D Gaussian model under that viewpoint.

[0071] In this step, the rendered RGB image and depth information provide strong geometric and color priors for the diffusion model, so that the final geometric structure and approximate color layout of the diffusion model are the same as the rendered result.

[0072] Step S53: Prepare the input conditions for the diffusion model and process the control conditions to obtain a noisy latent representation.

[0073] In this step, the preparation of input conditions for the diffusion model first requires assembling a conditional information package as an instruction to control the generation process of the diffusion model. This conditional information package includes at least the rendered RGB image and depth information, camera parameters (target pose, intrinsic parameters), and may also include text prompts and initially selected images. By inputting the above conditions into the control network of the diffusion model, the 3D Gaussian model can strictly adhere to these conditions in each denoising step of the generation process.

[0074] In this step, the control conditions are first processed by encoding the rendered 2D image (or the RGB image rendered by the 3D Gaussian model) of the new viewpoint selected in step S51 into the latent space of the diffusion model to obtain a noisy latent representation. Then, all control conditions (camera parameters and the RGB image and depth map rendered by the 3D Gaussian model) are input into the control network (such as ControlNet or multiple T2I-Adapters) to generate a series of feature maps and attention biases. These feature maps encode conditional information (e.g., edges of the depth map, color blocks of the rendered image), and these generated feature maps are added or connected to various skip connections and intermediate layers of UNet. The attention biases modify the cross-attention mechanism in UNet, making UNet pay more attention to the conditional information, thereby realizing the processing of control conditions.

[0075] Step S54: Refine and upsample the noisy latent representation to obtain the refined image.

[0076] In this step, UNet propagates forward using the current noisy latent image and the time step embedding as input. UNet continuously fuses with feature maps from the control network. During this process, each layer of UNet not only acquires the noisy latent image but also the geometric and color blueprints provided by 3DGS, which continuously guides the inference direction of UNet until it finally predicts the noise that should be removed from the current latent image. Then, based on the sampler algorithm of the diffusion model (such as DDIM), a more denoised latent image is calculated using the predicted noise, thereby realizing the latent image update iteration until the time step converges, resulting in a clean latent representation. Finally, the VAE decoder is used to convert the above clean latent representation back to the pixel space to obtain the refined image.

[0077] Step S55: Repeat steps S52, S53 and S54 multiple times to obtain images from different perspectives after refinement.

[0078] Step S56: Optimize the 3D Gaussian model again using the newly refined images from multiple different perspectives until the 3D Gaussian model and the diffusion model converge to an optimal state, resulting in highly consistent multi-view images and detailed, high-quality 3D assets.

[0079] In this step, the 3D Gaussian model provides more precise geometric constraints for the diffusion model, and the diffusion model provides higher quality and more consistent training images for the 3D Gaussian model. Ultimately, the diffusion model generates highly consistent multi-view images, and the 3D Gaussian model reconstructs a detailed and high-quality 3D asset.

[0080] Please refer to Figures 1-6. In step S6, the process of using a recurrent neural network to process the multi-view images and 3D assets in step 5 to obtain consistent temporal trajectory images and laser point cloud data includes: Step S61: Inputting images from different perspectives at a certain time point into a pre-trained CNN, obtaining the output of the last convolutional layer or global average pooling layer of the CNN, and obtaining a feature vector of fixed length.

[0081] In this step, each time step is labeled with several feature vectors. The number of feature vectors is the same as the number of viewpoints. These feature vectors are highly abstract representations of the image, containing its key information, while greatly reducing the data dimensionality, which is convenient for RNN processing.

[0082] Step S62: Use an attention mechanism to weightedly fuse features from different perspectives at a certain time point to obtain a single, highly representative joint feature vector that can represent that time point. Repeat the above operation to process all time points to obtain a time series.

[0083] In this step, the features of each viewpoint at a certain time point are first projected through different linear layers based on linear projection to generate the Query, Key, and Value vectors required by the attention mechanism (the Query is a learnable vector, and the Key and Value are generated from the viewpoint features). The similarity between the Query and each Key is calculated using the dot product method, and the scores are normalized to a probability distribution using the Softmax function to obtain the attention weights (the weights represent the importance of the corresponding viewpoint to the current task). Then, the attention weights are used to perform a weighted summation of all Value vectors to obtain the final fused feature. The above operation is repeated until a fused feature is generated at each time point. Then, the fused features generated at each time step are collected in chronological order to form a complete sequence tensor, which perfectly meets the input requirements of RNN.

[0084] Step S63: Input the time series obtained in step S62 into a preset RNN unit (LSTM variant or GRU variant) so that the RNN updates the hidden state A at each time point in sequence to obtain a hidden state A sequence.

[0085] In this step, the input data for the current time step (the joint feature vector in step 62) needs to be sliced ​​from the input sequence. At the same time, the hidden state A of the previous time step is extracted (a mathematical vector that encodes the compressed information and contextual relationships of all input data processed by the RNN from the beginning of the sequence to the previous time step). Then, the input data and the hidden state A of the previous time step are input into the RNN unit to update the internal state of the current time step. After the entire sequence is processed, a sequence of hidden states A can be obtained.

[0086] Step S64: Map the abstract features in the RNN hidden space to the real output space through the neural network layer to obtain the trajectory prediction sequence, and output the time-series trajectory image based on the trajectory prediction sequence through the standard scientific plotting library.

[0087] In this step, the neural network layer is a fully connected neural network used for decoding, so as to perform a linear transformation to map the high-dimensional hidden state A of each time step to the low-dimensional trajectory coordinates, thereby realizing the decoding process, so as to collect the decoding output of each time step in chronological order and form a trajectory prediction sequence.

[0088] In this step, the standard scientific plotting library is the Matplotlib module in Python, which allows for the direct generation of an image depicting the entire time series trajectory.

[0089] Step S65: Process the current sensor pose and hidden state B in each time step through a preset RNN unit (LSTM variant or GRU variant), output the new hidden state B and the predicted pose increment to realize the beam pose update iteration, and generate a continuously changing laser beam pose sequence during the iteration process.

[0090] This step also includes setting the initial attitude of the virtual LiDAR and the initial hidden state B of the RNN. The initial attitude of the virtual LiDAR and the initial hidden state B of the RNN are the inputs in the first time step processing of the preset RNN unit. The attitude of the LiDAR includes the azimuth angle and the elevation angle. The initial hidden state B of the RNN is a compressed, dense vector representation that encodes all relevant information in the entire laser beam scanning history up to the current time step.

[0091] Step S66: Construct a laser ray based on the attitude of the lidar within the current time step, and simultaneously set the sampling range and the initial settings of the laser ray.

[0092] In this step, laser ray construction includes converting the lidar's attitude into a unit direction vector and defining the ray equation. The unit direction vector is determined by the lidar's scanning mechanism (e.g., a rotating mirror) and is typically calculated using azimuth (az) and elevation (zl). Setting the sampling range refers to limiting reasonable maximum and minimum detection distances. The initial settings for the laser beam refer to the settings of parameters such as cumulative transparency (T), step distance (l), final color (C), and final depth (D).

[0093] The formula for transforming the unit direction vector is as follows:

[0094]

[0095]

[0096] Where d represents the unit direction vector of the laser beam. It is the length of the projection of vector d onto the X-axis of the world coordinate system. It is the length of the projection of vector d onto the Y-axis of the world coordinate system. It is the length of the projection of vector d onto the Z-axis of the world coordinate system. Represents azimuth. Let represent the azimuth angle of the laser beam in the i-th time step. Represents pitch angle, This represents the elevation angle of the laser beam within the i-th time step.

[0097] The ray equation is as follows:

[0098] in, It is a vector-valued function representing a ray function, where o is the origin of the ray, representing the 3D position of the optical center of the lidar sensor in the global coordinate system (world coordinate system), l is the ray parameter or distance parameter, representing the precise distance from the lidar to the surface of the object being measured, and d is the ray direction vector, representing the direction of a single laser beam in three-dimensional space.

[0099] Step S67: Perform voxel rendering along the ray.

[0100] In this step, the first step is to perform a stepping motion, which means starting from the minimum detection distance and moving in steps of a certain size. l moves in the direction of maximum detection distance, and calculates the sample point P on the ray after each step. Simultaneously, spatial acceleration structures such as KD-Tree or BVH are used to quickly find Gaussians whose center points are within a certain neighborhood of sample point P, avoiding traversing every Gaussian in the scene. For each neighboring Gaussian, the probability density function value PDF of sample point P under the Gaussian distribution is calculated. Then, the contributions of all neighboring Gaussians are mixed to calculate the blending opacity δ of the current sample point P. At the same time, the colors of the neighboring Gaussians are mixed to obtain the color C of the current sample point P. Finally, the probability density of the current sample point being the location where the laser beam is ultimately reflected or hit is calculated. This continues until the transmittance falls below a threshold or the step distance reaches the set maximum detection distance, ultimately yielding a set of discrete sampling points. and their corresponding weight values , where k is the sequence of the number of sampling points.

[0101] The method for calculating the Gaussian center point in the above content is as follows:

[0102] in, Let be the mean of the j-th Gaussian distribution, and represent the world coordinates of the center point of the j-th Gaussian ellipsoid. Represents a row vector. These are the components of the vector along the X, Y, and Z axes, respectively, with the superscript "T" indicating the transpose operation; the probability density function (PDF) value of the sample point under a Gaussian distribution is calculated as follows:

[0103] in, Let be the value of the probability density function at point P, representing the relative probability density that point P belongs to the j-th Gaussian distribution; the formula for calculating the mixing opacity δ is as follows:

[0104] in Let the opacity (or "contribution") learned by the j-th Gaussian be denoted by itself. Let be the effective opacity of the j-th Gaussian at the current sample point; the formula for calculating the color C of the current sample point is as follows:

[0105] in, Let be the color of the j-th Gaussian sample as viewed from the current ray direction. The weighted contribution of the j-th Gaussian pair to the final color at point P. It is a very small positive number to prevent division by zero.

[0106] The cumulative effect of colors satisfies the following conditions:

[0107] The attenuation of transparency satisfies the following condition:

[0108] The formula for calculating the contribution weight of the current sampling point to the final hit point is as follows:

[0109] Contribution weight This represents a small interval near the sampling point P(l) on the ray. Within this point, the probability density is the location where the laser beam is ultimately reflected or struck.

[0110] Step S68: Based on the weight distribution function calculated along the ray in step S67 (l) Determine an optimal depth estimate. And calculate the world coordinates and reflection intensity of that point.

[0111] In this step, the calculation process for the time coordinate is as follows: First, calculate the sum of all weights. Then perform the desired depth Calculations can then be performed, and the desired depth can then be utilized. Calculate 3D coordinates If during this process If the value is very small, it is considered an invalid ray and subsequent calculations are skipped.

[0112] In this step, the calculation process for reflection intensity is as follows: the cumulative color value C is an RGB vector (R, G, B). First, the RGB colors need to be converted to grayscale values, and then the reflection intensity estimate is obtained. Then, the surface normal n and the incident angle are estimated. After cosine calculation, the incident angle effect can be simulated using Lambert's cosine law to obtain a high-fidelity scalar of reflection intensity. .

[0113] All weights and The calculation formula is as follows:

[0114] Expected depth The calculation formula is as follows:

[0115] 3D coordinates The calculation formula is as follows:

[0116] In the above formula, i represents the laser beam being processed. Let d be the desired depth and d be the direction vector.

[0117] Reflection intensity estimation The acquisition method is as follows:

[0118] The coefficients in the above formula are derived from the ITU-R BT.601 standard. The human eye is most sensitive to green light, so the green channel has the highest weight (0.587); followed by red light (0.299); and least sensitive to blue light (0.114).

[0119] The surface normal n is obtained as follows: Analyze the Gaussians that contribute the most to color; the eigenvectors of their covariance matrices suggest their orientation. Typically, the eigenvector corresponding to the smallest eigenvalue can be approximated as the surface normal at that point. We then take a weighted average of all the major contributing Gaussian normals to obtain the final normal estimate n.

[0120] Angle of incidence It is the angle between the surface normal n and the laser beam direction d, and its cosine value. The formula for applying the Lambert factor modulation intensity is as follows:

[0121] In the above formula, It is the dot product of the laser beam direction vector d and the surface normal vector n, and since d and n are direction vectors (of length 1), therefore... ,and This means taking the maximum of the dot product result and zero, to avoid... It is a negative number, in addition, By combining inherent properties with observational geometry, a physical error is eliminated, therefore As the final strength value .

[0122] Step S69: Use a preset noise model to convert the original point sequence into high-fidelity, directly usable virtual laser point cloud data.

[0123] In this step, the preset noise models include distance noise model, beam divergence noise model and intensity noise model.

[0124] Ranging error is typically related to the signal-to-noise ratio (SNR) of the received signal, and SNR is inversely proportional to the square of the distance. Therefore, the range noise model needs to satisfy the following formula:

[0125] in, is the standard deviation of the distance noise, and f is a coefficient determined based on the specific performance of the lidar. The larger the value of f, the lower the accuracy of the sensor. This is the original ideal distance value of the current point (the expected depth in step 68); the application of distance noise first requires starting from a value with a mean of 0 and a standard deviation of... Sample a noise value from a Gaussian distribution Then, a noise value is applied, and the noise application formula is as follows:

[0126] in, The noise value is given, and the noise value follows a normal distribution. .

[0127] The laser beam is not an ideal line, but has a small divergence angle. This means that the distance it measures is the average distance of several surface points within the beam's coverage area, which can lead to positioning ambiguity, especially at the edges. Therefore, the construction of the beam divergence noise model needs to satisfy the following formula:

[0128] in, To understand distance The radius of the laser beam represents the maximum possible horizontal displacement range of the current point. The total divergence angle of the lidar beam (in radians rad) is an inherent parameter of the sensor and can usually be found in its datasheet.

[0129] Applying beam divergence noise first requires generating a random offset. This value can be sampled from a uniform distribution from 0 to r, and Meet the following requirements Then, a random angle is collected as the offset direction. This angle can be sampled from a uniform distribution between 0 and 2π, and meets the following usage requirements. Then the random offset needs to be set. and offset direction Combined into 2D offset ( , Then, the 2D offset is transformed to the world coordinate system to obtain the offset vector ΔP, and finally, beam divergence noise is applied according to the coordinate offset formula.

[0130] 2D offset ( , The calculation formula is as follows:

[0131]

[0132] The coordinate offset formula is as follows:

[0133] Wherein, the offset vector ΔP is the 2D offset ( , The result is obtained by transforming it to the world coordinate system.

[0134] The intensity value is affected by photon shot noise and electronic thermal noise, resulting in slight random fluctuations. Therefore, the intensity noise model needs to satisfy the following formula:

[0135] in, It is a random noise value. It is the standard deviation of intensity noise, representing the intensity and amplitude of the noise. The value of is a constant that needs to be set. This value can be determined by fitting the intensity fluctuations of real radar data, and it usually ranges from 0.01 to 0.05.

[0136] Intensity noise The application of refers to the generation of random noise. Add to original strength value The application of intensity noise must satisfy the following formula:

[0137] in, The final intensity value obtained in step S68; after the intensity noise is applied, it is also ensured that the intensity value after adding noise still falls within the valid physical range, when the updated intensity value Exceed After determining the range, perform a cropping operation according to the formula below:

[0138] The above formula can exceed The values ​​after the range are pulled back to 0 or 1 to ensure the validity of the data.

[0139] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The computer program is executed by the processor to perform the above-described method for generating scene and sensor data based on remote sensing images.

[0140] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of this disclosure, and all such changes and modifications will fall within the scope of protection of this invention.

Claims

1. A method for generating scene and sensor data based on remote sensing imagery, characterized in that, include: Step S1: Acquire several single high-resolution images using drone aerial photography data or short-cycle high-resolution optical remote sensing satellite data; Step S2: Acquire multi-source modal data based on the single high-resolution images, wherein the multi-source modal data includes embedded features of multi-view images, virtual driving trajectories within the coverage area of ​​remote sensing data, and scene text descriptions. Step S3: Obtain multi-layer feature self-attention for the corresponding modality by encoding different modal data, and perform mutual attention feature fusion of different modal data based on multi-layer feature self-attention to obtain the updated feature representation; Step S4: Process the new feature representation under the pose using the DPT model to obtain a dense depth map and a high-precision 3D point cloud with GS flow embedding features; Step S5: Based on the dense depth map and high-precision 3D point cloud in Step S4, generate multi-view images and high-quality 3D assets observed by the same sensor configuration under the next pose using a diffusion model; Step S6: Process the multi-view images and high-quality 3D assets in Step 5 using a recurrent neural network to obtain consistent temporal trajectory images and laser point cloud data.

2. The method for generating scene and sensor data based on remote sensing imagery according to claim 1, characterized in that, In step S2, the acquisition of multi-source modal data based on a single high-resolution image includes: step S21: generating multi-view images based on a single high-resolution image; step S22: generating embedded features of scene text descriptions for each view based on the multi-view images; step S23: generating virtual driving trajectories within the coverage area for each view based on the multi-view images.

3. The method for generating scene and sensor data based on remote sensing imagery according to claim 2, characterized in that, In step S21, generating a multi-view image based on a single high-resolution image includes: step S211: using a deep learning model to predict the depth information of each pixel and generating a depth map; step S212: reconstructing the sparse three-dimensional geometry of the image scene; step S213: using the reconstructed three-dimensional geometry to simulate a camera and re-render the scene from any new perspective, simulating a new view of the scene from different perspectives.

4. The method for generating scene and sensor data based on remote sensing imagery according to claim 3, characterized in that, In step S22, the embedding features for generating scene text descriptions for each view based on multi-view images include: Step S221: confirming the backbone network for feature extraction and the scene description model used; Step S222: inputting a single high-resolution remote sensing image and its new view into the scene description model used, so that the backbone network outputs a high-dimensional feature tensor rich in semantic information; Step S223: using a self-attention mechanism to integrate the visual features extracted in step S222 into a structured representation rich in contextual semantic information; Step S224: using a decoder to progressively and autoregressively convert the contextually rich, fixed-size visual representation from step S223 into a word sequence, forming the final image scene text description.

5. The method for generating scene and sensor data based on remote sensing imagery according to claim 4, characterized in that, In step S23, generating a virtual driving trajectory within the coverage area of ​​each view based on multi-view images includes: Step S231: Creating a realistic and accurate digital environment using remote sensing data; Step S232: Converting the processed remote sensing data into a vector road network model that can be understood and simulated by a computer; Step S233: Adding simulation rules to the generated vector road network, including road grade, number of lanes, speed limit, traffic rules, and gradient; Step S234: Configuring simulation parameters for the preset simulation model using a genetic algorithm and parameter calibration tools, and performing path planning based on the shortest time and shortest path, including traffic demand, vehicle characteristics, and driving behavior patterns; Step S235: Outputting trajectory data and loading the trajectory file using professional visualization analysis tools to overlay it onto the remote sensing image to obtain the virtual driving trajectory.

6. The method for generating scene and sensor data based on remote sensing imagery according to claim 1, characterized in that, In step S3, the step of encoding different modal data to obtain the corresponding multi-layer feature self-attention and performing mutual attention feature fusion of different modal data based on the multi-layer feature self-attention to obtain the updated feature representation includes: Step S31: Encoding multi-source modal data obtained based on a single high-resolution image, wherein the multi-view image uses frame encoding, the virtual driving trajectory within the remote sensing data coverage area uses pose encoding, and the embedded features of the scene text description use CLIP encoding; Step S32: Performing information fusion of multi-layer feature self-attention of different modal data based on a preset mutual attention model to obtain the updated feature representation.

7. The method for generating scene and sensor data based on remote sensing imagery according to claim 1, characterized in that, In step S4, the process of obtaining a dense depth map and a high-precision 3D point cloud with GS flow embedding features by processing the new feature representation under the pose using the DPT model includes: Step S41: Using the DPT model to read the new feature representation from step S3, and obtaining a dense depth map through encoding fusion; Step S42: Converting each pixel and its depth value in the optimized depth map into a 3D point in world coordinates through back projection to obtain an initial dense point cloud; Step S43: Denoising and simplifying the initial dense point cloud using a point cloud optimization algorithm to obtain the final dense point cloud; Step S44: Inferring the complete geometric and semantic properties of the 3D space based on the Transformer decoder, thereby converting the 3D points into data with optimizable features and a high-dimensional, learnable feature vector; this feature vector is the GS flow embedding feature.

8. The method for generating scene and sensor data based on remote sensing imagery according to claim 1, characterized in that, In step S5, the step of generating multi-view images and high-quality 3D assets observed by the same sensor configuration in the next pose using a diffusion model based on the dense depth map and high-precision 3D point cloud from step S4 includes: Step S51: Generating a 3D Gaussian model based on the dense depth map and high-precision 3D point cloud with GS flow embedding features from step S4; Step S52: Performing differentiable rendering using the 3D Gaussian model in the target pose to obtain the RGB image and depth information rendered by the 3D Gaussian model in that viewpoint; Step S53: Preparing the diffusion model... Input conditions are processed and control conditions are applied to obtain a noisy latent representation; Step S54: The noisy latent representation is refined and upsampled to obtain a refined image; Step S55: Steps S52, S53 and S54 are repeated multiple times to obtain refined images from different viewpoints; Step S56: The 3D Gaussian model is optimized again using the newly refined images from multiple different viewpoints until the 3D Gaussian model and the diffusion model converge to an optimal state, resulting in highly consistent multi-view images and detailed, high-quality 3D assets.

9. The method for generating scene and sensor data based on remote sensing imagery according to claim 1, characterized in that, In step S6, the process of using a recurrent neural network to process the multi-view images and 3D assets in step 5 to obtain consistent temporal trajectory images and laser point cloud data includes: Step S61: Inputting images from different perspectives at a certain time point into a pre-trained CNN, obtaining the output of the last convolutional layer or global average pooling layer of the CNN, and obtaining a fixed-length feature vector; Step S62: Using an attention mechanism to weightedly fuse the features from different perspectives at a certain time point, obtaining a single, highly representative joint feature vector that can represent that time point, repeating the above operation to process all time points, and obtaining a time series; Step S63: Inputting the time series obtained in step S62 into a preset RNN unit, so that the RNN updates the hidden state A at each time point in sequence, and obtaining a hidden state A sequence; Step S64: Mapping the abstract features in the RNN hidden space through neural network layers. Step S65: Obtain the trajectory prediction sequence in the real output space, and output the time-series trajectory image based on the trajectory prediction sequence through a standard scientific plotting library; Step S66: Process the current sensor attitude and hidden state in each time step through a preset RNN unit, output the new hidden state B and the predicted attitude increment to realize the beam attitude update iteration, and generate a continuously changing laser beam attitude sequence during the iteration process; Step S67: Construct a laser ray based on the attitude of the lidar in the current time step, and set the sampling range and the initial settings of the laser ray; Step S68: Perform voxel rendering along the ray to obtain a set of discrete sampling points and corresponding weight values; Step S69: Based on the weight distribution function obtained in Step S67, determine an optimal depth estimate, and calculate the world coordinates and reflection intensity of the point; Step S60: Use a preset noise model to convert the original point sequence into high-fidelity, directly usable virtual laser point cloud data.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the scene and sensor data generation method based on remote sensing imagery as described in any one of claims 1 to 9.

Images