Scene and sensor data generation method based on remote sensing image and electronic device

Abstract

The application discloses a scene and sensor data generation method based on remote sensing images and electronic equipment, relates to the technical field of data generation, and acquires multi-source modal data by taking remote sensing data collected by a unmanned aerial vehicle or a satellite as input, so that consistent time sequence trajectory images and laser point cloud data can be obtained based on derivation, fusion, reconstruction and generation of the multi-source modal data, so as to obtain three-dimensional space information and visual features of the environment around the vehicle. The scene and sensor data generation method based on remote sensing images can generate time sequence trajectory images and laser point cloud data by using a small amount of unmanned aerial vehicle aerial photography data or short-period high-resolution optical remote sensing satellite data, can not only reduce the difficulty of data acquisition in complex or extreme traffic scenes, but also effectively reduce the collection cost, realize comprehensive coverage of different terrains, road structures and environmental conditions, and solve the problem that real vehicle collection is limited by routes, traffic flow and safety factors.

Description

Technical Field

[0001] This invention relates to the field of data generation technology, and more specifically, to a method and electronic device for generating scene and sensor data based on remote sensing imagery. Background Technology

[0002] With the rapid development of autonomous driving technology, vehicles' perception and decision-making regarding the environment are highly dependent on the support of multimodal data. The realization of high-precision autonomous driving systems requires the collection of core sensor data, including LiDAR point clouds and camera images, to obtain three-dimensional spatial information and visual features of the vehicle's surrounding environment.

[0003] Existing data collection methods mainly rely on real-vehicle data collection, which usually requires a large number of vehicles, sensor devices and professional personnel. This is not only costly, but also limited by driving routes and traffic conditions, making it difficult to cover complex or extreme traffic scenarios. As a result, the scope of data acquisition is limited, which restricts the performance improvement of autonomous driving systems in diverse environments. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a method and electronic device for generating scene and sensor data based on remote sensing imagery, so as to increase the coverage and diversity of the collected data while reducing the data acquisition cost.

[0005] To achieve the above objectives, the present invention provides a method for generating scene and sensor data based on remote sensing imagery, comprising: Step S1: Acquire several single high-resolution images using drone aerial photography data or short-cycle high-resolution optical remote sensing satellite data; Step S2: Obtain multi-source modal data based on a single high-resolution image. The multi-source modal data includes embedded features of multi-view images, virtual driving trajectories within the coverage area of ​​remote sensing data, and scene text descriptions. Step S3: Obtain multi-layer feature self-attention for the corresponding modality by encoding different modal data, and perform mutual attention feature fusion of different modal data based on multi-layer feature self-attention to obtain the updated feature representation; Step S4: Process the new feature representation under this pose using the DPT model to obtain a dense depth map and a high-precision 3D point cloud with GS flow embedding features; Step S5: Based on the dense depth map and high-precision 3D point cloud in step S4, generate multi-view images and high-quality 3D assets observed by the same sensor configuration in the next pose using a diffusion model; Step S6: Use a recurrent neural network to process the multi-view images and high-quality 3D assets from step 5 to obtain consistent temporal trajectory images and laser point cloud data.

[0006] Furthermore, in step S2, the acquisition of multi-source modal data based on a single high-resolution image includes: Step S21: Generate multi-view images based on a single high-resolution image; Step S22: Generate embedded features of scene text descriptions for each view based on the multi-view image; Step S23: Generate a virtual driving trajectory within the coverage area for each of the multi-view images.

[0007] Further, in step S21, the generation of multi-view images based on a single high-resolution image includes: Step S211: Use a deep learning model to predict the depth information of each pixel and generate a depth map; Step S212: Reconstruct the sparse 3D geometry of the image scene; Step S213: Use the reconstructed 3D geometry to create a virtual camera and re-render the scene from any new perspective to simulate a new view of the scene from different angles.

[0008] Furthermore, in step S22, the embedding features for generating scene text descriptions for each view based on the multi-view image include: Step S221: Confirm the backbone network for feature extraction and the scene description model used; Step S222: Input a single high-resolution remote sensing image and its new view into the scene description model used, so that the backbone network outputs a high-dimensional feature tensor rich in semantic information; Step S223: Use a self-attention mechanism to integrate the visual features extracted in step 222 into a structured representation rich in contextual semantic information; Step S224: The decoder is used to progressively and autoregressively convert the context-rich, fixed-size visual representation from step S223 into a word sequence, forming the final image scene text description.

[0009] Furthermore, in step S23, generating a virtual driving trajectory within the coverage area for each of the multiple view images includes: Step S231: Create a realistic and accurate digital environment using remote sensing data; Step S232: Convert the processed remote sensing data into a vector road network model that can be understood and simulated by a computer; Step S233: Add simulation rules to the generated vector road network, the simulation rules including road grade, number of lanes, speed limit, traffic rules and gradient; Step S234: Configure simulation parameters for the preset simulation model using a genetic algorithm and parameter calibration tool, and perform path planning based on the shortest time and shortest path. The simulation parameters include traffic demand, vehicle characteristics, and driving behavior patterns. Step S235: Output the trajectory data and load the trajectory file using a professional visualization analysis tool to overlay it onto the remote sensing image to obtain a virtual driving trajectory.

[0010] Further, in step S3, the step of encoding different modal data to obtain corresponding multi-layer feature self-attention, and performing mutual attention feature fusion of different modal data based on multi-layer feature self-attention to obtain updated feature representations, includes: Step S31: Encode the multi-source modal data acquired based on a single high-resolution image, with frame encoding used for the multi-view images, pose encoding used for the virtual vehicle trajectory within the coverage area of ​​the remote sensing data, and CLIP encoding used for the embedded features of the scene text description. Step S32: Based on the preset mutual attention model, perform multi-layer feature self-attention information fusion of different modal data and obtain the updated feature representation.

[0011] Further, in step S4, the step of processing the new feature representation under this pose using the DPT model to obtain a dense depth map and a high-precision 3D point cloud with GS flow embedding features includes: Step S41: Use the DPT model to read the new feature representation from step S3, and obtain a dense depth map through encoding fusion; Step S42: Convert each pixel and its depth value in the optimized depth map into a 3D point in world coordinates through back projection to obtain the initial dense point cloud; Step S43: Use a point cloud optimization algorithm to denoise and simplify the initial dense point cloud to obtain the final dense point cloud; Step S44: Based on the Transformer decoder, infer the complete geometric and semantic properties of the 3D space, thereby transforming the 3D points into optimizable data and a high-dimensional, learnable feature vector; this feature vector is the GS stream embedding feature.

[0012] Furthermore, in step S5, the generation of multi-view images and high-quality 3D assets observed by the same sensor configuration in the next pose, based on the dense depth map and high-precision 3D point cloud from step S4, through a diffusion model, includes: Step S51: Generate a 3D Gaussian model based on the dense depth map and the high-precision 3D point cloud with GS flow embedding features from step S4; Step S52: Perform differentiable rendering using a 3D Gaussian model under the target pose to obtain the RGB image and depth information rendered by the 3D Gaussian model under this viewpoint; Step S53: Prepare the input conditions for the diffusion model and process the control conditions to obtain a noisy latent representation; Step S54: Refine and upsample the noisy latent representation to obtain a refined image; Step S55: Repeat steps S52, S53 and S54 multiple times to obtain images from different perspectives after refinement; Step S56: Optimize the 3D Gaussian model again using the newly refined images from multiple different perspectives until the 3D Gaussian model and the diffusion model converge to an optimal state, resulting in highly consistent multi-view images and detailed, high-quality 3D assets.

[0013] Furthermore, in step S6, the process of using a recurrent neural network to process the multi-view images and 3D assets from step 5 to obtain consistent temporal trajectory images and laser point cloud data includes: Step S61: Input images from different perspectives at a certain time point into the pre-trained CNN, obtain the output of the last convolutional layer or global average pooling layer of the CNN, and obtain a feature vector of fixed length. Step S62: Use an attention mechanism to weightedly fuse features from different perspectives at a certain time point to obtain a single, highly representative joint feature vector that can represent that time point. Repeat the above operation to process all time points to obtain a time series. Step S63: Input the time series obtained in step S62 into the preset RNN unit so that the RNN updates the hidden state A at each time point in sequence to obtain a hidden state A sequence; Step S64: Map the abstract features in the RNN hidden space to the real output space through the neural network layer to obtain the trajectory prediction sequence, and output the time-series trajectory image based on the trajectory prediction sequence through the standard scientific plotting library; Step S65: Process the current sensor pose and hidden state in each time step through the preset RNN unit, output the new hidden state B and the predicted pose increment, so as to realize the beam pose update iteration and generate a continuously changing laser beam pose sequence during the iteration process. Step S66: Construct a laser ray based on the attitude of the lidar within the current time step, and simultaneously set the sampling range and the initial settings of the laser ray; Step S67: Perform voxel rendering along the ray to obtain a set of discrete sampling points and corresponding weight values; Step S68: Based on the weight distribution function obtained in step S67, determine an optimal depth estimate and calculate the world coordinates and reflection intensity of the point; Step S69: Use a preset noise model to convert the original point sequence into high-fidelity, directly usable virtual laser point cloud data.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the scene and sensor data generation method based on remote sensing imagery.

[0015] Compared with the prior art, the present invention has the following advantages and effects: 1. The scene and sensor data generation method based on remote sensing imagery in this invention can generate time-series trajectory images and laser point cloud data using a small amount of UAV aerial photography data or short-cycle high-resolution optical remote sensing satellite data. This not only reduces the difficulty of data acquisition in complex or extreme traffic scenarios, but also effectively reduces the acquisition cost, achieves comprehensive coverage of different terrains, road structures and environmental conditions, and solves the problem of real vehicle data acquisition being limited by routes, traffic flow and safety factors.

[0016] 2. The scene and sensor data generation method based on remote sensing images in this invention can generate road scenes under extreme weather conditions such as rain, snow, night, and fog, as well as high-density traffic or accident scenes through remote sensing images, making up for the rare situations that are difficult to reach by real vehicle data collection and improving the robustness of the autonomous driving system.

[0017] 3. Compared with the scene and sensor data generation method based on remote sensing imagery in this invention, the large-scale deployment of real vehicle data collection, using drones or satellites to collect remote sensing images and combining them with algorithms to generate other sensor data such as lidar point clouds and camera images, eliminates the need to deploy real vehicles for every route, saving vehicle, sensor and manpower costs, while also enabling the rapid generation of large-scale training datasets.

[0018] 4. The scene and sensor data generation method based on remote sensing images in this invention utilizes remote sensing images to generate data for the same road segment under different weather (rain, snow, fog, etc.) or different time (day, night, dawn and dusk) conditions, without waiting for the right opportunity to repeatedly collect data on-site. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the method for generating scene and sensor data based on remote sensing imagery in an embodiment of the present invention. Figure 2 for Figure 1 A flowchart illustrating step S2; Figure 3 for Figure 1 A flowchart illustrating step S3; Figure 4 for Figure 1 A flowchart illustrating step S4 in the middle section; Figure 5 for Figure 1 A flowchart illustrating step S5; Figure 6 for Figure 1 A flowchart illustrating step S6. Detailed Implementation

[0020] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can also refer to the internal connection of two components; and they can refer to a wireless connection or a wired connection. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0022] Please see Figure 1-6 As shown, this embodiment of the invention provides a method and electronic device for generating scene and sensor data based on remote sensing imagery, including: Step S1: Acquire several single high-resolution images using drone aerial photography data or short-cycle high-resolution optical remote sensing satellite data.

[0023] Step S2: Obtain multi-source modal data based on a single high-resolution image. The multi-source modal data includes embedded features of multi-view images, virtual driving trajectories within the coverage area of ​​remote sensing data, and scene text descriptions. Step S3: Obtain multi-layer feature self-attention for the corresponding modality by encoding data of different modalities, and perform mutual attention feature fusion of data of different modalities based on multi-layer feature self-attention to obtain the updated feature representation; Step S4: Process the new feature representation under this pose using the DPT model to obtain a dense depth map and a high-precision 3D point cloud with GS flow embedding features; Step S5: Based on the predicted image information in step S4, generate multi-view images and high-quality 3D assets observed by the same sensor configuration in the next pose using a diffusion model; Step S6: Use a recurrent neural network to process the multi-view images and high-quality 3D assets from step 5 to obtain consistent temporal trajectory images and laser point cloud data.

[0024] Please see Figure 1-2 As shown, in step S2, multi-source modal data is acquired based on a single high-resolution image, including: Step S21: Generate multi-view images based on a single high-resolution image; Step S22: Generate embedded features of scene text descriptions for each view based on the multi-view image; Step S23: Generate a virtual driving trajectory within the coverage area for each of the multi-view images.

[0025] Specifically, in step S21, generating multiple view images based on a single high-resolution image includes: Step S211: Use a deep learning model to predict the depth information of each pixel and generate a depth map.

[0026] In this step, the deep learning model is a pre-trained monocular depth estimation model, such as the MiDaS model, the DPT (Dense Prediction Transformer) model, the AdaBins model, etc. Monocular depth estimation models are suitable for inferring depth from cues such as texture, shadow, perspective, and target size. The pre-processed single high-resolution image (RGB image) is input into the monocular depth estimation model, and the model outputs a depth map registered with the input image.

[0027] Step S212: Reconstruct the sparse 3D geometry of the image scene.

[0028] In this step, based on the camera's intrinsic and extrinsic parameters, each image pixel (u,v) and its depth value d are transformed into a point (X,Y,Z) in the three-dimensional world coordinate system through back projection; the transformation formula is as follows:

[0029]

[0030]

[0031] Where f is the focal length in the camera parameters. The x-coordinate of the principal point in the camera parameters. The ordinate of the principal point in the camera parameters; , The principal point is the location of the principal point in the camera parameters. The principal point refers to the intersection of the camera optical axis and the image sensor plane (image plane) in the camera intrinsic parameter matrix. Based on the above formula, the position information of each point in the image is calculated, and a three-dimensional point cloud is generated.

[0032] Step S213: Use the reconstructed 3D geometry to create a virtual camera and re-render the scene from any new perspective to simulate a new view of the scene from different angles.

[0033] In this step, the camera parameters for the new image to be generated must first be confirmed, including the coordinates of the new camera center in the world coordinate system (X_c', Y_c', Z_c'), the rotation angle of the new camera (yaw, pitch, roll), the focal length of the new image, and the sensor size. Each 3D point on the 3D point cloud or mesh is projected onto the new 2D image plane according to the parameters of the new camera. Using the standard rasterization or ray tracing workflow of computer graphics, the textured mesh is rendered into a new 2D image to obtain the new view.

[0034] Specifically, in step S22, the embedding features for generating scene text descriptions for each view based on the multi-view image include: Step S221: Confirm the backbone network for feature extraction and the scene description model used.

[0035] In this step, the backbone network refers to a convolutional neural network (CNN) or a Vision Transformer (ViT) pre-trained on a large natural image dataset (such as ImageNet), and the scene description model used is one of ResNet, VGG, DenseNet, and EfficientNet.

[0036] Step S222: Input a single high-resolution remote sensing image and its new view into the scene description model used, so that the backbone network outputs a high-dimensional feature tensor rich in semantic information.

[0037] In this step, the high-dimensional feature tensor can be a global feature vector or a feature map.

[0038] Step S223: Use a self-attention mechanism to integrate the visual features extracted in step 222 into a structured representation rich in contextual semantic information.

[0039] In this step, the spatial feature map extracted in step 222 is first flattened and regarded as a sequence. Then, this sequence is input into a network composed of multiple Transformer Encoder modules. Inside the Encoder, each image region token interacts with all other tokens in the sequence (calculating attention weights) so that each token absorbs global information and updates its own representation. Finally, a context-aware visual token sequence is output, and each vector in the sequence already contains information about its relationship with all other regions. A token is a numerical vector representing a specific part or region in the image.

[0040] Step S224: The decoder is used to progressively and autoregressively convert the context-rich, fixed-size visual representation from step S223 into a word sequence, forming the final image scene text description.

[0041] In this step, the decoder architecture is Transformer Decoder, which uses masked self-attention to focus on the generated text sequence and cross-attention to focus on the features extracted from the image; the specific process is as follows: The visual context vector and the currently generated sequence are input into the decoder; the decoder outputs a probability score on all words in the vocabulary; the corresponding words are generated using beam search; the newly generated words are appended to the sequence as input for the next step; the above steps are repeated until a complete word sequence is generated or the maximum generation length is reached, and finally a natural language description is generated.

[0042] Specifically, in step S23, generating a virtual driving trajectory within the coverage area for each of the multiple view images includes: Step S231: Create a realistic and accurate digital environment using remote sensing data.

[0043] In this step, the geographical boundaries of the virtual trajectory generation (i.e., the "remote sensing data coverage area") need to be clearly defined first. Then, multi-source remote sensing data is acquired based on image information. Specifically, this includes acquiring true-color or false-color surface images from optical satellite / aerial imagery, acquiring high-precision digital surface models (DSM) and digital elevation models (DEM) from lidar data, and acquiring supplementary optical data from radar data. The digital surface models (DSM) and digital elevation models (DEM) can provide three-dimensional elevation information including buildings and trees, which is key to calculating slope, curvature, and field of view. After that, data preprocessing is required to improve spectral accuracy, geometric precision, and information richness.

[0044] Specifically, data preprocessing methods include radiometric calibration and atmospheric correction, geometric correction and orthorectification, and data registration and fusion. Radiometric calibration refers to using calibration coefficients (gain and offset) provided by the sensor manufacturer to perform linear transformation, converting the raw digital quantization values ​​(DN values) recorded by the sensor into physically meaningful apparent radiance or apparent reflectance. The conversion formula is as follows:

[0045] in, Represents radiance, This represents the conversion factor, used to scale DN values ​​to physical quantities such as radiance or reflectivity. These are the values ​​originally recorded by the sensor, representing the grayscale values ​​of the pixels (raw data without calibration). It is the offset, used to adjust the reference.

[0046] Atmospheric correction refers to the processing of remote sensing images using the dark pixel method to eliminate the influence of atmospheric scattering, absorption, and reflection on the reflectance signals of ground objects, thereby obtaining the true surface reflectance of ground objects to accurately reflect their spectral characteristics.

[0047] Geometric correction refers to using ground control points (GCPs) and polynomial models to correct systematic distortions caused by sensor attitude (tilt), Earth curvature, etc. Orthorectification refers to using digital elevation models (DEMs) to correct projection differences caused by terrain undulations. An image that has undergone orthorectification is called an orthophoto, and each pixel has accurate geographic coordinates, just like a map with a uniform scale.

[0048] Image registration refers to the process of selecting a large number of corresponding feature points on an image (reference image) and an image to be registered, and achieving precise alignment through polynomial transformation or triangular mesh distortion correction, so as to accurately align images from different sources, times, and angles to the same coordinate system in space. Image fusion refers to the process of using algorithms such as Brovy transform, PCA (principal component analysis) transform, and Gram-Schmidt transform to combine image data with different spatial and spectral resolutions to generate a new image with both high spatial and high spectral resolution, so as to improve the image's clarity and interpretation capabilities, and make details such as road edges and lane lines more clearly distinguishable.

[0049] Step S232: Convert the processed remote sensing data into a vector road network model that can be understood and simulated by a computer.

[0050] In this step, the first step is to extract a high-precision network and use a deep learning semantic segmentation model to perform pixel-level classification of the remote sensing image, automatically identify road areas, and then convert them into vector lines (road center lines) through post-processing. In this application, the deep learning semantic segmentation model is U-Net as an example. Before use, the deep learning semantic segmentation model needs to be trained with a large amount of labeled "image-road label" data using a deep convolutional neural network (CNN) so that the model can learn the deep abstract features of the road and enable the model to have strong recognition and generalization capabilities.

[0051] Step S233: Add simulation rules to the generated vector road network, the simulation rules including road grade, number of lanes, speed limit, traffic rules and gradient; As a further description of the above steps, the simulation rules are stored in the GIS attribute table, which are the "traffic rules" and "physical laws" of the simulation.

[0052] Step S234: Configure simulation parameters for the preset simulation model using a genetic algorithm and parameter calibration tool, and perform path planning based on the shortest time and shortest path. The simulation parameters include traffic demand, vehicle characteristics, and driving behavior patterns. This step involves using a genetic algorithm to generate paths for tens of thousands of points in batches, setting road attributes in batches, and processing the OD matrix. Specifically, this includes using scripts to transmit data estimated through traffic surveys, mobile signaling data, GPS big data, etc., to simulation software to configure simulation parameters. The OD matrix defines the number of vehicles departing from each traffic zone to other traffic zones within a specific time interval. During this process, a calibration tool can be used to automatically perform multiple iterative simulations to find the optimal parameter combination, thereby automatically adjusting model parameters (such as maximum acceleration, expected vehicle distance, etc.) to plan the shortest time and shortest path while making the simulation output results as close as possible to the real data. The calibration tool in this application uses SUMO's calibrator tool as an example.

[0053] Step S235: Output the trajectory data and load the trajectory file using a professional visualization analysis tool to overlay it onto the remote sensing image to obtain a virtual driving trajectory.

[0054] In this step, the processed road network files, vehicle files, and configuration files are imported into microscopic traffic simulation software (such as SUMO, VISSIM, Aimsun). The simulation engine then proceeds at a fixed time step (e.g., 0.1 seconds) so that, based on all configured models and rules, the engine calculates the acceleration, velocity, and new position (x, y) of each vehicle within each time step until the simulation ends. After the simulation, detailed trajectory data for each vehicle is output, including vehicle ID, timestamp, planar coordinates, elevation, velocity, acceleration, and lane ID.

[0055] Please see Figure 1-3 As shown, in step S3, the step of obtaining multi-layer feature self-attention for the corresponding modality by encoding different modal data, and performing mutual attention feature fusion of different modal data based on multi-layer feature self-attention to obtain the updated feature representation includes: Step S31: Encode the multi-source modal data acquired based on a single high-resolution image, with frame encoding used for the multi-view images, pose encoding used for the virtual vehicle trajectory within the remote sensing data coverage area, and CLIP encoding used for the embedded features of the scene text description.

[0056] In this step, the frame encoding of the multi-view image is suitable for converting the original image into another smaller data format through a compression algorithm. The process is as follows: ① Divide the frame-coded image into 16×16 pixel macroblocks. Then, based on the richness of the image details, each macroblock can be further divided into smaller subblocks. The more detailed and complex the motion, the smaller and finer the subblocks should be, so as to process them more accurately. ② Based on the values ​​of the pixels of the adjacent encoded blocks above and to the left of the current frame image, the value of the current block is predicted through a specific directional pattern (such as horizontal, vertical, diagonal, etc.) to eliminate redundancy in the image space; ③ Search between the block in the current frame and the reference frame to find the best matching region; the encoder will calculate a motion vector to describe where the block moved from in the reference frame to the current position, in order to eliminate temporal redundancy in the video sequence. ④ Subtracting the predicted block from the original block yields the residual block; the more accurate the prediction, the smaller the residual value, the less data redundancy, and the better for subsequent compression. ⑤ Use Discrete Cosine Transform (DCT) to concentrate the image energy into the low-frequency region. After the transformation, the upper left corner of the matrix contains low-frequency coefficients (high energy, most important) and the lower right corner contains high-frequency coefficients (low energy, usually details and noise, not important). This achieves the effect of transforming the residual block from the spatial domain to the frequency domain, rearranging the information and making it easier to compress. ⑥ Use algorithms like CAVLC or CABAC to convert the quantized coefficients (as well as motion vectors, prediction modes, and all other information) into the final binary bitstream to eliminate statistical redundancy and further compress the data, so that the compressed data (bitstream) can be output or transmitted.

[0057] In this step, the pose encoding of the virtual vehicle trajectory within the remote sensing data coverage area is suitable for converting a high-precision, continuous pose sequence into a compact, discrete data representation, thereby saving storage space, improving transmission efficiency, and facilitating processing. The process is as follows: ① Use filtering algorithms (such as Kalman filtering, low-pass filtering, particle filtering) to remove noise and outliers from the output data of step 235; ensure the physical rationality of the trajectory, avoid sudden changes in acceleration and curvature, and make the motion smoother; ② Determine a unified global reference coordinate system and transform all pose data to this unified coordinate system to ensure data consistency; ③ While preserving the trajectory shape and kinematic characteristics as much as possible, keyframe extraction and uniform resampling are used to reduce the number of data points that need to be processed. The keyframe extraction method retains more points in areas of rapid trajectory change (such as curves) and fewer points in areas of gentle change (such as straight lines) based on indicators such as curvature and speed changes, so as to significantly reduce the amount of data without significantly losing accuracy. The uniform resampling method refers to discarding a large number of intermediate points in the processing of simple trajectory shapes such as straight road segments, and retaining only a few points at fixed time intervals to define the path, so as to make the data more regular. ④ Use a parametric mathematical model to fit the whole or segmented trajectory. This application adopts a segmented parameterization method, which divides the whole trajectory into multiple segments, fits each segment with a low-order polynomial, and needs to ensure that these segments are smoothly connected at the connection points. For example, a smooth curve is generated by using a series of control points and polynomial parameters through cubic splines or B-splines. ⑤ The encoded digital data is packaged into a storable or transmittable binary stream or file using precision quantization and compression. The quantization process is as follows: First, determine the possible range of each variable to be quantized and determine the required precision according to the application requirements. Then, convert each floating-point number into an integer. After that, add a text header and metadata to each encoded file. Finally, select a sequence number format and package it. The above sequence number format is one of the following: custom binary format, Protocol Buffers (Protobuf), JSON / XML, and a specific standard format. Compression refers to the application of a general compression algorithm (such as Delta Encoding + ZigZag + entropy encoding) to perform secondary compression on the quantized integer sequence.

[0058] In this step, CLIP encoding of the scene text description embedding features is used to convert a text describing the scene into a high-dimensional numerical vector that can represent its semantic information. The specific process is as follows: ① Configure the Python environment and install dependent libraries (such as openai-clip or transformers library) and deep learning frameworks (such as PyTorch or TensorFlow). ② Load the CLIP model and its corresponding preprocessing functions; ③ Import the text file into the CLIP model, so that the CLIP model converts the text into a format called "tokens"; ④ Input the tokenized text into the text encoder of the model to obtain the normalized feature vector.

[0059] Step S32: Based on the preset mutual attention model, perform multi-layer feature self-attention information fusion of different modal data to obtain the updated feature representation; In this step, the features of different modalities are concatenated at the input level to form a unified, hybrid representation, which is then fed into the model (using Transformer as an example in this application) to automatically learn all the complex relationships within and between all modalities.

[0060] ① The three modal information representations are converted into feature sequences, and the feature sequences of the three modalities are simply concatenated end to end to form a long, mixed multimodal sequence; ② Add modality type embedding to enable the model to distinguish different modalities, and add positional encoding to add a standard positional code to each position of the constant sequence to inject sequence information; so as to facilitate the generation of the final input representation; ③ Input the prepared mixed sequence into a standard Transformer encoder consisting of multiple layers to achieve fully arbitrary, dense cross-modal interaction; ④ After processing by L layers of Transformers, the final output sequence is obtained. Each vector in this sequence is a "multimodal vector"; it no longer purely represents the original modal information, but contains relevant information from all other modalities.

[0061] Please see Figure 1-4 As shown, in step S4, the process of obtaining a dense depth map and a high-precision 3D point cloud with GS flow embedding features by processing the new feature representation under the pose using the DPT model includes: Step S41: Use the DPT model to read the new feature representation from step S3, and obtain a dense depth map through encoding fusion; In this step, the feature representation sequence obtained in step S3 above is input into the ViT encoder of the DPT model to read all the multi-scale features. The ViT encoder is composed of multiple Transformer blocks stacked together. In this process, not all Transformer blocks will be used. Usually, 4 to 5 representative layers are selected. These layers are evenly distributed or strategically distributed in the encoder. The corresponding initial feature vector is extracted from the output of each selected target layer. A small fully connected layer (Linear Layer) is used to project each initial feature vector to a uniform number of channels (D). Each vector with a uniform number of channels is regarded as a pseudo-feature map. Then, deconvolution or bilinear upsampling is used to upsample each pseudo-feature map to a specific spatial size, thereby forming multiple feature maps with different spatial resolutions but rich information. This set of feature maps is then used to construct the multi-scale feature pyramid required by the decoder. Finally, a simple convolutional layer (usually a 1x1 convolution) is used as the prediction head to act on the feature vector of each pixel, mapping the high-dimensional features to a scalar value, namely the predicted depth value of the pixel, and outputting a dense depth map.

[0062] Step S42: Convert each pixel and its depth value in the optimized depth map into a 3D point in world coordinates through back projection to obtain the initial dense point cloud; In this step, the essence of generating point clouds is to calculate the 3D coordinates P of each pixel (u,v) in the world coordinate system in order to achieve alignment of point clouds from multiple perspectives.

[0063] In the depth map, the depth value of each pixel is d=D(u,v), where D is a hyperparameter representing the number of special channels; the formula for calculating the 3D coordinates of a pixel in the world coordinate system is as follows:

[0064] in, It is the inverse of the camera intrinsic parameter matrix. These are the homogeneous coordinates of the pixels. This is the depth value of that pixel. The superscript of T indicates the transpose sign. All of these are part of the camera's external parameters. Represents the translation vector. Represents a rotation matrix.

[0065] Step S43: Use a point cloud optimization algorithm to denoise and simplify the initial dense point cloud to obtain the final dense point cloud.

[0066] In this step, the point cloud optimization algorithm of this application includes a denoising algorithm and a simplification algorithm. The denoising algorithm is one or a combination of statistical filtering algorithm, radius filtering algorithm, Gaussian filtering algorithm and bilateral filtering algorithm, while the simplification algorithm is one of voxel grid filtering algorithm, curvature sampling algorithm and feature preservation sampling algorithm. Step S44: Based on the Transformer decoder, infer the complete geometric and semantic properties of the 3D space, thereby transforming 3D points into data with optimizable features and a high-dimensional, learnable feature vector, which is the GS stream embedding feature. In this step, after obtaining the 3D coordinates in the world coordinate system, the Transformer decoder calculates the correlation between the 3D points and each element in the input sequence (these elements are associated with image locations, text words, and trajectory points) to obtain the GS stream embedding features corresponding to the 3D points, thereby enriching the point cloud.

[0067] Please see Figure 1-5 As shown, in step S5, the generation of multi-view images and high-quality 3D assets observed by the same sensor configuration in the next pose, based on the dense depth map and high-precision 3D point cloud from step S4, through a diffusion model, includes: Step S51: Generate a 3D Gaussian model based on the dense depth map and the high-precision 3D point cloud with GS flow embedding features from step S4.

[0068] In this step, a viewpoint is first selected from the initial remote sensing image and the multi-view images it produces, or a new viewpoint is randomly generated within a reasonable range. A differentiable renderer is used to render a 2D image from this viewpoint based on the current 3D representation and appearance parameters. The rendered image is compared with the pre-selected image, and the rendering loss and regularization loss are calculated. The gradient of the total loss with respect to all optimizable parameters is calculated. Since the renderer is differentiable, this gradient can propagate backward from the pixel space of the 2D image to the geometric and appearance parameters in the 3D space. An optimizer (such as Adam) can be used to update the parameters (position, shape, color, transparency) based on the gradient, so that when the loss function converges, the final optimized high-precision 3D asset, i.e., a high-precision 3D Gaussian model, can be output.

[0069] As a preferred embodiment of the above scheme, the rendering loss includes RGB L1 / L2 Loss (a loss function used to measure the difference between the predicted and actual values ​​of an RGB image), perceptual loss (used to compare whether the high-level semantic features of two images are similar), and adversarial loss (referring to the introduction of a discriminator to determine whether the rendered image is "real" or "generated" in order to improve the realism of the rendered image). The regularization loss includes geometric smoothness loss (preventing unnatural bumps on the surface by encouraging smooth changes in adjacent vertices or SDF values) and sparsity loss (making the surface simpler by encouraging unnecessary geometry to be pruned).

[0070] Step S52: Perform differentiable rendering using a 3D Gaussian model under the target pose to obtain the RGB image and depth information rendered by the 3D Gaussian model under that viewpoint.

[0071] In this step, the rendered RGB image and depth information provide strong geometric and color priors for the diffusion model, so that the final geometric structure and approximate color layout of the diffusion model are the same as the rendered result.

[0072] Step S53: Prepare the input conditions for the diffusion model and process the control conditions to obtain a noisy latent representation.

[0073] In this step, the preparation of input conditions for the diffusion model first requires assembling a conditional information package as an instruction to control the generation process of the diffusion model. This conditional information package includes at least the rendered RGB image and depth information, camera parameters (target pose, intrinsic parameters), and may also include text prompts and initially selected images. By inputting the above conditions into the control network of the diffusion model, the 3D Gaussian model can strictly adhere to these conditions in each denoising step of the generation process.

[0074] In this step, the control conditions are first processed by encoding the rendered 2D image (or the RGB image rendered by the 3D Gaussian model) of the new viewpoint selected in step S51 into the latent space of the diffusion model to obtain a noisy latent representation. Then, all control conditions (camera parameters and the RGB image and depth map rendered by the 3D Gaussian model) are input into the control network (such as ControlNet or multiple T2I-Adapters) to generate a series of feature maps and attention biases. These feature maps encode conditional information (e.g., edges of the depth map, color blocks of the rendered image), and these generated feature maps are added or connected to various skip connections and intermediate layers of UNet. The attention biases modify the cross-attention mechanism in UNet, making UNet pay more attention to the conditional information, thereby realizing the processing of control conditions.

[0075] Step S54: Refine and upsample the noisy latent representation to obtain the refined image.

[0076] In this step, UNet propagates forward using the current noisy latent image and the time step embedding as input. UNet continuously fuses with feature maps from the control network. During this process, each layer of UNet not only acquires the noisy latent image but also the geometric and color blueprints provided by 3DGS, which continuously guides the inference direction of UNet until it finally predicts the noise that should be removed from the current latent image. Then, based on the sampler algorithm of the diffusion model (such as DDIM), a more denoised latent image is calculated using the predicted noise, thereby realizing the latent image update iteration until the time step converges, resulting in a clean latent representation. Finally, the VAE decoder is used to convert the above clean latent representation back to the pixel space to obtain the refined image.

[0077] Step S55: Repeat steps S52, S53 and S54 multiple times to obtain images from different perspectives after refinement.

[0078] Step S56: Optimize the 3D Gaussian model again using the newly refined images from multiple different perspectives until the 3D Gaussian model and the diffusion model converge to an optimal state, resulting in highly consistent multi-view images and detailed, high-quality 3D assets.

[0079] In this step, the 3D Gaussian model provides more precise geometric constraints for the diffusion model, and the diffusion model provides higher quality and more consistent training images for the 3D Gaussian model. Ultimately, the diffusion model generates highly consistent multi-view images, and the 3D Gaussian model reconstructs a detailed and high-quality 3D asset.

[0080] Please see Figure 1-6 As shown, in step S6, the process of using a recurrent neural network to process the multi-view images and 3D assets from step 5 to obtain consistent temporal trajectory images and laser point cloud data includes: Step S61: Input images from different perspectives at a certain time point into the pre-trained CNN, obtain the output of the last convolutional layer or global average pooling layer of the CNN, and obtain a feature vector of fixed length.

[0081] In this step, each time step is labeled with several feature vectors. The number of feature vectors is the same as the number of viewpoints. These feature vectors are highly abstract representations of the image, containing its key information, while greatly reducing the data dimensionality, which is convenient for RNN processing.

[0082] Step S62: Use an attention mechanism to weightedly fuse features from different perspectives at a certain time point to obtain a single, highly representative joint feature vector that can represent that time point. Repeat the above operation to process all time points to obtain a time series.

[0083] In this step, the features of each viewpoint at a certain time point are first projected through different linear layers based on linear projection to generate the Query, Key, and Value vectors required by the attention mechanism (the Query is a learnable vector, and the Key and Value are generated from the viewpoint features). The similarity between the Query and each Key is calculated using the dot product method, and the scores are normalized to a probability distribution using the Softmax function to obtain the attention weights (the weights represent the importance of the corresponding viewpoint to the current task). Then, the attention weights are used to perform a weighted summation of all Value vectors to obtain the final fused feature. The above operation is repeated until a fused feature is generated at each time point. Then, the fused features generated at each time step are collected in chronological order to form a complete sequence tensor, which perfectly meets the input requirements of RNN.

[0084] Step S63: Input the time series obtained in step S62 into a preset RNN unit (LSTM variant or GRU variant) so that the RNN updates the hidden state A at each time point in sequence to obtain a hidden state A sequence.

[0085] In this step, the input data for the current time step (the joint feature vector in step 62) needs to be sliced ​​from the input sequence. At the same time, the hidden state A of the previous time step is extracted (a mathematical vector that encodes the compressed information and contextual relationships of all input data processed by the RNN from the beginning of the sequence to the previous time step). Then, the input data and the hidden state A of the previous time step are input into the RNN unit to update the internal state of the current time step. After the entire sequence is processed, a sequence of hidden states A can be obtained.

[0086] Step S64: Map the abstract features in the RNN hidden space to the real output space through the neural network layer to obtain the trajectory prediction sequence, and output the time-series trajectory image based on the trajectory prediction sequence through the standard scientific plotting library.

[0087] In this step, the neural network layer is a fully connected neural network used for decoding, so as to perform a linear transformation to map the high-dimensional hidden state A of each time step to the low-dimensional trajectory coordinates, thereby realizing the decoding process, so as to collect the decoding output of each time step in chronological order and form a trajectory prediction sequence.

[0088] In this step, the standard scientific plotting library is the Matplotlib module in Python, which allows for the direct generation of an image depicting the entire time series trajectory.

[0089] Step S65: Process the current sensor pose and hidden state B in each time step through a preset RNN unit (LSTM variant or GRU variant), output the new hidden state B and the predicted pose increment to realize the beam pose update iteration, and generate a continuously changing laser beam pose sequence during the iteration process.

[0090] This step also includes setting the initial attitude of the virtual LiDAR and the initial hidden state B of the RNN. The initial attitude of the virtual LiDAR and the initial hidden state B of the RNN are the inputs in the first time step processing of the preset RNN unit. The attitude of the LiDAR includes the azimuth angle and the elevation angle. The initial hidden state B of the RNN is a compressed, dense vector representation that encodes all relevant information in the entire laser beam scanning history up to the current time step.

[0091] Step S66: Construct a laser ray based on the attitude of the lidar within the current time step, and simultaneously set the sampling range and the initial settings of the laser ray.

[0092] In this step, laser ray construction includes converting the lidar's attitude into a unit direction vector and defining the ray equation. The unit direction vector is determined by the lidar's scanning mechanism (e.g., a rotating mirror) and is typically calculated using azimuth (az) and elevation (zl). Setting the sampling range refers to limiting reasonable maximum and minimum detection distances. The initial settings for the laser beam refer to the settings of parameters such as cumulative transparency (T), step distance (l), final color (C), and final depth (D).

[0093] The formula for transforming the unit direction vector is as follows:

[0094]

[0095]

[0096] Where d represents the unit direction vector of the laser beam. It is the length of the projection of vector d onto the X-axis of the world coordinate system. It is the length of the projection of vector d onto the Y-axis of the world coordinate system. It is the length of the projection of vector d onto the Z-axis of the world coordinate system. Represents azimuth. Let represent the azimuth angle of the laser beam in the i-th time step. Represents pitch angle, This represents the elevation angle of the laser beam within the i-th time step.

[0097] The ray equation is as follows:

[0098] in, It is a vector-valued function representing a ray function, where o is the origin of the ray, representing the 3D position of the optical center of the lidar sensor in the global coordinate system (world coordinate system), l is the ray parameter or distance parameter, representing the precise distance from the lidar to the surface of the object being measured, and d is the ray direction vector, representing the direction of a single laser beam in three-dimensional space.

[0099] Step S67: Perform voxel rendering along the ray.

[0100] In this step, the first step is to perform a stepping motion, which means starting from the minimum detection distance and moving in steps of a certain size. l moves in the direction of maximum detection distance, and calculates the sample point P on the ray after each step. Simultaneously, spatial acceleration structures such as KD-Tree or BVH are used to quickly find Gaussians whose center points are within a certain neighborhood of sample point P, avoiding traversing every Gaussian in the scene. For each neighboring Gaussian, the probability density function value PDF of sample point P under the Gaussian distribution is calculated. Then, the contributions of all neighboring Gaussians are mixed to calculate the blending opacity δ of the current sample point P. At the same time, the colors of the neighboring Gaussians are mixed to obtain the color C of the current sample point P. Finally, the probability density of the current sample point being the location where the laser beam is ultimately reflected or hit is calculated. This continues until the transmittance falls below a threshold or the step distance reaches the set maximum detection distance, ultimately yielding a set of discrete sampling points. and their corresponding weight values , where k is the sequence of the number of sampling points.

[0101] The method for calculating the Gaussian center point in the above content is as follows:

[0102] in, Let be the mean of the j-th Gaussian distribution, and represent the world coordinates of the center point of the j-th Gaussian ellipsoid. Represents a row vector. These are the components of the vector along the X, Y, and Z axes, respectively, with the superscript "T" indicating the transpose operation; The formula for calculating the probability density function (PDF) value of a sample point under a Gaussian distribution is as follows:

[0103] in, Let be the value of the probability density function at point P, representing the relative probability density of point P belonging to the j-th Gaussian distribution; The formula for calculating the blending opacity δ is as follows:

[0104] in Let the opacity (or "contribution") learned by the j-th Gaussian be denoted by itself. Let be the effective opacity of the j-th Gaussian at the current sample point; The formula for calculating the color C of the current sample point is as follows:

[0105] in, Let be the color of the j-th Gaussian sample as viewed from the current ray direction. The weighted contribution of the j-th Gaussian pair to the final color at point P. It is a very small positive number to prevent division by zero.

[0106] The cumulative effect of colors satisfies the following conditions:

[0107] The attenuation of transparency satisfies the following condition:

[0108] The formula for calculating the contribution weight of the current sampling point to the final hit point is as follows:

[0109] Contribution weight This represents a small interval near the sampling point P(l) on the ray. Within this point, the probability density is the location where the laser beam is ultimately reflected or struck.

[0110] Step S68: Based on the weight distribution function calculated along the ray in step S67 (l) Determine an optimal depth estimate. And calculate the world coordinates and reflection intensity of that point.

[0111] In this step, the calculation process for the time coordinate is as follows: First, calculate the sum of all weights. Then perform the desired depth Calculations can then be performed, and the desired depth can then be utilized. Calculate 3D coordinates If during this process If the value is very small, it is considered an invalid ray and subsequent calculations are skipped.

[0112] In this step, the calculation process for reflection intensity is as follows: the cumulative color value C is an RGB vector (R, G, B). First, the RGB colors need to be converted to grayscale values, and then the reflection intensity estimate is obtained. Then, the surface normal n and the incident angle are estimated. After cosine calculation, the incident angle effect can be simulated using Lambert's cosine law to obtain a high-fidelity scalar of reflection intensity. .

[0113] All weights and The calculation formula is as follows:

[0114] Expected depth The calculation formula is as follows:

[0115] 3D coordinates The calculation formula is as follows:

[0116] In the above formula, i represents the laser beam being processed. Let d be the desired depth and d be the direction vector.

[0117] Reflection intensity estimation The acquisition method is as follows:

[0118] The coefficients in the above formula are derived from the ITU-R BT.601 standard. The human eye is most sensitive to green light, so the green channel has the highest weight (0.587); followed by red light (0.299); and least sensitive to blue light (0.114).

[0119] The surface normal n is obtained as follows: Analyze the Gaussians that contribute the most to color; the eigenvectors of their covariance matrices suggest their orientation. Typically, the eigenvector corresponding to the smallest eigenvalue can be approximated as the surface normal at that point. We then take a weighted average of the Gaussian normals that contribute to all the major contributions to obtain the final normal estimate n.

[0120] Angle of incidence It is the angle between the surface normal n and the laser beam direction d, and its cosine value. The formula for applying the Lambert factor modulation intensity is as follows:

[0121] In the above formula, It is the dot product of the laser beam direction vector d and the surface normal vector n, and since d and n are direction vectors (of length 1), therefore... ,and This means taking the maximum of the dot product result and zero, to avoid... It is a negative number, in addition, By combining inherent properties with observational geometry, a physical error is eliminated, therefore As the final strength value .

[0122] Step S69: Use a preset noise model to convert the original point sequence into high-fidelity, directly usable virtual laser point cloud data.

[0123] In this step, the preset noise models include distance noise model, beam divergence noise model and intensity noise model.

[0124] Ranging error is typically related to the signal-to-noise ratio (SNR) of the received signal, and SNR is inversely proportional to the square of the distance. Therefore, the range noise model needs to satisfy the following formula:

[0125] in, is the standard deviation of the distance noise, and f is a coefficient determined based on the specific performance of the lidar. The larger the value of f, the lower the accuracy of the sensor. It is the original ideal distance value of the current point (the expected depth in step 68); The application of distance noise first requires starting with a mean of 0 and a standard deviation of... Sample a noise value from a Gaussian distribution Then, a noise value is applied, and the noise application formula is as follows:

[0126] in, The noise value is given, and the noise value follows a normal distribution. .

[0127] The laser beam is not an ideal line, but has a small divergence angle. This means that the distance it measures is the average distance of several surface points within the beam's coverage area, which can lead to positioning ambiguity, especially at the edges. Therefore, the construction of the beam divergence noise model needs to satisfy the following formula:

[0128] in, To understand distance The radius of the laser beam represents the maximum possible horizontal displacement range of the current point. The total divergence angle of the lidar beam (in radians rad) is an inherent parameter of the sensor and can usually be found in its datasheet.

[0129] Applying beam divergence noise first requires generating a random offset. This value can be sampled from a uniform distribution from 0 to r, and Meet the following requirements Then, a random angle is collected as the offset direction. This angle can be sampled from a uniform distribution between 0 and 2π, and meets the following usage requirements. Then the random offset needs to be set. and offset direction Combined into 2D offset ( , Then, the 2D offset is transformed to the world coordinate system to obtain the offset vector ΔP, and finally, beam divergence noise is applied according to the coordinate offset formula.

[0130] 2D offset ( , The calculation formula is as follows:

[0131]

[0132] The coordinate offset formula is as follows:

[0133] Wherein, the offset vector ΔP is the 2D offset ( , The result is obtained by transforming it to the world coordinate system.

[0134] The intensity value is affected by photon shot noise and electronic thermal noise, resulting in slight random fluctuations. Therefore, the intensity noise model needs to satisfy the following formula:

[0135] in, It is a random noise value. It is the standard deviation of intensity noise, representing the intensity and amplitude of the noise. The value of is a constant that needs to be set. This value can be determined by fitting the intensity fluctuations of real radar data, and it usually ranges from 0.01 to 0.05.

[0136] Intensity noise The application of refers to the generation of random noise. Add to original strength value The application of intensity noise must satisfy the following formula:

[0137] in, This refers to the final strength value obtained in step S68; After applying the intensity noise, it is also ensured that the intensity value after adding noise still falls within the valid physical range. (This is followed by a seemingly unrelated sentence about updating the intensity value.) Exceed After determining the range, perform a cropping operation according to the formula below:

[0138] The above formula can exceed The values ​​after the range are pulled back to 0 or 1 to ensure the validity of the data.

[0139] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The computer program is executed by the processor to perform the above-described method for generating scene and sensor data based on remote sensing images.

[0140] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of this disclosure, and all such changes and modifications will fall within the scope of protection of this invention.

Claims

1. A method for generating scene and sensor data based on remote sensing imagery, characterized in that, include: Step S1: Acquire several single high-resolution images using drone aerial photography data or short-cycle high-resolution optical remote sensing satellite data; Step S2: Obtain multi-source modal data based on a single high-resolution image. The multi-source modal data includes embedded features of multi-view images, virtual driving trajectories within the coverage area of ​​remote sensing data, and scene text descriptions. Step S3: Obtain multi-layer feature self-attention for the corresponding modality by encoding data of different modalities, and perform mutual attention feature fusion of data from different modalities based on multi-layer feature self-attention to obtain the updated feature representation; Step S3 includes: Step S31: Encode the multi-source modal data acquired based on a single high-resolution image, with frame encoding used for the multi-view images, pose encoding used for the virtual vehicle trajectory within the coverage area of ​​the remote sensing data, and CLIP encoding used for the embedded features of the scene text description. Step S4: Process the new feature representation under this pose using the DPT model to obtain a dense depth map and a high-precision 3D point cloud with GS flow embedding features; Step S5: Based on the dense depth map and high-precision 3D point cloud in step S4, generate multi-view images and high-quality 3D assets observed by the same sensor configuration in the next pose using a diffusion model; Step S6: Process the multi-view images and high-quality 3D assets from Step 5 using a recurrent neural network to obtain consistent temporal trajectory images and laser point cloud data; Step S6 specifically includes: Step S61: Input images from different perspectives at a certain time point into the pre-trained CNN, obtain the output of the last convolutional layer or global average pooling layer of the CNN, and obtain a feature vector of fixed length. Step S62: Use an attention mechanism to weightedly fuse features from different perspectives at a certain time point to obtain a single, highly representative joint feature vector that can represent that time point. Repeat the above operation to process all time points to obtain a time series. Step S63: Input the time series obtained in step S62 into the preset RNN unit so that the RNN updates the hidden state A at each time point in sequence to obtain a hidden state A sequence; Step S64: Map the abstract features in the RNN hidden space to the real output space through the neural network layer to obtain the trajectory prediction sequence, and output the time-series trajectory image based on the trajectory prediction sequence through the standard scientific plotting library; Step S65: Process the current sensor pose and hidden state in each time step through the preset RNN unit, output the new hidden state B and the predicted pose increment, so as to realize the beam pose update iteration and generate a continuously changing laser beam pose sequence during the iteration process. Step S66: Construct a laser ray based on the attitude of the lidar within the current time step, and simultaneously set the sampling range and the initial settings of the laser ray; Step S67: Perform voxel rendering along the ray to obtain a set of discrete sampling points and corresponding weight values; Step S68: Based on the weight distribution function obtained in step S67, determine an optimal depth estimate and calculate the world coordinates and reflection intensity of the point; Step S69: Use a preset noise model to convert the original point sequence into high-fidelity, directly usable virtual laser point cloud data.

2. The method for generating scene and sensor data based on remote sensing imagery according to claim 1, characterized in that, In step S2, acquiring multi-source modal data based on a single high-resolution image includes: Step S21: Generate multi-view images based on a single high-resolution image; Step S22: Generate embedded features of scene text descriptions for each view based on the multi-view image; Step S23: Generate a virtual driving trajectory within the coverage area for each of the multi-view images.

3. The method for generating scene and sensor data based on remote sensing imagery according to claim 2, characterized in that, In step S21, generating multi-view images based on a single high-resolution image includes: Step S211: Use a deep learning model to predict the depth information of each pixel and generate a depth map; Step S212: Reconstruct the sparse 3D geometry of the image scene; Step S213: Use the reconstructed 3D geometry to create a virtual camera and re-render the scene from any new perspective to simulate a new view of the scene from different angles.

4. The method for generating scene and sensor data based on remote sensing imagery according to claim 3, characterized in that, In step S22, the embedding features for generating scene text descriptions for each view based on the multi-view image include: Step S221: Confirm the backbone network for feature extraction and the scene description model used; Step S222: Input a single high-resolution remote sensing image and its new view into the scene description model used, so that the backbone network outputs a high-dimensional feature tensor rich in semantic information; Step S223: Use a self-attention mechanism to integrate the visual features extracted in step 222 into a structured representation rich in contextual semantic information; Step S224: The decoder is used to progressively and autoregressively convert the context-rich, fixed-size visual representation from step S223 into a word sequence, forming the final image scene text description.

5. The method for generating scene and sensor data based on remote sensing imagery according to claim 4, characterized in that, In step S23, generating a virtual driving trajectory within the coverage area for each of the multiple view images includes: Step S231: Create a realistic and accurate digital environment using remote sensing data; Step S232: Convert the processed remote sensing data into a vector road network model that can be understood and simulated by a computer; Step S233: Add simulation rules to the generated vector road network, the simulation rules including road grade, number of lanes, speed limit, traffic rules and gradient; Step S234: Configure simulation parameters for the preset simulation model using a genetic algorithm and parameter calibration tool, and perform path planning based on the shortest time and shortest path. The simulation parameters include traffic demand, vehicle characteristics, and driving behavior patterns. Step S235: Output the trajectory data and load the trajectory file using a professional visualization analysis tool to overlay it onto the remote sensing image to obtain a virtual driving trajectory.

6. The method for generating scene and sensor data based on remote sensing imagery according to claim 1, characterized in that, In step S3, the step of encoding different modal data to obtain corresponding multi-layer feature self-attention, and performing mutual attention feature fusion of different modal data based on multi-layer feature self-attention to obtain the updated feature representation, further includes: Step S32: Based on the preset mutual attention model, perform multi-layer feature self-attention information fusion of different modal data and obtain the updated feature representation.

7. The method for generating scene and sensor data based on remote sensing imagery according to claim 1, characterized in that, In step S4, the step of processing the new feature representation under this pose using the DPT model to obtain a dense depth map and a high-precision 3D point cloud with GS flow embedding features includes: Step S41: Use the DPT model to read the new feature representation from step S3, and obtain a dense depth map through encoding fusion; Step S42: Convert each pixel and its depth value in the optimized depth map into a 3D point in world coordinates through back projection to obtain the initial dense point cloud; Step S43: Use a point cloud optimization algorithm to denoise and simplify the initial dense point cloud to obtain the final dense point cloud; Step S44: Based on the Transformer decoder, infer the complete geometric and semantic properties of the 3D space, thereby transforming the 3D points into optimizable data and a high-dimensional, learnable feature vector; this feature vector is the GS stream embedding feature.

8. The method for generating scene and sensor data based on remote sensing imagery according to claim 1, characterized in that, In step S5, based on the dense depth map and high-precision 3D point cloud from step S4, the generation of multi-view images and high-quality 3D assets observed by the same sensor configuration in the next pose using a diffusion model includes: Step S51: Generate a 3D Gaussian model based on the dense depth map and the high-precision 3D point cloud with GS flow embedding features from step S4; Step S52: Perform differentiable rendering using a 3D Gaussian model under the target pose to obtain the RGB image and depth information rendered by the 3D Gaussian model under this viewpoint; Step S53: Prepare the input conditions for the diffusion model and process the control conditions to obtain a noisy latent representation; Step S54: Refine and upsample the noisy latent representation to obtain a refined image; Step S55: Repeat steps S52, S53 and S54 multiple times to obtain images from different perspectives after refinement; Step S56: Optimize the 3D Gaussian model again using the newly refined images from multiple different perspectives until the 3D Gaussian model and the diffusion model converge to an optimal state, resulting in highly consistent multi-view images and detailed, high-quality 3D assets.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the scene and sensor data generation method based on remote sensing imagery as described in any one of claims 1 to 8.

Images