A satellite-borne load system based on co-loading of synthetic aperture radar and lidar and a data fusion processing method thereof

Abstract

The application discloses a satellite load system based on a synthetic aperture radar and a laser radar and a data fusion processing method thereof, and the system comprises a satellite platform, a SAR load module and a LiDAR load module installed on the satellite platform, a time-space reference synchronization module connected to the two load modules and used for realizing time, frequency and space pointing synchronization, a data acquisition and control module used for controlling a working mode, and a satellite data processing module used for data preprocessing and fusion processing. Through the unified time-space reference, the application realizes the on-satellite cooperative observation of the SAR and the LiDAR, uses the LiDAR point cloud to assist the SAR imaging processing, uses the SAR multi-polarization characteristics to assist the LiDAR point cloud classification, and realizes the multi-modal feature fusion through a deep learning network, so that the problems of difficult cooperative observation and low fusion precision of the heterogeneous data in the prior art are solved, and the high-precision three-dimensional earth observation under all-weather conditions is realized.

Description

Technical Field

[0001] This invention relates to the field of satellite remote sensing technology, and in particular to a satellite payload system based on the co-carrying of synthetic aperture radar and lidar, and its data fusion processing method. Background Technology

[0002] In the field of Earth observation, synthetic aperture radar (SAR) satellites, as active microwave remote sensing devices, work by transmitting microwave signals to the Earth's surface. They utilize range pulse compression technology to improve range resolution and combine it with azimuth synthetic aperture technology (which uses satellite platform motion to create an equivalent large-aperture antenna) to enhance azimuth resolution, ultimately acquiring high-resolution two-dimensional radar images. Because microwave signals have strong penetrating power, effectively penetrating complex meteorological conditions such as clouds, rain, fog, and dust storms, and are not limited by day or night lighting, SAR satellites possess continuous all-weather, all-day observation capabilities. This makes them a key technology for achieving large-scale, high-resolution Earth observation and is widely used in fields such as national land surveys, marine monitoring, and military reconnaissance. However, SAR satellites use a side-looking imaging mode, and their imaging geometry is closely related to the slope of the terrain. In areas with dramatic topographic relief (such as mountains, canyons, and plateau edges), the radar echo paths of surface targets differ, which can easily lead to geometric distortions in radar images (such as perspective shrinkage, overlay, and shadow stretching). At the same time, microwave signals cannot form effective echoes in obscured areas, resulting in radar shadows. This causes the loss of intensity and phase information in that area, seriously affecting the accuracy of SAR images in topographic representation and limiting their direct application in high-precision 3D mapping scenarios.

[0003] LiDAR, another mainstream active remote sensing technology, works by emitting a high-energy, narrow-pulse laser beam towards a surface target using a laser emission module. A photoelectric detection module precisely measures the round-trip time of the laser pulse, and by combining this with parameters such as the laser beam's emission angle and the platform's position, the three-dimensional spatial coordinates (x, y, z) of the surface target are calculated using triangulation principles, forming high-density three-dimensional point cloud data. Thanks to the high collimation, high monochromaticity, and nanosecond-level pulse width of the laser beam, LiDAR's three-dimensional measurement accuracy can reach centimeter or even millimeter levels. It can accurately characterize the elevation information and spatial morphology of surface targets, giving it irreplaceable advantages in high-precision terrain modeling, 3D building reconstruction, and forest resource surveys. However, spaceborne LiDAR technology still has significant limitations: on the one hand, the manufacturing process of spaceborne LiDAR payloads is complex (such as the stringent requirements for core components like high-power laser emitters and high-precision time measurement units), resulting in high research and development and launch costs for satellite platforms; on the other hand, limited by the scanning range of the laser beam and the satellite's transit trajectory, the observation coverage area of ​​spaceborne LiDAR is distributed in a strip, resulting in discontinuous coverage and low efficiency for large-scale observations; in addition, laser signals are easily affected by attenuation from weather factors such as clouds, precipitation, and atmospheric aerosols, making it impossible to achieve effective observation under adverse weather conditions, and its adaptability to working environments is far lower than that of SAR technology.

[0004] In existing technologies, SAR and LiDAR are typically deployed as independent remote sensing payloads, carried on different satellite platforms. This independent deployment makes it difficult for them to conduct simultaneous (same time point or very short time interval) collaborative observations of the same observation area: due to differences in the transit times and orbital parameters of different satellites, the acquired SAR images and LiDAR point cloud data often exhibit significant spatiotemporal discrepancies, failing to reflect the surface condition at the same moment. To achieve complementary advantages between the two types of data, existing technologies mostly employ post-processing fusion methods, that is, after acquiring SAR images and LiDAR point clouds separately, a comprehensive observation result is generated through steps such as data registration, format conversion, and feature fusion. However, this type of post-processing solution faces multiple technical bottlenecks: First, the imaging mechanisms of SAR and LiDAR differ significantly (SAR is based on microwave scattering characteristics, while LiDAR is based on laser reflection characteristics), resulting in fundamental differences in data types (two-dimensional images vs. three-dimensional point clouds) and information representation methods (intensity / phase vs. elevation / reflectivity), increasing the difficulty of fusion; Second, the spatiotemporal references of the two types of data are not unified. The geographic coordinates of SAR images depend on orbital parameters and imaging geometric correction, while the coordinates of LiDAR point clouds are based on satellite positioning systems (such as GPS and BeiDou). The coordinate references and timestamp accuracy of the two are prone to deviation, causing registration errors; Third, due to the differences in imaging mechanisms and the deviation in spatiotemporal references, existing registration algorithms are unable to achieve high-precision feature matching, resulting in problems such as spatial misalignment, information redundancy, or missing information in the fused data, failing to fully leverage the synergistic advantages of SAR's all-weather penetration capability and LiDAR's high-precision three-dimensional measurement capability.

[0005] Therefore, how to overcome the limitations of independent deployment and post-processing fusion of SAR and LiDAR in existing technologies, achieve collaborative observation and efficient fusion of the two types of payloads, and achieve the observation goal of "one pass, multiple gains", while solving technical problems such as differences in imaging mechanisms, inconsistent spatiotemporal references, and low registration accuracy, has become a key technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] The present invention aims to provide a satellite payload system based on the co-carrying of synthetic aperture radar and lidar and its data fusion processing method, so as to solve the problems of difficult coordinated observation of SAR and LiDAR and low data fusion accuracy in the prior art, and realize high-precision three-dimensional Earth observation under all-weather conditions.

[0007] This invention provides a satellite payload system based on the co-carrying of synthetic aperture radar and lidar, comprising: The satellite platform is equipped with a SAR payload module and a LiDAR payload module. A spatiotemporal reference synchronization module is connected to the SAR payload module and the LiDAR payload module, and is used to realize time synchronization, frequency synchronization and spatial pointing synchronization between the SAR payload module and the LiDAR payload module. The data acquisition and control module is connected to the SAR payload module and the LiDAR payload module, and is used to control the working mode of the SAR payload module and the LiDAR payload module according to the observation mission requirements. The onboard data processing module is connected to the spatiotemporal reference synchronization module and the data acquisition and control module, and is used to preprocess and fuse the SAR echo data of the SAR payload module and the LiDAR point cloud data of the LiDAR payload module. The data transmission module is connected to the onboard data processing module and is used to transmit the fused data to the ground station.

[0008] In one embodiment of the present invention, the SAR payload module includes: The SAR antenna subsystem includes an active phased array antenna for multi-polarization operation. SAR transmit / receive subsystem, used to transmit microwave pulses and receive ground echo signals; The SAR imaging processing unit is used to perform range compression and azimuth synthetic aperture processing on ground echo signals to generate SAR complex image data. The SAR payload module operates in the X-band, C-band, or L-band and has multiple imaging modes.

[0009] In one embodiment of the present invention, the LiDAR payload module includes: Laser emitters, including high-repetition-rate lasers, are used to emit laser pulses in the near-infrared band; The scanning mechanism is configured to achieve cross-track wide-swath scanning via a tilting mirror or fiber optic scanning method; The echo detection and timing unit is configured to measure the round-trip time of a laser pulse using single-photon detection or multi-channel parallel detection techniques. The point cloud generation unit is configured to combine satellite orbital attitude data to generate geocoded 3D point cloud data.

[0010] In one embodiment of the present invention, the spatiotemporal reference synchronization module includes: The time synchronization unit is configured to achieve timing alignment between SAR transmission pulses and laser transmission pulses by triggering pulse signals; The frequency synchronization unit provides a unified frequency reference for the SAR transmit / receive subsystem and the laser timing unit through a shared frequency reference source, thereby eliminating measurement errors introduced by frequency drift. The space synchronization unit is configured to adjust the platform attitude through the satellite attitude control system based on the pointing of the active phased array antenna of the SAR payload module and the instantaneous field of view of the scanning mechanism, so as to ensure that the laser footprint falls within the SAR imaging coverage area and achieve coordinated airspace coverage.

[0011] In one embodiment of the present invention, the operating mode includes: The SAR payload module and the LiDAR payload module operate independently according to their respective preset parameters; According to the requirements of the observation mission, control the two payload modules to conduct synchronous observations of the same target area; The target is identified in real time using SAR wide-area imaging results, triggering the LiDAR payload module to scan the target area and acquire three-dimensional information.

[0012] In one embodiment of the present invention, the onboard data processing module includes: SAR data preprocessing unit, which is configured to generate single-look complex images; The LiDAR data preprocessing unit is configured to generate a 3D point cloud. The joint localization and registration unit is configured to perform pixel-level registration between SAR imagery and LiDAR point clouds. The data fusion processing unit is configured to perform joint processing based on the registered multi-source data.

[0013] In one embodiment of the present invention, the data fusion processing unit includes: The terrain-aided SAR imaging module is configured to generate digital elevation models from LiDAR point clouds for geocoding, orthorectification, terrain-adaptive matched filtering, or terrain phase removal for interferometric measurements of SAR images. The LiDAR point cloud classification enhancement module is configured to perform ground feature classification and recognition by fusing the multi-polarization features of SAR images and the three-dimensional geometric features of LiDAR point clouds through a deep learning network. The joint 3D reconstruction module is configured to fuse coherent information from SAR with geometric information from LiDAR to construct a 3D surface model.

[0014] In one embodiment of the present invention, the onboard data processing module further includes a deep learning-based intelligent fusion module, the intelligent fusion module comprising: The shallow feature extraction submodule is configured to extract features from SAR images and LiDAR point cloud projection feature maps. The deep feature extraction submodule is configured to extract multi-scale deep features through a residual network structure. A dual-domain feature fusion submodule is configured to perform feature fusion alternately in the frequency domain and the spatial domain; The reconstruction submodule is configured to reconstruct a 3D surface product based on fusion features.

[0015] The present invention also provides a data fusion processing method based on the above system, comprising: The time, frequency, and space synchronization of the SAR payload module and the LiDAR payload module are achieved through the spatiotemporal reference synchronization module. The SAR payload module and the LiDAR payload module are controlled to conduct coordinated observations of the same target area to acquire raw echo data. The raw data is preprocessed to generate SAR single-view complex images and LiDAR 3D point clouds; Fine-register SAR single-view complex images with LiDAR 3D point clouds; The registered multi-source data is then fused to generate a fused product.

[0016] In one embodiment of the present invention, the fusion processing of the registered multi-source data includes: Digital elevation models generated from LiDAR 3D point clouds are used to assist in the imaging processing of SAR single-look complex images; or Utilizing the multi-polarization features of SAR single-view complex images to assist in the classification and recognition of LiDAR 3D point clouds; or Joint 3D reconstruction is performed by fusing coherent information from SAR with geometric information from LiDAR.

[0017] The present invention has the following beneficial effects: (1) By using the time and space reference synchronization module, the problem of unifying the time, frequency and space reference of SAR and LiDAR heterogeneous data was solved, laying the foundation for subsequent high-precision fusion and overcoming the defect that traditional satellite-separated observations are difficult to achieve simultaneous phase observation.

[0018] (2) High-precision DEM generated by LiDAR point cloud is used to assist SAR imaging processing, which effectively eliminates geometric distortion and shadow effects caused by terrain, and significantly improves the SAR image quality of mountainous areas.

[0019] (3) By combining the all-weather penetration capability of SAR with the high-precision ranging capability of LiDAR, high-precision three-dimensional surface information can be obtained even under cloud and rain conditions, solving the observation limitations of a single sensor in complex environments.

[0020] (4) By using on-board intelligent fusion processing, the amount of data transmission is reduced. At the same time, the multimodal fusion method based on deep learning makes full use of the complementary characteristics of different sensors. The generated surface products are superior to single sensor products in terms of geometric accuracy and richness of attribute information.

[0021] (5) The guided observation mode realizes the organic combination of wide-area search and fine scanning, which improves the response capability to emergency observation tasks such as dynamic targets and disaster areas. Attached Figure Description

[0022] Figure 1 A block diagram of a satellite payload system based on the co-carrying of synthetic aperture radar and lidar is shown in one embodiment of the present invention; Figure 2 A schematic diagram illustrating the working principle of a time-space reference synchronization module in one embodiment of the present invention is shown. Figure 3 A flowchart of data fusion processing in one embodiment of the present invention is shown; Figure 4 The diagram illustrates a dual-domain feature fusion network architecture based on deep learning in one embodiment of the present invention. Detailed Implementation

[0023] In the following description, the invention is described with reference to various embodiments. However, those skilled in the art will recognize that the embodiments may be practiced without one or more specific details or with other alternatives and / or additional methods, materials, or components. In other instances, well-known structures, materials, or operations are not shown or described in detail so as not to obscure the inventive points of the invention. Similarly, for illustrative purposes, specific quantities, materials, and configurations are set forth to provide a comprehensive understanding of embodiments of the invention. However, the invention is not limited to these specific details.

[0024] In this invention, the various embodiments are merely intended to illustrate the solutions of the invention and should not be construed as limiting.

[0025] In this specification, references to "an embodiment" or "this embodiment" mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment of the invention. The phrase "in one embodiment" appearing throughout this specification does not necessarily refer to the same embodiment in all instances.

[0026] Furthermore, the numbering of the steps in the methods of the present invention does not limit the execution order of the method steps. Unless otherwise specified, the method steps may be executed in different orders.

[0027] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0028] Figure 1 A block diagram of a satellite payload system based on the co-carrying of synthetic aperture radar and lidar is shown in one embodiment of the present invention.

[0029] like Figure 1 As shown, in this embodiment, the satellite payload system based on the co-carrying of synthetic aperture radar and lidar includes: Satellite platform 100, carrying SAR payload module 10 and LiDAR payload module 20.

[0030] SAR payload module 10 is used to implement the transmission, reception and imaging processing of satellite SAR signals, including: SAR antenna subsystem 11, using an active phased array antenna, supporting multi-polarization working modes; SAR transmit / receive subsystem 12, used to transmit microwave pulses and receive ground echo signals; SAR imaging processing unit 13, used to perform range compression and azimuth synthetic aperture processing on the echo signals to generate SAR complex image data.

[0031] The working band of the SAR payload module 10 is X-band, C-band or L-band, and it has multiple imaging modes such as strip, scan, and spotlight.

[0032] LiDAR payload module 20 is used to implement lidar earth exploration and obtain three-dimensional point cloud data, including: Laser transmitter 21, using a high-repetition-rate laser to emit near-infrared laser pulses; Scanning mechanism 22, using a galvanometer or fiber optic scanning method to achieve cross-track wide-swath scanning; Echo detection and timing unit 23, using single-photon detection or multi-channel parallel detection technology to accurately measure the round-trip time of laser pulses; Point cloud generation unit 24, combining satellite orbit attitude data to generate geocoded three-dimensional point cloud data.

[0033] Space-time reference synchronization module 30 is used to provide a unified time, frequency and space reference for the SAR payload module 10 and the LiDAR payload module 20, achieve multi-payload synchronous observation, and interact with the satellite platform attitude control system, including: Time synchronization unit 31, using a high-stability crystal oscillator as the clock source, achieving the timing alignment of the SAR transmit pulse and the laser transmit pulse through trigger pulse signals, and the time synchronization accuracy is better than the nanosecond level; Frequency synchronization unit 32, using a shared frequency reference source to provide a unified frequency reference for the SAR transmit / receive subsystem 12 and the echo detection and timing unit 23, eliminating the measurement error introduced by frequency drift; Space synchronization unit 33, according to the antenna pointing of the SAR antenna subsystem 11 and the instantaneous field of view angle of the scanning mechanism 22, adjusts the platform attitude through the satellite attitude control system to ensure that the laser footprint falls within the SAR imaging coverage area, achieving the coordination of airspace coverage.

[0034] The data acquisition and control module 40 is used for command control and status monitoring of SAR and LiDAR payloads, realizing two-way interactive management. The data acquisition and control module 40 supports the following operating modes: Independent observation mode: SAR payload module 10 and LiDAR payload module 20 operate independently according to their respective preset parameters; Collaborative observation mode: Based on the requirements of the observation mission, control the two payload modules to conduct synchronous observations of the same target area; Guided observation mode: Real-time identification of targets of interest using SAR wide-area imaging results, triggering the LiDAR payload module 20 to perform high-precision scanning of the target area and acquire detailed three-dimensional information.

[0035] The onboard data processing module 50 is used for comprehensive preprocessing, positioning registration, and fusion processing of SAR and LiDAR data. It is the core of the system's data processing and includes: SAR data preprocessing unit 51 is configured to perform the following operations: Range and azimuth compression are performed on the raw SAR echo data to generate single-look complex images; Perform radiation calibration and coarse geometric correction to eliminate the effects of system gain and antenna pattern.

[0036] LiDAR data preprocessing unit 52 is configured to perform the following operations: Analyze the laser echo waveform to extract the ground echo signal; By combining satellite orbital attitude data, the laser ranging values ​​are converted into a three-dimensional point cloud in geocentric rectangular coordinates; Perform noise point filtering and point cloud thinning.

[0037] The joint positioning and registration unit 53 is configured to perform the following operations: Using the unified spatiotemporal reference provided by the spatiotemporal reference synchronization module, the SAR image pixels are precisely registered with the LiDAR point cloud, specifically including: Based on the SAR imaging geometric model and the LiDAR ranging geometric model, collinearity equations are established. Simulated SAR images are generated using LiDAR point clouds, and corresponding points are obtained through cross-correlation matching. The registration parameters are calculated to achieve pixel-level fine registration.

[0038] The data fusion processing unit 54 is used to perform preliminary fusion of multi-source payload data, providing a foundation for subsequent intelligent fusion.

[0039] The data transmission module 60 is connected after the onboard data processing module 50 and ultimately transmits the data to the ground station.

[0040] The system, based on satellite platform 100, integrates a SAR payload module 10 and a LiDAR payload module 20. A spatiotemporal reference synchronization module 30 ensures strict synchronization of the two payloads in time, frequency, and spatial orientation using a high-precision clock and frequency reference, and transmits the synchronization information to the onboard data processing module 50. The data acquisition and control module 40 coordinates and controls the operating modes of the SAR and LiDAR according to mission requirements. The acquired raw data is sent to the onboard data processing module 50 for preprocessing, registration, and deep fusion. Finally, the fused data product is transmitted to the ground station via the data transmission module 60. This system architecture lays the hardware foundation for real-time, high-precision fusion processing of multi-source data onboard the satellite.

[0041] Figure 2 A schematic diagram illustrating the working principle of a time-space reference synchronization module in one embodiment of the present invention is shown.

[0042] like Figure 2 As shown, in this embodiment, the working logic of the spatiotemporal reference synchronization module 30 revolves around three core dimensions: time synchronization, frequency synchronization, and spatial synchronization. Multi-dimensional synchronization accuracy is ensured through hardware link connections and closed-loop control, as detailed below: The time synchronization unit 31 is equipped with a shared atomic frequency standard (or a high-stability crystal oscillator) as the core time reference source. The frequency reference signal output by the reference source is divided into two paths: one path is transmitted to the clock input terminal of the SAR transmit / receive subsystem 12 of the SAR payload module 10, and the other path is transmitted to the clock input terminal of the echo detection and timing unit 23 of the LiDAR payload module 20, providing a common frequency reference for the time counting modules of the two payloads, and eliminating the time deviation caused by frequency drift from the source.

[0043] Meanwhile, the time synchronization unit 31 has a built-in synchronization trigger pulse generator. The synchronization trigger pulse signal generated by the generator is also divided into two paths, which are respectively pointed to the transmission pulse control terminal of the SAR transmission / reception subsystem 12 and the pulse trigger terminal of the laser transmitter 21 of the LiDAR payload module 20. The leading edges of the two pulse signals are strictly aligned. According to actual measurement, its time synchronization accuracy is better than 1ns, ensuring that the transmission timing of the SAR transmission pulse and the laser pulse is completely synchronized.

[0044] The frequency synchronization unit 32 and the time synchronization unit 31 share the same atomic frequency standard reference source. Through a dedicated frequency allocation link, the standardized frequency signal is stably transmitted to the transmit / receive link (including the SAR transmit / receive subsystem 12) of the SAR payload module 10 and the timing link (including the echo detection and timing unit 23) of the LiDAR payload module 20.

[0045] This design ensures that the phase measurement and signal modulation / demodulation process of the SAR payload and the laser round-trip time measurement process of the LiDAR payload are based on a unified frequency reference, avoiding the loss of ranging accuracy and phase measurement deviation caused by differences in frequency references, and guaranteeing the frequency consistency of the original data of the two payloads.

[0046] The core of the spatial synchronization unit 33 is the spatial synchronization calculation unit, which receives two data streams in real time: one is the SAR beam pointing data (including beam center latitude and longitude and swath width parameters) from the SAR payload module 10, and the other is the laser footprint position data from the LiDAR payload module 20 (obtained by jointly calculating the attitude parameters of the scanning mechanism 22 and the laser ranging data).

[0047] The space synchronous calculation unit compares the two sets of data in real time to determine whether the laser footprint is within the SAR imaging swath. If the laser footprint is detected to be deviating from the SAR swath area, an attitude adjustment command is immediately generated. This command is transmitted to the attitude control system of the satellite platform 100. By adjusting the satellite attitude, the laser footprint is brought back to the SAR imaging swath, thus forming a space synchronous closed-loop control.

[0048] In this embodiment, the spatiotemporal reference synchronization module 30 constructs a three-in-one synchronization system of "time-frequency-space" through the collaborative work of the time synchronization unit 31, the frequency synchronization unit 32 and the spatial synchronization unit 33. This provides a unified spatiotemporal reference framework for the observation data of SAR and LiDAR payloads, effectively supporting the subsequent onboard data processing module 50 to perform pixel-level multi-source data fusion, and significantly improving the spatiotemporal consistency and accuracy of the fused data products.

[0049] Figure 3 A flowchart of data fusion processing in one embodiment of the present invention is shown.

[0050] In this embodiment, the data fusion processing unit 54 includes: The terrain-aided SAR imaging module 541 is configured to generate a digital elevation model from a LiDAR point cloud for geocoding, orthorectification, terrain-adaptive matched filtering, or terrain phase removal for interferometric measurements of SAR images. The LiDAR point cloud classification enhancement module 542 is configured to perform ground feature classification and recognition by fusing the multi-polarization features of SAR images and the three-dimensional geometric features of LiDAR point clouds through a deep learning network. The joint 3D reconstruction module 543 is configured to fuse coherent information from SAR with geometric information from LiDAR to construct a 3D surface model.

[0051] like Figure 3As shown, firstly, the raw SAR echo data and LiDAR full waveform data are preprocessed to generate SAR single-look complex images and LiDAR 3D point clouds. Then, the joint positioning and registration unit utilizes a unified spatiotemporal reference and matching technology based on point cloud-simulated SAR images to achieve pixel-level precise registration of SAR and LiDAR data. Based on this, the workflow is divided into three paths for deep fusion; the three processing paths can be executed in parallel or selected individually, ultimately converging to form the final fused data product. This workflow fully leverages the complementary advantages of multi-source data. The specific processing steps are as follows: Input layer: Input the raw data sources, including: SAR raw echo data, LiDAR full waveform data, and satellite orbital attitude data.

[0052] Preprocessing layer: 1) The SAR data preprocessing unit 51 receives the raw SAR echo data, processes it, and outputs a SAR single-look complex image (SLC). 2) The LiDAR data preprocessing unit 52 receives LiDAR full waveform data and satellite orbit attitude data, and outputs a three-dimensional point cloud after calculation.

[0053] Core processing layer: The joint positioning and registration unit 53 simultaneously receives SAR single-view complex images and three-dimensional point clouds. Relying on a unified spatiotemporal reference and using a point cloud-simulated SAR image matching algorithm, it completes pixel-level fine registration of the data and outputs the registered SAR image and the registered LiDAR point cloud. At the same time, a high-precision DEM is generated from the registered LiDAR point cloud and fed back to the SAR image to complete geocoding and orthorectification.

[0054] Fusion processing layer: This layer sets up three deep fusion processing paths that can be executed in parallel or selectively: 1) First path: Terrain-assisted SAR imaging module 541, which uses high-precision DEM generated by LiDAR point cloud to assist SAR image in completing terrain radiometric correction, geometric fine correction, terrain adaptive matched filtering or interferometric terrain phase removal, and generating terrain-corrected SAR products. 2) Second path: LiDAR point cloud classification enhancement module 542, constructs a deep learning network, integrates the multi-polarization texture features of SAR images with the three-dimensional geometric features of LiDAR point clouds, realizes ground feature classification enhancement, and generates a high-precision ground feature classification map; 3) Third path: Combine 3D reconstruction module 543 to fuse SAR coherent information and LiDAR geometric information to generate a hole-free, high-precision, high-resolution fused DEM or 3D surface model.

[0055] Output layer: The various data products output from the convergence processing layer are aggregated and integrated to form the final converged data product, which is then transmitted to the ground station via the data transmission module 60.

[0056] Figure 4 The diagram illustrates a dual-domain feature fusion network architecture based on deep learning in one embodiment of the present invention.

[0057] In this embodiment, the onboard data processing module 50 further includes an intelligent fusion module 55, which uses a deep neural network to achieve end-to-end fusion of SAR and LiDAR data. The network architecture includes: The shallow feature extraction module 551 performs shallow feature extraction on SAR images and LiDAR point cloud projection feature maps, respectively. The deep feature extraction module 552 uses a residual network structure to extract multi-scale deep features; The dual-domain feature fusion module 553 performs feature fusion alternately in the frequency domain and spatial domain to achieve complementary enhancement of cross-modal information; Reconstruction module 554 reconstructs high-resolution 3D surface products based on fusion features.

[0058] like Figure 4 As shown, this network architecture belongs to the intelligent fusion module 55 in the onboard data processing module 50. It adopts a structure of "dual-branch input, multi-level fusion, and single output". The data processing flow proceeds from left to right to complete feature extraction, deep mining, dual-domain alternating fusion, and product reconstruction. The complete network structure and working principle are as follows: The network has two input branches on the left, corresponding to the input processing of different modalities of SAR and LiDAR data, respectively. The upper branch inputs SAR imagery into the SAR shallow feature extraction network, which performs preliminary extraction of shallow features of SAR imagery through multiple convolutional layers. The lower branch inputs the height map and intensity map obtained by projection processing of LiDAR point cloud into the LiDAR shallow feature extraction network, realizing the extraction of shallow geometric and intensity features of LiDAR projection data. The two branches simultaneously complete the preliminary mining of shallow features of multi-source data.

[0059] The dual-modal features output by the shallow feature extraction module 551 are input into the corresponding deep feature extraction networks. These networks are constructed using ResNet residual network blocks to perform multi-scale deep feature mining on the shallow features of SAR images and LiDAR point cloud projection data. This further extracts the deep semantic features and texture features of SAR images and the deep geometric structure features of LiDAR point clouds, providing a high-dimensional and high-information feature foundation for subsequent cross-modal fusion.

[0060] The dual-domain feature fusion module 553 is the core innovation of this network. It achieves deep cross-modal feature fusion by alternating between spatial domain fusion and frequency domain fusion between two deep feature branches. The spatial domain fusion module introduces an attention mechanism to perform spatial dimension-weighted fusion of the feature maps from both branches, focusing on effective feature regions and suppressing redundant noise information. The frequency domain fusion module transforms the feature maps to the frequency domain through FFT transformation, interacting and enhancing the amplitude and phase information of the features in the frequency domain, and then returning them to the spatial domain through inverse transformation. This effectively captures and fuses the inherent structural information of different modal data, such as the texture periodicity of SAR images and the spatial geometric distribution of LiDAR point clouds. The alternating iterative fusion of spatial and frequency domains fully realizes the deep complementarity and feature enhancement of SAR and LiDAR cross-modal information.

[0061] The joint feature map obtained after multi-level dual-domain alternating fusion is input into the reconstruction module 554, which consists of upsampling layers and convolutional layers. The reconstruction module decodes and reconstructs the deeply fused joint feature map, and finally outputs fused products such as high-resolution 3D surface models, high-precision DEMs, or land cover classification maps.

[0062] This network employs a dual-branch structure to perform shallow feature extraction on SAR imagery and LiDAR point cloud projection data respectively. Its core innovation lies in the dual-domain feature fusion module 553: this module not only includes traditional spatial domain attention fusion but also innovatively introduces frequency domain fusion. By interacting and enhancing the amplitude and phase information of features in the frequency domain, the network can more effectively capture and fuse the intrinsic structural information of different modalities. The alternating fusion of spatial and frequency domains achieves deep complementarity and enhancement of cross-modal information. Finally, the reconstruction module decodes the deeply fused feature map into a final high-precision 3D surface product or classification product, significantly improving the accuracy and robustness of the fusion results.

[0063] In one embodiment of the present invention, a specific implementation scheme for a SAR and LiDAR co-carrying satellite payload system is provided.

[0064] The satellite platform is a medium-sized remote sensing satellite platform with three-axis stabilization control capability, attitude determination accuracy better than 0.01°, and attitude stability better than 0.001° / s. The platform provides a continuous power supply of no less than 1000W and supports simultaneous operation of SAR and LiDAR.

[0065] The SAR payload module employs an X-band active phased array antenna with dimensions of 5m × 1.5m. It features three operating modes: strip mode (3m resolution, 50km swath width), spotlight mode (1m resolution, 10km swath width), and scanning mode (20m resolution, 200km swath width). The peak transmit power is 2000W, and the system noise equivalent backscattering coefficient is better than -22dB.

[0066] The LiDAR payload module uses a 1064nm wavelength laser with a pulse repetition frequency of 100kHz and a single pulse energy of 0.5mJ. The scanning mechanism employs a tilting mirror design, with a scanning swath matched to the SAR swath (typically 50km). The laser footprint diameter is 5m at a 500km orbital height. Ranging accuracy is better than 0.5m (including orbital attitude error).

[0067] The time-space reference synchronization module uses a shared atomic clock as the frequency reference, with a frequency stability better than 1×10⁻¹¹ / s. The time synchronization unit sends synchronization trigger pulses to the two payload modules via cable, with a trigger delay of less than 1ns. The space synchronization unit calculates the laser footprint position in real time using the spaceborne computer and compares it with the SAR beam pointing. When the deviation exceeds a threshold, it compensates through attitude adjustment.

[0068] The onboard data processing module adopts a high-performance FPGA + multi-core DSP architecture, with a fixed-point computing capability of 2 TFLOPS, and supports real-time processing of SAR imaging, point cloud generation and multimodal data fusion.

[0069] In another embodiment of the present invention, specific steps of a data fusion processing method based on the above system are provided, including: Step 1: Data Acquisition and Preprocessing The SAR payload module acquires raw echo data, performs range and azimuth compression, and generates single-view complex imagery (SLC). The LiDAR payload module acquires full laser waveform data, extracts the surface echo time through waveform decomposition, and combines it with satellite orbital attitude data to generate a 3D point cloud.

[0070] Step 2: Spatiotemporal registration: Using the precise timestamps recorded by the spatiotemporal reference synchronization module, SAR image pixels and LiDAR point clouds are aligned within the same spatiotemporal reference frame. The specific steps are as follows: Convert the pixel coordinates of the SAR image to slant range-azimuth time coordinates; Based on the geographic coordinates of the LiDAR point cloud, the corresponding pixel position in the SAR image is calculated. Simulated SAR intensity images are generated using LiDAR point clouds and cross-correlation matching is performed with actual SAR images to obtain sub-pixel level registration offset. SAR images are resampled to achieve accurate registration with LiDAR point clouds.

[0071] Step 3: LiDAR-based terrain-aided SAR processing: The registered LiDAR point cloud is interpolated to generate a high-resolution DEM (grid spacing matched to SAR pixels) for use in: Geocoding of SAR imagery: Based on SAR imaging geometry and DEM, slant range images are projected onto a geographic coordinate system; Topographic radiation correction: Radiation correction is performed based on the topographic influence factor of the backscattering coefficient calculated according to the local incident angle; InSAR Terrain Phase Removal: For repeating orbital interferometric data, terrain phase is simulated using a DEM and removed from the interferogram.

[0072] Step 4: Multimodal Feature Extraction and Fusion Construct a two-branch deep learning network to process SAR imagery and LiDAR point clouds respectively: SAR branch: The input is a multi-polarization SAR image, and texture and polarization features are extracted through a convolutional neural network; LiDAR branch: The input is the height map, intensity map and density map generated by the point cloud projection, and geometric features are extracted through a convolutional neural network; Feature fusion layer: Employs an attention mechanism to weightedly fuse features from different modalities, generating a joint feature map; Task Header: Set a classification head, regression head, or reconstruction head according to specific application requirements, and output the fused product.

[0073] Step 5: Product Generation Generate one of the following products based on the task requirements: Fusion of high-resolution DEM: Combining high-precision LiDAR elevation points and SAR coherent information to generate a hole-free high-resolution DEM; Land cover classification map: Land cover classification is based on fused features, and the classification accuracy is better than that of a single sensor; Three-dimensional deformation field: High-precision surface deformation field is obtained by combining SAR interferometry and LiDAR registration information.

[0074] In another embodiment of the present invention, a specific implementation of the guided observation mode is provided. The data acquisition and control module 40 has the following pre-configured logic: Step A: The SAR payload module performs rapid imaging of a wide area in scanning mode (20m resolution, 200km swath width); Step B: The onboard data processing module processes SAR images in real time and identifies regions of interest (such as newly added landslides, earthquake-damaged buildings, etc.) through change detection or target recognition algorithms. Step C: Based on the identification results, the data acquisition and control module re-plans the observation task and controls the satellite attitude and payload parameters; Step D: The LiDAR payload module performs a high-density scan of the identification area in a focused mode to obtain a 3D point cloud with centimeter-level accuracy; Step E: The SAR payload module may optionally perform spotlight imaging of the same area to acquire high-resolution SAR images; Step F: The onboard data processing module fuses the acquired high-precision data to generate refined products, which are then prioritized for download.

[0075] This mode achieves a closed loop of "wide-area search - fine observation", which greatly improves the observation efficiency of dynamic targets and emergency events.

[0076] Although various embodiments of the invention have been described above, it should be understood that they are presented by way of example only and not as limitations. It will be apparent to those skilled in the art that various combinations, modifications, and alterations can be made without departing from the spirit and scope of the invention. Therefore, the breadth and scope of the invention disclosed herein should not be limited by the exemplary embodiments disclosed above, but should be defined solely by the appended claims and their equivalents.

Claims

1. A satellite payload system based on the co-carrying of synthetic aperture radar and lidar, characterized in that, include: The satellite platform is equipped with a SAR payload module and a LiDAR payload module. A spatiotemporal reference synchronization module is connected to the SAR payload module and the LiDAR payload module, and is used to realize time synchronization, frequency synchronization and spatial pointing synchronization between the SAR payload module and the LiDAR payload module. The data acquisition and control module is connected to the SAR payload module and the LiDAR payload module, and is used to control the working mode of the SAR payload module and the LiDAR payload module according to the observation mission requirements. The onboard data processing module is connected to the spatiotemporal reference synchronization module and the data acquisition and control module, and is used to preprocess and fuse the SAR echo data of the SAR payload module and the LiDAR point cloud data of the LiDAR payload module. The data transmission module is connected to the onboard data processing module and is used to transmit the fused data to the ground station.

2. The system according to claim 1, characterized in that, The SAR payload module includes: The SAR antenna subsystem includes an active phased array antenna for multi-polarization operation. SAR transmit / receive subsystem, used to transmit microwave pulses and receive ground echo signals; The SAR imaging processing unit is used to perform range compression and azimuth synthetic aperture processing on ground echo signals to generate SAR complex image data. The SAR payload module operates in the X-band, C-band, or L-band and has multiple imaging modes.

3. The system according to claim 1, characterized in that, The LiDAR payload module includes: Laser emitters, including high-repetition-rate lasers, are used to emit laser pulses in the near-infrared band; The scanning mechanism is configured to achieve cross-track wide-swath scanning via a tilting mirror or fiber optic scanning method; The echo detection and timing unit is configured to measure the round-trip time of a laser pulse using single-photon detection or multi-channel parallel detection techniques. The point cloud generation unit is configured to combine satellite orbital attitude data to generate geocoded 3D point cloud data.

4. The system according to claim 1, characterized in that, The spatiotemporal reference synchronization module includes: The time synchronization unit is configured to achieve timing alignment between SAR transmission pulses and laser transmission pulses by triggering pulse signals; The frequency synchronization unit provides a unified frequency reference for the SAR transmit / receive subsystem and the laser timing unit through a shared frequency reference source, thereby eliminating measurement errors introduced by frequency drift. The space synchronization unit is configured to adjust the platform attitude through the satellite attitude control system based on the pointing of the active phased array antenna of the SAR payload module and the instantaneous field of view of the scanning mechanism, so as to ensure that the laser footprint falls within the SAR imaging coverage area and achieve coordinated airspace coverage.

5. The system according to claim 1, characterized in that, The working modes include: The SAR payload module and the LiDAR payload module operate independently according to their respective preset parameters; According to the requirements of the observation mission, control the two payload modules to conduct synchronous observations of the same target area; The target is identified in real time using SAR wide-area imaging results, triggering the LiDAR payload module to scan the target area and acquire three-dimensional information.

6. The system according to claim 1, characterized in that, The onboard data processing module includes: SAR data preprocessing unit, which is configured to generate single-look complex images; The LiDAR data preprocessing unit is configured to generate a 3D point cloud. The joint localization and registration unit is configured to perform pixel-level registration between SAR imagery and LiDAR point clouds. The data fusion processing unit is configured to perform joint processing based on the registered multi-source data.

7. The system according to claim 6, characterized in that, The data fusion processing unit includes: The terrain-aided SAR imaging module is configured to generate digital elevation models from LiDAR point clouds for geocoding, orthorectification, terrain-adaptive matched filtering, or terrain phase removal for interferometric measurements of SAR images. The LiDAR point cloud classification enhancement module is configured to perform ground feature classification and recognition by fusing the multi-polarization features of SAR images and the three-dimensional geometric features of LiDAR point clouds through a deep learning network. The joint 3D reconstruction module is configured to fuse coherent information from SAR with geometric information from LiDAR to construct a 3D surface model.

8. The system according to claim 1, characterized in that, The onboard data processing module further includes a deep learning-based intelligent fusion module, which includes: The shallow feature extraction submodule is configured to extract features from SAR images and LiDAR point cloud projection feature maps. The deep feature extraction submodule is configured to extract multi-scale deep features through a residual network structure. A dual-domain feature fusion submodule is configured to perform feature fusion alternately in the frequency domain and the spatial domain; The reconstruction submodule is configured to reconstruct a 3D surface product based on fusion features.

9. A data fusion processing method based on the system according to any one of claims 1-8, characterized in that, include: The time, frequency, and space synchronization of the SAR payload module and the LiDAR payload module are achieved through the spatiotemporal reference synchronization module. The SAR payload module and the LiDAR payload module are controlled to conduct coordinated observations of the same target area to acquire raw echo data. The raw data is preprocessed to generate SAR single-view complex images and LiDAR 3D point clouds; Fine-register SAR single-view complex images with LiDAR 3D point clouds; The registered multi-source data is then fused to generate a fused product.

10. The method according to claim 9, characterized in that, The fusion processing of the registered multi-source data includes: Digital elevation models generated from LiDAR 3D point clouds are used to assist in the imaging processing of SAR single-look complex images; or Utilizing the multi-polarization features of SAR single-view complex images to assist in the classification and recognition of LiDAR 3D point clouds; or Joint 3D reconstruction is performed by fusing coherent information from SAR with geometric information from LiDAR.

Images