GNSS-R soil humidity inversion method and system based on multi-modal feature fusion

Through the multimodal cross-branch fusion network method, the DDM image, delay waveform and corrected reflectivity characteristics are combined, and the problem of low soil moisture inversion accuracy in the prior art is solved, and more efficient soil moisture inversion results are achieved .

CN119004393AActive Publication Date: 2024-11-22CHINA UNIV OF PETROLEUM (EAST CHINA)

Patent Information

Application Number
CN202411488678.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-24
Publication Date
2024-11-22
Estimated Expiration
2044-10-24

AI Technical Summary

Technical Problem

The existing satellite-borne GNSS-R soil moisture inversion method ignores the delay waveform and the corrected surface reflectivity, resulting in low inversion accuracy, and the CNN model cannot capture global information when processing DDM images, and has high computational complexity.

Method used

The multimodal cross-branch fusion network (MCFN) method is adopted, combined with 1D-CNN, Transformer and BP modules, and the DDM image, delay waveform and corrected reflectivity characteristics are fused, and soil moisture-related characteristics are extracted through the multimodal feature fusion module. .

Benefits of technology

It improves the accuracy and robustness of soil moisture inversion, enhances the model's understanding of surface scattering characteristics, reduces the computational complexity, and achieves a more comprehensive soil moisture inversion result.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119004393A_ABST
    Figure CN119004393A_ABST
Patent Text Reader

Abstract

The invention discloses a GNSS-R soil humidity inversion method and system based on multi-modal feature fusion, and relates to the field of remote sensing image processing and soil humidity inversion, and the method comprises the steps: collecting original data to be inverted and carrying out the preprocessing, and obtaining the processed data; obtaining model input features based on the processed data; constructing a multi-modal cross-branch fusion network model; training the multi-modal cross-branch fusion network model to obtain a final model; and completing the inversion of the soil humidity by using the final model. The multi-modal cross-branch fusion network processes multi-modal data through the 1D-CNN module, the Transform module and the BP module, integrates features through the cross fusion module and the superposition fusion module, and effectively improves the precision and efficiency of soil humidity inversion. Meanwhile, time delay waveform features are deeply mined through 1D-CNN, and model robustness is enhanced in combination with earth surface features.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image processing and soil moisture inversion, and in particular to a GNSS-R soil moisture inversion method and system based on multi-modal feature fusion. Background Art

[0002] The existing soil moisture inversion methods for satellite-borne GNSS-R are mainly divided into two categories: traditional statistical regression methods and soil moisture inversion methods based on machine learning. The traditional statistical regression method is simple in principle and easy to implement, but this type of method considers fewer factors, the inversion model is simple, and the inversion accuracy is relatively low; the soil moisture inversion method based on machine learning uses observation data and labels for end-to-end learning, with strong learning ability and good robustness. However, the input data of the current machine learning inversion algorithm is usually DDM images and auxiliary features, ignoring another important observation data of GNSS-R: the time delay waveform. The time delay waveform is a function of the scattered power delayed with time, and is also a one-dimensional representation of the DDM image. The time delay waveform is of great significance for analyzing the scattering characteristics of ground targets. By observing the time delay waveform, explicit features such as the leading edge slope, trailing edge slope, reflected signal delay and power distribution can be obtained, but artificial extraction has the disadvantage of missing some information. Using deep learning can fully explore the deep hidden features of the time delay waveform.

[0003] For DDM images, most scholars still use CNN models to extract their features. When CNN performs convolution operations on DDM images, it has a strong reliance on local features and cannot capture the global information of the image. Secondly, the computational complexity of CNN increases significantly with the increase of network depth. In recent years, with the rapid rise of Transformer in the fields of natural language processing (NLP) and computer vision (CV), Transformer has shown significant advantages in processing long-distance dependencies, achieving parallel computing, versatility, model performance and architectural flexibility, and has achieved performance that exceeds CNN in many tasks. Compared with CNN, the Transformer model can connect the information under different delays and Dopplers in DDM through the self-attention mechanism, thereby extracting more refined global feature information.

[0004] In addition, studies have shown that there is a strong connection between reflectivity and soil moisture. In previous studies, the corrected surface reflectivity was often used to establish statistical regression models, or surface auxiliary parameters were added to the deep learning model to eliminate its impact on reflectivity. In summary, both approaches ignore the role of the corrected surface reflectivity in the machine learning model. Summary of the invention

[0005] In view of the above problems, the present invention proposes a GNSS-R soil moisture inversion method based on a multimodal cross-branch fusion network (MCFN). This method uses auxiliary information such as DDM images, time-delay waveforms, and corrected reflectivity as input features, fully integrates three types of features: image, waveform, and auxiliary information, and more comprehensively describes the surface scattering. MCFN consists of three parallel branch modules and two multimodal feature fusion modules. Each branch module is used to process different types of data. The 1D-CNN module is good at processing one-dimensional sequence data and is used to extract potential information in the time-delay waveform; the Transformer module perceives the global features of DDMs through a self-attention mechanism to better extract abstract feature information in DDMs; the BP module is used to mine auxiliary data to effectively supplement the feature information of DDMs and time-delay waveforms. It should be noted that a multimodal feature fusion module is embedded in MCFN to extract and fuse the features of each branch, thereby enhancing information exchange between modules and reducing information loss. The multimodal cross-branch fusion network model can simultaneously realize the feature mining of DDM images, time-delay waveforms and auxiliary information, and based on the multimodal feature fusion module, it can realize soil moisture inversion using multiple types of features to obtain higher-precision soil moisture inversion results.

[0006] To achieve the above object, the present invention provides a GNSS-R soil moisture inversion method based on multimodal feature fusion, the steps comprising:

[0007] The raw data to be inverted are collected and preprocessed to obtain processed data, wherein the raw data include CYGNSS data sets and SMAP data sets; the preprocessing includes correcting the reflectivity; specifically, firstly, a number of observation results are screened out by performing standard quality control, CYGNSS observation data with an incident angle greater than 65° are not considered, observation data with a receiver antenna gain less than 0 dB are eliminated, and outliers with an SNR less than 0 are removed, and finally, observation data with a DDM peak within the 7th to 10th time delay frame are selected as the processed data;

[0008] Based on the processed data, a model input feature is obtained, the steps comprising: extracting a feature set associated with soil moisture from the processed data as a model input, the model input feature consisting of three parts: a DDM image, a time-delay waveform and an auxiliary feature;

[0009] Constructing a multimodal cross-branch fusion network model; the multimodal cross-branch fusion network model includes: a 1D-CNN module, a Transformer module, a BP module and two multimodal feature fusion modules;

[0010] Training the multimodal cross-branch fusion network model to obtain a final model; the training step includes: inputting the model input features into the multimodal cross-branch fusion network model, and training with the soil moisture results provided by SMAP as reference values;

[0011] The final model is used to complete the inversion of soil moisture.

[0012] Preferably, the method for correcting the reflectivity includes: correcting the surface reflectivity using a parameterized water cloud model:

[0013] ,

[0014] ,

[0015] ,

[0016] In the formula, Represents the surface reflectivity; represents the part of scattering caused by vegetation; represents the scattering contribution of the soil layer; represents the bidirectional vegetation transmittance; Represents vegetation moisture data extracted by SMAP; represents the satellite incidence angle; A and B represent parameters related to vegetation. Here, the values ​​applicable to all land are adopted, where A=0.0012 and B=0.091, and the final corrected surface reflectivity is the square of the square:

[0017] ,

[0018] In the formula, Represents the surface reflectivity after correction by the water cloud model.

[0019] Preferably, the surface roughness coefficient in the SMAP data is used to further correct the reflectivity:

[0020] ,

[0021] In the formula, Indicates the roughness coefficient Corrected surface reflectivity; the exponential term represents the signal attenuation caused by roughness; is the surface roughness coefficient in SMAP data; is the satellite incidence angle; It represents the surface reflectivity after being corrected by the water cloud model; Indicates the roughness coefficient Corrected surface reflectivity.

[0022] Preferably, the DDM images include: a raw count DDM image, a simulated power DDM image, a bistatic radar cross section DDM image, and an effective scattering area DDM image;

[0023] There are three types of auxiliary features. The first type is features related to DDM, including corrected surface reflectivity, incident angle, DDM mean, DDM peak and signal-to-noise ratio; the second type is features related to the time-delay waveform, including leading slope, trailing slope, integrated area, number of maximum points, number of minimum points, skewness, kurtosis and average slope; the third type is features related to the surface, including longitude, latitude, vegetation optical thickness, surface roughness, surface temperature and vegetation moisture content.

[0024] Preferably, the 1D-CNN module consists of three parts: a convolution layer, a pooling layer, and a normalization layer; the 1D-CNN module is used to process one-dimensional data and extract potential information in the delay waveform. The specific workflow includes: connecting the three delay waveforms end to end and inputting them into the 1D-CNN; then, normalizing the input data and passing it to the convolution layer; the convolution layer performs a convolution operation on the input data to generate a feature image, and performs a maximum pooling operation with a size of 2×1 and a step size of 2 on the obtained feature map to achieve downsampling; in this process, the first convolution layer uses 16 convolution kernels with a size of 3×1; the second convolution layer uses 32 convolution kernels with a size of 3×1.

[0025] Preferably, in the Transformer module, a block operation is first performed, that is, the size of the patch is set, the patch is set to 1×1 size, and encoding is performed pixel by pixel to capture and process the feature information of each pixel; then each patch is flattened and combined to form a sequence.

[0026] Preferably, the BP module comprises: an input layer and a fully connected layer;

[0027] Among them, the input layer is responsible for receiving a single feature, passing it to the fully connected layer after normalization. The fully connected layer is responsible for linear combination and feature extraction of the features, and learning the complex relationship between the features by adjusting the weights and biases.

[0028] Preferably, the multimodal feature fusion module includes a cross fusion module and a superposition fusion module;

[0029] The cross-fusion module is used to extract the intermediate layer features of each branch and unify the feature dimensions of each branch through linear mapping to ensure that these features can be effectively fused in the same space;

[0030] The superposition fusion module is used to directly combine features from different sources or types head to tail to form a feature vector with a larger dimension, thereby improving the expressiveness of the features.

[0031] The present invention also provides a GNSS-R soil moisture inversion system based on multi-modal feature fusion, the system is used to implement the above method, including: an acquisition module, an extraction module, a construction module, a training module and an inversion module;

[0032] The acquisition module is used to acquire the raw data to be inverted and perform preprocessing to obtain processed data, wherein the raw data includes a CYGNSS data set and a SMAP data set; the preprocessing includes correcting the reflectivity; specifically, firstly, a standard quality control is performed to screen out a number of observation results, CYGNSS observation data with an incident angle greater than 65° is not considered, observation data with a receiver antenna gain less than 0 dB is eliminated, and outliers with an SNR less than 0 are removed, and finally, observation data with a DDM peak within the 7th to 10th time delay frame are selected as the processed data;

[0033] The extraction module is used to obtain model input features based on the processed data, and the steps include: extracting a feature set associated with soil moisture from the processed data as the input of the model, and the model input features are composed of three parts: DDM image, time delay waveform and auxiliary features;

[0034] The construction module is used to construct a multimodal cross-branch fusion network model; the multimodal cross-branch fusion network model includes: a 1D-CNN module, a Transformer module, a BP module and two multimodal feature fusion modules;

[0035] The training module is used to train the multimodal cross-branch fusion network model to obtain a final model; the training step includes: inputting the model input features into the multimodal cross-branch fusion network model, and training with the soil moisture results provided by SMAP as reference values;

[0036] The inversion module is used to complete the inversion of soil moisture using the final model.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] The present invention proposes a multimodal cross-branch fusion network for spaceborne GNSS-R soil moisture inversion. The network mainly includes three branch modules, which are used to process input data of different modes respectively. Among them, the 1D-CNN module can effectively extract features from sequence data, and performs well in processing one-dimensional data, and is particularly suitable for feature extraction of one-dimensional data of time-delay waveform; the transformer module has the advantage of capturing global features and can effectively extract deep features in DDM images; the BP module is based on the back propagation algorithm and is good at multivariate nonlinear processing. This module can integrate the contribution of multiple single features to soil moisture inversion. It should be noted that a multimodal feature fusion module is designed in the three branch modules, including a cross fusion module and a superposition fusion module. Through the designed fusion loss function, the consistency and effectiveness of different modal features in the fusion process can be ensured. The network model can simultaneously realize feature mining of DDM images, time-delay waveforms and auxiliary information, and based on the multimodal feature fusion module, effectively integrate multimodal information, enhance the synergy between branches, and realize deep feature extraction and fusion of multimodal data through a multi-branch architecture, providing a more comprehensive and efficient solution for soil moisture inversion.

[0039] In addition, the present invention particularly considers the important role of time-delay waveforms in soil moisture inversion. On the one hand, the 1D-CNN module can be used to deeply explore the deep potential features in the time-delay waveform to more accurately reflect the scattering characteristics of surface targets. It is worth noting that morphological statistical features are extracted from the time-delay waveform for the first time to better characterize the morphological characteristics of the time-delay waveform. The morphological statistical features include: integral area, number of maximum points, number of minimum points, skewness, kurtosis and average slope. They can effectively reflect the change of related power with time delay. At the same time, combined with other surface features (such as vegetation moisture content, surface roughness, etc.), the robustness and generalization ability of the model are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0041] Figure 1 A schematic diagram of a model structure of an embodiment of the present invention;

[0042] Figure 2 A schematic diagram of a cross-fusion module according to an embodiment of the present invention;

[0043] Figure 3 This is a schematic diagram of a superposition and fusion module according to an embodiment of the present invention;

[0044] Figure 4 The figure is a flowchart of the overall process of the method according to the embodiment of the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0047] Before describing the invention, the following terms are explained:

[0048] GNSS-R: Global Navigation Satellite System-Reflectometry is a new type of remote sensing technology. GNSS-R technology adopts dual (multi) base radar mode and uses the globally shared GNSS constellation as a multi-source microwave signal transmission source. The receiver is located on another platform different from the transmitter, which can be a shore-based platform, airborne platform, or satellite platform.

[0049] CYGNSS: The full name is Cyclone Global Navigation Satellite System. The constellation consists of 8 small satellites, each of which is equipped with a 4-channel GNSS-R receiver, which can simultaneously receive direct signals from 4 GNSS satellites and reflected signals from the surface, and generate DDM images of important observation data of GNSS-R satellites.

[0050] SM: Soil Moisture, also known as soil moisture content, is a physical quantity that indicates the degree of soil dryness or wetness.

[0051] DDM: A two-dimensional correlation power waveform with time delay as the horizontal axis and Doppler frequency shift as the vertical axis. For the CYGNSS constellation, the DDM consists of 17 time delay rows and 11 Doppler columns.

[0052] Time delay waveform: The time delay waveform is a function of the scattered power with time delay. It is one of the basic observation quantities of GNSS-R and it is also a one-dimensional representation of the DDM image.

[0053] 1D-CNN: One-dimensional Convolutional Neural Network Model (1D-CNN model for short) has the advantages of local feature extraction, parameter sharing and parallel computing, and is often used to process one-dimensional data.

[0054] Transformer: It is a deep learning architecture with self-attention mechanism as its core, which can effectively extract global feature information and long-range dependencies in images and realize the overall semantic information modeling of images.

[0055] Embodiment 1

[0056] The following will describe in detail how the present invention solves the technical problems in practical work in conjunction with this embodiment.

[0057] S1. Collect the original data to be inverted, and pre-process the original data to obtain processed data.

[0058] In this embodiment, the selected raw data include CYGNSS data sets and Soil Moisture Active and Passive (SMAP) data sets. The CYGNSS data product used is Level-1V3.1 data. The CYGNSS data set provides DDM images, time-delay waveforms, and single features. The SMAP data product used is enhanced L3-level data, which is a data product of SMAP. It has improved accuracy compared to earlier versions and is generally described as SMAPEnhanced L3 or SMAP L3 V5. The grid resolution of the data is 9km×9km, and the format is HDF5. The SMAP data set provides vegetation optical thickness, surface roughness coefficient, surface temperature, vegetation moisture content, and soil moisture data, of which soil moisture data is used as reference data, and other data are also used as single features.

[0059] The purpose of data preprocessing is mainly to extract the input features of the model from the original data set. It is particularly worth noting that the above embodiment adds a two-step correction of the reflectivity in the preprocessing stage, and uses the corrected reflectivity as the input feature of the network model. First, standard quality control is performed to screen out some observation results, excluding CYGNSS observation data with an incident angle greater than 65°, eliminating observation data with a receiver antenna gain less than 0dB, removing outliers with an SNR less than 0, and finally selecting observation data with DDM peaks in the 7th-10th delay frames as processed data. The specific steps are as follows:

[0060] Studies have shown that there is a strong connection between reflectivity and soil moisture. Since most land surfaces are relatively smooth, the GNSS-R signal is mainly a coherent reflection component. Based on this assumption, the surface reflectivity can be calculated according to formula (1):

[0061] , (1)

[0062] In the formula, Indicates the power at the peak point of the DDM image; is the power of the transmitted signal of the transmitter; is the antenna gain of the transmitter; Indicates the distance between the transmitter and the specular reflection point; Indicates the distance between the mirror reflection point and the receiver; represents the antenna gain of the receiver; Indicates the wavelength of the GNSS navigation satellite signal (19 cm); is the surface reflectivity. After all items are converted to dB, the surface reflectivity The following formula (2) can be used to solve it:

[0063] , (2)

[0064] However, in actual situations, GNSS-R signals are also affected by factors such as vegetation cover, surface roughness, and angle of incidence. Therefore, it is necessary to use SMAP auxiliary data to correct the surface reflectivity for vegetation attenuation and roughness. In previous studies, the corrected surface reflectivity was often used to establish a statistical regression model. Alternatively, by adding surface auxiliary parameters to the deep learning model, the effect of the reflectivity on the reflectivity is considered to be eliminated. In summary, both approaches ignore the role of the corrected surface reflectivity in the machine learning model.

[0065] Therefore, this embodiment uses a parameterized water cloud model to correct the surface reflectivity to eliminate the influence of vegetation. The water cloud model represents the incoherent combination of the backscattering power of the entire canopy, that is, the contribution of vegetation and the contribution of the underlying soil. Under the assumption that the influence of vegetation scattering can be ignored, the relevant formula of the water cloud model is as follows:

[0066] , (3)

[0067] , (4)

[0068] , (5)

[0069] In the formula, Represents the surface reflectivity; represents the part of scattering caused by vegetation; represents the scattering contribution of the soil layer; represents the bidirectional vegetation transmittance; represents the vegetation water content (VWC) data extracted by SMAP; represents the satellite incidence angle; A and B represent parameters related to vegetation. Here, the values ​​applicable to all land are adopted, where A=0.0012 and B=0.091, and the final corrected surface reflectivity is the square of the square:

[0070] , (6)

[0071] In the formula, Represents the surface reflectivity after correction by the water cloud model.

[0072] At the same time, the surface roughness also has a certain attenuation effect on the reflected signal. This embodiment uses the surface roughness coefficient in the SMAP data to further correct the reflectivity:

[0073] , (7)

[0074] In the formula, It represents the surface reflectivity after being corrected by the roughness coefficient h; the exponential term represents the signal attenuation caused by the roughness; is the surface roughness coefficient in SMAP data; is the satellite incidence angle; It represents the surface reflectivity after being corrected by the water cloud model; Represents the surface reflectivity after correction by the roughness coefficient h.

[0075] The corrected reflectance removes the influence of surface vegetation and roughness on reflectance, and can better characterize soil moisture changes as the input feature of the network model.

[0076] S2. Based on the processed data, obtain model input features.

[0077] A feature set associated with soil moisture is extracted from the processed data as the input of the model. The input features of the model consist of three parts: DDM image, time-delay waveform and auxiliary features.

[0078] The above-mentioned DDM images have four types of products, namely, raw count DDM images, analog power DDM images, bistatic radar cross-section DDM images, and effective scattering area DDM images. DDM is obtained by analyzing the delay and Doppler shift information of the reflected signal, which characterizes the time delay and frequency shift characteristics of the reflected signal. Therefore, when the water content in the soil changes, the DDM will also change accordingly. This embodiment uses the analog power (Power Analog) DDM image as the input of the transformer module.

[0079] The time delay waveform is usually defined as the profile of DDM at 0Hz Doppler frequency, also called zero Doppler time delay waveform (Central Doppler time-Delay Waveform), with the horizontal axis being the delay value and the vertical axis being the related power. The time delay waveform can comprehensively reflect the distribution of power and provide more comprehensive reflection surface information (such as surface roughness, vegetation coverage, etc.). Therefore, the time delay waveform is of great significance for analyzing the scattering characteristics of ground targets. The time delay waveform usually includes three categories. In this embodiment, the following three time delay waveforms are connected in series as the input of 1D-CNN.

[0080] 1. Zero Doppler delay waveform (CDW): CDW is the profile of DDM at 0Hz Doppler frequency.

[0081] 2. Integrated Doppler delay waveform (IDW): IDW is obtained by summing the DDM along the Doppler dimension.

[0082] 3. Extended Delay Doppler Waveform (DDW): It is expressed as the deviation of IDW and CDW at corresponding delays.

[0083] The auxiliary features include three categories. The first category is the features related to DDM, including the corrected surface reflectivity, incident angle, DDM mean, DDM peak and signal-to-noise ratio; the second category is the features related to the time delay waveform, including the leading slope, the trailing slope, the integrated area, the number of maximum points, the number of minimum points, the skewness, the kurtosis, and the average slope; the third category is the features related to the surface, including longitude, latitude, vegetation optical thickness, surface roughness, surface temperature, and vegetation moisture content. Table 1 gives a detailed list of all the auxiliary features used in this embodiment:

[0084] Table 1 Summary of auxiliary features

[0085] .

[0086] The leading edge slope (LES) and trailing edge slope (TES) are related to the coherent and incoherent components of the GNSS-R signal, and can reflect the roughness of the reflecting surface to a certain extent. Generally, the rougher the reflecting surface, the more incoherent scattering components there are in the GNSS-R signal, and the smaller the absolute values ​​of LES and TES. Conversely, the larger the LES and TES are. Therefore, LES and TES can be used as one of the indicators for evaluating the roughness of the reflecting surface, and the calculation method is as follows:

[0087] , (8)

[0088] , (9)

[0089] Where, LES represents the leading edge slope; TES represents the trailing edge slope; NIDW represents the normalized integrated Doppler delay waveform; Indicates the power value at the peak point; and Respectively represent the power values ​​of the two pixels before and after the peak point; Indicates the time delay resolution of the GNSS-R receiver. Its value is 0.2552 chips. Chips represents the duration of a chip, which is converted into seconds. .

[0090] In addition to using LES and TES, this embodiment further extracts features from IDW and calculates the statistical features of IDW: integral area (IA), number of peaks (NP), number of valleys (NV), skewness (SK), kurtosis (KU), and average slope (AS). The following is a brief introduction to these features:

[0091] Integrated area: The integrated area is a measure of the area under the delayed waveform and can reflect the overall power of the waveform.

[0092] The number of maximum and minimum points: can be used to analyze the oscillation frequency or periodicity of the waveform.

[0093] Skewness: describes the asymmetry of data distribution and reflects the degree of skewness of power distribution.

[0094] Kurtosis: A higher kurtosis indicates that the data has a peaked nature, while a lower kurtosis indicates that the data is flatter.

[0095] Average slope: The average slope is the average value of the slope of the delay waveform, reflecting the changing trend of the waveform.

[0096] The calculated delay waveform features can well reflect the changes in the relevant power. As global features of the delay waveform, they are less sensitive to noise than single-point features (such as peak point power), and therefore have stronger noise resistance. By introducing the integral area, number of maximum points, number of minimum points, skewness, kurtosis, average slope, LES and TES as input features in the BP module, the robustness and generalization ability of the model can be improved, the existing feature information can be supplemented, and a more comprehensive and stable description of the reflection surface features can be provided, thus playing an important role in model training and prediction.

[0097] The DDM mean refers to the average value of the pixel power in the area near the peak point in the DDM image. Here, a 3×3 area around the peak point is selected. When the navigation satellite signal is reflected by the surface, the GNSS-R signal is mainly recorded in the pixels near the peak point of the DDM image. Therefore, the DDM mean can be used to measure the degree of concentration and dispersion of the GNSS-R signal. The higher the DDM mean, the more concentrated the GNSS-R signal energy and the smoother the surface; conversely, the lower the DDM mean, the more dispersed the GNSS-R signal energy and the rougher the surface.

[0098] The DDM peak, incident angle, longitude and latitude of the specular reflection point, and signal-to-noise ratio data are stored in a NetCDF file, which can be downloaded from the CYGNSS data website.

[0099] Other auxiliary data include surface roughness, vegetation coverage, vegetation moisture content, and surface temperature, all of which can be obtained from SMAP satellite data. In addition, this embodiment uses the corrected reflectivity instead of the uncorrected surface reflectivity, and inputs it into the model constructed subsequently as a single feature.

[0100] S3. Construct a multimodal cross-branch fusion network model.

[0101] The multimodal cross-branch fusion network model consists of a 1D-CNN module, a Transformer module, a BP module and two multimodal feature fusion modules. Among them, the 1D-CNN module is good at processing one-dimensional sequence data, which is used to extract potential information in the delay waveform; the Transformer module perceives the global features of DDMs through the self-attention mechanism to better extract the abstract feature information in DDMs; the BP module is used to mine auxiliary data to effectively supplement the feature information of DDMs and delay waveforms. The auxiliary features are shown in Table 1 above. In addition, this embodiment designs a feature fusion module and fusion loss, which can extract and fuse the features of each branch and strengthen the information exchange between different branches.

[0102] The multimodal cross-branch fusion network model of this embodiment is as follows Figure 2As shown in the figure, the network is mainly composed of four parts, namely 1D-CNN module, Transformer module, BP module and multimodal feature fusion part. The following is a detailed introduction to each part:

[0103] The first part is the 1D-CNN module. Since the 1D-CNN module has the advantages of local feature extraction, parameter sharing and parallel computing, it is often used to process one-dimensional data. The module consists of three parts: convolution layer, pooling layer and normalization layer. The convolution layer obtains feature information by performing convolution operations on the input data, while the pooling layer uses a pooling window to retain the key information of the feature map in order to reduce the complexity of the data.

[0104] The workflow of the 1D-CNN module includes: First, the three time-delay waveforms are connected end to end and input into the 1D-CNN. Since each waveform has the same size of 1×17, the input size is 1×51. Then, the input data is normalized and passed to the convolution layer. The convolution layer performs a convolution operation on the input data to generate a feature image, and performs a maximum pooling operation of size 2×1 and stride 2 on the obtained feature map to achieve downsampling. In this process, the first convolution layer uses 16 convolution kernels of size 3×1; the second convolution layer uses 32 convolution kernels of size 3×1.

[0105] Patch embedding is for better feature input. In the Transformer module, the first step is to perform a block operation, that is, to set the size of the patch. This embodiment fully takes into account that each pixel of the DDM image contains its unique time delay and Doppler frequency shift information. Therefore, in order to accurately capture the information of each pixel, the patch is set to a size of 1×1 and encoded pixel by pixel to ensure that the feature information of each pixel is accurately captured and processed. The size of the CYGNSS data image in this implementation is 17×11, and the total number of patches is N=H×W=17×11=187. Each patch is denoted as , where i = 1, 2, ..., N. According to formula (10), each patch is flattened to form a vector Formula (11) is the sum of all the flattened vectors Combine into a sequence X.

[0106] The main function of position encoding is to provide the model with the position information of the elements in the sequence. In order to enhance the model's ability to perceive global information, the delayed Doppler coordinate P with the same size as X is introduced by formula (12) for position marking. Finally, according to formula (13), the input vector Z of the Transformer encoder is the sum of the feature vector X and the position marker P.

[0107] , (10)

[0108] , (11)

[0109] , (12)

[0110] , (13)

[0111] Among them, Flatten represents a flattening operation; i represents a natural number.

[0112] The purpose of multi-head self-attention is to fully consider the similarity between the input vectors to achieve global dependencies, so that the network can better capture the contextual information in the image. The specific working principle of multi-head self-attention is: First, use the linear transformation matrix W Q , W K , W V After multiplying with the input vector Z, we get Q (Query), K (Key), and V (Value):

[0113] , (14)

[0114] , (15)

[0115] , (16)

[0116] Among them, Q represents the query vector, which is used to determine the content to be focused on in the attention mechanism; K represents the key vector, which is used to calculate the correlation between Q and other positions in Z and provide additional contextual information; V represents the value vector, which contains the position information of each element in Z; W Q , W K , W V Randomly generated and updated during training:

[0117] , (17)

[0118] Where W represents the weight matrix corresponding to the output result; learning_rate represents the learning rate; d loss Represents the change of loss function; d W Represents the change of loss function under weight W.

[0119] In the Transformer module, Q, K, and V are the key elements to implement the self-attention mechanism. The output of the self-attention mechanism is:

[0120] , (18)

[0121] in, represents the attention weight; Indicates the dimension size of the input vector. The essence of the multi-head self-attention mechanism is the linear transformation after the splicing of multiple attention calculation results. This mechanism enables the model to use different feature information obtained at different positions, thereby increasing the diversity of features. The calculation method of multi-head self-attention is as follows:

[0122] , (19)

[0123] , (20)

[0124] In the formula, j represents a natural number; h represents the total number of heads, and the number of heads in the experiment of this embodiment is 4; represents the features of the j-th head, It represents the weight matrix corresponding to the output result, and Concat represents the concatenation operation of the vectors.

[0125] The purpose of layer normalization is to keep the feature value within a smaller range and speed up the convergence of the model. The multi-layer perceptron consists of two fully connected layers and an activation function layer. The purpose is to linearly transform and nonlinearly activate each input feature to enhance the nonlinear expression ability of the model.

[0126] The third part is the BP module. The input layer of the BP module consists of 19 neurons, which input 19 features, namely, the longitude of the mirror reflection point, the latitude of the mirror reflection point, the corrected surface reflectivity, the integrated area, the number of maximum points, the number of minimum points, the skewness, the kurtosis, the average slope, the leading slope, the trailing slope, the DDM mean, the DDM peak, the signal-to-noise ratio, the incident angle, the surface roughness coefficient, the vegetation optical thickness, the vegetation moisture content and the surface temperature. The input layer is responsible for receiving a single feature, passing it to the fully connected layer after normalization. The fully connected layer is responsible for linear combination and feature extraction of the features, and learning the complex relationship between the features by adjusting the weights and biases. For example, in formula (21), the input feature The output of the i-th neuron in the fully connected layer for:

[0127] , (twenty one)

[0128] In the formula, i and j represent natural numbers; x j represents the jth input feature of the neuron, w ij represents the weight value of the connection between the jth feature and the i-th neuron, b i Indicates the bias value, Represents the activation function.

[0129] The fourth part is the multimodal feature fusion part, which aims to effectively integrate feature information from different sources. It mainly includes two specific modules: Cross Fusion Module, such as Figure 2 and the Stacked Fusion Module, as shown Figure 3 The following is a detailed introduction to the cross fusion module and the superposition fusion module:

[0130] (1) Cross-fusion module

[0131] This module is embedded in three branches (Transformer module, 1D-CNN module, BP module). This module focuses on extracting the intermediate layer features of each branch. The extraction positions of the intermediate layers of the three branches are as follows: Figure 1 As shown in the figure, the feature dimensions of each branch are unified through linear mapping to ensure that these features can be effectively fused in the same space. Subsequently, these intermediate layer features are fused through feature addition (Add operation). Feature addition refers to adding two feature maps element by element to increase the amount of information without changing the dimension. The fused features are passed back to their respective branch networks for further processing. It should be noted that the multimodal feature fusion loss Loss2 (see Figure 2 ). The specific calculation method is: concatenate the three input features (f1, f2, f3) and the three output features (f1', f2', f3'), and then use Loss2 to calculate their loss values. The multimodal feature fusion loss function designed in this embodiment aims to ensure that features from different sources can maintain the integrity and similarity of information during the fusion process, and enhance the synergy between the features of each branch, so that the fused features are more accurate. By introducing the cross-fusion module, each branch can not only process its own data independently, but also share and interact with other branches during the fusion process, and effectively absorb useful information from other branches. This design not only enhances the feature interaction between branches, but also reduces the loss of information, thereby achieving more efficient multimodal feature fusion.

[0132] (2) Overlay fusion module

[0133] After each branch network completes feature extraction and fusion, the model uses the early fusion method to concatenate the output features (F1, F2, F3) from the three branches (Concat operation), as shown in the following figure: Figure 3As shown in the figure. Feature concatenation directly combines features from different sources or types head to tail to form a feature vector with a larger dimension, which improves the expressiveness of the features. The concatenated features are input into the last part of MCFN, which is a fully connected module consisting of three layers of neurons (256, 128, and 64) and a loss Loss1 for evaluating the predicted value.

[0134] In this embodiment, both Loss1 and Loss2 use mean square error (MSE) as the loss function during the training process. MSE is a commonly used numerical standard and is calculated as follows:

[0135] , (twenty two)

[0136] Where, L MES represents the mean square error loss value; n represents the total number of sample points; f i represents the actual observed value; y i Represents the model prediction value.

[0137] The two loss functions are respectively established in the feature fusion module and the fully connected module, which together constitute the loss function of this embodiment. The total loss LT can be expressed as:

[0138] , (twenty three)

[0139] This design not only optimizes the feature expression within the branch, but also achieves unified processing of diverse features at the overall level of the model, enabling the model to capture the complex characteristics of soil moisture more accurately.

[0140] In addition, in order to prevent the network from overfitting during the training process, the method of this embodiment adds a Dropout processing module between each fully connected layer of the network, which reduces the complexity of the neural network by randomly discarding neurons and improves the generalization ability of the model. Through the design of these two parts, the MCFN model realizes the effective integration of different feature sources and improves the accuracy of soil moisture inversion.

[0141] S4. Train the multimodal cross-branch fusion network model to obtain the final model.

[0142] Model data input model input features, and the soil moisture results provided by SMAP are used as reference values. Among them, the data set based on SMAP data contains a total of 103,122 sample points. The model is trained and 70% of the data set is selected as the training set. During the training process, the model hyperparameters are first set: learning rate: 0.001; epoch: 100; batch size: 500, optimizer: Adam; loss function: MSE; Dropout parameter: 0.4; the experimental development environment is the pytorch library.

[0143] The MCFN model first obtains the prediction results by passing the data from the input layer to the output layer through layer-by-layer calculations between each network layer, and then adjusts the network parameters based on the gradient descent algorithm to reduce the error index between the prediction results and the reference values. This embodiment uses the Adam optimization algorithm during the training process. The Adam optimizer can automatically adjust the values ​​of the learning rate and momentum according to the current training state, better cope with different data dimensions and model complexity, and thus significantly improve the training effect of the model. Compared with other optimization algorithms, the Adam optimizer obtains momentum by calculating the mean of the gradient. By introducing momentum, the Adam optimizer can consider the influence of the previous gradient when updating the parameters, so that it can be carried out more stably during the training process. The adaptive learning rate dynamically adjusts the size of the learning rate and the update step size of the parameters by calculating the variance of the gradient to meet the update requirements of different parameters. The update principle of the Adam optimizer is:

[0144] , (twenty four)

[0145] Where L represents the loss function; g t Represents the current gradient; m t and v t Represent the mean and variance of the gradient respectively; and Represents the correction result of the estimated value; α represents the learning rate, and the default value is 0.001; β1 and β2 represent the decay exponents of the mean and variance of the gradient respectively; represents a smoothing term, which is used to prevent the denominator from being 0; t is the number of iterations. Finally, the multimodal cross-branch fusion network model is trained to become the final model.

[0146] S5. Use the final model to complete the inversion of soil moisture.

[0147] This embodiment selects 30% of the data set as the test set, inputs it into the trained model, and uses the correlation coefficient (R) and root mean square error (RMSE) as two performance indicators to evaluate the inversion accuracy of the multimodal cross-branch fusion network model. Root mean square error (RMSE) is a commonly used indicator, which is often used to measure the size of the error between predicted data and reference data; and the correlation coefficient (R) is an indicator used to evaluate the degree of association between two variables, and its value range is between -1 and 1. The calculation methods of the root mean square error (RMSE) and the correlation coefficient (R) are:

[0148] , (25)

[0149] , (26)

[0150] Among them, f i represents the actual observed value; y i Represents the model prediction value; n represents the total number of sample points; and represent the mean of the predicted values ​​and the mean of the actual observed values, respectively.

[0151] Since the prediction results of the model are in the form of discrete distribution, they cannot be directly visualized, resulting in the inability to accurately calculate the error with the reference data. Therefore, this embodiment converts the prediction results into a raster image with the same resolution as the SMAP data according to the latitude and longitude positions of the prediction results. Due to the difference in temporal resolution between CYGNSS satellites and SMAP satellites, there will be a problem of multiple prediction results falling into the same SMAP grid during the conversion process. To address the above problem, this embodiment uses the average value of all data falling into the grid to fill in. By converting the prediction results, the distribution law of the prediction results can be intuitively observed, and the accuracy of the prediction results can be calculated. The process framework of this embodiment is as follows: Figure 4 shown.

[0152] Embodiment 2

[0153] This embodiment also provides a GNSS-R soil moisture inversion system based on multimodal feature fusion, including: an acquisition module, an extraction module, a construction module, a training module and an inversion module;

[0154] The acquisition module is used to collect the raw data to be inverted and perform preprocessing to obtain processed data. The raw data includes CYGNSS data sets and SMAP data sets; the preprocessing includes the correction of reflectivity; specifically, firstly, standard quality control is performed to screen out several observation results, CYGNSS observation data with an incident angle greater than 65° is not considered, observation data with a receiver antenna gain less than 0dB is eliminated, and outliers with an SNR less than 0 are removed. Finally, the observation data with a DDM peak in the 7th to 10th time delay frame is selected as the processed data;

[0155] The extraction module is used to obtain model input features based on the processed data, and the steps include: extracting a feature set associated with soil moisture from the processed data as the input of the model, and the model input features are composed of three parts: DDM image, time delay waveform and auxiliary features;

[0156] The construction module is used to construct a multimodal cross-branch fusion network model; the multimodal cross-branch fusion network model includes: a 1D-CNN module, a Transformer module, a BP module and two multimodal feature fusion modules;

[0157] The training module is used to train the multimodal cross-branch fusion network model to obtain the final model; the training steps include: inputting the model input features into the multimodal cross-branch fusion network model, and training with the soil moisture results provided by SMAP as reference values;

[0158] The inversion module is used to complete the inversion of soil moisture using the final model.

[0159] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. GNSS-R soil moisture inversion method based on multi-modal feature fusion, It is characterized in that the steps include: Collecting raw data to be inverted and preprocessing them to obtain processed data, wherein the raw data includes a CYGNSS data set and a SMAP data set; The preprocessing includes correcting the reflectivity; specifically, firstly, a standard quality control is performed to screen out a number of observation results, CYGNSS observation data with an incident angle greater than 65° is not considered, observation data with a receiver antenna gain less than 0 dB is eliminated, and outliers with an SNR less than 0 are removed, and finally, observation data with a DDM peak within the 7th to 10th delay frame is selected as the processed data; Based on the processed data, a model input feature is obtained, the steps comprising: extracting a feature set associated with soil moisture from the processed data as a model input, the model input feature consisting of three parts: a DDM image, a time-delay waveform and an auxiliary feature; Constructing a multimodal cross-branch fusion network model; the multimodal cross-branch fusion network model includes: a 1D-CNN module, a Transformer module, a BP module and two multimodal feature fusion modules; Training the multimodal cross-branch fusion network model to obtain a final model; the training step includes: inputting the model input features into the multimodal cross-branch fusion network model, and training with the soil moisture results provided by SMAP as reference values; The final model is used to complete the inversion of soil moisture.

2. According to the GNSS-R soil moisture inversion method based on multimodal feature fusion according to claim 1, It is characterized in that The method of correcting the reflectivity includes: using a parameterized water cloud model to correct the surface reflectivity: , , , In the formula, Represents the surface reflectivity; represents the part of scattering caused by vegetation; represents the scattering contribution of the soil layer; represents the bidirectional vegetation transmittance; Represents vegetation moisture data extracted by SMAP; represents the satellite incidence angle; A and B represent parameters related to vegetation; take the values ​​applicable to all land, where A=0.0012, B=0.091, and the final corrected surface reflectance is its square: , In the formula, Represents the surface reflectivity after correction by the water cloud model.

3. According to the GNSS-R soil moisture inversion method based on multi-modal feature fusion according to claim 2, It is characterized in that The surface roughness coefficient in the SMAP data is used to further correct the reflectivity: , In the formula, Indicates the roughness coefficient Corrected surface reflectivity; the exponential term represents the signal attenuation caused by roughness; is the surface roughness coefficient in SMAP data; is the satellite incidence angle.

4. The GNSS-R soil moisture inversion method based on multimodal feature fusion according to claim 1, It is characterized in that DDM images include: original count DDM image, analog power DDM image, bistatic radar cross section DDM image and effective scattering area DDM image; There are three types of auxiliary features. The first type is features related to DDM, including corrected surface reflectivity, incident angle, DDM mean, DDM peak and signal-to-noise ratio; the second type is features related to the time-delay waveform, including leading slope, trailing slope, integrated area, number of maximum points, number of minimum points, skewness, kurtosis and average slope; the third type is features related to the surface, including longitude, latitude, vegetation optical thickness, surface roughness, surface temperature and vegetation moisture content.

5. The GNSS-R soil moisture inversion method based on multimodal feature fusion according to claim 1, It is characterized in that The 1D-CNN module consists of three parts: a convolution layer, a pooling layer, and a normalization layer; the 1D-CNN module is used to process one-dimensional data and extract potential information from the delay waveform. The specific workflow includes: connecting the three delay waveforms end to end and inputting them into the 1D-CNN, then normalizing the input data and passing it to the convolution layer; the convolution layer performs a convolution operation on the input data to generate a feature image, and performs a maximum pooling operation with a size of 2×1 and a step size of 2 on the obtained feature map to achieve downsampling; in this process, the first convolution layer uses 16 convolution kernels with a size of 3×1; the second convolution layer uses 32 convolution kernels with a size of 3×1.

6. The GNSS-R soil moisture inversion method based on multimodal feature fusion according to claim 1, It is characterized in that In the Transformer module, a block operation is first performed, that is, the size of the patch is set, the patch is set to 1×1 size, and encoded pixel by pixel to capture and process the feature information of each pixel; then each patch is flattened and combined to form a sequence.

7. The GNSS-R soil moisture inversion method based on multimodal feature fusion according to claim 1, It is characterized in that The BP module includes: an input layer and a fully connected layer; Among them, the input layer is responsible for receiving a single feature, passing it to the fully connected layer after normalization. The fully connected layer is responsible for linear combination and feature extraction of the features, and learning the complex relationship between the features by adjusting the weights and biases.

8. The GNSS-R soil moisture inversion method based on multi-modal feature fusion according to claim 1, It is characterized in that The multimodal feature fusion module includes a cross fusion module and a superposition fusion module; The cross-fusion module is used to extract the intermediate layer features of each branch and unify the feature dimensions of each branch through linear mapping to ensure that these features can be effectively fused in the same space; The superposition fusion module is used to directly combine features from different sources or types head to tail to form a feature vector with a larger dimension, thereby improving the expressiveness of the features.

9. A GNSS-R soil moisture inversion system based on multimodal feature fusion, the system being used to implement the method described in any one of claims 1 to 8, It is characterized in that include: Acquisition module, extraction module, construction module, training module and inversion module; The acquisition module is used to acquire the raw data to be inverted and perform preprocessing to obtain processed data, wherein the raw data includes a CYGNSS data set and a SMAP data set; the preprocessing includes correcting the reflectivity; specifically, firstly, a standard quality control is performed to screen out a number of observation results, CYGNSS observation data with an incident angle greater than 65° is not considered, observation data with a receiver antenna gain less than 0 dB is eliminated, and outliers with an SNR less than 0 are removed, and finally, observation data with a DDM peak within the 7th to 10th time delay frame are selected as the processed data; The extraction module is used to obtain model input features based on the processed data, and the steps include: extracting a feature set associated with soil moisture from the processed data as the input of the model, and the model input features are composed of three parts: DDM image, time delay waveform and auxiliary features; The construction module is used to construct a multimodal cross-branch fusion network model; the multimodal cross-branch fusion network model includes: a 1D-CNN module, a Transformer module, a BP module and two multimodal feature fusion modules; The training module is used to train the multimodal cross-branch fusion network model to obtain a final model; the training step includes: inputting the model input features into the multimodal cross-branch fusion network model, and training with the soil moisture results provided by SMAP as reference values; The inversion module is used to complete the inversion of soil moisture using the final model.

Citation Information

Patent Citations

  • Method for improving GNSS-R altimetry inversion precision based on multi-hidden-layer neural network

    CN115293198A

  • Classification identification method based on convolutional neural network and multi-modal fusion

    CN115438708A

  • Soil humidity inversion method based on multi-branch neural network and sectional model

    CN116821694A

  • UWB fuze target identification method based on physical feature extraction and SIFCDN model

    CN116883724A

  • Method for collaboratively inverting surface soil moisture and vegetation water content by fusing satellite-borne GNSS-R and multi-source RS data

    CN117076819A

Cited By

  • Chinese region soil humidity remote sensing inversion method based on CYGNSS satellite data

    CN119203065A

  • Multi-modal fusion alertness assessment method and system based on cross attention

    CN120124011A

  • Satellite-borne polarization SMAP-R data gap filling method based on multi-polarization SAR data

    CN120763263A

  • A method for filling gaps in spaceborne polarimetric SMAP-R data based on multipolarimetric SAR data

    CN120763263B

  • GNSS-R soil salinity inversion method and system based on cross-modal fusion

    CN121859581A