Remote sensing image reconstruction method, system, equipment and medium based on ground object structure

Abstract

The present invention discloses a method, system, device, and medium for reconstructing remote sensing images based on ground object structure. The method acquires historical satellite multispectral remote sensing data of a monitored area; extracts the ground object structure endmember abundance matrix of the high-temporal-resolution image data based on a pixel purity index algorithm; extracts the global dependency matrix of the historical satellite multispectral remote sensing data based on a multi-channel self-attention mechanism; processes the global dependency matrix based on a feedforward neural network model to obtain global features; fuses the global features with the ground object structure endmember abundance matrix based on a convolutional neural network model to obtain high-resolution feature map data; and iteratively optimizes parameters based on a loss function calculated from the high-resolution feature map data and the high-spatial-resolution image data. The real-time high-temporal-resolution image data is then input into the optimized target image reconstruction model to obtain a reconstructed image. The method provided by the present application reduces the impact of mixed pixels on image reconstruction and improves reconstruction accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of image reconstruction, and in particular to a method, system, equipment and medium for reconstructing remote sensing images based on ground object structure. Background Art

[0002] Agricultural land is land used directly or indirectly for agricultural production and includes a variety of types, including arable land, orchards, woodlands, and pastures. Refined monitoring of agricultural land is crucial for improving agricultural production efficiency, resource management, and ecological protection. However, agricultural land is generally composed of complex land features, such as different types of vegetation, soil, and water bodies. This leads to significant pixel mixing in remote sensing imagery, especially in low-resolution imagery. This poses significant challenges to the refined management of agricultural land monitoring.

[0003] With the development of remote sensing technology, satellite remote sensing has provided a new means for agricultural land monitoring, capable of capturing information on soil changes over large areas in a short period of time. In particular, multispectral remote sensing satellites, which are currently the most widely used, possess a wealth of spatial, temporal, and spectral information. However, when satellite multispectral remote sensing data is applied to agricultural land monitoring, it faces a key challenge: the contradiction between spatial and temporal resolution. Existing technologies achieve a balance between spatial and temporal resolution through high-resolution reconstruction. Existing high-resolution reconstruction methods can improve the spatial resolution of images to a certain extent by extracting spatial features from low-resolution images and performing upsampling reconstruction. However, this method directly processes satellite multispectral remote sensing images and improves spatial resolution based on the geometric features of satellite multispectral remote sensing images. It does not fully consider the structural composition of objects within the same pixel, resulting in large deviations or ambiguity in object classification and distribution, making it difficult to accurately reflect the true situation of agricultural land.

[0004] It can be seen that how to improve the accuracy of remote sensing image reconstruction and thus improve the accuracy of agricultural land monitoring has become a technical problem that needs to be urgently solved by those skilled in the art. Summary of the Invention

[0005] The present invention provides a method, system, device and medium for remote sensing image reconstruction based on land object structure to solve the technical problem of how to improve the accuracy of remote sensing image reconstruction and thus improve the accuracy of agricultural land monitoring, thereby achieving the effect of obtaining high-resolution image data that can accurately reflect the land object structure of agricultural land and improving the accuracy of agricultural land monitoring.

[0006] In a first aspect, the present invention provides a method for reconstructing remote sensing images based on ground object structure, the method comprising: obtaining historical satellite multispectral remote sensing data of a same time period in a to-be-monitored area, the historical satellite multispectral remote sensing data comprising at least high spatial resolution image data and high temporal resolution image data;

[0007] In the process of building the image reconstruction model, each ground object structure end member in the high temporal resolution image data is extracted based on the pixel purity index algorithm;

[0008] Designing an unmixing algorithm for the surface feature structure end members based on the least squares method to obtain a surface feature structure end member abundance matrix of the high temporal resolution image data;

[0009] A global dependency matrix of the historical satellite multispectral remote sensing data is extracted based on a multi-channel self-attention mechanism; the global dependency matrix is ​​linearly transformed and nonlinearly processed based on a selected feedforward neural network model to obtain global features; the global features are fused with the endmember abundance matrix of the ground feature structure based on a selected convolutional neural network model to obtain high-resolution feature map data;

[0010] Constructing a loss function according to the high-resolution feature map data and the high-spatial resolution image data, and performing forward propagation iterative optimization on the multi-channel self-attention mechanism and the feedforward neural network model based on feedback of the loss function to obtain a target image reconstruction model;

[0011] In the actual remote sensing image reconstruction process, the collected high-temporal-resolution image data of the area to be monitored is input into the target image reconstruction model to obtain a reconstructed image of the area to be monitored.

[0012] Preferably, the extraction of each ground object structure end member in the high temporal resolution image data based on the pixel purity index algorithm includes:

[0013] Projecting each pixel spectrum in the high temporal resolution image data onto different pre-set random unit vectors, and calculating the number of times each pixel spectrum falls on both ends of the random unit vector;

[0014] The average value of each of the calculated times is used as the pixel purity index;

[0015] Each ground object structure end member in the high temporal resolution image data is determined according to the pixel purity index and a preset threshold range.

[0016] Preferably, the least squares method is used to design the unmixing algorithm for the surface feature structure endmembers to obtain the surface feature structure endmember abundance matrix of the high temporal resolution image data, including:

[0017] Constructing a ground object structure end member spectral matrix based on the spectral reflectance of each band of each ground object structure end member;

[0018] According to the surface feature structure end-member spectral matrix, an unmixing algorithm is designed by the least squares method. The unmixing algorithm is:

[0019] R(x, y) = E·F+∈(x, y)

[0020] Among them, E represents the surface feature structure end-member spectral matrix, F represents the surface feature structure end-member abundance matrix, ∈(x, y) represents the error term, and R(x, y) represents the reflectance matrix of the corresponding pixel in each band;

[0021] Based on the set endmember abundance constraints, the unmixing algorithm is used to estimate the endmember abundance matrix of each ground feature structure in the high temporal resolution image data.

[0022] Preferably, the extracting the global dependency matrix of the historical satellite multispectral remote sensing data based on the multi-channel self-attention mechanism includes:

[0023] Extracting the reflectivity of each pixel in each band in the historical satellite multispectral remote sensing data, and constructing a reflectivity feature vector based on the reflectivity;

[0024] Performing a linear transformation on the reflectivity feature vector to generate a query vector, a key vector, and a value vector;

[0025] Obtaining an attention weight according to the query vector and the key vector;

[0026] The value vectors are weighted and summed according to the attention weights to obtain a global dependency matrix, where the global dependency matrix at least includes: mutual relationships between adjacent pixels and mutual relationships between different bands.

[0027] Preferably, the linear transformation and nonlinear processing of the global dependency matrix based on the selected feedforward neural network model to obtain global features includes:

[0028] The global dependency matrix is ​​linearly transformed using the first-layer linear transformation of the feedforward neural network model, and nonlinear processing of the activation function is performed to obtain feature variables to be processed;

[0029] The second layer linear transformation of the feedforward neural network model is used to perform linear transformation on the feature variables to be processed to obtain global features.

[0030] Preferably, the global features and the surface feature structure endmember abundance matrix are fused based on the selected convolutional neural network model to obtain high-resolution feature map data, including:

[0031] Splicing the global features and the end-member abundance matrix of the ground feature structure in the channel dimension to obtain a matrix to be fused;

[0032] The matrix to be fused is fused using a point-by-point convolution algorithm to obtain a fused feature matrix to be processed;

[0033] Performing nonlinear processing on the fused feature matrix to be processed to obtain a fused feature matrix;

[0034] Performing multi-layer convolution on the fused feature matrix to obtain low-resolution feature map data;

[0035] The low-resolution feature map data is upsampled to obtain high-resolution feature map data.

[0036] Preferably, the step of constructing a loss function based on the high-resolution feature map data and the high-spatial-resolution image data, and performing forward propagation iterative optimization on the multi-channel self-attention mechanism and the feedforward neural network model based on feedback of the loss function to obtain a target image reconstruction model includes:

[0037] Taking the high spatial resolution image data as a true value, and constructing a loss function of the image reconstruction model according to the true value and the high resolution feature map data;

[0038] The multi-channel self-attention mechanism and the feedforward neural network model are forward propagated and iteratively optimized according to the feedback of the loss function to obtain a target image reconstruction model.

[0039] In a second aspect, the present invention further provides a remote sensing image reconstruction system based on ground object structure, which implements the above-mentioned remote sensing image reconstruction method based on ground object structure, and the system includes: a data acquisition unit, a ground object structure end member extraction unit, a ground object structure end member abundance calculation unit, a feature extraction and fusion unit, a model training unit and a real-time image reconstruction unit;

[0040] The data acquisition unit is used to obtain historical satellite multispectral remote sensing data of the same time period of the monitored area, and the historical satellite multispectral remote sensing data at least includes: high spatial resolution image data and high temporal resolution image data;

[0041] The ground object structure end member extraction unit is used to extract each ground object structure end member in the high temporal resolution image data based on the pixel purity index algorithm during the image reconstruction model construction process;

[0042] The ground object structure end member abundance calculation unit is used to design an unmixing algorithm for the ground object structure end members based on the least squares method to obtain a ground object structure end member abundance matrix of the high temporal resolution image data;

[0043] The feature extraction and fusion unit is used to extract the global dependency matrix of the historical satellite multispectral remote sensing data based on a multi-channel self-attention mechanism; perform linear transformation and nonlinear processing on the global dependency matrix based on a selected feedforward neural network model to obtain global features; and fuse the global features with the endmember abundance matrix of the ground feature structure based on a selected convolutional neural network model to obtain high-resolution feature map data;

[0044] The model training unit is configured to construct a loss function based on the high-resolution feature map data and the high-spatial-resolution image data, and perform forward propagation iterative optimization on the multi-channel self-attention mechanism and the feedforward neural network model based on feedback from the loss function to obtain a target image reconstruction model;

[0045] The real-time image reconstruction unit is used to input the collected real-time high-temporal-resolution image data of the area to be monitored into the target image reconstruction model during the actual remote sensing image reconstruction process to obtain a reconstructed image of the area to be monitored.

[0046] In a third aspect, the present invention also provides a computer device, which includes a memory, a processor and a transceiver, which are connected via a bus; the memory is used to store a set of computer program instructions and data, and transmit the stored data to the processor, and the processor executes the program instructions stored in the memory to execute the above-mentioned remote sensing image reconstruction method based on ground object structure.

[0047] In a fourth aspect, the present invention further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed, the above-mentioned L remote sensing image reconstruction method based on the ground object structure is implemented.

[0048] The present invention provides a method, system, device, and medium for reconstructing remote sensing images based on ground object structure. Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0049] (1) The satellite multispectral remote sensing data with high temporal resolution but low spatial resolution is improved through super-resolution technology, while retaining the high temporal resolution. The resulting reconstructed remote sensing image has both high temporal and high spatial resolution.

[0050] (2) In view of the complex land object structure of agricultural land images, a land object structure end-member abundance acquisition module was designed, and the global features and land object structure end-member abundance matrix were fused, which greatly reduced the impact of mixed pixels on image reconstruction, further improved the accuracy of super-resolution image reconstruction, and could better capture the detailed features of agricultural land. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a schematic diagram of the steps of a method for reconstructing a remote sensing image based on ground object structure provided by a preferred embodiment of the present invention;

[0052] Figure 2 This is a schematic structural diagram of a remote sensing image reconstruction system based on ground object structure provided by a preferred embodiment of the present invention;

[0053] Figure 3 It is a structural diagram of a computer device provided in one embodiment of the present invention. DETAILED DESCRIPTION

[0054] The following is a detailed explanation of the embodiments of the present invention in conjunction with the accompanying drawings. The embodiments are provided for illustrative purposes only and cannot be understood as limitations on the present invention. The accompanying drawings are for reference and illustration purposes only and do not constitute a limitation on the scope of protection of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In the description of the present invention, the terms "first", "second", "third", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first", "second", "third", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, the meaning of "multiple" is two or more.

[0055] In the description of the present invention, it should be noted that, unless otherwise expressly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be a communication between the two components. The terms "vertical", "horizontal", "left", "right", "up", "down" and similar expressions used herein are for illustrative purposes only, and do not indicate or imply that the device or component referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0056] In describing the present invention, it should be noted that, unless otherwise defined, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. Those skilled in the art will understand the specific meanings of the above terms in the present invention in specific circumstances.

[0057] Satellite multispectral remote sensing data with high spatial resolution can accurately capture details of agricultural land, but its low temporal resolution (16-day revisit period) cannot promptly respond to dynamic changes in crop growth, especially in scenarios requiring rapid decision-making, such as farmland management and pest control. Conversely, remote sensing data with low spatial resolution images, while having a high temporal resolution (1-2 days), often lacks sufficient spatial resolution (250 / 500 meters) to provide sufficient ground feature details. This is especially true in complex mixed ground feature areas of agricultural land, such as the interface between vegetation and soil, or irrigation and farmland. This low spatial resolution of satellite multispectral remote sensing data cannot accurately describe ground feature distribution, resulting in suboptimal monitoring results.

[0058] In view of this, in an embodiment of the present invention, a remote sensing image reconstruction method based on ground object structure is provided. Figure 1 , the method comprising:

[0059] S1. Acquire historical satellite multispectral remote sensing data of the same time period of the area to be monitored, wherein the historical satellite multispectral remote sensing data at least includes: high spatial resolution image data and high temporal resolution image data.

[0060] S2. During the image reconstruction model construction process, each ground object structure end member in the high temporal resolution image data is extracted based on a pixel purity index algorithm.

[0061] S3. Designing an unmixing algorithm for the surface feature structure end members based on the least squares method to obtain a surface feature structure end member abundance matrix of the high temporal resolution image data.

[0062] S4. Extract the global dependency matrix of the historical satellite multispectral remote sensing data based on a multi-channel self-attention mechanism; perform linear transformation and nonlinear processing on the global dependency matrix based on a selected feedforward neural network model to obtain global features; and fuse the global features and the end-member abundance matrix of the land feature structure based on a selected convolutional neural network model to obtain high-resolution feature map data.

[0063] S5. Construct a loss function according to the high-resolution feature map data and the high-spatial resolution image data, and perform forward propagation iterative optimization on the multi-channel self-attention mechanism and the feedforward neural network model based on the value of the loss function to obtain a target image reconstruction model.

[0064] S6. In the actual remote sensing image reconstruction process, the collected real-time high-temporal-resolution image data of the area to be monitored is input into the target image reconstruction model to obtain a reconstructed image of the area to be monitored.

[0065] The method for reconstructing remote sensing images based on ground feature structure disclosed in the preferred embodiment of the present invention is primarily applied to agricultural land. Therefore, this method uses agricultural land as an example for illustration. Satellite multispectral remote sensing data is acquired for the agricultural land area to be monitored. Historical satellite multispectral remote sensing data includes at least high spatial resolution image data and high temporal resolution image data. The satellite multispectral remote sensing data used in the present invention comes from two sources: MODIS data and Landsat-8 data.

[0066] MODIS data has a high temporal resolution of 1-2 days, which is suitable for dynamic monitoring of large-scale agricultural land, but the spatial resolution is relatively low, at 250m / 500m. The selected bands include blue, green, red, near-infrared and short-wave infrared, a total of 5 bands.

[0067] Landsat-8 data has a high spatial resolution of 30 meters, which is used to obtain detailed structural information of agricultural land. However, its temporal resolution is lower, at 16 days. Similarly, five bands were selected: blue, green, red, near-infrared, and short-wave infrared.

[0068] Both the MODIS and Landsat-8 data can be downloaded through Google Earth Engine (GEE). During download, MODIS and Landsat-8 data were selected for overlapping periods in the same area. These data were clipped using a polygon vector file of agricultural land areas and masked for cloud and shadow effects to ensure consistent quality and coverage between the two data types. All MODIS data bands were normalized to 250 meters.

[0069] Furthermore, in a preferred embodiment of the present invention, during the image reconstruction model construction process, the constructed image reconstruction model includes a surface feature structure endmember abundance acquisition module, a multi-channel self-attention mechanism, a feedforward neural network model, and a convolutional neural network model. The surface feature structure endmember abundance acquisition module extracts individual surface feature structure endmembers from high-temporal-resolution image data, i.e., individual surface feature structure endmembers in MODIS data, based on pixel purity index, and designs a surface feature structure endmember unmixing algorithm based on the least squares method to obtain a surface feature structure endmember abundance matrix for the high-temporal-resolution image data.

[0070] Due to the limited spatial resolution of remote sensing satellites, a single pixel of satellite multispectral remote sensing data usually cannot contain only one type of land object, especially in agricultural land, woodland or areas with complex vegetation coverage, but rather contains mixed pixels of multiple land object types. Mixed pixels will contain spectral information of multiple land object structure types, which will lead to a decrease in accuracy during image classification and reconstruction. In a preferred embodiment of the present invention, a land object structure endmember abundance matrix is ​​introduced. The land object structure endmember abundance matrix is ​​a matrix composed of the proportions of different land object structure types in each mixed pixel. Before obtaining the land object structure endmember abundance matrix, it is necessary to extract each land object structure endmember in the high temporal resolution image data. The extraction of each land object structure endmember is based on the pixel purity index. The pixel purity index (PPI) is a geometry-based method used to identify the "purest" pixels in the image. These pixels often represent a single land object type, that is, each land object structure endmember, rather than a mixed land object.

[0071] The pixel purity index algorithm treats each pixel in the pixel purity index as an n-dimensional vector, with all pixels forming a vector space. While a basis in a vector space is not unique, it must exist entirely composed of vectors located at the boundary. These boundary vectors, when projected onto a randomly generated unit vector, have the highest probability of appearing at either end of the random unit vector. By calculating the mean of multiple iterative projections, the purity index of each pixel can be calculated, thereby identifying the end members of each feature structure.

[0072] In a preferred embodiment of the present invention, a threshold range is constructed based on the pixel purity index of all land object types contained in the agricultural land. The high temporal resolution image data is subjected to dimensionality reduction processing to reduce the amount of calculation and remove noise. Further, the spectrum of each pixel in the high temporal resolution image data after dimensionality reduction processing is projected onto a set of pre-set random unit vectors, and the number of times each pixel spectrum falls on both ends of the random unit vector is calculated. Using different random unit vectors, the average of the number of times each pixel spectrum falls on both ends of the random unit vector is calculated, and the average is used as the pixel purity index. The structural end members of each land object in the high temporal resolution image data are determined based on the pixel purity index and the pre-constructed threshold range. Specifically, the structural end members of each land object in the high temporal resolution image data are determined based on the overlapping area of ​​the pixel purity index and the threshold range.

[0073] After determining each feature structure endmember in the high temporal resolution image data, a feature structure endmember unmixing algorithm is designed using the least squares method for the feature structure endmembers, and the feature structure endmember abundance matrix of the high temporal resolution image data is estimated using the unmixing algorithm. Specifically, a feature structure endmember spectral matrix is ​​constructed based on the spectral reflectance of each band of each feature structure endmember. Based on the feature structure endmember spectral matrix and the characteristics of agricultural land, an unmixing algorithm is designed using the least squares method. The unmixing algorithm is a linear equation, and the expression of the linear equation is:

[0074] R(x, y) = E·F+∈(x, y)

[0075] Where E represents the surface feature endmember spectral matrix, F represents the surface feature endmember abundance matrix, ∈(x, y) represents the error term, and R(x, y) represents the reflectance matrix of the corresponding pixel in each band. The surface feature endmember spectral matrix has the shape of C × A, where C is the number of input bands and A is the number of surface feature endmembers.

[0076] The surface feature structure end-member abundance matrix represents the matrix composed of the proportions of each surface feature structure end-member in the high temporal resolution image data. The surface feature structure end-member abundance matrix is ​​obtained by solving the linear equation by minimizing the error term.

[0077] In addition, to ensure the physical rationality of the unmixing results, the feature structure end-member abundance matrix of the present invention needs to meet the following two constraints:

[0078] 1) Non-negativity constraint: the abundance f of each surface feature structure end member i ≥0, indicating that the feature scale cannot be negative.

[0079] 2) Sum constraint: the sum of the abundances of all end members of the ground feature structure must be 1.

[0080] The final output of the surface feature structure end member abundance matrix F is:

[0081]

[0082] Furthermore, a global dependency matrix of historical satellite multispectral remote sensing data is extracted based on a multi-channel self-attention mechanism. This global dependency matrix includes at least the interrelationships between adjacent pixels and the interrelationships between different bands. The interrelationships between adjacent pixels refer to the spatial dependencies between adjacent pixels, including the correlation or mutual information between adjacent pixels. The interrelationships between different bands refer to the spectral dependencies or correlations between different bands in the historical satellite multispectral remote sensing data.

[0083] The global dependency matrix emphasizes the integrity and correlation between regions in historical satellite multispectral remote sensing data. Compared with local dependencies, the global dependency matrix focuses more on capturing the global structure and contextual information in the image. This dependency typically covers a wider spatial range and can therefore provide richer semantic information.

[0084] The multi-channel self-attention mechanism calculates the similarity between pixel features, generates a weight matrix, and then weights the input satellite multispectral remote sensing data. This mechanism can focus on local and long-range feature correlations in satellite multispectral remote sensing data and incorporate correlation information between different bands into feature extraction, effectively improving the performance of the proposed remote sensing image reconstruction method based on ground object structure in areas with complex ground object backgrounds.

[0085] Specifically, the reflectivity of each pixel in each band in the historical satellite multispectral remote sensing data is extracted, and the reflectivity feature vector is constructed according to the reflectivity.

[0086] Historical satellite multispectral remote sensing data has multiple spectral bands. Multiple channels of the multi-channel self-attention mechanism are input, and the input matrix is ​​set as:

[0087] X∈R H×W×C

[0088] Where H and W are the height and width of the image respectively.

[0089] For each pixel of the historical satellite multispectral remote sensing data input, define its reflectance feature vector X i for:

[0090] X i =[X i (1) , X i (2) ,...,X i (C) ]

[0091] Among them, X i (C) Represents the reflectance of the i-th pixel in the C-th band.

[0092] Furthermore, the reflectivity feature vector is linearly transformed to generate the query vector, key vector and value vector, which are:

[0093] Q i =W Q ·X i +b Q

[0094] K i =W k ·Xi +b K

[0095] V i =W v ·X i +b v

[0096] Among them, Q i , K i and V i Represent the query vector, key vector and value vector of the i-th pixel respectively, W Q 、W K 、W V represents the trainable weight matrix, b Q 、b K 、b V represents the bias term.

[0097] Furthermore, the attention score between each pixel is calculated based on the dot product of the query vector and the key vector. The calculation formula of the attention score is:

[0098]

[0099] Here, dk is the dimension of the key vector.

[0100] Furthermore, the attention score is normalized to obtain the attention weight. The calculation formula of the attention weight is:

[0101]

[0102] Among them, N represents the number of pixels, α ij It represents the attention weight of the i-th pixel to the j-th pixel, reflecting the correlation between the two pixels. The higher the attention weight, the stronger the dependency between the two pixels.

[0103] Furthermore, the value vectors are weighted and summed according to the attention weights to obtain the global dependency matrix. The expression of the global dependency matrix is:

[0104]

[0105] Among them, V j Represents the value vector of the j-th pixel, Z i Represents the global dependency matrix output of the i-th pixel. The global dependency matrix combines the information of other pixels, including not only the pixel's own information, but also the information associated with other pixels and the pixel.

[0106] At the same time, in order to enhance the ability to capture multi-scale features, the multi-channel self-attention mechanism of this application is designed as a multi-head self-attention mechanism. Specifically, the input historical satellite multispectral remote sensing data is divided into multiple subspaces, and the attention weight is calculated in each subspace separately. Finally, the attention weights of each subspace are spliced. The processing process expression is:

[0107] MultiHead(Q,K,V)=Concat(head1,...,head h )·W O

[0108] Among them, head i represents the output of the i-th attention head, h is the number of attention heads, W O A weight matrix representing a linear transformation.

[0109] Through the multi-head attention mechanism, the image reconstruction model can focus on different features in different subspaces in parallel.

[0110] Among them, the linear change matrix W O It is a trainable weight matrix with the dimension of (h×d)×d′, where d is the dimension of each attention head output and d′ is the dimension of the final output feature.

[0111] The final output global dependency matrix is ​​expressed as:

[0112] Z=MultiHead(Q,K,V)=[Z1,Z2,...,Z M ]

[0113] Among them, Z M represents the global dependency matrix of the Mth subspace.

[0114] Furthermore, based on the selected feedforward neural network model, the global dependency matrix is ​​linearly transformed and nonlinearly processed to obtain global features. The role of the feedforward neural network model is to further enhance the expression ability of the global dependency matrix extracted by the multi-channel self-attention mechanism, and to improve the model's ability to distinguish complex land features through nonlinear transformation. The feedforward neural network in the embodiment of the present invention is composed of two layers of linear transformation and an activation function ReLU, wherein the first layer of linear transformation performs a linear transformation on the global dependency matrix, and then performs nonlinear processing of the activation function to obtain the feature variables to be processed. The global dependency matrix of each pixel will pass through the feedforward neural network model respectively, and the processing process expression is:

[0115] h i =ReLU(W1Z i +b1)

[0116] Among them, h i represents the feature variable to be processed of the i-th pixel, W1 and b1 are the weight matrix and bias vector of the first layer linear transformation, Z i represents the global dependency matrix of the i-th pixel, and ReLU represents the activation function.

[0117] Furthermore, the second-layer linear transformation performs a linear transformation on the feature variables to be processed to obtain global features. The processing process expression is:

[0118] Z′ i =W2h i +b2

[0119] Among them, W2 and b2 are the weight matrix and bias vector of the second layer linear transformation.

[0120] Furthermore, the global features and the end-member abundance matrix of the ground feature structure are fused based on the selected convolutional neural network model to obtain high-resolution feature map data.

[0121] In a preferred embodiment of the present invention, the key to remote sensing image reconstruction is to fuse the global features output by the feedforward neural network model with the endmember abundance matrix of the ground feature structure after hybrid decomposition. The specific fusion process is as follows:

[0122] First, the global features and the endmember abundance matrix of the feature structure are spliced ​​in the channel dimension to obtain the matrix to be fused. The expression of the matrix to be fused is:

[0123] H = [Z′, F]

[0124] Among them, the shape of the spliced ​​matrix H to be fused is N×(C+A), that is, each pixel position contains global features and local biomass abundance information.

[0125] Furthermore, the point-by-point convolution algorithm is used to fuse the matrix to be fused. The point-by-point convolution algorithm can learn the relationship between the global features and the end-member abundance matrix of the ground feature structure, and obtain the fused feature matrix to be processed. The expression of the fused feature matrix to be processed is:

[0126] F fused =Conv 1×1 (H)

[0127] The point-by-point convolution algorithm maps the C+A dimensional features into a new feature space to generate the fused feature matrix F to be processed. fused Represents a complex association matrix that contains both global features and abundance information.

[0128] Furthermore, the activation function ReLU is used to perform nonlinear processing on the fused feature matrix to be processed, thereby enhancing the ability of the fused feature matrix to express complex features and obtaining the fused feature matrix. The expression of the fused feature matrix is:

[0129] F′ fused =ReLU(F fused )

[0130] Furthermore, multi-layer convolution is performed on the fused feature matrix to perform further feature extraction to obtain low-resolution feature map data. The low-resolution feature map data is expressed as:

[0131] F conv =Conv(F′ fused )

[0132] Among them, F conv It is the low-resolution feature map data after multiple layers of convolution. Although the feature matrix of low-resolution feature map data is smaller in size, it still retains spatial structure and detail information. Low-resolution feature map data can be regarded as a low-dimensional representation of the image.

[0133] Furthermore, the low-resolution feature map data is upsampled by the upsampling layer to obtain high-resolution feature map data. Specifically, the low-resolution feature map data is input to the upsampling layer for deconvolution. Upsampling is the inverse operation of the convolution layer. A specific convolution kernel is learned to amplify the low-resolution feature map into a high-resolution feature map. The high-resolution feature map data is represented as:

[0134] I high-res =TransposedConv(F conv )

[0135] Among them, I high-res Represents high-resolution feature map data.

[0136] After obtaining the high-resolution feature map data, a loss function is constructed based on the high-resolution feature map data and high-spatial resolution image data, and the multi-channel self-attention mechanism and feedforward neural network model are forward propagated and iteratively optimized based on the feedback of the loss function to obtain the target image reconstruction model.

[0137] In a preferred embodiment of the present invention, high spatial resolution image data is used as the true value of the image reconstruction model, and a loss function is constructed based on the true value and high-resolution feature map data to perform forward propagation iterative optimization on the multi-channel self-attention mechanism and feedforward neural network model.

[0138] Using high spatial resolution image data as the ground truth of the image reconstruction model Landsat , use the high-resolution feature map as the initial reconstruction value I pred, the loss function is calculated based on the true value and the initial reconstruction value. The role of the loss function is to guide the image reconstruction model to gradually adjust the weight parameters of the multi-channel self-attention mechanism and the feedforward neural network model in each forward propagation and backpropagation. The loss function used in the embodiment of the present invention includes:

[0139] L1 loss (absolute error):

[0140]

[0141] L2 loss (mean square error):

[0142]

[0143] The values ​​of L1 and L2 will serve as a feedback signal to guide the update of the weight parameters of the image reconstruction model in back propagation. That is, the error calculated by the loss function will be transmitted back to the multi-channel self-attention mechanism and the feedforward neural network model through the chain rule to update the weight parameters of the multi-channel self-attention mechanism and the feedforward neural network model, so that the high-resolution feature map data generated during the next forward propagation will be closer to the true value.

[0144] When the loss function finally converges, the optimization of the image reconstruction model is completed, and the target image reconstruction model is obtained for subsequent actual remote sensing image reconstruction.

[0145] During the actual remote sensing image reconstruction process, the real-time, high-temporal-resolution image data collected from the agricultural land area to be monitored is input into the target image reconstruction model to obtain a reconstructed image of the agricultural land to be monitored. Specifically, the collected real-time, high-spatial-resolution image data is super-reconstructed through a surface feature structure endmember abundance acquisition module, a multi-channel self-attention mechanism, a feedforward neural network model, and a convolutional neural network model. Ultimately, a remote sensing image of the agricultural land with both high spatial and high temporal resolution is generated. This improves the ability to capture the dynamic changes of different surface features in the agricultural land, providing key data support for precision agriculture, land management, and ecological monitoring.

[0146] In a preferred embodiment of the present invention, historical satellite multispectral remote sensing data of the same time period of the monitored area is obtained, and the historical satellite multispectral remote sensing data includes at least high spatial resolution image data and high temporal resolution image data; in the process of constructing an image reconstruction model, each ground object structure end member in the high temporal resolution image data is extracted based on a pixel purity index algorithm; an unmixing algorithm for the ground object structure end member is designed based on the least squares method to obtain a ground object structure end member abundance matrix of the high temporal resolution image data; a global dependency matrix of the historical satellite multispectral remote sensing data is extracted based on a multi-channel self-attention mechanism; the global dependency matrix is ​​linearly transformed and nonlinearly processed based on a selected feedforward neural network model to obtain global features; the global features and the ground object structure end member abundance matrix are fused based on a selected convolutional neural network model to obtain high-resolution feature map data; a loss function is calculated based on the high-resolution feature map data and the high spatial resolution image data, and the multi-channel self-attention mechanism and the feedforward neural network model are forward propagated and iteratively optimized based on the loss function to obtain a target image reconstruction model; in the actual remote sensing image reconstruction process, the collected real-time high temporal resolution image data of the monitored area are input into the target image reconstruction model to obtain a reconstructed image of the monitored area. The remote sensing image reconstruction method based on land object structure provided in this application generates reconstructed remote sensing images with both high temporal and spatial resolution. In view of the characteristics of the complex land object structure of agricultural land images, a land object structure end-member abundance acquisition module is designed to fuse the global features and the land object structure end-member abundance matrix, greatly reducing the impact of mixed pixels on image reconstruction, further improving the accuracy of super-resolution image reconstruction, and better capturing the detailed features of agricultural land.

[0147] Accordingly, if Figure 2 As shown, based on a method for reconstructing a remote sensing image based on a ground object structure, an embodiment of the present invention further provides a remote sensing image reconstruction system based on a ground object structure, which implements the method for reconstructing a remote sensing image based on a ground object structure disclosed in an embodiment of the present invention. The system is applied to agricultural land monitoring, and the system includes: a data acquisition unit 1, a ground object structure end member extraction unit 2, a ground object structure end member abundance calculation unit 3, a feature extraction and fusion unit 4, a model training unit 5, and a real-time image reconstruction unit 6;

[0148] The data acquisition unit 1 is used to obtain historical satellite multispectral remote sensing data of the same time period of the monitored area, and the historical satellite multispectral remote sensing data at least includes: high spatial resolution image data and high temporal resolution image data;

[0149] The ground object structure end member extraction unit 2 is used to extract each ground object structure end member in the high temporal resolution image data based on the pixel purity index algorithm during the image reconstruction model construction process;

[0150] The surface feature structure end member abundance calculation unit 3 is used to design an unmixing algorithm for the surface feature structure end members based on the least squares method to obtain a surface feature structure end member abundance matrix of the high temporal resolution image data;

[0151] The feature extraction and fusion unit 4 is used to extract the global dependency matrix of the historical satellite multispectral remote sensing data based on a multi-channel self-attention mechanism; perform linear transformation and nonlinear processing on the global dependency matrix based on a selected feedforward neural network model to obtain global features; and fuse the global features with the endmember abundance matrix of the ground feature structure based on a selected convolutional neural network model to obtain high-resolution feature map data;

[0152] The model training unit 5 is used to construct a loss function based on the high-resolution feature map data and the high-spatial resolution image data, and perform forward propagation iterative optimization on the multi-channel self-attention mechanism and the feedforward neural network model based on feedback from the loss function to obtain a target image reconstruction model;

[0153] The real-time image reconstruction unit 6 is used to input the collected real-time high-temporal-resolution image data of the area to be monitored into the target image reconstruction model during the actual remote sensing image reconstruction process to obtain a reconstructed image of the area to be monitored.

[0154] For the specific definition of a remote sensing image reconstruction system based on the structure of land objects, please refer to the above-mentioned definition of a remote sensing image reconstruction method based on the structure of land objects, which will not be repeated here. A person of ordinary skill in the art will appreciate that the modules and steps described in conjunction with the embodiments disclosed in the present invention can be implemented in hardware, software, or a combination of both. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0155] like Figure 3 As shown, an embodiment of the present invention provides a computer device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the steps in the above-mentioned embodiment of the remote sensing image reconstruction method based on the structure of the object are implemented, for example Figure 1 Steps S1 to S6 described in .

[0156] Those skilled in the art will understand that the schematic Figure 3These are merely examples of computer devices and do not constitute limitations on the computer device. The computer device may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device may also include input and output devices, network access devices, buses, etc.

[0157] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the computer device, connecting various parts of the entire computer device using various interfaces and lines.

[0158] The memory can be used to store the computer programs and / or modules, and the processor implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created based on the use of the mobile phone (such as audio data, a phone book, etc.). In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0159] Wherein, if the module integrated in the computer device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0160] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0161] Accordingly, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the steps of the method for reconstructing remote sensing images based on the structure of objects in the above embodiment, for example Figure 1 Steps S1 to S6 described in .

[0162] The present embodiment provides a method, system, device, and medium for reconstructing remote sensing images based on ground object structure, which are used to solve the technical problem of improving the accuracy of remote sensing image reconstruction and thereby improving the accuracy of agricultural land monitoring. Acquire historical satellite multispectral remote sensing data for the same time period of the area to be monitored, the historical satellite multispectral remote sensing data including at least high spatial resolution image data and high temporal resolution image data; extract each ground object structure end member in the high temporal resolution image data based on a pixel purity index algorithm; design a ground object structure end member unmixing algorithm based on the least squares method to obtain a ground object structure end member abundance matrix of the high temporal resolution image data; extract a global dependency matrix of the historical satellite multispectral remote sensing data based on a multi-channel self-attention mechanism; perform linear transformation and nonlinear processing on the global dependency matrix based on a selected feedforward neural network model to obtain global features; fuse the global features and the ground object structure end member abundance matrix based on a selected convolutional neural network model to obtain high-resolution feature map data; construct a loss function based on the high-resolution feature map data and the high spatial resolution image data, and perform forward propagation iterative optimization on the multi-channel self-attention mechanism and the feedforward neural network model based on feedback of the loss function to obtain a target image reconstruction model; in the actual remote sensing image reconstruction process, input the collected real-time high temporal resolution image data of the area to be monitored into the target image reconstruction model to obtain a reconstructed image of the area to be monitored. The remote sensing image reconstruction method based on land object structure provided in this application generates reconstructed remote sensing images with both high temporal and spatial resolution. In view of the characteristics of the complex land object structure of agricultural land images, a land object structure end-member abundance acquisition module is designed to fuse the global features and the land object structure end-member abundance matrix, greatly reducing the impact of mixed pixels on image reconstruction, further improving the accuracy of super-resolution image reconstruction, and better capturing the detailed features of agricultural land.

[0163] Each embodiment in this specification is described in a progressive manner, and the same or similar parts of each embodiment can be directly referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. It should be noted that the various technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the various technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0164] The above-described embodiments merely represent several preferred embodiments of the present invention. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that a person skilled in the art could make several improvements and substitutions without departing from the technical principles of the present invention, and such improvements and substitutions should also be considered within the scope of the present invention. Therefore, the scope of the present invention should be determined by the scope of the claims.

Claims

1. A remote sensing image reconstruction method based on ground object structure, characterized in that: The method comprises: Acquire historical satellite multispectral remote sensing data of the same time period of the area to be monitored, wherein the historical satellite multispectral remote sensing data includes at least: high spatial resolution image data and high temporal resolution image data; In the process of building the image reconstruction model, each ground object structure end member in the high temporal resolution image data is extracted based on the pixel purity index algorithm; Designing an unmixing algorithm for the surface feature structure end members based on the least squares method to obtain a surface feature structure end member abundance matrix of the high temporal resolution image data; A global dependency matrix of the historical satellite multispectral remote sensing data is extracted based on a multi-channel self-attention mechanism; the global dependency matrix is ​​linearly transformed and nonlinearly processed based on a selected feedforward neural network model to obtain global features; the global features are fused with the endmember abundance matrix of the ground feature structure based on a selected convolutional neural network model to obtain high-resolution feature map data; Constructing a loss function according to the high-resolution feature map data and the high-spatial resolution image data, and performing forward propagation iterative optimization on the multi-channel self-attention mechanism and the feedforward neural network model based on feedback of the loss function to obtain a target image reconstruction model; In the actual remote sensing image reconstruction process, the collected real-time high-time resolution image data of the area to be monitored is input into the target image reconstruction model to obtain a reconstructed image of the area to be monitored.

2. The method for reconstructing remote sensing images based on ground object structure according to claim 1, wherein: The method of extracting each ground object structure end member in the high temporal resolution image data based on the pixel purity index algorithm includes: Projecting each pixel spectrum in the high temporal resolution image data onto different pre-set random unit vectors, and calculating the number of times each pixel spectrum falls on both ends of the random unit vector; The average value of each of the calculated times is used as the pixel purity index; Each ground object structure end member in the high temporal resolution image data is determined according to the pixel purity index and a preset threshold range.

3. The method for reconstructing remote sensing images based on ground object structure according to claim 1, wherein: The least squares method is used to design the unmixing algorithm for the surface feature structure end members to obtain the surface feature structure end member abundance matrix of the high temporal resolution image data, including: Constructing a ground object structure end member spectral matrix based on the spectral reflectance of each band of each ground object structure end member; According to the surface feature structure end-member spectral matrix, an unmixing algorithm is designed by the least squares method. The unmixing algorithm is: R(x, y) = E·F+∈(x, y), where E represents the surface feature structure end-member spectral matrix, F represents the surface feature structure end-member abundance matrix, ∈(x, y) represents the error term, and R(x, y) represents the reflectance matrix of the corresponding pixel in each band. Based on the set endmember abundance constraints, the unmixing algorithm is used to estimate the endmember abundance matrix of each ground feature structure in the high temporal resolution image data.

4. The method for reconstructing remote sensing images based on ground object structure according to claim 1, wherein: The method of extracting the global dependency matrix of the historical satellite multispectral remote sensing data based on the multi-channel self-attention mechanism includes: Extracting the reflectivity of each pixel in each band in the historical satellite multispectral remote sensing data, and constructing a reflectivity feature vector based on the reflectivity; Performing a linear transformation on the reflectivity feature vector to generate a query vector, a key vector, and a value vector; Obtaining an attention weight according to the query vector and the key vector; The value vectors are weighted and summed according to the attention weights to obtain a global dependency matrix, where the global dependency matrix at least includes: mutual relationships between adjacent pixels and mutual relationships between different bands.

5. The method for reconstructing remote sensing images based on ground object structure according to claim 1, wherein: The linear transformation and nonlinear processing of the global dependency matrix based on the selected feedforward neural network model to obtain global features include: The global dependency matrix is ​​linearly transformed using the first-layer linear transformation of the feedforward neural network model, and nonlinear processing of the activation function is performed to obtain feature variables to be processed; The second layer linear transformation of the feedforward neural network model is used to perform linear transformation on the feature variables to be processed to obtain global features.

6. The method for reconstructing remote sensing images based on ground object structure according to claim 1, wherein: The global features and the end-member abundance matrix of the ground feature structure are fused based on the selected convolutional neural network model to obtain high-resolution feature map data, including: Splicing the global features and the end-member abundance matrix of the ground feature structure in the channel dimension to obtain a matrix to be fused; The matrix to be fused is fused using a point-by-point convolution algorithm to obtain a fused feature matrix to be processed; Performing nonlinear processing on the fused feature matrix to be processed to obtain a fused feature matrix; Performing multi-layer convolution on the fused feature matrix to obtain low-resolution feature map data; The low-resolution feature map data is upsampled to obtain high-resolution feature map data.

7. The method for reconstructing remote sensing images based on ground object structure according to claim 1, wherein: The method constructs a loss function based on the high-resolution feature map data and the high-spatial resolution image data, and performs forward propagation iterative optimization on the multi-channel self-attention mechanism and the feedforward neural network model based on feedback of the loss function to obtain a target image reconstruction model, including: Taking the high spatial resolution image data as a true value, and constructing a loss function of the image reconstruction model according to the true value and the high resolution feature map data; The multi-channel self-attention mechanism and the feedforward neural network model are forward propagated and iteratively optimized according to the feedback of the loss function to obtain a target image reconstruction model.

8. A remote sensing image reconstruction system based on ground object structure, which implements the remote sensing image reconstruction method based on ground object structure according to any one of claims 1 to 7, characterized in that: The system includes: a data acquisition unit, a ground object structure end member extraction unit, a ground object structure end member abundance calculation unit, a feature extraction and fusion unit, a model training unit and a real-time image reconstruction unit; The data acquisition unit is used to obtain historical satellite multispectral remote sensing data of the same time period of the monitored area, and the historical satellite multispectral remote sensing data at least includes: high spatial resolution image data and high temporal resolution image data; The ground object structure end member extraction unit is used to extract each ground object structure end member in the high temporal resolution image data based on the pixel purity index algorithm during the image reconstruction model construction process; The ground object structure end member abundance calculation unit is used to design an unmixing algorithm for the ground object structure end members based on the least squares method to obtain a ground object structure end member abundance matrix of the high temporal resolution image data; The feature extraction and fusion unit is used to extract the global dependency matrix of the historical satellite multispectral remote sensing data based on a multi-channel self-attention mechanism; perform linear transformation and nonlinear processing on the global dependency matrix based on a selected feedforward neural network model to obtain global features; and fuse the global features with the endmember abundance matrix of the ground feature structure based on a selected convolutional neural network model to obtain high-resolution feature map data; The model training unit is configured to construct a loss function based on the high-resolution feature map data and the high-spatial-resolution image data, and perform forward propagation iterative optimization on the multi-channel self-attention mechanism and the feedforward neural network model based on feedback from the loss function to obtain a target image reconstruction model; The real-time image reconstruction unit is used to input the collected real-time high-temporal-resolution image data of the area to be monitored into the target image reconstruction model during the actual remote sensing image reconstruction process to obtain a reconstructed image of the area to be monitored.

9. A computer device, characterized in that: The computer device includes a memory, a processor and a transceiver, which are connected via a bus; the memory is used to store a set of computer program instructions and data, and transmit the stored data to the processor, and the processor executes the program instructions stored in the memory to perform the remote sensing image reconstruction method based on ground object structure as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the remote sensing image reconstruction method based on ground object structure according to any one of claims 1 to 7 is implemented.

Images