Remote sensing image reconstruction method, system and equipment based on ground object structure and medium

Abstract

The invention discloses a remote sensing image reconstruction method, system and device based on a ground object structure and a medium. Historical satellite multispectral remote sensing data of a to-be-monitored area are acquired; extracting a surface feature structure end member abundance matrix of the high-time-resolution image data based on a pixel purity index algorithm; extracting a global dependency relationship matrix of historical satellite multispectral remote sensing data based on a multi-channel self-attention mechanism; processing the global dependency relationship matrix based on a feedforward neural network model to obtain global features; fusing the global features and the surface feature structure end member abundance matrix based on a convolutional neural network model to obtain high-resolution feature map data; and performing parameter iterative optimization according to a loss function calculated by the high-resolution feature map data and the high-spatial-resolution image data, and inputting the real-time high-temporal-resolution image data into the optimized target image reconstruction model to obtain a reconstructed image. According to the method provided by the invention, the influence of the mixed pixels on image reconstruction is reduced, and the reconstruction precision is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image reconstruction, and in particular to a remote sensing image reconstruction method, system, device and medium based on ground object structure. Background Technique

[0002] Agricultural land is land directly or indirectly used for agricultural production, including various types such as cultivated land, orchard land, forest land and pasture land. The refined monitoring of agricultural land is of great significance for improving agricultural production efficiency, resource management and ecological protection. However, agricultural land is generally composed of complex ground object structures, such as different types of vegetation, soil and water bodies, resulting in a serious mixed pixel effect in remote sensing images, especially in low-resolution images, which poses a huge challenge to the refined management of agricultural land monitoring.

[0003] With the development of remote sensing technology, satellite remote sensing provides a new means for agricultural land monitoring and can capture large-area soil change information in a short time. Especially the currently most widely used multi-spectral remote sensing satellites have rich spatial, temporal and spectral information. However, when satellite multi-spectral remote sensing data is applied to agricultural land monitoring, a key problem is faced: the contradiction between spatial and temporal resolutions. Existing technologies achieve a balance between spatial and temporal resolutions through high-resolution reconstruction. Existing high-resolution reconstruction methods perform upsampling reconstruction by extracting spatial features from low-resolution images, which can improve the spatial resolution of images to a certain extent. However, this method directly processes satellite multi-spectral remote sensing images and improves the spatial resolution based on the geometric features of satellite multi-spectral remote sensing images, without fully considering the ground object structure composition within the same pixel, resulting in large deviations or ambiguities in ground object classification and distribution, and it is difficult to accurately reflect the true situation of agricultural land.

[0004] Therefore, how to improve the accuracy of remote sensing image reconstruction and thus improve the accuracy of agricultural land monitoring has become a technical problem that needs to be urgently solved by those skilled in the art. Summary of the Invention

[0005] The present invention provides a remote sensing image reconstruction method, system, device and medium based on ground object structure to solve the technical problem of how to improve the accuracy of remote sensing image reconstruction and thus improve the accuracy of agricultural land monitoring, and achieve the effect of obtaining high-resolution image data that can accurately reflect the ground object structure of agricultural land and improving the accuracy of agricultural land monitoring.

[0006] In a first aspect, the present invention provides a remote sensing image reconstruction method based on ground object structure, the method comprising: obtaining historical satellite multi-spectral remote sensing data of the same time period in the area to be monitored, the historical satellite multi-spectral remote sensing data at least including: high spatial resolution image data and high temporal resolution image data; During the construction of the image reconstruction model, each ground object structure endmember in the high temporal resolution image data is extracted based on the pixel purity index algorithm; Based on the least squares method, an unmixing algorithm for the ground object structure endmembers is designed to obtain the abundance matrix of the ground object structure endmembers in the high temporal resolution image data; Based on the multi-channel self-attention mechanism, a global dependency matrix of the historical satellite multi-spectral remote sensing data is extracted; based on the selected feed-forward neural network model, linear transformation and non-linear processing are performed on the global dependency matrix to obtain global features; based on the selected convolutional neural network model, the global features and the abundance matrix of the ground object structure endmembers are fused to obtain high-resolution feature map data; According to the high-resolution feature map data and the high spatial resolution image data, a loss function is constructed, and based on the feedback of the loss function, forward propagation iterative optimization is performed on the multi-channel self-attention mechanism and the feed-forward neural network model to obtain the target image reconstruction model; During the actual remote sensing image reconstruction process, the high temporal resolution image data of the to-be-monitored area collected is input into the target image reconstruction model to obtain the reconstructed image of the to-be-monitored area.

[0007] Preferably, the extraction of each ground object structure endmember in the high temporal resolution image data based on the pixel purity index algorithm includes: Project the spectra of each pixel in the high temporal resolution image data onto different pre-set random unit vectors, and calculate the number of times each pixel spectrum falls at both ends of the random unit vector; Take the mean value of the calculated numbers as the pixel purity index; Determine each ground object structure endmember in the high temporal resolution image data according to the pixel purity index and the pre-set threshold range.

[0008] Preferably, the design of the unmixing algorithm for the ground object structure endmembers based on the least squares method to obtain the abundance matrix of the ground object structure endmembers in the high temporal resolution image data includes: Construct a ground object structure endmember spectral matrix based on the spectral reflectance of each band of each ground object structure endmember; According to the ground object structure endmember spectral matrix, an unmixing algorithm is designed by the least squares method, and the unmixing algorithm is: R(x, y) = E·F + ∈(x, y) where E represents the ground object structure endmember spectral matrix, F represents the abundance matrix of the ground object structure endmembers, ∈(x, y) represents the error term, and R(x, y) represents the reflectance matrix of the corresponding pixel in each band; Based on the set endmember abundance constraints, use the unmixing algorithm to estimate the endmember abundance matrix of each ground object structure in the high temporal resolution image data.

[0009] Preferably, the extraction of the global dependency matrix of the historical satellite multispectral remote sensing data based on the multi-channel self-attention mechanism includes: Extract the reflectance of each pixel in each band of the historical satellite multispectral remote sensing data, and construct a reflectance feature vector according to the reflectance; Perform a linear transformation on the reflectance feature vector to generate a query vector, a key vector, and a value vector; According to the query vector and the key vector, obtain the attention weights; Perform a weighted sum on the value vector according to the attention weights to obtain the global dependency matrix, and the global dependency matrix includes at least: the mutual relationship between adjacent pixels and the mutual relationship between different bands.

[0010] Preferably, the linear transformation and non-linear processing of the global dependency matrix based on the selected feed-forward neural network model to obtain the global features includes: Perform a linear transformation on the global dependency matrix using the first-layer linear transformation of the feed-forward neural network model, and perform non-linear processing of the activation function to obtain the feature variable to be processed; Perform a linear transformation on the feature variable to be processed using the second-layer linear transformation of the feed-forward neural network model to obtain the global features.

[0011] Preferably, the fusion of the global features and the endmember abundance matrix of the ground object structure based on the selected convolutional neural network model to obtain the high-resolution feature map data includes: Concatenate the global features and the endmember abundance matrix of the ground object structure in the channel dimension to obtain the matrix to be fused; Use the pointwise convolution algorithm to fuse the matrix to be fused to obtain the fusion feature matrix to be processed; Perform non-linear processing on the fusion feature matrix to be processed to obtain the fusion feature matrix; Perform multi-layer convolution on the fusion feature matrix to obtain the low-resolution feature map data; Perform upsampling on the low-resolution feature map data to obtain the high-resolution feature map data.

[0012] Preferably, the construction of the loss function according to the high-resolution feature map data and the high spatial resolution image data, and the forward propagation iterative optimization of the multi-channel self-attention mechanism and the feed-forward neural network model based on the feedback of the loss function to obtain the target image reconstruction model includes: Taking the high-spatial-resolution image data as the ground truth, a loss function of the image reconstruction model is constructed according to the ground truth and the high-resolution feature map data; The multi-channel self-attention mechanism and the feed-forward neural network model are iteratively optimized by forward propagation according to the feedback of the loss function to obtain a target image reconstruction model.

[0013] In a second aspect, the present invention also provides a remote sensing image reconstruction system based on the ground object structure, which implements the remote sensing image reconstruction method based on the ground object structure described above. The system includes: a data acquisition unit, a ground object structure endmember extraction unit, a ground object structure endmember abundance calculation unit, a feature extraction and fusion unit, a model training unit, and a real-time image reconstruction unit; The data acquisition unit is used to obtain historical satellite multi-spectral remote sensing data of the area to be monitored in the same time period. The historical satellite multi-spectral remote sensing data at least includes: high-spatial-resolution image data and high-temporal-resolution image data; The ground object structure endmember extraction unit is used to extract each ground object structure endmember in the high-temporal-resolution image data based on the pixel purity index algorithm during the construction of the image reconstruction model; The ground object structure endmember abundance calculation unit is used to design an unmixing algorithm for the ground object structure endmember based on the least squares method to obtain the ground object structure endmember abundance matrix of the high-temporal-resolution image data; The feature extraction and fusion unit is used to extract the global dependence matrix of the historical satellite multi-spectral remote sensing data based on the multi-channel self-attention mechanism; perform linear transformation and non-linear processing on the global dependence matrix based on a selected feed-forward neural network model to obtain global features; fuse the global features and the ground object structure endmember abundance matrix based on a selected convolutional neural network model to obtain high-resolution feature map data; The model training unit is used to construct a loss function according to the high-resolution feature map data and the high-spatial-resolution image data, and iteratively optimize the multi-channel self-attention mechanism and the feed-forward neural network model by forward propagation based on the feedback of the loss function to obtain a target image reconstruction model; The real-time image reconstruction unit is used to input the real-time high-temporal-resolution image data of the area to be monitored collected during the actual remote sensing image reconstruction process into the target image reconstruction model to obtain the reconstructed image of the area to be monitored.

[0014] In a third aspect, the present invention further provides a computer device, which includes a memory, a processor, and a transceiver, and they are connected through a bus; the memory is used to store a set of computer program instructions and data, and transmit the stored data to the processor, and the processor executes the program instructions stored in the memory to execute the above-mentioned remote sensing image reconstruction method based on ground object structure.

[0015] In a fourth aspect, the present invention further provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is run, the above-mentioned remote sensing image reconstruction method based on ground object structure is implemented.

[0016] The present invention provides a remote sensing image reconstruction method, system, device and medium based on ground object structure. Compared with the prior art, the beneficial effects of the embodiments of the present invention are at least one of the following: (1) The satellite multi-spectral remote sensing data with high temporal resolution but low spatial resolution is improved in spatial resolution through super-resolution technology, while retaining high temporal resolution, and the generated reconstructed remote sensing image has both high temporal and high spatial resolution.

[0017] (2) In view of the characteristics of the complex ground object structure of agricultural land images, a ground object structure end-member abundance acquisition module is designed, and the global features and the ground object structure end-member abundance matrix are fused, which greatly reduces the influence of mixed pixels on image reconstruction, further improves the accuracy of super-resolution image reconstruction, and can better capture the detailed features of agricultural land. Description of the Drawings

[0018] Figure 1 It is a schematic diagram of the steps of a remote sensing image reconstruction method based on ground object structure provided by a preferred embodiment of the present invention; Figure 2 It is a schematic diagram of the structure of a remote sensing image reconstruction system based on ground object structure provided by a preferred embodiment of the present invention; Figure 3 It is a schematic diagram of the structure of a computer device provided in one of the embodiments of the present invention. Detailed Embodiments

[0019] The embodiments of the present invention will be specifically described below in conjunction with the accompanying drawings. The given embodiments are only for illustrative purposes and should not be construed as a limitation of the present invention. The accompanying drawings are for reference and illustration only and do not constitute a limitation on the protection scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention. In the description of the present invention, the terms "first", "second", "third", etc. are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first", "second", "third", etc. may explicitly or implicitly include one or more of such features. In the description of the present invention, unless otherwise specified, the meaning of "a plurality" is two or more.

[0020] In the description of the present invention, it should be noted that, unless otherwise clearly defined and limited, the terms "mounted", "connected", "coupled" shall be construed in a broad sense. For example, it may be a fixed connection, a detachable connection, or an integral connection; it may be a mechanical connection or an electrical connection; it may be directly connected or indirectly connected through an intermediate medium, and it may be the internal communication of two elements. The terms "vertical", "horizontal", "left", "right", "up", "down" and similar expressions used herein are only for illustrative purposes and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as a limitation of the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0021] In the description of the present invention, it should be noted that, unless otherwise defined, all technical and scientific terms used in the present invention have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0022] In satellite multispectral remote sensing data, remote sensing data with high spatial resolution can accurately capture the details of agricultural land, but its low temporal resolution (a revisit cycle of 16 days) cannot respond in a timely manner to the dynamic changes in crop growth, especially in scenarios such as farmland management and pest control that require rapid decision-making. On the contrary, although remote sensing data with low spatial resolution images has a high temporal resolution (1-2 days), its spatial resolution (250 / 500 meters) often fails to provide sufficient details of ground objects. Especially in complex ground object mixing areas of agricultural land, such as the boundaries between vegetation and soil, and irrigation and farmland, this type of satellite multispectral remote sensing data with low spatial resolution cannot accurately describe the distribution of ground objects, resulting in unsatisfactory monitoring effects.

[0023] In view of this, in an embodiment of the present invention, a remote sensing image reconstruction method based on ground object structure is provided. Please refer to Figure 1 , and the method includes: S1. Obtain historical satellite multispectral remote sensing data of the area to be monitored in the same time period. The historical satellite multispectral remote sensing data at least includes: high spatial resolution image data and high temporal resolution image data.

[0024] S2. During the construction of the image reconstruction model, extract each ground object structure endmember in the high temporal resolution image data based on the pixel purity index algorithm.

[0025] S3. Design an unmixing algorithm for the ground object structure endmembers based on the least squares method to obtain the ground object structure endmember abundance matrix of the high temporal resolution image data.

[0026] S4. Extract the global dependence matrix of the historical satellite multispectral remote sensing data based on the multi-channel self-attention mechanism; perform linear transformation and non-linear processing on the global dependence matrix based on the selected feed-forward neural network model to obtain global features; fuse the global features and the ground object structure endmember abundance matrix based on the selected convolutional neural network model to obtain high-resolution feature map data.

[0027] S5. Construct a loss function based on the high-resolution feature map data and the high spatial resolution image data, and perform forward propagation iterative optimization on the multi-channel self-attention mechanism and the feed-forward neural network model based on the value of the loss function to obtain the target image reconstruction model.

[0028] S6. During the actual remote sensing image reconstruction process, input the real-time high temporal resolution image data of the area to be monitored into the target image reconstruction model to obtain the reconstructed image of the area to be monitored.

[0029] In the remote sensing image reconstruction method based on ground object structure disclosed in the preferred embodiment of the present invention, it is mainly applied to agricultural land. Therefore, in the present invention, agricultural land is taken as an example for illustration. Satellite multispectral remote sensing data within the agricultural land area to be monitored is obtained. The historical satellite multispectral remote sensing data at least includes: high-spatial-resolution image data and high-temporal-resolution image data. The satellite multispectral remote sensing data selected in the present invention comes from two sources, namely MODIS data and Landsat-8 data.

[0030] MODIS data has a high temporal resolution of 1 - 2 days, which is suitable for dynamic monitoring of large-scale agricultural land. However, its spatial resolution is relatively low, being 250m / 500m. The selected bands include a total of 5 bands: blue, green, red, near-infrared, and short-wave infrared.

[0031] Landsat-8 data has a high spatial resolution of 30m, which is used to obtain detailed structural information of agricultural land. However, its temporal resolution is relatively low, being 16 days. Similarly, 5 bands of blue, green, red, near-infrared, and short-wave infrared are selected.

[0032] The above-mentioned MODIS data and Landsat-8 data can both be downloaded through Google Earth Engine (GEE). When downloading, the overlapping period data of MODIS data and Landsat-8 data in the same area is screened, and the agricultural land area vector file is used for cropping and cloud and shadow masking to ensure the consistency of the quality and coverage range of the two types of data. At the same time, each band of MODIS data is unified to 250m.

[0033] Furthermore, in the preferred embodiment of the present invention, during the construction of the image reconstruction model, the constructed image reconstruction model includes a ground object structure endmember abundance acquisition module, a multi-channel self-attention mechanism, a feed-forward neural network model, and a convolutional neural network model. Among them, the ground object structure endmember abundance acquisition module extracts each ground object structure endmember in the high-temporal-resolution image data based on the pixel purity index, that is, each ground object structure endmember in the MODIS data, and designs an unmixing algorithm for the ground object structure endmember based on the least squares method to obtain the ground object structure endmember abundance matrix of the high-temporal-resolution image data.

[0034] Due to the limited spatial resolution of remote sensing satellites, a single pixel of satellite multispectral remote sensing data usually cannot contain only one type of ground object. Especially in areas with complex agricultural land, forest land, or vegetation cover, it contains mixed pixels of multiple ground object types. The mixed pixels will contain spectral information of multiple ground object structure types, which will lead to a decrease in accuracy during image classification and reconstruction. In the preferred embodiment of the present invention, a ground object structure endmember abundance matrix is introduced. The ground object structure endmember abundance matrix is a matrix composed of the proportions of different ground object structure types in each mixed pixel. Before obtaining the ground object structure endmember abundance matrix, it is necessary to extract each ground object structure endmember from the high temporal resolution image data. The extraction of each ground object structure endmember is based on the pixel purity index. The pixel purity index (PPI) is a geometry-based method used to identify the "purest" pixels in the image. These pixels often represent a single ground object type, that is, each ground object structure endmember, rather than a mixed ground object.

[0035] The pixel purity index algorithm takes each pixel in the pixel purity index as an n-dimensional vector, and all pixels form a vector space. In the vector space, the basis is not unique, but there must be a basis composed entirely of vectors located at the boundary positions. When these boundary vectors are projected onto a randomly generated unit vector, the probability of appearing at both ends of the random unit vector is the highest. By calculating the mean through multiple iterative projection calculations, the purity index of each pixel can be calculated, thereby determining each ground object structure endmember.

[0036] In the preferred embodiment of the present invention, a threshold range is constructed according to the pixel purity indices of all ground object types contained in the agricultural land. The high temporal resolution image data is subjected to dimensionality reduction processing to reduce the computational amount and remove noise. Further, the spectra of each pixel in the high temporal resolution image data after dimensionality reduction processing are projected onto a set of pre-set random unit vectors, and the number of times each pixel spectrum falls at both ends of the random unit vector is calculated. Different random unit vectors are used, and the mean value of the number of times each pixel spectrum falls at both ends of the random unit vector is calculated and used as the pixel purity index. Each ground object structure endmember in the high temporal resolution image data is determined according to the pixel purity index and the pre-constructed threshold range. Specifically, each ground object structure endmember in the high temporal resolution image data is determined according to the overlapping region of the pixel purity index and the threshold range.

[0037] After determining each feature structure endmember in the high temporal resolution image data, for the feature structure endmember, a unmixing algorithm for the feature structure endmember is designed by the least squares method, and the abundance matrix of the feature structure endmember in the high temporal resolution image data is estimated by using the unmixing algorithm. Specifically, a spectral matrix of the feature structure endmember is constructed based on the spectral reflectances of each band of each feature structure endmember. According to the spectral matrix of the feature structure endmember and the characteristics of agricultural land, a unmixing algorithm is designed by the least squares method. The unmixing algorithm is a linear equation, and the expression of the linear equation is: R(x, y) = E·F + ∈(x, y) where E represents the spectral matrix of the feature structure endmember, F represents the abundance matrix of the feature structure endmember, ∈(x, y) represents the error term, and R(x, y) represents the reflectance matrix of the corresponding pixel in each band. The shape of the spectral matrix of the feature structure endmember is C×A, where C is the number of input bands and A represents the number of feature structure endmembers.

[0038] The abundance matrix of the feature structure endmember represents a matrix composed of the proportions of each feature structure endmember in the high temporal resolution image data. By minimizing the error term, the linear equation is solved to obtain the abundance matrix of the feature structure endmember.

[0039] In addition, to ensure the physical rationality of the unmixing result, the abundance matrix of the feature structure endmember of the present invention needs to satisfy the following two constraint conditions: 1) Non-negativity constraint: That is, the abundance f of each feature structure endmember i ≥0, indicating that the feature proportion cannot be negative.

[0040] 2) Sum constraint: That is, the sum of the abundances of all feature structure endmembers must be 1,

[0041] The finally output abundance matrix F of the feature structure endmember is: Furthermore, based on the multi-channel self-attention mechanism, a global dependency matrix of historical satellite multispectral remote sensing data is extracted. The global dependency matrix at least includes: the mutual relationship between adjacent pixels and the mutual relationship between different bands. The mutual relationship between adjacent pixels is the spatial dependency relationship between adjacent pixels, including the correlation or mutual information between adjacent pixels. The mutual relationship between different bands is the spectral dependency relationship or correlation between different bands in the historical satellite multispectral remote sensing data.

[0042] The global dependency matrix emphasizes the integrity and relevance between regions in the historical satellite multispectral remote sensing data. Compared with the local dependency relationship, the global dependency matrix pays more attention to capturing the global structure and context information in the image. This dependency relationship usually involves a wider spatial range, so it can provide richer semantic information.

[0043] The multi-channel self-attention mechanism generates a weight matrix by calculating the similarity between pixel features, and then weights the input satellite multi-spectral remote sensing data. This mechanism can pay attention to the local and remote feature associations in the satellite multi-spectral remote sensing data, and incorporate the association information between different bands into the feature extraction, which can effectively improve the performance of the remote sensing image reconstruction method based on ground object structure of the present invention in areas with complex ground object backgrounds.

[0044] Specifically, extract the reflectance of each pixel in each band of the historical satellite multi-spectral remote sensing data, and construct a reflectance feature vector based on the reflectance.

[0045] The historical satellite multi-spectral remote sensing data has multiple spectral bands, which are input into multiple channels of the multi-channel self-attention mechanism, and the input matrix is set as: X ∈ R H×W×C where H and W are the height and width of the image respectively.

[0046] For each pixel of the input historical satellite multi-spectral remote sensing data, define its reflectance feature vector X i as: X i = [X i (1) , X i (2) ,..., X i (C) where X i (C) represents the reflectance of the i-th pixel in the C-th band.

[0047] Furthermore, perform a linear transformation on the reflectance feature vector to generate a query vector, a key vector, and a value vector, which are respectively: Q i = W Q · X i + b Q K i = W k · X i + b K V i = W v · X i + b v where Q i , K i and V i respectively represent the query vector, key vector, and value vector of the i-th pixel, W Q , W​K , W V represents a trainable weight matrix, and b Q , b K , b V represents the bias term.

[0048] Furthermore, the attention scores between each pixel are calculated based on the dot product of the query vector and the key vector. The calculation formula for the attention scores is: where dk is the dimension of the key vector.

[0049] Furthermore, the attention scores are normalized to obtain the attention weights. The calculation formula for the attention weights is: where N represents the number of pixels, and α ij represents the attention weight of the i-th pixel to the j-th pixel, reflecting the correlation between the two pixels. The higher the attention weight, the stronger the dependence relationship between the two pixels.

[0050] Furthermore, the value vectors are weighted and summed according to the attention weights to obtain the global dependence relationship matrix. The expression of the global dependence relationship matrix is: where V j represents the value vector of the j-th pixel, and Z i represents the output of the global dependence relationship matrix of the i-th pixel. The global dependence relationship matrix combines the information of other pixels, not only including the self-information of this pixel, but also integrating the correlation information between other pixels and this pixel.

[0051] At the same time, in order to enhance the ability to capture multi-scale features, the multi-channel self-attention mechanism of this application is designed as a multi-head self-attention mechanism. Specifically, the input historical satellite multi-spectral remote sensing data is divided into multiple sub-spaces, and the attention weights are calculated separately in each sub-space. Finally, the attention weights of each sub-space are concatenated. The expression of the processing process is: MultiHead(Q, K, V) = Concat(head1,..., head h ) · W O where head i represents the output of the i-th attention head, h is the number of attention heads, and W O represents the weight matrix of the linear transformation.

[0052] Through the multi-head attention mechanism, the image reconstruction model can concurrently focus on different features in different sub-spaces.

[0053] Among them, the linear transformation matrix W O is a trainable weight matrix with a dimension of (h×d)×d′, where d is the dimension of the output of each attention head, and d′ is the dimension of the final output feature.

[0054] The globally dependent relationship matrix of the final output is expressed as: Z = MultiHead(Q, K, V) = [Z 1 , Z 2 ,..., Z M where Z M represents the globally dependent relationship matrix of the Mth subspace.

[0055] Furthermore, based on the selected feed-forward neural network model, linear transformation and non-linear processing are performed on the globally dependent relationship matrix to obtain global features. The role of the feed-forward neural network model is to further enhance the expressive ability of the globally dependent relationship matrix extracted by the multi-channel self-attention mechanism, and improve the discrimination ability of the model for complex ground object structures through non-linear transformation. The feed-forward neural network in the embodiment of the present invention consists of two layers of linear transformation and an activation function ReLU. Among them, the first layer of linear transformation performs linear transformation on the globally dependent relationship matrix, and then performs non-linear processing of the activation function to obtain the feature variable to be processed. The globally dependent relationship matrix of each pixel will pass through the feed-forward neural network model respectively, and the expression of the processing process is: h i = ReLU(W 1 Z i + b 1 ) where h i represents the feature variable to be processed of the ith pixel, W 1 and b 1 are the weight matrix and bias vector of the first layer of linear transformation, Z i represents the globally dependent relationship matrix of the ith pixel, and ReLU represents the activation function.

[0056] Furthermore, the second layer of linear transformation performs linear transformation on the feature variable to be processed to obtain global features, and the expression of the processing process is: Z′ i = W 2 h i + b 2 where W 2 and b 2 are the weight matrix and bias vector of the second layer of linear transformation.

[0057] ​Further, based on the selected convolutional neural network model, the global features and the endmember abundance matrix of the ground object structure are fused to obtain high-resolution feature map data.

[0058] In a preferred embodiment of the present invention, the key to remote sensing image reconstruction lies in fusing the global features output by the feedforward neural network model and the endmember abundance matrix of the ground object structure after hybrid decomposition. The specific fusion process is as follows: First, the global features and the endmember abundance matrix of the ground object structure are concatenated in the channel dimension to obtain a matrix to be fused. The expression of the matrix to be fused is: H = [Z′, F] Among them, the shape of the concatenated matrix H to be fused is N×(C + A), that is, each pixel position contains global features and the abundance information of each ground object.

[0059] Further, a pointwise convolution algorithm is used to fuse the matrix to be fused. The pointwise convolution algorithm can learn the relationship between the global features and the endmember abundance matrix of the ground object structure to obtain a matrix of fusion features to be processed. The expression of the matrix of fusion features to be processed is: F fused = Conv 1×1 (H) The pointwise convolution algorithm maps the C + A-dimensional features to a new feature space, and the generated matrix of fusion features to be processed F fused represents a complex correlation matrix that contains both global features and abundance information.

[0060] Further, the activation function ReLU is used to perform nonlinear processing on the matrix of fusion features to be processed to enhance the expression ability of the matrix of fusion features to be processed for complex features, and a matrix of fused features is obtained. The expression of the matrix of fused features is: F′ fused = ReLU(F fused ) Further, multi-layer convolution is performed on the matrix of fused features for further feature extraction to obtain low-resolution feature map data, which is expressed as: F conv = Conv(F′ fused ) Among them, F conv is the low-resolution feature map data after multi-layer convolution. Although the feature matrix of the low-resolution feature map data is smaller in size, it still retains the spatial structure and detail information. The low-resolution feature map data can be regarded as a low-dimensional expression of the image.

[0061] Furthermore, the low-resolution feature map data is upsampled through an upsampling layer to obtain high-resolution feature map data. Specifically, the low-resolution feature map data is input into the upsampling layer for deconvolution processing. Upsampling is the inverse operation of the convolutional layer. By learning a specific convolutional kernel, the low-resolution feature map is enlarged into a high-resolution feature map. The high-resolution feature map data is expressed as: I high-res =TransposedConv(F conv ) where I high-res represents the high-resolution feature map data.

[0062] After obtaining the high-resolution feature map data, a loss function is constructed based on the high-resolution feature map data and the high-spatial-resolution image data, and the multi-channel self-attention mechanism and the feed-forward neural network model are iteratively optimized through forward propagation based on the feedback of the loss function to obtain the target image reconstruction model.

[0063] In a preferred embodiment of the present invention, the high-spatial-resolution image data is used as the ground truth of the image reconstruction model, and a loss function is constructed based on the ground truth and the high-resolution feature map data to iteratively optimize the multi-channel self-attention mechanism and the feed-forward neural network model through forward propagation.

[0064] Taking the high-spatial-resolution image data as the ground truth I Landsat , and taking the high-resolution feature map as the initial reconstruction value I pred , a loss function is calculated based on the ground truth and the initial reconstruction value. The role of the loss function is to guide the image reconstruction model to gradually adjust the weight parameters of the multi-channel self-attention mechanism and the feed-forward neural network model in each forward propagation and backpropagation. The loss functions used in the embodiments of the present invention include: L1 loss (absolute error): L2 loss (mean square error): The values of L1 and L2 will be used as a feedback signal to guide the update of the weight parameters of the image reconstruction model in backpropagation, that is, the error calculated by the loss function will be passed back to the multi-channel self-attention mechanism and the feed-forward neural network model through the chain rule to update the weight parameters of the multi-channel self-attention mechanism and the feed-forward neural network model, so that in the next forward propagation, the generated high-resolution feature map data is closer to the ground truth.

[0065] When the loss function finally converges, the optimization of the image reconstruction model is completed, and the target image reconstruction model is obtained for subsequent actual remote sensing image reconstruction.

[0066] In the actual remote sensing image reconstruction process, the real-time high temporal resolution image data of the agricultural land area to be monitored collected is input into the target image reconstruction model to obtain the reconstructed image of the agricultural land to be monitored. Specifically, the collected real-time high spatial resolution image data is subjected to super-resolution reconstruction through the endmember abundance acquisition module of the ground object structure, the multi-channel self-attention mechanism, the feed-forward neural network model, and the convolutional neural network model, and finally an agricultural land remote sensing image with both high spatial and high temporal resolutions is generated, improving the ability to capture the dynamic changes of different ground object structures of agricultural land and providing key data support for precision agriculture, land management, and ecological monitoring.

[0067] In a preferred embodiment of the present invention, historical satellite multi-spectral remote sensing data of the same time period in the area to be monitored is obtained. The historical satellite multi-spectral remote sensing data at least includes: high spatial resolution image data and high temporal resolution image data; in the process of constructing the image reconstruction model, each endmember of the ground object structure in the high temporal resolution image data is extracted based on the pixel purity index algorithm; a unmixing algorithm for the endmember of the ground object structure is designed based on the least squares method to obtain the endmember abundance matrix of the high temporal resolution image data; the global dependence matrix of the historical satellite multi-spectral remote sensing data is extracted based on the multi-channel self-attention mechanism; the global dependence matrix is linearly transformed and non-linearly processed based on the selected feed-forward neural network model to obtain global features; the global features and the endmember abundance matrix of the ground object structure are fused based on the selected convolutional neural network model to obtain high-resolution feature map data; a loss function is calculated based on the high-resolution feature map data and the high spatial resolution image data, and the multi-channel self-attention mechanism and the feed-forward neural network model are iteratively optimized by forward propagation based on the loss function to obtain the target image reconstruction model; in the actual remote sensing image reconstruction process, the real-time high temporal resolution image data of the area to be monitored collected is input into the target image reconstruction model to obtain the reconstructed image of the area to be monitored. The remote sensing image reconstruction method based on the ground object structure provided by the present application generates a reconstructed remote sensing image with both high temporal and high spatial resolutions, and designs an endmember abundance acquisition module for the ground object structure of the agricultural land image, fuses the global features and the endmember abundance matrix of the ground object structure, greatly reducing the influence of mixed pixels on image reconstruction, further improving the accuracy of super-resolution image reconstruction, and being able to better capture the detailed features of agricultural land.

[0068] Correspondingly, as Figure 2As shown, based on a remote sensing image reconstruction method based on ground object structure, an embodiment of the present invention also provides a remote sensing image reconstruction system based on ground object structure to implement the remote sensing image reconstruction method based on ground object structure disclosed in the embodiment of the present invention. The system is applied to agricultural land monitoring and includes: a data acquisition unit 1, a ground object structure endmember extraction unit 2, a ground object structure endmember abundance calculation unit 3, a feature extraction and fusion unit 4, a model training unit 5, and a real-time image reconstruction unit 6; The data acquisition unit 1 is configured to obtain historical satellite multispectral remote sensing data of the area to be monitored in the same time period. The historical satellite multispectral remote sensing data at least includes: high-spatial-resolution image data and high-temporal-resolution image data; The ground object structure endmember extraction unit 2 is configured to extract each ground object structure endmember in the high-temporal-resolution image data based on the pixel purity index algorithm during the construction of the image reconstruction model; The ground object structure endmember abundance calculation unit 3 is configured to design an unmixing algorithm for the ground object structure endmembers based on the least squares method to obtain the ground object structure endmember abundance matrix of the high-temporal-resolution image data; The feature extraction and fusion unit 4 is configured to extract the global dependence matrix of the historical satellite multispectral remote sensing data based on the multi-channel self-attention mechanism; perform linear transformation and non-linear processing on the global dependence matrix based on the selected feed-forward neural network model to obtain global features; fuse the global features and the ground object structure endmember abundance matrix based on the selected convolutional neural network model to obtain high-resolution feature map data; The model training unit 5 is configured to construct a loss function according to the high-resolution feature map data and the high-spatial-resolution image data, and perform forward propagation iterative optimization on the multi-channel self-attention mechanism and the feed-forward neural network model based on the feedback of the loss function to obtain the target image reconstruction model; The real-time image reconstruction unit 6 is configured to input the collected real-time high-temporal-resolution image data of the area to be monitored into the target image reconstruction model during the actual remote sensing image reconstruction process to obtain the reconstructed image of the area to be monitored.

[0069] For the specific limitations of a remote sensing image reconstruction system based on ground object structure, reference may be made to the above limitations on a remote sensing image reconstruction method based on ground object structure, which will not be elaborated here. Those of ordinary skill in the art can realize that, in combination with the modules and steps described in the embodiments disclosed in the present invention, they can be implemented by hardware, software, or a combination of both. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0070] As Figure 3 shown, a computer device provided by an embodiment of the present invention includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the steps in the embodiments of the remote sensing image reconstruction method based on ground object structure as described above, such as Figure 1 the steps S1 to S6 described in

[0071] Those skilled in the art can understand that the Figure 3 illustration is only an example of a computer device and does not constitute a limitation on the computer device. It may include more or fewer components than shown in the figure, or combine some components, or different components. For example, the computer device may also include input / output devices, network access devices, buses, etc.

[0072] The processor may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The processor is the control center of the computer device, connecting various parts of the entire computer device through various interfaces and lines.

[0073] The memory can be used to store the computer programs and / or modules. By running or executing the computer programs and / or modules stored in the memory, and invoking the data stored in the memory, the processor can implement various functions of the computer device. The memory mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created according to the use of the mobile phone (such as audio data, phone book, etc.). In addition, the memory can include high-speed random access memory, and can also include non-volatile memory, such as a hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage devices.

[0074] Among them, if the modules integrated in the computer device are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the above embodiment methods of the present invention, it can also be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of the above various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0075] Those of ordinary skill in the art can understand that to implement all or part of the processes in the above embodiment methods, it can be completed by instructing relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above method embodiments. Among them, the storage medium can be a magnetic disk, optical disc, read-only memory (ROM), or random access memory (RAM), etc.

[0076] Accordingly, an embodiment of the present invention provides a computer-readable storage medium, which includes a stored computer program. When the computer program runs, it controls the device where the computer-readable storage medium is located to execute the steps in the remote sensing image reconstruction method based on ground object structure in the above embodiment, such as Figure 1 the steps S1 to S6 described therein.

[0077] A remote sensing image reconstruction method, system, device and medium based on ground object structure provided in this embodiment are used to solve the technical problem of improving the accuracy of remote sensing image reconstruction and further improving the accuracy of agricultural land monitoring. Obtain historical satellite multispectral remote sensing data of the area to be monitored in the same time period. The historical satellite multispectral remote sensing data at least includes: high-spatial-resolution image data and high-temporal-resolution image data; extract each ground object structure endmember in the high-temporal-resolution image data based on the pixel purity index algorithm; design an unmixing algorithm for the ground object structure endmember based on the least squares method to obtain the abundance matrix of the ground object structure in the high-temporal-resolution image data; extract the global dependence matrix of the historical satellite multispectral remote sensing data based on the multi-channel self-attention mechanism; perform linear transformation and non-linear processing on the global dependence matrix based on the selected feed-forward neural network model to obtain global features; fuse the global features and the abundance matrix of the ground object structure based on the selected convolutional neural network model to obtain high-resolution feature map data; construct a loss function based on the high-resolution feature map data and the high-spatial-resolution image data, and perform forward propagation iterative optimization on the multi-channel self-attention mechanism and the feed-forward neural network model based on the feedback of the loss function to obtain a target image reconstruction model; in the actual remote sensing image reconstruction process, input the real-time high-temporal-resolution image data of the area to be monitored into the target image reconstruction model to obtain the reconstructed image of the area to be monitored. The remote sensing image reconstruction method based on ground object structure provided in this application generates a reconstructed remote sensing image with both high temporal and high spatial resolutions. In view of the characteristics of the complex ground object structure of agricultural land images, a ground object structure endmember abundance acquisition module is designed to fuse the global features and the abundance matrix of the ground object structure, greatly reducing the impact of mixed pixels on image reconstruction, further improving the accuracy of super-resolution image reconstruction, and being able to better capture the detailed features of agricultural land.

[0078] Each embodiment in this specification is described in a progressive manner. For parts that are the same or similar in each embodiment, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for system embodiments, since they are basically similar to method embodiments, the description is relatively simple. For relevant parts, reference can be made to the partial description of method embodiments. It should be noted that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as within the scope described in this specification.

[0079] The above-described embodiments only represent several preferred embodiments of the present invention. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention. It should be pointed out that for those of ordinary skill in the art in this technical field, without departing from the technical principle of the present invention, several improvements and substitutions can be made, and these improvements and substitutions should also be regarded as within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A remote sensing image reconstruction method based on ground object structure, characterized in that: The method comprises: Acquire historical satellite multispectral remote sensing data of the same time period of the area to be monitored, wherein the historical satellite multispectral remote sensing data at least includes: high spatial resolution image data and high temporal resolution image data; In the process of constructing the image reconstruction model, each ground object structure end member in the high temporal resolution image data is extracted based on a pixel purity index algorithm; Designing an unmixing algorithm for the surface feature structure endmembers based on the least squares method to obtain a surface feature structure endmember abundance matrix of the high temporal resolution image data; Extracting the global dependency matrix of the historical satellite multispectral remote sensing data based on a multi-channel self-attention mechanism; performing linear transformation and nonlinear processing on the global dependency matrix based on a selected feedforward neural network model to obtain global features; fusing the global features with the endmember abundance matrix of the ground feature structure based on a selected convolutional neural network model to obtain high-resolution feature map data; Constructing a loss function according to the high-resolution feature map data and the high-spatial-resolution image data, and performing forward propagation iterative optimization on the multi-channel self-attention mechanism and the feedforward neural network model based on feedback of the loss function to obtain a target image reconstruction model; In the actual remote sensing image reconstruction process, the collected real-time high-temporal resolution image data of the area to be monitored is input into the target image reconstruction model to obtain a reconstructed image of the area to be monitored.

2. The remote sensing image reconstruction method based on ground object structure according to claim 1, characterized in that: The step of extracting each ground object structure end member in the high temporal resolution image data based on a pixel purity index algorithm includes: Projecting each pixel spectrum in the high time resolution image data onto different preset random unit vectors, and calculating the number of times each pixel spectrum falls on both ends of the random unit vector; The average value of each of the calculated times is used as the pixel purity index; Each ground object structure end member in the high temporal resolution image data is determined according to the pixel purity index and a preset threshold range.

3. The remote sensing image reconstruction method based on ground object structure according to claim 1, characterized in that: The least squares method is used to design the unmixing algorithm of the surface feature structure end members to obtain the surface feature structure end member abundance matrix of the high temporal resolution image data, including: Constructing a ground object structure end member spectral matrix based on the spectral reflectance of each band of each ground object structure end member; According to the end-member spectrum matrix of the ground feature structure, an unmixing algorithm is designed by the least squares method. The unmixing algorithm is: R(x, y) = E·F+∈(x, y), where E represents the surface feature structure end-member spectrum matrix, F represents the surface feature structure end-member abundance matrix, ∈(x, y) represents the error term, and R(x, y) represents the reflectance matrix of the corresponding pixel in each band; Based on the set endmember abundance constraints, the unmixing algorithm is used to estimate the endmember abundance matrix of each ground feature structure in the high temporal resolution image data.

4. The remote sensing image reconstruction method based on ground object structure according to claim 1, characterized in that: The method of extracting the global dependency matrix of the historical satellite multispectral remote sensing data based on the multi-channel self-attention mechanism includes: Extracting the reflectivity of each pixel in each band in the historical satellite multispectral remote sensing data, and constructing a reflectivity feature vector according to the reflectivity; Performing a linear transformation on the reflectivity feature vector to generate a query vector, a key vector, and a value vector; Obtaining an attention weight according to the query vector and the key vector; The value vectors are weighted and summed according to the attention weights to obtain a global dependency matrix, wherein the global dependency matrix includes at least: mutual relationships between adjacent pixels and mutual relationships between different bands.

5. The remote sensing image reconstruction method based on ground object structure according to claim 1, characterized in that: The linear transformation and nonlinear processing of the global dependency matrix based on the selected feedforward neural network model to obtain global features includes: The first layer linear transformation of the feedforward neural network model is used to perform linear transformation on the global dependency matrix, and nonlinear processing of the activation function is performed to obtain feature variables to be processed; The second layer linear transformation of the feedforward neural network model is used to perform linear transformation on the feature variables to be processed to obtain global features.

6. The remote sensing image reconstruction method based on ground object structure according to claim 1, characterized in that: The global features and the end-member abundance matrix of the ground feature structure are fused based on the selected convolutional neural network model to obtain high-resolution feature map data, including: The global feature and the end-member abundance matrix of the ground feature structure are spliced ​​in the channel dimension to obtain a matrix to be fused; The matrix to be fused is fused by using a point-by-point convolution algorithm to obtain a fused feature matrix to be processed; Performing nonlinear processing on the fused feature matrix to be processed to obtain a fused feature matrix; Performing multi-layer convolution on the fused feature matrix to obtain low-resolution feature map data; The low-resolution feature map data is upsampled to obtain high-resolution feature map data.

7. The remote sensing image reconstruction method based on ground object structure according to claim 1, characterized in that: The loss function is constructed according to the high-resolution feature map data and the high-spatial resolution image data, and forward propagation iterative optimization is performed on the multi-channel self-attention mechanism and the feedforward neural network model based on feedback of the loss function to obtain a target image reconstruction model, including: Taking the high spatial resolution image data as a true value, and constructing a loss function of the image reconstruction model according to the true value and the high resolution feature map data; The multi-channel self-attention mechanism and the feedforward neural network model are forward propagated and iteratively optimized according to the feedback of the loss function to obtain a target image reconstruction model.

8. A remote sensing image reconstruction system based on ground object structure, which implements the remote sensing image reconstruction method based on ground object structure according to any one of claims 1 to 7, characterized in that: The system comprises: a data acquisition unit, a ground object structure end member extraction unit, a ground object structure end member abundance calculation unit, a feature extraction and fusion unit, a model training unit and a real-time image reconstruction unit; The data acquisition unit is used to obtain historical satellite multispectral remote sensing data of the same time period of the monitored area, and the historical satellite multispectral remote sensing data at least includes: high spatial resolution image data and high temporal resolution image data; The ground object structure end member extraction unit is used to extract each ground object structure end member in the high temporal resolution image data based on the pixel purity index algorithm during the image reconstruction model construction process; The ground object structure end member abundance calculation unit is used to design an unmixing algorithm for the ground object structure end members based on the least squares method to obtain a ground object structure end member abundance matrix of the high temporal resolution image data; The feature extraction and fusion unit is used to extract the global dependency matrix of the historical satellite multispectral remote sensing data based on a multi-channel self-attention mechanism; perform linear transformation and nonlinear processing on the global dependency matrix based on a selected feedforward neural network model to obtain global features; and fuse the global features and the end member abundance matrix of the ground feature structure based on a selected convolutional neural network model to obtain high-resolution feature map data; The model training unit is used to construct a loss function according to the high-resolution feature map data and the high-spatial resolution image data, and to perform forward propagation iterative optimization on the multi-channel self-attention mechanism and the feedforward neural network model based on feedback of the loss function to obtain a target image reconstruction model; The real-time image reconstruction unit is used to input the collected real-time high-temporal resolution image data of the area to be monitored into the target image reconstruction model during the actual remote sensing image reconstruction process to obtain a reconstructed image of the area to be monitored.

9. A computer device, characterized in that: The computer device includes a memory, a processor and a transceiver, which are connected via a bus; the memory is used to store a set of computer program instructions and data, and transmit the stored data to the processor, and the processor executes the program instructions stored in the memory to execute the remote sensing image reconstruction method based on the ground object structure as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the remote sensing image reconstruction method based on the ground object structure as claimed in any one of claims 1 to 7 is implemented.

Images