Multi-source remote sensing data fusion method and system for power grid survey

Abstract

The invention relates to the technical field of power grid multi-source data processing, and discloses a power grid survey-oriented multi-source remote sensing data fusion method, which comprises the following steps of: performing linear projection on remote sensing data after image block segmentation, and adding position embedding to a vector after projection to obtain vector representation; projecting the vector representation to obtain a first projection sequence and a second projection sequence; performing feature enhancement on the state space prediction result of the first projection sequence according to the activation output value of the second projection sequence and the gating vector to obtain a feature enhancement vector; focusing on a local dependency relationship in a feature enhancement vector based on a masked multi-attention mechanism to obtain a feature sequence containing local and global information; and obtaining a gating weight by using the gating activation code, and performing feature fusion on the feature sequence according to the gating weight to obtain a final fusion result. The method has the beneficial effects that feature information in different remote sensing data is fully utilized, and the accuracy and detail richness of a final fusion result are improved.

Description

Technical Field

[0001] The present application relates to the technical field of multi-source data processing for power grids, and in particular to a multi-source remote sensing data fusion method and system for power grid survey. Background Art

[0002] Grid survey is a key link in ensuring the stable supply and efficient operation of the power system. In the process of grid line design, the integration of multi-source survey data can help provide a multi-dimensional environmental consideration framework for the design of intelligent grid lines and enhance the comprehensive grasp of surrounding environmental factors during line design. By integrating multi-source data, it is possible to accurately assess the construction challenges brought by topography and avoid potential geological disaster risks, thereby optimizing line path selection and ensuring the long-term stable operation and efficient maintenance of the power grid. In related technologies, the utilization rate of the characteristic information contained in remote sensing survey data still needs to be improved. Summary of the invention

[0003] The present application provides a multi-source remote sensing data fusion method and system for power grid survey, which solves the technical problem of low utilization rate of feature information in remote sensing data, achieves the technical effect of making full use of feature information in different remote sensing data and improving the accuracy and detail richness of the final fusion result.

[0004] In order to achieve the above objectives, the main technical solutions adopted in this application include:

[0005] In a first aspect, an embodiment of the present application provides a multi-source remote sensing data fusion method for power grid survey, the method comprising:

[0006] Performing linear projection on any remote sensing data in the multi-source remote sensing data after image block segmentation, adding position embedding to the projected vector, and obtaining a vector representation corresponding to the any remote sensing data;

[0007] Projecting the vector representation to a first direction to obtain a first projection sequence, and projecting the vector representation to a second direction to obtain a second projection sequence, wherein the first direction and the second direction are different directions;

[0008] According to the activation output value and the gating vector of the second projection sequence, the state space prediction result of the first projection sequence is feature enhanced to obtain the feature enhancement vector of the any remote sensing data; wherein the state space prediction result is obtained by state conversion prediction according to the first projection sequence, the activation output value is obtained by nonlinear transformation and activation processing according to the second projection sequence, and the gating vector is obtained by gating the second projection sequence;

[0009] Extracting local dependencies from the feature enhancement vector based on a masked multi-head attention mechanism to obtain a feature sequence of any remote sensing data, wherein the feature sequence is used to represent the local dependencies of any remote sensing data;

[0010] According to the feature sequence, the gating weight of any remote sensing data is obtained using gated activation coding, and according to the gating weight of each remote sensing data in the multi-source remote sensing data, the feature sequence of each remote sensing data is feature fused to obtain the final fusion result.

[0011] The multi-source remote sensing data fusion method for power grid survey proposed in the embodiment of the present application, by combining state space and attention mechanism, performs feature extraction and feature enhancement on remote sensing data in two directions, thereby improving the ability to extract feature information in remote sensing data. On this basis, according to the importance of feature information in different remote sensing data, a gating mechanism is used to perform weighted fusion of multi-source remote sensing data, thereby making full use of the different advantages of multi-source remote sensing data, improving the accuracy and detail richness of the final fusion result, and providing high-quality data support for power grid survey.

[0012] In a second aspect, an embodiment of the present application provides a multi-source remote sensing data fusion system for power grid survey, the system comprising:

[0013] A segmentation and embedding module, used for performing linear projection on any remote sensing data in the multi-source remote sensing data after image block segmentation, adding position embedding to the projected vector, and obtaining a vector representation corresponding to any remote sensing data;

[0014] A feature projection module, configured to project the vector representation to a first direction to obtain a first projection sequence, and project the vector representation to a second direction to obtain a second projection sequence, wherein the first direction and the second direction are different directions;

[0015] a feature enhancement module, configured to perform feature enhancement on the state space prediction result of the first projection sequence according to the activation output value and the gating vector of the second projection sequence, so as to obtain a feature enhancement vector of the any remote sensing data; wherein the state space prediction result is obtained by predicting the state conversion of the first projection sequence, the activation output value is obtained by performing nonlinear transformation and activation processing on the second projection sequence, and the gating vector is obtained by performing a gating operation on the second projection sequence;

[0016] A feature learning module, used for extracting local dependencies from the feature enhancement vector based on a masked multi-head attention mechanism to obtain a feature sequence of any remote sensing data, wherein the feature sequence is used to represent the local dependencies of any remote sensing data;

[0017] The feature fusion module is used to obtain the gating weight of any remote sensing data according to the feature sequence by using gated activation coding, and to perform feature fusion on the feature sequence of each remote sensing data according to the respective gating weight of each remote sensing data in the multi-source remote sensing data to obtain a final fusion result. In a third aspect, an embodiment of the present application provides a computer device, including: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the multi-source remote sensing data fusion method for power grid survey described in any one of the above embodiments by executing the computer instructions.

[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions are used to enable a computer to execute the multi-source remote sensing data fusion method for power grid survey described in any one of the above embodiments.

[0019] In a fifth aspect, an embodiment of the present application provides a computer program product, including computer instructions, wherein the computer instructions are used to enable a computer to execute the multi-source remote sensing data fusion method for power grid survey described in any one of the above embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 A step diagram of a multi-source remote sensing data fusion method for power grid survey provided in an embodiment of the present application;

[0022] Figure 2 A diagram showing the steps for obtaining a feature enhancement vector in an embodiment of the present application;

[0023] Figure 3 A diagram showing the steps for obtaining state space observation results in an embodiment of the present application;

[0024] Figure 4 A diagram showing the steps of obtaining an activation output value and a gating vector in an embodiment of the present application;

[0025] Figure 5 A diagram showing the steps for obtaining the final fusion result in the embodiment of the present application;

[0026] Figure 6 A schematic diagram of expanding a remote sensing image into a vector mapping in an embodiment of the present application;

[0027] Figure 7 This is a module structure diagram of RS-Mambaformer in the embodiment of this application;

[0028] Figure 8 A network architecture diagram of an embodiment of the present application;

[0029] Fig. 9 A module diagram of a multi-source remote sensing data fusion system for power grid survey provided in an embodiment of the present application.

[0030] Among them, the figure numbers of the drawings in the specification are as follows: 100. Segmentation embedding module, 200. Feature projection module, 300. Feature enhancement module, 400. Feature learning module, 500. Feature fusion module. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0032] Grid survey is a key link in ensuring the stable supply and efficient operation of the power system. In the process of grid line design, the integration of multi-source survey data helps to provide a multi-dimensional environmental consideration framework for the design of intelligent grid lines, and enhances the comprehensive grasp of surrounding environmental factors during line design. By integrating multi-source data, it is possible to accurately assess the construction challenges brought by topography and avoid potential geological disaster risks, thereby optimizing line path selection and ensuring the long-term stable operation and efficient maintenance of the power grid.

[0033] In power grid survey, panchromatic image, multispectral image and hyperspectral image are three important data sources. Panchromatic image usually provides high-resolution black and white images, which are suitable for extracting macroscopic landform views such as roads, buildings, bridges, etc. Multispectral image can capture spectral information of different bands, which helps to identify and classify different surface features, such as vegetation types, water bodies, etc. Hyperspectral image provides more detailed spectral resolution, which can reveal the chemical and physical properties of substances, which is particularly important for geological survey and environmental monitoring. By fusing the above three types of survey data, designers can be provided with more comprehensive and detailed environmental information around the power grid survey area. In addition, data fusion can enhance the ability to capture complex environmental factors, such as vegetation cover, soil moisture, etc., which may affect the stability and security of the power grid.

[0034] There is an inherent trade-off in the development of optical remote sensing technology, namely the trade-off between spatial resolution and spectral resolution. This trade-off stems from the physical limitations of sensor design. Specifically, when a remote sensing sensor is designed to have high spatial resolution, it is able to provide finer details of surface features, but often at the expense of spectral resolution. The reduction in spectral resolution means that the sensor can capture fewer types of spectral information, which limits the ability to perform detailed analysis of surface material properties. Conversely, if the sensor is designed to have high spectral resolution, it is able to identify and distinguish more spectral bands, thereby providing richer material property information. However, such high spectral resolution sensors are usually accompanied by lower spatial resolution, resulting in limited ability to identify small-scale surface features.

[0035] In order to enrich the information expressed by remote sensing images, the hyperspectral image (HSI) and multispectral image (MSI) or panchromatic image (PAN) of the same area can be fused to ensure that sufficient surface details can be obtained while maintaining accurate identification of material properties. Currently, the commonly used fusion methods can be divided into three types: fusion of high and medium resolution images (FHM), fusion of high, medium and panchromatic images (FHP) and deep learning-based methods.

[0036] Early FHM methods can be divided into three categories: extended pan-sharpening methods, matrix decomposition-based methods, and tensor decomposition-based methods. In general, existing FHP methods are mainly derived from PanSharpening methods and can be divided into three categories: component replacement (CS), multiresolution analysis (MRA), and model-based methods. The CS method replaces the spatial components of the HS (or MS) image with the PAN image, so a fused HS (or MS) image with high spatial quality can be generated. In order to inject the spatial details of the PAN image into the HS image in the MRA method, various multiscale decomposition methods are used, such as decimated wavelet transform, smoothing filter-based intensity modulation, and modulation transfer function based on generalized Laplacian pyramid (GLP). The model-based method is based on the fusion model of different physical observation data. However, due to the large difference in spectral (or spatial) resolution between HS images and PAN images, this method is prone to severe spectral distortion.

[0037] In recent years, deep learning has been gradually applied to the fusion task of HSI, MSI and PAN, called super network. By learning from the data, excellent fusion results can be obtained. Because PAN can provide relatively higher spatial resolution than MSI. However, due to the huge difference in spectral and spatial resolution between HSI and PAN, the fusion algorithm based on deep learning has always been very unstable and there is significant spectral distortion.

[0038] Based on the above problems, the present application provides a multi-source remote sensing data fusion method and system for power grid survey, the method comprising: performing image block segmentation and linear projection on any remote sensing data, adding position embedding to the projected vector to obtain a vector representation; projecting the vector representation to obtain a first projection sequence and a second projection sequence respectively; performing feature enhancement on the state space prediction result of the first projection sequence according to the activation output value and the gating vector of the second projection sequence to obtain a feature enhancement vector; focusing on the local dependencies in the feature enhancement vector based on a masked multi-head attention mechanism to obtain a feature sequence containing local and global information; obtaining a gating weight using gated activation coding, and performing feature fusion on the feature sequence according to the respective gating weights in the multi-source remote sensing data to obtain a final fusion result.

[0039] This application combines state space and attention mechanisms to perform weighted fusion of feature sequences of multi-source remote sensing data, thereby solving the technical problem of low utilization of feature information in remote sensing data, and achieving the technical effect of making full use of feature information in different remote sensing data to improve the accuracy and detail richness of the final fusion result.

[0040] This specification provides a multi-source remote sensing data fusion method and system for power grid survey, which can be applied to the fusion of multi-source remote sensing survey data, wherein the multi-source remote sensing survey data is collected for the target area where the power grid line design is required. The types of remote sensing survey data may include panchromatic images, multi-spectral images, hyperspectral images and other types of remote sensing data. By fusing multi-source remote sensing data, relevant personnel can grasp the surface features and topographic features of interest in the target area, thereby providing a data basis for power grid line design.

[0041] In this embodiment, a multi-source remote sensing data fusion method for power grid survey is provided, which can be used to fuse multi-source remote sensing survey data of a target area before power grid line design, thereby providing data support for power grid line design. Figure 1 As shown, Figure 1 The step diagram of the multi-source remote sensing data fusion method for power grid survey provided in the embodiment of the present application is shown in the figure. The method includes:

[0042] S100. Performing linear projection on any remote sensing data in the multi-source remote sensing data after image block segmentation, adding position embedding to the projected vector, and obtaining a vector representation corresponding to any remote sensing data;

[0043] S200. Projecting the vector representation to a first direction to obtain a first projection sequence, and projecting the vector representation to a second direction to obtain a second projection sequence, wherein the first direction and the second direction are different directions;

[0044] S300. Feature enhancement is performed on the state space prediction result of the first projection sequence according to the activation output value and the gating vector of the second projection sequence to obtain a feature enhancement vector of any remote sensing data; wherein the state space prediction result is obtained by state conversion prediction based on the first projection sequence, the activation output value is obtained by nonlinear transformation and activation processing based on the second projection sequence, and the gating vector is obtained by gating the second projection sequence.

[0045] S400. Extract local dependencies from the feature enhancement vector based on the masked multi-head attention mechanism to obtain a feature sequence of any remote sensing data, and the feature sequence is used to represent the local dependencies of any remote sensing data.

[0046] S500. According to the feature sequence, the gating weight of any remote sensing data is obtained by using gated activation coding. According to the gating weight of each remote sensing data in the multi-source remote sensing data, the feature sequence of each remote sensing data is subjected to feature fusion to obtain the final fusion result.

[0047] Specifically, the multi-source remote sensing data used for fusion are all in a unified data format, and different remote sensing data all use the same boundaries and scales, and use the same coordinate system to represent the surface features of the same area in the same space-time framework, thereby ensuring the consistency of multi-source remote sensing data, improving the feature quality of the final fusion result, and enhancing the reliability of the features in the final fusion result. Exemplarily, the format of multi-source remote sensing data can be GeoTIFF format, and CGCS2000 is used as a unified coordinate system. Among them, the GeoTIFF format is an image format extended from TIFF (Tagged Image File Format), which is used to store geospatial data. It combines geographic information with image data so that image data can be associated with a geographic coordinate system. CGCS2000 (China Geodetic Coordinate System 2000, 2000 National Geodetic Coordinate System) is China's latest national geodetic coordinate system. Its origin is the center of mass of the entire earth including the ocean and atmosphere; the Z axis points from the origin to the direction of the earth's reference pole at epoch 2000.0. The direction of this epoch is calculated from the initial direction given by the International Time Bureau at epoch 1984.0. The directional time evolution ensures that there is no residual global rotation relative to the earth's crust. The X axis points from the origin to the intersection of the Greenwich reference meridian and the earth's equatorial plane at epoch 2000.0. The Y axis, Z axis and X axis form a right-handed orthogonal coordinate system, and adopts the scale in the sense of general relativity.

[0048] Furthermore, for the noise and interference appearing in the remote sensing data, the noise in the data is first removed by a filtering algorithm, and the filtering algorithm includes but is not limited to: Gaussian filtering, median filtering and mean filtering; then, the outliers in the remote sensing data are identified by statistical methods, where the outliers are points that significantly deviate from the normal range; then, the correlation between the outliers and the interference is judged, and the outliers that are correlated with the interference are processed through data cleaning, and the processing methods include but are not limited to: according to the set threshold, removing the outliers that do not meet the threshold; or replacing the outliers with statistical quantities such as the mean and median, etc.

[0049] Specifically, the obtained vector representation is input into the RS-Mambaformer module, and the step S200 of obtaining the first projection sequence and the second projection sequence, the step S300 of feature enhancement of the projection sequence, and the step S400 of obtaining the feature sequence are completed in the RS-Mambaformer module. The RS-Mambaformer module includes a Mamba state space model and a Transformer model, and combines the multi-head attention in the state space and Transformer models to improve the ability to extract long-term and short-term local dependencies, thereby better extracting local dependencies between features; in addition, the RS-Mambaformer module can improve computational efficiency while ensuring feature extraction capabilities, thereby reducing computational costs.

[0050] Furthermore, the first direction and the second direction are two different projection directions. By projecting the vector representation to different projection directions, the feature information in the vector representation can be extracted, thereby improving the learning ability of the feature information; at the same time, the dimension of each image block in the vector representation is reduced by projection, thereby reducing the amount of data to be processed and improving the data processing speed. Exemplarily, in this embodiment, the first direction is the X direction, that is, the horizontal direction, in which the edge features in the remote sensing data can be reflected, thereby identifying the linear features of the surface; the second direction is the Z direction, that is, the depth direction, in which the height changes in the remote sensing data can be reflected, thereby identifying the terrain undulation features. In other embodiments, a specific projection direction can be selected according to the feature type required by the actual scene.

[0051] In some embodiments, the vector representation may be dimensionally expanded by linear projection during projection. Exemplarily, in this embodiment, the vector representation is projected to the X direction and the Z direction by Linearx and Linearz, and both are dimensionally expanded to obtain a first projection sequence and a second projection sequence whose dimensions are the expanded state dimensions, wherein the size of the expanded state dimension is twice the size of the dimension of the vector representation. By expanding the dimension of the projection sequence, the level of detail of the image blocks contained in the projection sequence is increased, thereby improving the ability to learn features in the vector representation in subsequent processes.

[0052] Furthermore, a state space prediction result is obtained through state transition prediction according to the first projection sequence, and the local dependency between the image blocks of the remote sensing data is introduced into the feature enhancement vector through the state space prediction result. An activation output value is obtained through nonlinear transformation and activation processing according to the second projection sequence, and a gated vector is obtained through a gating operation, wherein the activation output value introduces a nonlinear variable into the feature enhancement vector through a nonlinear transformation, thereby enhancing the feature enhancement vector's ability to express feature information. The gating operations for obtaining the gated vector include, but are not limited to, input gates, forget gates, and output gates, etc., and the selection and filtering of feature information in the second projection sequence are achieved through different gating operations, so that important features can be enhanced.

[0053] Furthermore, feature sequences are used to represent local dependencies of remote sensing data, that is, the correlation and influence between adjacent pixels or regions in remote sensing data. This relationship helps capture the details and structural features of remote sensing images. Feature sequences include local information and global information, where local information refers to the feature details within any small range in the remote sensing image, and global features refer to the overall features and structures in the remote sensing image.

[0054] Specifically, when the feature sequences of each remote sensing data are fused, a progressive fusion mechanism is adopted. First, the feature sequences of any two remote sensing data in the multi-source remote sensing data are fused, and then the fusion result is fused with the next remote sensing data until all remote sensing data are fused to obtain the final fusion result. Among them, the gating weight of the remote sensing data in each stage represents the contribution of the remote sensing data to the fusion result when it is fused at that stage. Through the progressive fusion mechanism and the calculation of the gating weight, the contribution of each remote sensing data in the final fusion result can be dynamically adjusted to achieve targeted attention to important feature information.

[0055] The multi-source remote sensing data fusion method for power grid survey proposed in the embodiment of the present application, by combining state space and attention mechanism, performs feature extraction and feature enhancement on remote sensing data in two directions, thereby improving the ability to extract feature information in remote sensing data. On this basis, according to the importance of feature information in different remote sensing data, a gating mechanism is used to perform weighted fusion of multi-source remote sensing data, thereby making full use of the different advantages of multi-source remote sensing data, improving the accuracy and detail richness of the final fusion result, and providing high-quality data support for power grid survey.

[0056] Reference Figure 2 As shown, Figure 2This is a step diagram for obtaining a feature enhancement vector in an embodiment of the present application. As shown in the figure, as an embodiment of the present application, the state space prediction result includes a forward prediction result and a backward prediction result; the gating vector includes a forward gating vector and a backward gating vector; the state space prediction result of the first projection sequence is feature enhanced according to the activation output value and the gating vector of the second projection sequence to obtain a feature enhancement vector of any remote sensing data, including:

[0057] S320. Perform a gating operation on the forward prediction result according to the forward gating vector to obtain a forward gating output.

[0058] S330. Perform a gating operation on the backward prediction result according to the backward gating vector to obtain a backward gating output.

[0059] S340. Add the activation output value, the forward gated output, and the backward gated output pixel by pixel to obtain a feature enhancement vector.

[0060] Specifically, the forward prediction result indicates the result obtained by predicting the future state of the first projection sequence in the state space in chronological order; the backward prediction result indicates the result obtained by predicting the past state of the first projection sequence when the future observation result of the first projection sequence is known. The obtained feature enhancement vector enhances the understanding of the local dependency relationship between each image block in the first projection sequence by combining the prediction results in different time directions, thereby improving the accuracy of feature extraction when extracting the features corresponding to each image block.

[0061] Furthermore, by performing a gating operation on the forward prediction result through a forward gating vector, and performing a gating operation on the backward prediction result through a backward gating vector, the prediction result can be processed in a targeted manner in the forward and backward processing, including retaining or enhancing important feature information and suppressing or weakening unimportant feature information, so that the features in the feature enhancement vector are more representative and can accurately represent the necessary topographic features and other features in the target area.

[0062] Exemplarily, the forms of the forward gated output and the backward gated output are as follows:

[0063] Y' {forward} =Z {forward} ·Y0

[0064] Y' {backward} =Z {backward} ·Y1

[0065] Among them, Y' {forwar}d represents the forward gated output, Y' {forward} ∈R B×M×D ; Y' {backward} Represents the backward gated output, Y'{backward} ∈R B×M×D ; Z {forward} represents the forward gating vector; Z {backward} represents the backward gating vector; Y0 represents the forward prediction result; Y1 represents the backward prediction result. B represents the number of images contained in any remote sensing data; M represents the length of the vector representation; D represents the dimension of the state space.

[0066] Furthermore, the feature enhancement vector is in the following form:

[0067]

[0068] Among them, H1 represents the feature enhancement vector, H1∈R B×M×D ; Z' represents the activated output value.

[0069] Reference Figure 3 As shown, Figure 3 The step diagram of obtaining the state space observation result in the embodiment of the present application is shown in the figure. As an embodiment of the present application, the state space prediction result of the first projection sequence is obtained according to the following method:

[0070] S302. Perform feature extraction on the first projection sequence using one-dimensional convolution to obtain a first feature vector.

[0071] S304. Perform a linear transformation on the first eigenvector using a linear regression model to obtain a state transition matrix, an input transfer matrix, and a prediction matrix.

[0072] S306. Update the implicit state matrix of the first eigenvector according to the state transition matrix and the input transfer matrix.

[0073] S308. Predict the updated implicit state matrix according to the prediction matrix to obtain a state space prediction result.

[0074] Specifically, the RS-Mambaformer module includes an OSSM block and an S6 block. The first feature vector is input into the OSSM block, and the first feature vector is scanned in multiple directions by the OSSM block, wherein the directions include horizontal, vertical, diagonal and opposite directions, and multiple groups of image patch sequences are generated according to the scanning results, and the image patch sequences are input into the S6 block. Each group of image patch sequences is processed independently by the selective scanning mechanism of the S6 block, so as to obtain the time series prediction results of the image patch sequence, and global modeling is performed according to the time series prediction results, and finally the state space prediction results of the first projection sequence are obtained.

[0075] The OSSM block (Omnidirectional Selective Scan Module) is used to selectively scan the image in multiple directions and extract features from the image. The S6 block (SelectiveScan Space State Sequential Model) is responsible for processing the input sequence through a series of linear transformations and discretization processes, which plays a key role in capturing the temporal dynamics of the sequence, which is a key aspect of sequence modeling tasks such as language modeling.

[0076] For example, taking the forward prediction result as an example, the above steps can be specifically described as follows: first, the first eigenvector is linearly transformed to obtain a state transition matrix, an input transfer matrix, and a prediction matrix, wherein the state transition matrix, the input transfer matrix, and the prediction matrix all correspond to the same time step. Here, the state transition matrix, the input transfer matrix, and the prediction matrix are in the following forms:

[0077]

[0078] Among them, A0 represents the forward state transition matrix obtained according to the first eigenvector, A0∈R N×N ; B0 represents the forward input transfer matrix obtained according to the first eigenvector, B0∈R N×1 ; C0 represents the forward prediction matrix obtained according to the first eigenvector, C0∈R N×1 ; Δ0 represents the time step; is the forward state transition matrix updated according to the time step, is the forward input transfer matrix updated according to the time step, represents the forward prediction matrix updated according to the time step, I is the identity matrix.

[0079] Furthermore, the state space model operation is performed on the first eigenvector according to the state transition matrix, the input transfer matrix and the prediction matrix to obtain the state space prediction result, which is in the following form:

[0080]

[0081] Among them, Y0 represents the forward prediction result; h t is the implicit state matrix in the state space at time t that contains all the information at previous times, h t ∈R N , which has the following form:

[0082]

[0083] Among them, X'0 represents the first eigenvector; h t-1 is the implicit state matrix in the state space at time t-1 that contains all the information at the previous time, that is, the implicit state matrix at the previous time, h t-1 ∈R N Correspondingly, the backward prediction result can be obtained in the same way.

[0084] Reference Figure 4 As shown, Figure 4 This is a step diagram for obtaining an activation output value and a gating vector in an embodiment of the present application. As shown in the figure, as an embodiment of the present application, the activation output value and the gating vector of the second projection sequence are obtained according to the following method:

[0085] S312. Perform a nonlinear transformation on the second projection sequence using a linear layer and a linear rectification activation function to obtain an activation output value.

[0086] S314. Perform a gating operation on the second projection sequence using the first gating function to obtain a forward gating vector.

[0087] S316. Perform a gating operation on the second projection sequence using a second gating function to obtain a backward gating vector.

[0088] Specifically, the linear rectifier activation function (ReLU activation function) is also called the rectified linear unit. It is a commonly used activation function in artificial neural networks, usually referring to nonlinear functions represented by ramp functions and their variants. The activation output value is as follows:

[0089] Z'=Linear(Z)

[0090] Wherein, Z represents the second projection sequence.

[0091] Furthermore, the forms of the forward gating vector and the backward gating vector are as follows:

[0092] Z {forward} =f θi (Z)

[0093] Z {backward} =f ωi (Z)

[0094] in, Indicates that the parameter in the linear layer is θ i The first gate function of Indicates that the parameter in the linear layer is ω i The first gate function and the second gate function include but are not limited to: input gate, forget gate and output gate.

[0095] As an embodiment of the present application, local dependencies are extracted from feature enhancement vectors based on a masked multi-head attention mechanism to obtain a feature sequence of any remote sensing data, including:

[0096] S410. The feature enhancement vector is transformed based on the weight data corresponding to the masked multi-head attention mechanism to obtain a query matrix, a key matrix and a value matrix.

[0097] S420. Perform scaling dot product processing on the query matrix, key matrix and value matrix to obtain a feature sequence.

[0098] Specifically, the query matrix, key matrix, and value matrix are in the following forms:

[0099] Q i =H1W i Q

[0100] K i =H1W i K

[0101] V i =H1W i V

[0102] Among them, Q i represents the query matrix corresponding to the i-th head; K i represents the key matrix corresponding to the i-th head; V i represents the value matrix corresponding to the i-th head; W i Q , W i K and W i V is the weight data, W i Q ,W i K , W i Q , W i K and W i V are all learnable matrices that can be obtained through optimization algorithms. k For the output dimension of each attention head, the following equation is satisfied:

[0103]

[0104] Among them, d v represents the dimension of the value matrix; h represents the hidden state in the attention mechanism.

[0105] Furthermore, the dot product attention form used to perform the scaled dot product processing is as follows:

[0106]

[0107] Among them, O i Indicates that each attention head is connected to a dimension hd v For the output vector O, the output of the attention mechanism is obtained with a learnable projection matrix, which is in the following form:

[0108] H2=OW O

[0109] Among them, H2 represents the output of the attention mechanism, H2∈R B×L×D ; W O represents the learnable projection matrix, W O It can be obtained through optimization algorithm.

[0110] Furthermore, after obtaining the output of the attention mechanism, the feature sequence corresponding to the remote sensing data is obtained through residual links and normalization layers.

[0111] As an embodiment of the present application, the multi-source remote sensing data includes panchromatic images, multispectral images and hyperspectral images; before obtaining the gating weight of any remote sensing data by using gated activation coding according to the feature sequence, and performing feature fusion on the feature sequence of each remote sensing data according to the respective gating weight of each remote sensing data in the multi-source remote sensing data, and obtaining the final fusion result, the method further includes:

[0112] S502. Encode the full-color image according to the activation function and the first weight parameter to obtain a first encoding feature.

[0113] S504. Encode the multispectral image according to the activation function and the second weight parameter to obtain a second encoding feature.

[0114] S506. Encode the hyperspectral image according to the activation function and the third weight parameter to obtain a third encoding feature.

[0115] Specifically, the multi-source remote sensing data include panchromatic images, multispectral images and hyperspectral images. The panchromatic images are taken by the GF-2 satellite with a resolution of 0.8 meters per pixel; the multispectral images are provided by the Landsat 8 satellite, including blue, green, red and near-infrared bands, with a resolution of 30 meters per pixel; the hyperspectral images are provided by the Hyperion sensor, with 242 bands and a resolution of 30 meters per pixel.

[0116] Furthermore, after obtaining the corresponding feature sequence according to each remote sensing data, the feature sequences of different remote sensing data are progressively fused. Among them, the activation function used for gated activation is a hyperbolic tangent function, which improves the learning ability of complex features by introducing nonlinear factors. Furthermore, the forms of the first coding feature, the second coding feature and the third coding feature are as follows:

[0117]

[0118] in, represents the first coding feature; represents the second coding feature; Indicates the third coding feature; F PAN Represents the characteristic sequence of full-color image; F MSI Represents the characteristic sequence of multispectral images; F HSI Represents the characteristic sequence of hyperspectral images; W PAN represents the first weight parameter; W MSI represents the second weight parameter; W HSI represents the third weight parameter; W PAN , W MSI and W HSI is a learnable weight parameter that can be obtained through an optimization algorithm.

[0119] In some embodiments, when remote sensing data is used for power grid survey, the first weight parameter, the second weight parameter, and the third weight parameter are related to features in the remote sensing data, including terrain, vegetation, surface materials, etc. The weight parameters are determined based on the degree of influence of the above features on the stability and safety of the power grid.

[0120] Reference Figure 5 As shown, Figure 5 The figure is a step diagram for obtaining the final fusion result in the embodiment of the present application. As shown in the figure, as an embodiment of the present application, according to the feature sequence, the gating weight of any remote sensing data is obtained by using gated activation coding, and according to the respective gating weights of each remote sensing data in the multi-source remote sensing data, the feature sequences of each remote sensing data are subjected to feature fusion to obtain the final fusion result, including:

[0121] S510. Obtain a first gating weight using a sigmoid function according to the first weight parameter, the first coding feature, the second weight parameter, and the second coding feature.

[0122] S512. Perform feature fusion on the feature sequence of the panchromatic image and the feature sequence of the multispectral image according to the first gating weight to obtain a first fusion result.

[0123] S514. According to the third weight parameter, the third coding feature, the fourth weight parameter and the first fusion result, a second gating weight is obtained using an S-type function; wherein the fourth weight parameter is obtained by learning according to the first fusion result.

[0124] S516. Perform feature fusion on the first fusion result and the feature sequence of the hyperspectral image according to the second gating weight to obtain a final fusion result.

[0125] Specifically, the S-type function is a Sigmoid function, which is often used as an activation function of a neural network to map variables between [0, 1]. In this embodiment, the Sigmoid function is used to reflect the contribution of the features contained in each remote sensing data to the final fusion result, and the output range of the Sigmoid function is between 0 and 1, ensuring the rationality of the gating weight.

[0126] Furthermore, the first gating weight is in the form of:

[0127]

[0128] Among them, S1 represents the first gating weight; σ represents the Sigmoid function; W1 is in the following form:

[0129]

[0130] The first fusion result can be obtained through the first gating weight, which is in the following form:

[0131]

[0132] Among them, F P,M Represents the first fusion result.

[0133] Furthermore, the second gating weight is in the form of:

[0134]

[0135] Among them, S2 represents the second gating weight; W2 is in the following form:

[0136]

[0137] Among them, W P,M is a learnable weight matrix that can be obtained through optimization algorithms. The final fusion result can be obtained through the second gating weight, which is in the following form:

[0138]

[0139] Among them, F final Represents the final fusion result.

[0140] In some embodiments, when remote sensing data is used for power grid survey, the first gating weight and the second gating weight may be determined according to the importance of the feature. For example, in a densely forested area, the hyperspectral feature of the vegetation type may be more important than the terrain feature. In this case, the second gating weight may be appropriately reduced to increase the contribution of the hyperspectral image in the final fusion result. By calculating the first gating weight and the second gating weight, the contribution of the multi-source remote sensing data to the final fusion result is dynamically adjusted, so that the features in the final fusion result that are more distinguishing for the power grid survey task can be strengthened, thereby providing targeted attention.

[0141] As an embodiment of the present application, linear projection is performed on any remote sensing data in the multi-source remote sensing data after image block segmentation, and position embedding is added to the projected vector to obtain a vector representation corresponding to any remote sensing data, including:

[0142] S110. Divide any remote sensing data into a number of image blocks, and project the image blocks into a state space to obtain an image block projection sequence.

[0143] S120. Perform position embedding on the image block projection sequence.

[0144] S130. Obtain a vector representation according to the image block projection sequence after position embedding and the category token corresponding to the remote sensing data.

[0145] Specifically, the image blocks are flat two-dimensional blocks with no overlapping areas. The vector representation is as follows:

[0146]

[0147] Among them, T0 represents the vector representation, T0∈R B×M×D ; is the image block, J is the number of image blocks, J=M-1; p is the size of the image block, indicating that the size of the image block is p×p; C is the number of channels of remote sensing data; x cls represents a class token, which is used to represent a sequence of image patches; is a learnable projection matrix that can be obtained through an optimization algorithm and used to convert image patches into a processable format; E pos represents the position embedding, E pos ∈R J+1×D .

[0148] As an embodiment of the present application, before linearly projecting any remote sensing data in the multi-source remote sensing data after image block segmentation and adding position embedding to the projected vector to obtain a vector representation corresponding to any remote sensing data, the method further includes:

[0149] S102. Preprocess the multi-source remote sensing data; wherein the preprocessing includes at least one of a unified data format, a unified coordinate system, data registration, data resampling, and filtering and denoising.

[0150] Specifically, there may be problems such as inconsistent formats and coordinate axes among the multi-source remote sensing data collected, and the resolutions of different remote sensing data are also different. Therefore, for multi-source remote sensing data, it is necessary to convert the data from different data sources into a unified data format through pre-processing such as format conversion, coordinate system unification, data registration, and data resampling, and eliminate the boundaries and scale differences between the data to unify them into the same spatiotemporal framework.

[0151] Exemplarily, the multi-source remote sensing data includes panchromatic images, multispectral images and hyperspectral images, and the multi-source remote sensing data are converted into a unified GeoTIFF format, and the coordinate system is unified to CGCS2000. For the multi-source remote sensing data after the unified format and coordinate system, firstly, the multispectral images and hyperspectral images are registered by geographic information system software to ensure the spatial alignment of the multispectral images and hyperspectral images with the high-resolution panchromatic images. Secondly, the multispectral images and hyperspectral images are resampled to match the higher resolution of the panchromatic images. The resampling methods include but are not limited to: bilinear interpolation and cubic convolution methods.

[0152] This embodiment provides a multi-source remote sensing data fusion method for power grid survey, which specifically includes the following steps:

[0153] S602. Preprocess the multi-source remote sensing data; wherein the multi-source remote sensing data includes panchromatic images, multispectral images and hyperspectral images, and the preprocessing includes at least one of unified data format, unified coordinate system, data alignment, data resampling and filtering denoising.

[0154] S604. Divide any remote sensing data in the multi-source remote sensing data into a number of image blocks, and project the image blocks into a state space to obtain an image block projection sequence.

[0155] S606. Perform position embedding on the image block projection sequence.

[0156] S608. Obtain a vector representation according to the image block projection sequence after position embedding and the category token corresponding to the remote sensing data.

[0157] Reference Figure 6 As shown, Figure 6This is a schematic diagram of expanding a remote sensing image into a vector mapping in an embodiment of the present application. As shown in the figure, in steps S602 to S608, for the input multi-source remote sensing data, first remote sensing data is selected, and image block segmentation is performed on the first remote sensing data to obtain multiple image blocks. Image block segmentation enhances the ability to capture feature details in remote sensing data and improves the accuracy of overall features.

[0158] Secondly, the segmented image blocks are linearly mapped to the state space, and the vector representation corresponding to the first remote sensing data is obtained after adding position embedding. Among them, position embedding enables the image blocks to have absolute and relative positions, providing the ability to perceive spatial information for subsequent processing, and can better utilize the dependencies in remote sensing data. The state space provides dynamic changes in the system state, so that the local dependencies between image blocks can be extracted in subsequent processing, thereby improving the accuracy of feature extraction.

[0159] S610. Project the vector representation to a first direction to obtain a first projection sequence, and project the vector representation to a second direction to obtain a second projection sequence.

[0160] S612. Use one-dimensional convolution to perform feature extraction on the first projection sequence to obtain a first feature vector.

[0161] S614. Use the linear regression model to perform linear transformation on the first eigenvector to obtain a state transition matrix, an input transfer matrix and a prediction matrix.

[0162] S618. Update the implicit state matrix of the first eigenvector according to the state transition matrix and the input transfer matrix.

[0163] S620. Predict the updated implicit state matrix according to the prediction matrix to obtain a state space prediction result; wherein the state space prediction result includes a forward prediction result and a backward prediction result.

[0164] S622. Perform a nonlinear transformation on the second projection sequence using a linear layer and a linear rectification activation function to obtain an activation output value.

[0165] S624. Perform a gating operation on the second projection sequence using the first gating function to obtain a forward gating vector.

[0166] S626. Perform a gating operation on the second projection sequence using a second gating function to obtain a backward gating vector.

[0167] S628. Perform a gating operation on the forward prediction result according to the forward gating vector to obtain a forward gating output.

[0168] S630. Perform a gating operation on the backward prediction result according to the backward gating vector to obtain a backward gating output.

[0169] S632. Add the activation output value, the forward gated output, and the backward gated output pixel by pixel to obtain a feature enhancement vector.

[0170] S634. Based on the weight data corresponding to the masked multi-head attention mechanism, the feature enhancement vector is transformed to obtain a query matrix, a key matrix and a value matrix.

[0171] S636. Perform scaling dot product processing on the query matrix, key matrix and value matrix to obtain a feature sequence.

[0172] Reference Figure 7 As shown, Figure 7 This is a structural diagram of the RS-Mambaformer module in an embodiment of the present application. As shown in the figure, for the vector representation of the first remote sensing data, after the vector representation is input into the RS-Mambaformer module, the vector representation is first normalized through the normalization layer to obtain a normalized vector representation T0'. The normalized vector representation is processed as in steps S610 to S632 to obtain a feature enhancement vector H1. For the feature enhancement vector H1, local dependencies are extracted from the feature enhancement vector H1 based on the masked multi-head attention mechanism to obtain a feature sequence of the first remote sensing data. Accordingly, the other remote sensing data in the multi-source remote sensing data are processed in the same way as the first remote sensing data to obtain a feature sequence of all remote sensing data in the multi-source remote sensing data.

[0173] In some embodiments, in order to obtain the characteristic sequence of the first remote sensing data, multiple RS-Mambaformer modules may be connected to process the characteristics of the first remote sensing data multiple times, thereby enhancing the characteristic information in the first remote sensing data and providing data support for power grid survey.

[0174] S638. Encode the full-color image according to the activation function and the first weight parameter to obtain a first encoding feature.

[0175] S640. Encode the multispectral image according to the activation function and the second weight parameter to obtain a second encoding feature.

[0176] S642. Encode the hyperspectral image according to the activation function and the third weight parameter to obtain a third encoding feature.

[0177] S644. Obtain a first gating weight using a S-type function according to the first weight parameter, the first coding feature, the second weight parameter, and the second coding feature.

[0178] S646. Perform feature fusion on the feature sequence of the panchromatic image and the feature sequence of the multispectral image according to the first gating weight to obtain a first fusion result.

[0179] S648. According to the third weight parameter, the third coding feature, the fourth weight parameter and the first fusion result, a second gating weight is obtained using an S-type function; wherein the fourth weight parameter is obtained by learning according to the first fusion result.

[0180] S650. Perform feature fusion on the first fusion result and the feature sequence of the hyperspectral image according to the second gating weight to obtain a final fusion result.

[0181] Reference Figure 8 , Figure 8 This is a network architecture diagram of an embodiment of the present application. As shown in the figure, in this embodiment, multi-source remote sensing data includes panchromatic images, multispectral images and hyperspectral images, among which panchromatic images can provide high-resolution black and white images, which are suitable for extracting macroscopic landform views such as roads, buildings, bridges, etc.; multispectral images can capture spectral information of different bands, which helps to identify and classify different surface features, such as vegetation types, water bodies, etc.; hyperspectral images can provide more detailed spectral resolution, which can reveal the chemical and physical properties of substances, and are particularly important for geological surveys and environmental monitoring. For the feature sequence of the obtained multi-source remote sensing data, the panchromatic image and the multispectral image are firstly fused; and based on the preliminary fusion result, the hyperspectral image is fused with the preliminary fusion result to obtain the final fusion result, thereby making full use of the spatial resolution advantage of the panchromatic image and the spectral resolution advantage of the hyperspectral image, so that the target area features contained in the final fusion result are more comprehensive and can highlight important features.

[0182] Furthermore, by calculating the gated weights using gated activation coding in steps S644 to S650, the contribution of different remote sensing data can be dynamically adjusted during the fusion process, thereby improving the applicability and flexibility of the present application, making the present application applicable to new or different data environments, and ensuring the wide applicability of the final fusion results.

[0183] In addition, it should be noted that this application proposes the RS-Mambaformer module by combining the Mamba state space model and the Transformer model. Compared with the method of only using the Transformer model in the related art, it can effectively reduce the complexity of the calculation, reduce the calculation overhead, and thus improve the calculation efficiency.

[0184] This embodiment provides a multi-source remote sensing data fusion system for power grid survey, please refer to Fig. 9 , Fig. 9 The module diagram of the multi-source remote sensing data fusion system for power grid survey provided in the embodiment of the present application is shown in the figure. The system includes:

[0185] The segmentation and embedding module 100 is used to perform linear projection on any remote sensing data in the multi-source remote sensing data after image block segmentation, add position embedding to the projected vector, and obtain a vector representation corresponding to any remote sensing data.

[0186] The feature projection module 200 is used to project the vector representation to a first direction to obtain a first projection sequence, and to project the vector representation to a second direction to obtain a second projection sequence, wherein the first direction and the second direction are different directions.

[0187] The feature enhancement module 300 is used to enhance the features of the state space prediction results of the first projection sequence according to the activation output value and the gating vector of the second projection sequence, so as to obtain the feature enhancement vector of any remote sensing data; wherein the state space prediction results are obtained by state conversion prediction based on the first projection sequence, the activation output value is obtained by nonlinear transformation and activation processing based on the second projection sequence, and the gating vector is obtained by gating the second projection sequence.

[0188] The feature learning module 400 is used to extract local dependencies from the feature enhancement vector based on the masked multi-head attention mechanism to obtain a feature sequence of any remote sensing data, and the feature sequence is used to represent the local dependencies of any remote sensing data.

[0189] The feature fusion module 500 is used to obtain the gating weight of any remote sensing data according to the feature sequence using gated activation coding, and to perform feature fusion on the feature sequence of each remote sensing data according to the gating weight of each remote sensing data in the multi-source remote sensing data to obtain the final fusion result.

[0190] In some optional implementations, the state space prediction result includes a forward prediction result and a backward prediction result; the gating vector includes a forward gating vector and a backward gating vector; and the feature enhancement module 300 includes:

[0191] The forward gating operation unit is used to perform a gating operation on the forward prediction result according to the forward gating vector to obtain a forward gating output.

[0192] The backward gating operation unit is used to perform a gating operation on the backward prediction result according to the backward gating vector to obtain a backward gating output.

[0193] The feature vector enhancement unit is used to perform pixel-by-pixel addition according to the activation output value, the forward gated output and the backward gated output to obtain a feature enhancement vector.

[0194] In some optional implementations, the feature enhancement module 300 further includes:

[0195] The first feature vector extraction unit is used to extract features from the first projection sequence by using one-dimensional convolution to obtain a first feature vector.

[0196] The first linear transformation unit is used to perform linear transformation on the first eigenvector by using a linear regression model to obtain a state conversion matrix, an input transfer matrix and a prediction matrix.

[0197] The first state transfer unit is used to update the implicit state matrix of the first eigenvector according to the state transition matrix and the input transfer matrix.

[0198] The first state prediction unit is used to predict the updated implicit state matrix according to the prediction matrix to obtain a state space prediction result.

[0199] In some optional implementations, the feature enhancement module 300 further includes:

[0200] The second activation transformation unit is used to perform a nonlinear transformation on the second projection sequence using a linear layer and a linear rectification activation function to obtain an activation output value.

[0201] The second forward gating unit is used to perform a gating operation on the second projection sequence with the first gating function to obtain a forward gating vector.

[0202] The second backward gating unit is used to perform a gating operation on the second projection sequence with a second gating function to obtain a backward gating vector.

[0203] In some optional implementations, the feature learning module 400 includes:

[0204] A vector conversion unit is used to convert the feature enhancement vector based on the weight data corresponding to the masked multi-head attention mechanism to obtain a query matrix, a key matrix and a value matrix.

[0205] The dot product processing unit is used to perform scaling dot product processing according to the query matrix, the key matrix and the value matrix to obtain a feature sequence.

[0206] In some optional implementations, the multi-source remote sensing data includes panchromatic images, multispectral images, and hyperspectral images; the feature fusion module 500 includes:

[0207] The first encoding unit is used to encode the full-color image according to the activation function and the first weight parameter to obtain a first encoding feature.

[0208] The second encoding unit is used to encode the multispectral image according to the activation function and the second weight parameter to obtain a second encoding feature.

[0209] The third encoding unit is used to encode the hyperspectral image according to the activation function and the third weight parameter to obtain a third encoding feature.

[0210] In some optional implementations, the feature fusion module 500 further includes:

[0211] The first weight calculation unit is used to obtain the first gating weight using a S-type function according to the first weight parameter, the first coding feature, the second weight parameter and the second coding feature.

[0212] The first fusion unit is used to perform feature fusion on the feature sequence of the panchromatic image and the feature sequence of the multispectral image according to the first gating weight to obtain a first fusion result.

[0213] The second weight calculation unit is used to obtain the second gating weight using an S-type function according to the third weight parameter, the third encoding feature, the fourth weight parameter and the first fusion result; wherein the fourth weight parameter is obtained by learning according to the first fusion result.

[0214] The second fusion unit is used to perform feature fusion on the first fusion result and the feature sequence of the hyperspectral image according to the second gating weight to obtain a final fusion result.

[0215] In some optional implementations, the segmentation embedding module 100 includes:

[0216] The image segmentation unit is used to segment any remote sensing data into a number of image blocks, and project the image blocks into the state space to obtain an image block projection sequence.

[0217] The position embedding unit is used to embed the position of the image block projection sequence.

[0218] The sequence acquisition unit is used to obtain a vector representation according to the image block projection sequence after position embedding and the category token corresponding to the remote sensing data.

[0219] In some optional implementations, the segmentation embedding module 100 further includes:

[0220] The preprocessing unit is used to preprocess the multi-source remote sensing data; wherein the preprocessing includes at least one of a unified data format, a unified coordinate system, data registration, data resampling and filtering and denoising.

[0221] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0222] The multi-source remote sensing data fusion system for power grid survey in this embodiment is presented in the form of functional units, where the units refer to ASIC (Application Specific Integrated Circuit) circuits, processors and memories that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0223] The embodiment of the present application also provides a computer device, which includes: one or more processors, memory, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system).

[0224] The processor may be a central processing unit, a network processor or a combination thereof. The processor may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0225] The memory stores instructions executable by at least one processor, so that the at least one processor executes the method shown in the above embodiment.

[0226] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0227] The memory may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid state drive; the memory may also include a combination of the above types of memory.

[0228] The computer device also includes a communication interface, which is used for the computer device to communicate with other devices or a communication network.

[0229] The embodiment of the present application also provides a computer-readable storage medium. The above method according to the embodiment of the present application can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0230] The embodiment of the present application provides a computer program product, which includes computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method of any embodiment of the present application.

[0231] Although the embodiments of the present application are described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present application, and such modifications and variations are all within the scope defined by the appended claims.

[0232] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A multi-source remote sensing data fusion method for power grid survey, characterized in that: The method comprises: Performing linear projection on any remote sensing data in the multi-source remote sensing data after image block segmentation, adding position embedding to the projected vector, and obtaining a vector representation corresponding to any remote sensing data; Projecting the vector representation to a first direction to obtain a first projection sequence, and projecting the vector representation to a second direction to obtain a second projection sequence, wherein the first direction and the second direction are different directions; According to the activation output value and the gating vector of the second projection sequence, the state space prediction result of the first projection sequence is feature enhanced to obtain the feature enhancement vector of the any remote sensing data; wherein the state space prediction result is obtained by state conversion prediction according to the first projection sequence, the activation output value is obtained by nonlinear transformation and activation processing according to the second projection sequence, and the gating vector is obtained by gating the second projection sequence; Extracting local dependencies from the feature enhancement vector based on a masked multi-head attention mechanism to obtain a feature sequence of any remote sensing data, wherein the feature sequence is used to represent the local dependencies of any remote sensing data; According to the feature sequence, the gating weight of any remote sensing data is obtained using gated activation coding, and according to the gating weight of each remote sensing data in the multi-source remote sensing data, the feature sequence of each remote sensing data is feature fused to obtain the final fusion result.

2. The method according to claim 1, characterized in that The state space prediction result includes a forward prediction result and a backward prediction result; the gating vector includes a forward gating vector and a backward gating vector; the feature enhancement of the state space prediction result of the first projection sequence according to the activation output value and the gating vector of the second projection sequence to obtain the feature enhancement vector of any remote sensing data includes: Performing a gating operation on the forward prediction result according to the forward gating vector to obtain a forward gating output; Performing a gating operation on the backward prediction result according to the backward gating vector to obtain a backward gating output; The feature enhancement vector is obtained by performing pixel-by-pixel addition on the activation output value, the forward gated output, and the backward gated output.

3. The method according to claim 2, characterized in that The state space prediction result of the first projection sequence is obtained in the following manner: Performing feature extraction on the first projection sequence using one-dimensional convolution to obtain a first feature vector; Performing a linear transformation on the first eigenvector using a linear regression model to obtain a state transition matrix, an input transfer matrix, and a prediction matrix; updating an implicit state matrix of the first eigenvector according to the state transition matrix and the input transfer matrix; The updated implicit state matrix is ​​predicted according to the prediction matrix to obtain the state space prediction result.

4. The method according to claim 2, characterized in that: The activation output value and the gating vector of the second projection sequence are obtained according to the following method: Performing a nonlinear transformation on the second projection sequence using a linear layer and a linear rectification activation function to obtain the activation output value; Performing a gating operation on the second projection sequence with a first gating function to obtain the forward gating vector; A gating operation is performed on the second projection sequence using a second gating function to obtain the backward gating vector.

5. The method according to claim 1, characterized in that The method of extracting local dependencies from the feature enhancement vector based on the masked multi-head attention mechanism to obtain a feature sequence of any remote sensing data includes: The feature enhancement vector is transformed based on the weight data corresponding to the masked multi-head attention mechanism to obtain the query matrix, key matrix and value matrix; The feature sequence is obtained by performing a scaling dot product process on the query matrix, the key matrix and the value matrix.

6. The method according to claim 1, characterized in that The multi-source remote sensing data includes panchromatic images, multispectral images and hyperspectral images; before obtaining the gating weight of any remote sensing data by gated activation coding according to the feature sequence, and performing feature fusion on the feature sequence of each remote sensing data according to the gating weight of each remote sensing data in the multi-source remote sensing data to obtain the final fusion result, the method further includes: Encoding the panchromatic image according to the activation function and the first weight parameter to obtain a first encoding feature; Encoding the multispectral image according to the activation function and the second weight parameter to obtain a second encoding feature; The hyperspectral image is encoded according to the activation function and the third weight parameter to obtain a third encoding feature.

7. The method according to claim 6, characterized in that The method of obtaining a gating weight of any remote sensing data by using gated activation coding according to the feature sequence, and performing feature fusion on the feature sequence of each remote sensing data according to the gating weight of each remote sensing data in the multi-source remote sensing data to obtain a final fusion result includes: Obtaining a first gating weight using a sigmoid function according to the first weight parameter, the first coding feature, the second weight parameter, and the second coding feature; According to the first gating weight, feature fusion is performed on the feature sequence of the panchromatic image and the feature sequence of the multispectral image to obtain a first fusion result; Obtaining a second gating weight using a S-type function according to the third weight parameter, the third encoding feature, the fourth weight parameter and the first fusion result; wherein the fourth weight parameter is obtained by learning according to the first fusion result; According to the second gating weight, feature fusion is performed on the first fusion result and the feature sequence of the hyperspectral image to obtain a final fusion result.

8. The method according to any one of claims 1 to 7, characterized in that The performing linear projection on any remote sensing data in the multi-source remote sensing data after image block segmentation, adding position embedding to the projected vector, and obtaining a vector representation corresponding to the any remote sensing data, comprises: Dividing any remote sensing data into a plurality of image blocks, and projecting the plurality of image blocks into a state space to obtain an image block projection sequence; Performing position embedding on the image block projection sequence; The vector representation is obtained according to the image block projection sequence after position embedding and the category token corresponding to the remote sensing data.

9. The method according to any one of claims 1 to 7, characterized in that Before performing linear projection on any remote sensing data in the multi-source remote sensing data after image block segmentation and adding position embedding to the projected vector to obtain the vector representation corresponding to the any remote sensing data, the method further includes: The multi-source remote sensing data is preprocessed; wherein the preprocessing includes at least one of a unified data format, a unified coordinate system, data registration, data resampling, and filtering and denoising.

10. A multi-source remote sensing data fusion system for power grid survey, characterized in that: The system comprises: A segmentation and embedding module, used for performing linear projection on any remote sensing data in the multi-source remote sensing data after image block segmentation, adding position embedding to the projected vector, and obtaining a vector representation corresponding to any remote sensing data; A feature projection module, configured to project the vector representation to a first direction to obtain a first projection sequence, and project the vector representation to a second direction to obtain a second projection sequence, wherein the first direction and the second direction are different directions; a feature enhancement module, configured to perform feature enhancement on the state space prediction result of the first projection sequence according to the activation output value and the gating vector of the second projection sequence, so as to obtain a feature enhancement vector of the any remote sensing data; wherein the state space prediction result is obtained by predicting the state conversion of the first projection sequence, the activation output value is obtained by performing nonlinear transformation and activation processing on the second projection sequence, and the gating vector is obtained by performing a gating operation on the second projection sequence; A feature learning module, used for extracting local dependencies from the feature enhancement vector based on a masked multi-head attention mechanism to obtain a feature sequence of any remote sensing data, wherein the feature sequence is used to represent the local dependencies of any remote sensing data; The feature fusion module is used to obtain the gating weight of any remote sensing data according to the feature sequence using gated activation coding, and to perform feature fusion on the feature sequence of each remote sensing data according to the gating weight of each remote sensing data in the multi-source remote sensing data to obtain the final fusion result.

11. The system according to claim 10, characterized in that The state space prediction result includes a forward prediction result and a backward prediction result; the gating vector includes a forward gating vector and a backward gating vector; The feature enhancement module comprises: A forward gating operation unit, configured to perform a gating operation on the forward prediction result according to the forward gating vector to obtain a forward gating output; A backward gating operation unit, configured to perform a gating operation on the backward prediction result according to the backward gating vector to obtain a backward gating output; A feature vector enhancement unit is used to perform pixel-by-pixel addition according to the activation output value, the forward gated output and the backward gated output to obtain the feature enhancement vector.

12. The system according to claim 11, characterized in that The feature enhancement module also includes: A first feature vector extraction unit, configured to extract features from the first projection sequence using one-dimensional convolution to obtain a first feature vector; A first linear transformation unit, used for performing a linear transformation on the first feature vector by using a linear regression model to obtain a state transition matrix, an input transfer matrix and a prediction matrix; a first state transfer unit, configured to update an implicit state matrix of the first eigenvector according to the state transition matrix and the input transfer matrix; The first state prediction unit is used to predict the updated implicit state matrix according to the prediction matrix to obtain the state space prediction result.

13. The system according to claim 11, characterized in that The feature enhancement module also includes: A second activation transformation unit, configured to perform a nonlinear transformation on the second projection sequence using a linear layer and a linear rectification activation function to obtain the activation output value; a second forward gating unit, configured to perform a gating operation on the second projection sequence using a first gating function to obtain the forward gating vector; The second backward gating unit is used to perform a gating operation on the second projection sequence with a second gating function to obtain the backward gating vector.

14. The system according to claim 10, characterized in that The feature learning module includes: A vector conversion unit, used to convert the feature enhancement vector based on the weight data corresponding to the masked multi-head attention mechanism to obtain a query matrix, a key matrix and a value matrix; A dot product processing unit is used to perform scaling dot product processing according to the query matrix, the key matrix and the value matrix to obtain the feature sequence.

15. The system according to claim 10, characterized in that The multi-source remote sensing data includes panchromatic images, multispectral images and hyperspectral images; the feature fusion module includes: A first encoding unit, used for encoding the full-color image according to an activation function and a first weight parameter to obtain a first encoding feature; A second encoding unit, used for encoding the multispectral image according to the activation function and the second weight parameter to obtain a second encoding feature; The third encoding unit is used to encode the hyperspectral image according to the activation function and the third weight parameter to obtain a third encoding feature.

16. The system according to claim 15, characterized in that The feature fusion module also includes: a first weight calculation unit, configured to obtain a first gating weight using a sigmoid function according to the first weight parameter, the first coding feature, the second weight parameter, and the second coding feature; A first fusion unit is used to perform feature fusion on the feature sequence of the panchromatic image and the feature sequence of the multispectral image according to the first gating weight to obtain a first fusion result; a second weight calculation unit, configured to obtain a second gating weight using a S-type function according to the third weight parameter, the third encoding feature, a fourth weight parameter and the first fusion result; wherein the fourth weight parameter is obtained by learning according to the first fusion result; The second fusion unit is used to perform feature fusion on the first fusion result and the feature sequence of the hyperspectral image according to the second gating weight to obtain a final fusion result.

17. The system according to any one of claims 10 to 16, characterized in that The segmentation embedding module comprises: An image segmentation unit, used for segmenting any remote sensing data into a plurality of image blocks, and projecting the plurality of image blocks into a state space to obtain an image block projection sequence; A position embedding unit, used for performing position embedding on the image block projection sequence; A sequence acquisition unit is used to obtain the vector representation according to the image block projection sequence after position embedding and the category token corresponding to the remote sensing data.

18. The system according to any one of claims 10 to 16, characterized in that The segmentation embedding module also includes: A preprocessing unit is used to preprocess the multi-source remote sensing data; wherein the preprocessing includes at least one of a unified data format, a unified coordinate system, data alignment, data resampling and filtering denoising.

19. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the multi-source remote sensing data fusion method for power grid survey according to any one of claims 1 to 9.

Images