Image fusion method, system and device based on curvature guidance, product and medium

Abstract

The invention relates to the field of computer image processing, and provides an image fusion method, system and device based on curvature guidance, a product and a medium, and the method comprises the steps: obtaining a to-be-processed image, and carrying out the feature extraction to obtain a shallow feature; the number of sub-encoder units is determined, the sub-encoder units are connected in series to form an encoder unit, and intermediate target encoding features are obtained; performing feature enhancement through a curvature space self-attention mechanism, a low-rank mixed attention mechanism and a curvature channel self-attention mechanism to obtain intermediate coding features, and performing updating to obtain intermediate target coding features; constructing sub-decoder units, connecting the sub-decoder units in series to form a decoder unit, and sequentially connecting the encoder unit and the decoder unit in series; and updating the intermediate target coding features through a low-rank mixed attention mechanism and up-sampling operation to obtain target decoding features, and fusing the target decoding features to obtain a target image. According to the invention, a high-quality target image can be output.

Description

Technical Field

[0001] The present invention relates to the field of computer image processing technology, and in particular to a curvature-guided image fusion method, system, device, product and medium. Background Art

[0002] HS (Hyperspectral) images are obtained by sampling hundreds of continuous narrow spectral bands using a spectral imaging system. They can provide rich spectral information and are often used to study the spectral difference characteristics of materials. However, due to the limitations of the optical hardware system design, traditional hyperspectral imaging always faces a trade-off between spectral and spatial resolution during signal acquisition. Single-spectral imaging systems cannot obtain HRHS (High Resolution Hyperspectral) images. This physical technical constraint causes conventional HS images to have an inherent problem of blurred spatial details. On the contrary, PAN (panchromatic) imaging systems can obtain higher spatial resolution under the same conditions because they only output single-band imaging PAN images, but they lack multi-spectral resolution capabilities. The hyperspectral panchromatic sharpening method deeply fuses the spectral information of HS images with the spatial information of PAN images, so that the generated HRHS images have both the spectral information of HS images and the spatial information of PAN images, significantly enhancing its practical value in precision agriculture, mineral exploration and other fields. Traditional hyperspectral pan-sharpening methods can be divided into two categories: traditional methods and deep learning-based methods. Traditional methods can be divided into four categories: component replacement-based methods, multi-resolution analysis-based methods, Bayesian-based methods, and model-based methods. Although these traditional methods are easy to implement and physically interpretable, they often suffer from spatial and spectral distortion problems due to inappropriate prior assumptions and imprecise manual feature extraction. Summary of the invention

[0003] The present invention aims to solve at least one of the technical problems existing in the related art. To this end, the present invention provides a curvature-guided image fusion method, system, device, product and medium to achieve the output of HRHS images with higher resolution and less distortion.

[0004] The present invention provides an image fusion method based on curvature guidance, comprising: S1: Obtain an image to be processed, and extract features of the image to be processed through a residual block to obtain shallow features; S2: construct sub-encoder units and determine the number of the sub-encoder units, and connect the sub-encoder units in series to form an encoder unit; S3: In the sub-encoder unit, feature enhancement is performed through the curvature space self-attention mechanism, the low-rank mixed attention mechanism and the curvature channel self-attention mechanism to obtain intermediate coding features, and in other sub-encoder units, the intermediate coding features are updated to obtain intermediate target coding features, and the shallow features are input into the encoder unit to obtain intermediate target coding features; S4: construct sub-decoder units and determine the number of the sub-decoder units, connect the sub-decoder units in series to form a decoder unit, and connect the encoder unit and the decoder unit in series; S5: In the sub-decoder unit, the intermediate target encoding features are updated through the low-rank mixed attention mechanism and upsampling operation to obtain target decoding features, and the target decoding features are fused to obtain a target image.

[0005] According to the curvature-guided image fusion method provided by the present invention, step S1 specifically includes: S11: Acquire the image to be processed including a hyperspectral image and a panchromatic image, extract the panchromatic image through a residual block, and obtain panchromatic shallow features; S12: upsampling the hyperspectral image and extracting the hyperspectral image through the residual block to obtain hyperspectral shallow features, and using the panchromatic shallow features and the hyperspectral shallow features as the shallow features.

[0006] According to the curvature-guided image fusion method provided by the present invention, in step S3, the implementation method of the curvature space self-attention mechanism and the curvature channel self-attention mechanism is: the implementation method of the curvature space self-attention mechanism and the curvature channel self-attention mechanism is: first calculate the first curvature parameter and the second curvature parameter, and calculate the differential curvature according to the first curvature parameter and the second curvature parameter, and then use the differential curvature and the maximum pooling operation and the average pooling operation to perform feature enhancement, so as to obtain the curvature space feature and the curvature channel feature respectively.

[0007] According to the curvature-guided image fusion method provided by the present invention, in step S3, the low-rank mixed attention mechanism is implemented as follows: determine a low-rank mixed attention input value, perform non-negative matrix decomposition and convolution on the low-rank mixed attention input value, and obtain a full-color low-rank output and a high-spectral low-rank output, respectively.

[0008] According to the curvature-guided image fusion method provided by the present invention, in step S4, the number of the sub-decoder units is equal to the number of the sub-encoder units.

[0009] According to the curvature-guided image fusion method provided by the present invention, in step S5, the intermediate target encoding features output by the last sub-decoder unit are used as target decoding features, and the target decoding features are spliced, and feature fusion and channel adjustment are performed through the residual block to obtain the target image.

[0010] The present invention also provides an image fusion system based on curvature guidance, comprising: Shallow feature module: used to obtain the image to be processed, extract features of the image to be processed through the residual block, and obtain shallow features; Encoder unit module: used to construct sub-encoder units and determine the number of the sub-encoder units, and connect the sub-encoder units in series to form an encoder unit; Intermediate coding feature module: used to perform feature enhancement in the sub-encoder unit through the curvature space self-attention mechanism, the low-rank mixed attention mechanism and the curvature channel self-attention mechanism to obtain intermediate coding features, and update the intermediate coding features in other sub-encoder units to obtain intermediate target coding features, and input the shallow features into the encoder unit to obtain intermediate target coding features; Decoder unit module: used to construct sub-decoder units and determine the number of the sub-decoder units, connect the sub-decoder units in series to form a decoder unit, and connect the encoder unit and the decoder unit in series; Target image module: used to update the intermediate target encoding features in the sub-decoder unit through the low-rank mixed attention mechanism and upsampling operation to obtain target decoding features, and fuse the target decoding features to obtain the target image.

[0011] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of any of the above-mentioned curvature-guided image fusion methods are implemented.

[0012] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the above-mentioned curvature-guided image fusion methods are implemented.

[0013] The present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can perform the steps of any of the curvature-guided image fusion methods described above.

[0014] The above one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects: The curvature-guided image fusion method, system, equipment, product and medium provided by the present invention effectively enhance the detail expression of the target image through the curvature space self-attention mechanism, the low-rank mixed attention mechanism and the curvature channel self-attention mechanism, so that the target image achieves the best balance between spatial texture clarity and spectral feature fidelity, and provides reliable data quality assurance for application scenarios such as precise classification of remote sensing objects and dynamic environmental monitoring.

[0015] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flow chart of the curvature-guided image fusion method provided by the present invention.

[0018] Figure 2 It is a comparison chart of experimental results of the curvature-guided image fusion method provided by the present invention.

[0019] Figure 3 It is a structural schematic diagram of the curvature-guided image fusion system provided by the present invention.

[0020] Figure 4 It is a structural schematic diagram of the curvature-guided image fusion device provided by the present invention.

[0021] Reference numerals: 100, shallow feature module; 200, encoder unit module; 300, intermediate encoding feature module; 400, decoder unit module; 500, target image module; 810, processor; 820, communication interface; 830, memory; 840, communication bus. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme in the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present invention. The following embodiments are used to illustrate the present invention, but cannot be used to limit the scope of the present invention.

[0023] In the description of the embodiments of the present invention, it should be noted that the terms “first”, “second” and “third” are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0024] In the description of the embodiments of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "connected" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the embodiments of the present invention can be understood according to specific circumstances.

[0025] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0026] Combine the following Figures 1 to 4 Describing embodiments of the present invention: Figure 1 The present invention provides a flow chart of the curvature-guided image fusion method, which first extracts features from the image to be processed to obtain shallow features, then constructs sub-encoder units and connects the sub-encoder units in series to form an encoder unit, then performs feature enhancement in the sub-encoder units to obtain intermediate encoding features and updates them to obtain intermediate target encoding features, then connects sub-decoder units in series to form a decoder unit, connects the decoder unit in series with the encoder unit, and finally updates the intermediate target encoding features in the sub-decoder units to obtain a target image.

[0027] The present invention provides an image fusion method based on curvature guidance, comprising: S1: Obtain an image to be processed, and extract features of the image to be processed through a residual block to obtain shallow features; Furthermore, the purpose of this stage is to extract features from the image to be processed, thereby obtaining shallow features to prepare for subsequent operations. Specifically, step S1 includes: S11: Acquire the image to be processed including a hyperspectral image and a panchromatic image, extract the panchromatic image through a residual block, and obtain panchromatic shallow features; S12: upsampling the hyperspectral image and extracting the hyperspectral image through the residual block to obtain hyperspectral shallow features, and using the panchromatic shallow features and the hyperspectral shallow features as the shallow features.

[0028] With respect to the above steps, the specific implementation methods in this embodiment are as follows: First, the image to be processed including the hyperspectral image LRHS and the panchromatic image PAN is obtained by shooting, and then the panchromatic image is extracted through the residual block to obtain the panchromatic shallow feature : Among them, RB() inputs the contents in the brackets into the residual block structure.

[0029] The hyperspectral image is then upsampled by 2 times and extracted through the residual block to obtain the hyperspectral shallow features. : Wherein, U() is upsampling the content in the brackets, and in this embodiment, the upsampling ratio is 2. The panchromatic shallow features and the hyperspectral shallow features are taken as shallow features.

[0030] S2: construct sub-encoder units and determine the number of the sub-encoder units, and connect the sub-encoder units in series to form an encoder unit; Furthermore, the purpose of this stage is to determine the number of sub-encoder units and connect the sub-encoder units in series to form an encoder unit. Specifically, in this stage, it is necessary to determine the total downsampling ratio in the encoder unit, so as to determine the number of sub-encoder units according to the downsampling ratio, thereby constructing the sub-encoder unit and connecting the sub-encoder units in series in sequence to form an encoder unit.

[0031] S3: In the sub-encoder unit, feature enhancement is performed through the curvature space self-attention mechanism, the low-rank mixed attention mechanism and the curvature channel self-attention mechanism to obtain intermediate coding features, and in other sub-encoder units, the intermediate coding features are updated to obtain intermediate target coding features, and the shallow features are input into the encoder unit to obtain intermediate target coding features; Furthermore, the purpose of this stage is to obtain intermediate coding features through the curvature space self-attention mechanism, the low-rank mixed attention mechanism and the curvature channel self-attention mechanism, and update the intermediate coding features. Specifically, in step S3, the curvature space self-attention mechanism and the curvature channel self-attention mechanism are implemented as follows: The curvature space self-attention mechanism and the curvature channel self-attention mechanism are implemented as follows: First, the first curvature parameter and the second curvature parameter are calculated, and the differential curvature is calculated according to the first curvature parameter and the second curvature parameter, and then the differential curvature is used to perform feature enhancement using the maximum pooling operation and the average pooling operation to obtain the curvature space feature and the curvature channel feature respectively.

[0032] In step S3, the low-rank mixed attention mechanism is implemented as follows: determine a low-rank mixed attention input value, perform non-negative matrix decomposition and convolution on the low-rank mixed attention input value, and obtain a full-color low-rank output and a high-spectral low-rank output, respectively.

[0033] With respect to the above steps, the specific implementation methods in this embodiment are as follows: After obtaining the shallow features, the features can be enhanced through the curvature space self-attention mechanism, the low-rank mixed attention mechanism and the curvature channel self-attention mechanism, and the full color coding features including the i-th stage can be obtained. and the hyperspectral encoding features of the i-th stage Encoding features: Among them, CSSM() is to operate the content in the brackets through the curvature space self-attention mechanism, and CCSM() is to operate the content in the brackets through the curvature channel self-attention mechanism. To obtain full-color low-rank output through low-rank mixed attention mechanism, In order to obtain high-spectral low-rank output through low-rank mixed attention mechanism, is the intermediate full-color encoding feature of the i-1th stage, is the intermediate hyperspectral coding feature of the i-1th stage, and when i is 1, the intermediate full-color coding feature of the i-1th stage is the full-color shallow feature, and the intermediate hyperspectral coding feature of the i-1th stage is the hyperspectral shallow feature.

[0034] In the curvature space self-attention mechanism and the curvature channel self-attention mechanism, the differential curvature C needs to be calculated first: in, is the first curvature parameter, is the second curvature parameter, which is calculated as follows: in, is the horizontal first-order derivative convolution kernel, is the vertical first-order derivative convolution kernel, is the horizontal second-order derivative convolution kernel, Here, the horizontal first-order derivative convolution kernel, the vertical first-order derivative convolution kernel, the horizontal second-order derivative convolution kernel, and the vertical second-order derivative convolution kernel are all preset known parameters.

[0035] After obtaining the differential curvature, the feature enhancement can be performed by using the maximum pooling operation and the average pooling operation through the differential curvature, so as to obtain the curvature space features of the i-th stage respectively. and the curvature channel characteristics of the i-th stage : Among them, Cur() is the attention weight of the channel and space calculated by differential curvature, Sig() is activated by the sigmoid function, MP() represents the maximum pooling operation on the content in the brackets, and AP() represents the average pooling operation on the content in the brackets. The curvature space feature is The output result of the curvature channel feature is The output result is:

[0036] The implementation method of the low-rank mixed attention mechanism is as follows: determine the low-rank mixed attention input value, where the low-rank mixed attention input value is the intermediate full-color encoding feature of the i-1th stage and the intermediate hyperspectral encoding feature of the i-1th stage. Then perform non-negative matrix decomposition and convolution on the low-rank mixed attention input value to obtain the full-color low-rank output of the i-th stage and the hyperspectral low-rank output of the i-th stage : Among them, Conv() is a 1×1 convolution operation, and MD() is a non-negative matrix decomposition of the content in the brackets. The panchromatic low-rank output of the i-th stage is the result of obtaining the panchromatic low-rank output through the low-rank mixed attention mechanism, and the hyperspectral low-rank output of the i-th stage is the result of obtaining the hyperspectral low-rank output through the low-rank mixed attention mechanism.

[0037] Finally, the panchromatic coding features of the i-th stage and the hyperspectral coding features of the i-th stage are downsampled respectively to obtain the intermediate panchromatic coding features including the i-th stage. and the intermediate hyperspectral encoding features of the i-th stage The intermediate encoding features of the i-th stage are: Among them, D() is a downsampling of the content in the brackets by a factor of 2. After obtaining the intermediate coding features of the i-th stage, it can be input into the next sub-encoder unit to increase its stage number by 1, and feature enhancement is performed through the curvature space self-attention mechanism, the low-rank mixed attention mechanism, and the curvature channel self-attention mechanism to update the intermediate coding features, and the intermediate coding features output by the last sub-encoder unit in the encoder unit are used as the intermediate target coding features. The shallow features are input into the first sub-encoder unit of the encoder unit, and the intermediate target coding features can be obtained through the encoder unit.

[0038] S4: construct sub-decoder units and determine the number of the sub-decoder units, connect the sub-decoder units in series to form a decoder unit, and connect the encoder unit and the decoder unit in series; Furthermore, the purpose of this stage is to determine the number of sub-decoder units to form a decoder unit, and to connect the encoder unit and the decoder unit in series. Specifically, in step S4, the number of the sub-decoder units is equal to the number of the sub-encoder units.

[0039] With respect to the above steps, the specific implementation methods in this embodiment are as follows: First, construct sub-decoder units, and make the number of sub-decoder units equal to the number of sub-encoder units, so as to determine the number of sub-decoder units. Connect the sub-decoder units in series to form a decoder unit, and connect the encoder unit and the decoder unit in series in sequence.

[0040] S5: In the sub-decoder unit, the intermediate target encoding features are updated through the low-rank mixed attention mechanism and upsampling operation to obtain target decoding features, and the target decoding features are fused to obtain a target image.

[0041] Furthermore, the purpose of this stage is to update the intermediate target coding features, obtain the target decoding features, and fuse the target decoding features to obtain the target image. Specifically, in step S5, the intermediate target coding features output by the last sub-decoder unit are used as the target decoding features, and the target decoding features are spliced, and feature fusion and channel adjustment are performed through the residual block to obtain the target image.

[0042] With respect to the above steps, the specific implementation methods in this embodiment are as follows: First, the intermediate target encoding feature is obtained. If the sub-decoder unit is the first sub-decoder unit in the decoder unit, the intermediate target encoding feature is the intermediate target encoding feature output by the last sub-decoder unit in the encoder unit, otherwise it is the intermediate target encoding feature output by the previous sub-decoder unit. Then the intermediate target encoding feature is updated through the low-rank mixed attention mechanism and upsampling operation to obtain the intermediate target full color encoding feature including the i-th stage and the hyperspectral encoding features of the intermediate target in the i-th stage The intermediate target encoding features of the i-th stage are: Among them, U() means upsampling the content in the brackets by a factor of 2. is the full color coding feature of the intermediate target in the i-1th stage, It is the hyperspectral encoding feature of the intermediate target in the i-1th stage.

[0043] is the full-color decoding feature of the i-th stage, where the full-color decoding feature is the intermediate full-color coding feature obtained by the sub-encoder unit, and the specific value is determined according to the position of the sub-decoder unit. For example, when the sub-decoder unit is the first sub-decoder unit in the decoder unit, the full-color decoding feature is the intermediate target full-color coding feature of the last sub-encoder unit in the encoder unit; when the sub-decoder unit is the second sub-decoder unit in the decoder unit, the full-color decoding feature is the intermediate full-color coding feature of the second to last sub-encoder unit in the encoder unit; when the sub-decoder unit is the last sub-decoder unit in the decoder unit, the full-color decoding feature is the intermediate full-color coding feature of the first sub-encoder unit in the encoder unit. Similarly, is the hyperspectral decoding feature of the i-th stage, and the hyperspectral decoding feature is the intermediate hyperspectral coding feature obtained by the sub-encoder unit. The specific value is determined according to the position of the sub-decoder unit. For example, when the sub-decoder unit is the first sub-decoder unit in the decoder unit, the hyperspectral decoding feature is the intermediate target hyperspectral coding feature of the last sub-encoder unit in the encoder unit. When the sub-decoder unit is the second sub-decoder unit in the decoder unit, the hyperspectral decoding feature is the intermediate hyperspectral coding feature of the second to last sub-encoder unit in the encoder unit. When the sub-decoder unit is the last sub-decoder unit in the decoder unit, the hyperspectral decoding feature is the intermediate hyperspectral coding feature of the first sub-encoder unit in the encoder unit. Similarly, in the decoder unit, each time a sub-decoder unit is entered, the number of stages increases by 1.

[0044] Finally, the intermediate target encoding features output by the last sub-decoder unit are used as target decoding features, which include the target full-color decoding features and target hyperspectral decoding features , the target decoded features are spliced, and feature fusion and channel adjustment are performed through the residual block to obtain the target image HRHS: Among them, Cat() means splicing the contents in the brackets.

[0045] In order to verify the effectiveness of the present invention, the decoding output results of the method provided by the present invention and other methods are compared on the public data set Pavia center. The comparison results are as follows: Figure 2 As shown in Figure 2, FPFNet, HyperRefiner, and DCTransformer are all existing methods. Figure 2 In the figure, the more blue parts there are, the higher the resolution of the HRHS image and the less distortion it has. It can be seen that the method provided by the present invention achieves the best resolution and the least distortion. The specific experimental results are shown in Table 1: Table 1 Performance comparison of HRHS images obtained by the method provided by the present invention and other methods

[0046] Among them, CC is the correlation coefficient, SAM is the spectral angle mapping, RMSE is the root mean square error, ERGAS is the dimensionless global relative error, PSNR is the peak signal-to-noise ratio, the upward arrow indicates that the higher the value, the better, the downward arrow indicates that the lower the value, the better, the underline indicates that the value of this method has achieved the second best performance in this project, and the bold indicates that the value of this method has achieved the best performance in this project. It can be seen that the method provided by the present invention has achieved the best performance in most values.

[0047] The image fusion device based on curvature guidance provided by the present invention is described below. The image fusion device based on curvature guidance described below and the image fusion method based on curvature guidance described above can be referred to each other.

[0048] Figure 3 The structural diagram of the curvature-guided image fusion system is shown in FIG. Figure 3 As shown, the method for performing the image fusion method based on curvature guidance as described above includes: Shallow feature module 100: used to obtain an image to be processed, and extract features of the image to be processed through a residual block to obtain shallow features; Encoder unit module 200: used to construct sub-encoder units and determine the number of the sub-encoder units, and connect the sub-encoder units in series to form an encoder unit; The intermediate coding feature module 300 is used to perform feature enhancement in the sub-encoder unit through the curvature space self-attention mechanism, the low-rank mixed attention mechanism and the curvature channel self-attention mechanism to obtain the intermediate coding feature, and in other sub-encoder units, update the intermediate coding feature to obtain the intermediate target coding feature, and input the shallow feature into the encoder unit to obtain the intermediate target coding feature; Decoder unit module 400: used to construct sub-decoder units and determine the number of the sub-decoder units, connect the sub-decoder units in series to form a decoder unit, and connect the encoder unit and the decoder unit in series; Target image module 500: used to update the intermediate target encoding features in the sub-decoder unit through the low-rank mixed attention mechanism and upsampling operation to obtain target decoding features, and fuse the target decoding features to obtain the target image.

[0049] on the other hand, Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the image fusion method based on curvature guidance, and the method includes: S1: Obtain an image to be processed, and extract features of the image to be processed through a residual block to obtain shallow features; S2: construct sub-encoder units and determine the number of the sub-encoder units, and connect the sub-encoder units in series to form an encoder unit; S3: In the sub-encoder unit, feature enhancement is performed through the curvature space self-attention mechanism, the low-rank mixed attention mechanism and the curvature channel self-attention mechanism to obtain intermediate coding features, and in other sub-encoder units, the intermediate coding features are updated to obtain intermediate target coding features, and the shallow features are input into the encoder unit to obtain intermediate target coding features; S4: construct sub-decoder units and determine the number of the sub-decoder units, connect the sub-decoder units in series to form a decoder unit, and connect the encoder unit and the decoder unit in series; S5: In the sub-decoder unit, the intermediate target encoding features are updated through the low-rank mixed attention mechanism and upsampling operation to obtain target decoding features, and the target decoding features are fused to obtain a target image.

[0050] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0051] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, when the program instructions are executed by a computer, the computer can execute the image fusion method based on curvature guidance provided by the above methods, the method comprising: S1: Obtain an image to be processed, and extract features of the image to be processed through a residual block to obtain shallow features; S2: construct sub-encoder units and determine the number of the sub-encoder units, and connect the sub-encoder units in series to form an encoder unit; S3: In the sub-encoder unit, feature enhancement is performed through the curvature space self-attention mechanism, the low-rank mixed attention mechanism and the curvature channel self-attention mechanism to obtain intermediate coding features, and in other sub-encoder units, the intermediate coding features are updated to obtain intermediate target coding features, and the shallow features are input into the encoder unit to obtain intermediate target coding features; S4: construct sub-decoder units and determine the number of the sub-decoder units, connect the sub-decoder units in series to form a decoder unit, and connect the encoder unit and the decoder unit in series; S5: In the sub-decoder unit, the intermediate target encoding features are updated through the low-rank mixed attention mechanism and upsampling operation to obtain target decoding features, and the target decoding features are fused to obtain a target image.

[0052] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the curvature-guided image fusion method provided by the above methods, the method comprising: S1: Obtain an image to be processed, and extract features of the image to be processed through a residual block to obtain shallow features; S2: construct sub-encoder units and determine the number of the sub-encoder units, and connect the sub-encoder units in series to form an encoder unit; S3: In the sub-encoder unit, feature enhancement is performed through the curvature space self-attention mechanism, the low-rank mixed attention mechanism and the curvature channel self-attention mechanism to obtain intermediate coding features, and in other sub-encoder units, the intermediate coding features are updated to obtain intermediate target coding features, and the shallow features are input into the encoder unit to obtain intermediate target coding features; S4: construct sub-decoder units and determine the number of the sub-decoder units, connect the sub-decoder units in series to form a decoder unit, and connect the encoder unit and the decoder unit in series; S5: In the sub-decoder unit, the intermediate target encoding features are updated through the low-rank mixed attention mechanism and upsampling operation to obtain target decoding features, and the target decoding features are fused to obtain a target image.

[0053] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0054] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0055] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. The image fusion method based on curvature guidance is characterized by: include: S1: Obtain an image to be processed, and extract features of the image to be processed through a residual block to obtain shallow features; S2: construct sub-encoder units and determine the number of the sub-encoder units, and connect the sub-encoder units in series to form an encoder unit; S3: In the sub-encoder unit, feature enhancement is performed through the curvature space self-attention mechanism, the low-rank mixed attention mechanism and the curvature channel self-attention mechanism to obtain intermediate coding features, and in other sub-encoder units, the intermediate coding features are updated to obtain intermediate target coding features, and the shallow features are input into the encoder unit to obtain intermediate target coding features; S4: construct sub-decoder units and determine the number of the sub-decoder units, connect the sub-decoder units in series to form a decoder unit, and connect the encoder unit and the decoder unit in series; S5: In the sub-decoder unit, the intermediate target encoding features are updated through the low-rank mixed attention mechanism and upsampling operation to obtain target decoding features, and the target decoding features are fused to obtain a target image.

2. The curvature-guided image fusion method according to claim 1, characterized in that: Step S1 specifically includes: S11: Acquire the image to be processed including a hyperspectral image and a panchromatic image, extract the panchromatic image through a residual block, and obtain panchromatic shallow features; S12: upsampling the hyperspectral image and extracting the hyperspectral image through the residual block to obtain hyperspectral shallow features, and using the panchromatic shallow features and the hyperspectral shallow features as the shallow features.

3. The curvature-guided image fusion method according to claim 1, characterized in that: In step S3, the curvature space self-attention mechanism and the curvature channel self-attention mechanism are implemented as follows: first, the first curvature parameter and the second curvature parameter are calculated, and the differential curvature is calculated based on the first curvature parameter and the second curvature parameter, and then the feature is enhanced by the differential curvature and the maximum pooling operation and the average pooling operation to obtain the curvature space feature and the curvature channel feature respectively.

4. The curvature-guided image fusion method according to claim 1, characterized in that: In step S3, the low-rank mixed attention mechanism is implemented as follows: determine a low-rank mixed attention input value, perform non-negative matrix decomposition and convolution on the low-rank mixed attention input value, and obtain a full-color low-rank output and a high-spectral low-rank output, respectively.

5. The curvature-guided image fusion method according to claim 1, characterized in that: In step S4, the number of the sub-decoder units is equal to the number of the sub-encoder units.

6. The curvature-guided image fusion method according to claim 1, characterized in that: In step S5, the intermediate target encoding features output by the last sub-decoder unit are used as target decoding features, and the target decoding features are spliced, and feature fusion and channel adjustment are performed through the residual block to obtain the target image.

7. A curvature-guided image fusion system, used to execute the curvature-guided image fusion method according to any one of claims 1 to 6, characterized in that: include: Shallow feature module: used to obtain the image to be processed, extract features of the image to be processed through the residual block, and obtain shallow features; Encoder unit module: used to construct sub-encoder units and determine the number of the sub-encoder units, and connect the sub-encoder units in series to form an encoder unit; Intermediate coding feature module: used to perform feature enhancement in the sub-encoder unit through the curvature space self-attention mechanism, the low-rank mixed attention mechanism and the curvature channel self-attention mechanism to obtain intermediate coding features, and update the intermediate coding features in other sub-encoder units to obtain intermediate target coding features, and input the shallow features into the encoder unit to obtain intermediate target coding features; Decoder unit module: used to construct sub-decoder units and determine the number of the sub-decoder units, connect the sub-decoder units in series to form a decoder unit, and connect the encoder unit and the decoder unit in series; Target image module: used to update the intermediate target encoding features in the sub-decoder unit through the low-rank mixed attention mechanism and upsampling operation to obtain target decoding features, and fuse the target decoding features to obtain the target image.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the curvature-guided image fusion method according to any one of claims 1 to 6 are implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the curvature-guided image fusion method according to any one of claims 1 to 6 are implemented.

10. A computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, characterized in that: When the program instructions are executed by a computer, the computer can perform the steps of the curvature-guided image fusion method according to any one of claims 1 to 6.

Images