Image fusion method, system and device based on uncertainty, product and medium

Abstract

The invention relates to the technical field of computer image processing, and provides an uncertainty-based image fusion method, system, device, product and medium, comprising: obtaining hyperspectral information and panchromatic information, performing convolution activation, and constructing a stage information extraction unit; group image information output is obtained, image information output is obtained, an information space interaction unit is constructed, and the information extraction unit and the information space interaction unit are connected to form an image feature fusion unit; information shielding is carried out, variance is calculated, cognitive uncertainty loss is obtained, and a cognitive uncertainty loss unit is constructed; a cognitive uncertainty loss unit is constructed, the adaptive uncertainty loss output by the image information is solved, a decoding output result is obtained, an adaptive uncertainty loss unit is constructed, and the cognitive uncertainty loss unit and the adaptive uncertainty loss unit are connected to form an uncertainty unit; and connecting the encoder unit with the decoder unit, and outputting the target image. According to the invention, a high-quality target image can be output.

Description

Technical Field

[0001] The present invention relates to the field of computer image processing technology, and in particular to an uncertainty-based image fusion method, system, device, product and medium. Background Art

[0002] HS (Hyperspectral) images are obtained by sampling hundreds of continuous narrow spectral bands using a spectral imaging system. They can provide rich spectral information and are often used to study the spectral difference characteristics of materials. However, due to the physical design limitations of the optical imaging system, the imaging results need to compromise between spectral resolution and spatial resolution. Single-spectral imaging systems cannot obtain HRHS (High Resolution Hyperspectral) images, and their imaging results often have low spatial resolution, which limits the application of HS images in mineral detection, ecosystem monitoring, agricultural detection and other fields. PAN (panchromatic) imaging systems only output single-band imaging PAN images, which have high spatial resolution but low spectral resolution. The hyperspectral panchromatic sharpening method aims to make full use of the spectral information of HS images and the spatial information of PAN images to generate HRHS images. Due to the high market demand for HRHS images, this technology is currently receiving widespread attention and has shown good performance in many remote sensing image downstream tasks such as environmental monitoring, target recognition and classification.

[0003] Traditional hyperspectral pan-sharpening methods can be further divided into component substitution-based methods, multi-resolution analysis-based methods, Bayesian-based methods, and model-based methods. Although traditional methods can improve the spatial resolution of HS images, the quality of the fused image will be seriously degraded due to inappropriate prior knowledge modeling, unknown sensor characteristics, mismatched prior assumptions, and the fact that the features manually constructed from the dictionary do not match reality. In addition, traditional convolutional neural network-based methods ignore the data structure attribute of hyperspectral images as sequence-based, resulting in information loss. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems existing in the related art. To this end, the present invention provides an image fusion method, system, device, product and medium based on uncertainty to achieve the output of high-quality HRHS images.

[0005] The present invention provides an image fusion method based on uncertainty, comprising: S1: In the stage information extraction unit, feature map information is obtained and grouped to obtain hyperspectral information and panchromatic information, and convolution activation is performed on the hyperspectral information and panchromatic information respectively to obtain hyperspectral stage information and panchromatic stage information; S2: In the information space interaction unit, the hyperspectral information and the panchromatic information are fused to obtain the group image information output, and the image information output is obtained through the group image information output, the hyperspectral stage information and the panchromatic stage information. The information extraction unit and the information space interaction unit are connected to form an image feature fusion unit; S3: In the cognitive uncertainty loss unit, the image information output is convolved to obtain the input features, the input features are information-shielded and the variance is calculated to obtain the cognitive uncertainty loss; S4: in the well-posed uncertainty unit, the well-posed uncertainty loss of the image information output is solved by the branch estimation decoder, the decoding output result is obtained by the cognitive uncertainty loss and the well-posed uncertainty loss, and the cognitive uncertainty loss unit and the well-posed uncertainty loss unit are connected; S5: An encoder unit is formed by an image feature fusion unit, a decoder unit is formed by the image feature fusion unit and an uncertainty unit, the encoder unit is connected to the decoder unit, the image to be parsed is input into the encoder unit, and a target image is output from the decoder unit.

[0006] According to the uncertainty-based image fusion method provided by the present invention, step S1 specifically includes: S11: constructing a stage information extraction unit, in which characteristic graph information is acquired, spectral channel dimensions of the characteristic graph information are extracted, and the characteristic graph information is grouped according to the spectral channel dimensions to obtain the hyperspectral information and the panchromatic information; S12: Acquire stage fusion state information, perform convolution activation on the hyperspectral information and the panchromatic information respectively, obtain hyperspectral information stage parameters and panchromatic information stage parameters, perform stage fusion on the stage fusion state information, the hyperspectral information stage parameters and the panchromatic information stage parameters respectively, obtain the hyperspectral stage information and the panchromatic stage information, thereby completing the construction of the stage information extraction unit.

[0007] According to the uncertainty-based image fusion method provided by the present invention, step S2 specifically includes: S21: constructing an information space interaction unit, in which the hyperspectral information and the panchromatic information are fused through the stage fusion state information to obtain group image information output; S22: performing convolution on the hyperspectral stage information and the panchromatic stage information to obtain convolution information, performing global correlation modeling on the convolution information to obtain intermediate convolution information, performing convolution on the intermediate convolution information to obtain iterative stage fusion state information, and iterating the stage fusion state information through the iterative stage fusion state information; S23: Obtain image information output according to the stage fusion state information after iteration and the group image information output, thereby completing the construction of the information space interaction unit, and connecting the information extraction unit and the information space interaction unit in series to obtain an image feature fusion unit.

[0008] According to the uncertainty-based image fusion method provided by the present invention, step S3 specifically includes: S31: constructing a cognitive uncertainty loss unit, in which the image information output is upsampled, and the number of channels of the upsampled image information output is adjusted by convolution to obtain input features; S32: Obtain loss image information by performing random information shielding on the input features, calculate the mean of the loss image information, and calculate the variance of the loss image information by the mean to obtain the cognitive uncertainty loss, thereby completing the construction of the cognitive uncertainty loss unit.

[0009] According to the uncertainty-based image fusion method provided by the present invention, in step S4, after obtaining the well-posed uncertainty loss, the spectral attention and the spatial attention are respectively calculated using the cognitive uncertainty loss, a cognitive correction value is calculated based on the spectral attention and the spatial attention, and the image information output is uncertainty-guided using the cognitive correction value and the well-posed uncertainty loss to obtain the decoding output result.

[0010] According to the uncertainty-based image fusion method provided by the present invention, in step S5, the number of the image feature fusion units is determined and the image feature fusion units are connected in series to obtain an encoder unit, the image feature fusion unit and the uncertainty unit are connected in sequence to obtain a decoder sub-unit, the number of decoder sub-units is determined according to the number of the image feature fusion units, and the decoder sub-units are connected in sequence to obtain a decoder unit.

[0011] The present invention also provides an uncertainty-based image fusion system, comprising: Information activation module: used for obtaining feature map information and grouping the feature map information in the stage information extraction unit to obtain hyperspectral information and panchromatic information, and performing convolution activation on the hyperspectral information and panchromatic information respectively to obtain hyperspectral stage information and panchromatic stage information; Information fusion module: used to fuse the hyperspectral information and the panchromatic information in the information space interaction unit to obtain the group image information output, obtain the image information output through the group image information output, the hyperspectral stage information and the panchromatic stage information, and connect the information extraction unit and the information space interaction unit to form an image feature fusion unit; Epistemic uncertainty module: used to convolve the image information output to obtain input features in the epistemic uncertainty loss unit, perform information shielding on the input features and calculate the variance to obtain epistemic uncertainty loss; A well-posed uncertainty module: included in the well-posed uncertainty unit, the well-posed uncertainty loss of the decoder for solving the image information output is estimated through branches, the decoding output result is obtained through the cognitive uncertainty loss and the well-posed uncertainty loss, and the cognitive uncertainty loss unit is connected with the well-posed uncertainty loss unit; Image processing module: used to form an encoder unit through an image feature fusion unit, form a decoder unit through the image feature fusion unit and the uncertainty unit, connect the encoder unit with the decoder unit, input the image to be analyzed into the encoder unit, and output the target image from the decoder unit.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of any of the above-mentioned uncertainty-based image fusion methods are implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the above-mentioned uncertainty-based image fusion methods are implemented.

[0014] The present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the steps of any of the uncertainty-based image fusion methods described above.

[0015] The above one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects: The uncertainty-based image fusion method, system, device, product and medium provided by the present invention process hyperspectral information and panchromatic information separately, and guide the uncertainty of image information output through cognitive uncertainty and well-posed uncertainty, so as to accurately generate each part of the HRHS image and obtain a HRHS image with higher resolution and less distortion.

[0016] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flowchart of the uncertainty-based image fusion method provided by the present invention.

[0019] Figure 2 It is a comparison chart of experimental results of the uncertainty-based image fusion method provided by the present invention.

[0020] Figure 3 It is a structural schematic diagram of the uncertainty-based image fusion system provided by the present invention.

[0021] Figure 4 It is a structural schematic diagram of the uncertainty-based image fusion device provided by the present invention.

[0022] Reference numerals: 100, information activation module; 200, information fusion module; 300, cognitive uncertainty module; 400, well-posed uncertainty module; 500, image processing module; 810, processor; 820, communication interface; 830, memory; 840, communication bus. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme in the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present invention. The following embodiments are used to illustrate the present invention, but cannot be used to limit the scope of the present invention.

[0024] In the description of the embodiments of the present invention, it should be noted that the terms “first”, “second” and “third” are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0025] In the description of the embodiments of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "connected" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the embodiments of the present invention can be understood according to specific circumstances.

[0026] In the embodiments of the present invention, unless otherwise clearly specified and limited, the first feature being "above" or "below" the second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, the first feature being "above", "above" and "above" the second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. The first feature being "below", "below" and "below" the second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0027] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0028] Combine the following Figures 1 to 4 Describing embodiments of the present invention: Figure 1 The flowchart of the uncertainty-based image fusion method provided by the present invention is as follows. First, the hyperspectral information and panchromatic information are obtained in the stage information extraction unit, and convolution activation is performed on them; then, the hyperspectral information and panchromatic information are fused to obtain the group image information output, and an image feature fusion unit is constructed; then, the cognitive uncertainty loss is obtained through the image information output in the cognitive uncertainty loss unit; then, the decoding output result is obtained through the cognitive uncertainty loss and the well-posed uncertainty loss, and an uncertainty unit is constructed; finally, the encoder unit is connected to the decoder unit, the image to be analyzed is input into the encoder unit, and the target image is output from the decoder unit.

[0029] The present invention provides an image fusion method based on uncertainty, comprising: S1: In the stage information extraction unit, feature map information is obtained and grouped to obtain hyperspectral information and panchromatic information, and convolution activation is performed on the hyperspectral information and panchromatic information respectively to obtain hyperspectral stage information and panchromatic stage information; Furthermore, the purpose of this stage is to group and convolute the feature map information to obtain hyperspectral stage information and panchromatic stage information to construct a stage information extraction unit. Specifically, step S1 specifically includes: S11: constructing a stage information extraction unit, in which characteristic graph information is acquired, spectral channel dimensions of the characteristic graph information are extracted, and the characteristic graph information is grouped according to the spectral channel dimensions to obtain the hyperspectral information and the panchromatic information; S12: Acquire stage fusion state information, perform convolution activation on the hyperspectral information and the panchromatic information respectively, obtain hyperspectral information stage parameters and panchromatic information stage parameters, perform stage fusion on the stage fusion state information, the hyperspectral information stage parameters and the panchromatic information stage parameters respectively, obtain the hyperspectral stage information and the panchromatic stage information, thereby completing the construction of the stage information extraction unit.

[0030] With respect to the above steps, the specific implementation methods in this embodiment are as follows: First, we need to build a stage information extraction unit. In the stage information extraction unit, we first need to obtain feature map information, which includes the HS feature map of the i-th stage. And the PAN feature map of the i-th stage , and when i is 0, the HS feature map and PAN feature map are the images to be analyzed, that is, the original input HS image and PAN image. Then, the spectral channel dimension of the feature map information, that is, the number of channels of the feature map information, is extracted from the feature map information. Then, the feature map information is grouped according to the spectral channel dimension to obtain the hyperspectral information group and the panchromatic information group: Wherein, slicing() is a grouping operation for the contents in the brackets, and T is the number of groups of hyperspectral information and panchromatic information obtained after grouping. is the tth group of hyperspectral information, is the tth group of full color information.

[0031] Then obtain the stage fusion status information of the t-1th group Here, when t is 1, the value of the stage fusion state information is a fixed value determined based on experience. Convolution activation is performed on the t-th group of hyperspectral information to obtain the stage parameters including the first hyperspectral information of the t-th group and the second hyperspectral information stage parameters of the tth group Hyperspectral information phase parameters: Among them, sigmoid() represents the sigmoid activation function, It means a 3×3 convolution operation is performed on the content in the brackets, and Tanh() represents the tanh activation function.

[0032] Similarly, the t-th group of full-color information is convoluted and activated to obtain the parameters of the first full-color information stage of the t-th group and the tth group of second full-color information stage parameters Full color information phase parameters: Finally, the stage fusion state information, hyperspectral information stage parameters and panchromatic information stage parameters can be used to perform stage fusion respectively to obtain the tth group of hyperspectral stage information. and the tth group of full-color stage information : in, To multiply pixel by pixel, the construction of the stage information extraction unit is completed.

[0033] S2: In the information space interaction unit, the hyperspectral information and the panchromatic information are fused to obtain the group image information output, and the image information output is obtained through the group image information output, the hyperspectral stage information and the panchromatic stage information. The information extraction unit and the information space interaction unit are connected to form an image feature fusion unit; Furthermore, the purpose of this stage is to perform stage fusion to output image information, thereby completing the construction of the information space interaction unit, and connecting the information extraction unit and the information space interaction unit to form an image feature fusion unit. Specifically, step S2 specifically includes: S21: constructing an information space interaction unit, in which the hyperspectral information and the panchromatic information are fused through the stage fusion state information to obtain group image information output; S22: performing convolution on the hyperspectral stage information and the panchromatic stage information to obtain convolution information, performing global correlation modeling on the convolution information to obtain intermediate convolution information, performing convolution on the intermediate convolution information to obtain iterative stage fusion state information, and iterating the stage fusion state information through the iterative stage fusion state information; S23: Obtain image information output according to the stage fusion state information after iteration and the group image information output, thereby completing the construction of the information space interaction unit, and connecting the information extraction unit and the information space interaction unit in series to obtain an image feature fusion unit.

[0034] With respect to the above steps, the specific implementation methods in this embodiment are as follows: First, an information interaction unit is constructed. In the information space interaction unit, the hyperspectral information and panchromatic information are fused through the stage fusion state information to obtain the group image information output of the tth group. : in, Reset parameters for group t.

[0035] Then the hyperspectral phase information and the panchromatic phase information are concatenated and convolved to obtain the tth group of concatenated convolution information : Among them, Cat() is to splice the contents in the brackets. Then the global correlation modeling is performed on the spliced ​​convolution information to obtain the intermediate spliced ​​convolution information. , and then convolve the intermediate concatenated convolution information to obtain the fusion state information of the tth group of iterations : The iterative stage fusion state information is the stage fusion state information when the next set of hyperspectral information and panchromatic information are fused in stages. Therefore, the iteration of the stage fusion state information can be completed by iterating the stage fusion state information, so that the image information output is obtained according to the iterated stage fusion state information. Then, steps S11 to S22 are repeated to obtain the group image information output of all hyperspectral information and panchromatic information, and then the image information output of the i-th stage can be obtained. : In this way, the construction of the information space interaction unit is completed. The information extraction unit and the information space interaction unit are connected in series to obtain the image feature fusion unit. When the image feature fusion unit is connected to other image feature fusion units, the image information output of the i-th stage obtained by the image feature fusion unit is downsampled by 2 times to obtain the HS feature map and PAN feature map of the i+1th stage and input them into the next image feature fusion unit.

[0036] S3: In the cognitive uncertainty loss unit, the image information output is convolved to obtain the input features, the input features are information-shielded and the variance is calculated to obtain the cognitive uncertainty loss; Furthermore, the purpose of this stage is to perform convolution, information shielding and other operations on the image information output to obtain cognitive uncertainty loss, thereby constructing a cognitive uncertainty loss unit. Specifically, step S3 specifically includes: S31: constructing a cognitive uncertainty loss unit, in which the image information output is upsampled, and the number of channels of the upsampled image information output is adjusted by convolution to obtain input features; S32: Obtain loss image information by performing random information shielding on the input features, calculate the mean of the loss image information, and calculate the variance of the loss image information by the mean to obtain the cognitive uncertainty loss, thereby completing the construction of the cognitive uncertainty loss unit.

[0037] With respect to the above steps, the specific implementation methods in this embodiment are as follows: First, a cognitive uncertainty loss unit is constructed. In the cognitive uncertainty loss unit, the image information output is first upsampled by 2 times, and a 1×1 convolution is performed on the upsampled image information output, thereby adjusting the number of channels of the upsampled image information output to be the same as the set predetermined value, thereby obtaining the input feature of the i-th stage . Then, the input features are sampled V times. In each sampling process, the input features are randomly masked, that is, 2% of the channels and 2% of the pixel positions of the input features are randomly masked to simulate cognitive uncertainty loss. The input features after random information masking are convolved and activated to obtain the loss image information group. in, is the image information output after upsampling, To perform a 1×1 convolution on the contents in the brackets, To sample and randomly mask the contents in brackets, is the vth loss image information.

[0038] Then calculate the mean of all lost image information, that is, the mean of the i-th stage : Among them, mean() is the mean of the content in the brackets. Finally, the variance of the lost image information is calculated based on the mean, and the cognitive uncertainty loss of the i-th stage can be obtained. : Among them, var() is the variance of the content in the brackets, and the mean value needs to be used in the process of calculating the variance. This completes the construction of the cognitive uncertainty loss unit.

[0039] S4: in the well-posed uncertainty unit, the well-posed uncertainty loss of the image information output is solved by the branch estimation decoder, the decoding output result is obtained by the cognitive uncertainty loss and the well-posed uncertainty loss, and the cognitive uncertainty loss unit and the well-posed uncertainty loss unit are connected; Furthermore, the purpose of this stage is to calculate the well-posed uncertainty loss, and obtain the decoding output result through the known uncertainty loss and the well-posed uncertainty loss, and finally form the uncertainty unit. Specifically, in step S4, after obtaining the well-posed uncertainty loss, the spectral attention and the spatial attention are respectively calculated through the cognitive uncertainty loss, and the cognitive correction value is calculated according to the spectral attention and the spatial attention. The image information output is guided by the cognitive correction value and the well-posed uncertainty loss to obtain the decoding output result.

[0040] With respect to the above steps, the specific implementation methods in this embodiment are as follows: First, the well-posed uncertainty loss of the i-th stage of the image information output is calculated by the branch decoder ; After calculating the well-posed uncertainty loss, the spectral attention of the i-th stage is calculated by the cognitive uncertainty loss and the spatial attention of the i-th stage : Among them, GMP() means the global maximum pooling operation is performed on the brackets, MLP() means the use of multi-layer perceptron function on the brackets, and GAP() means the global average pooling operation is performed on the brackets. It means to perform a 7×7 convolution on the content in the brackets.

[0041] Then, the cognitive correction value of the i-th stage can be calculated based on spectral attention and spatial attention: : Then, the uncertainty of the upsampled image information output can be guided by the cognitive correction value and the appropriately posed uncertainty loss to obtain the decoding output result of the i-th stage: : Among them, UP() means upsampling the content in the brackets by 2 times, and UGM() means guiding the uncertainty of the upsampled image information output through cognitive correction value and well-posed uncertainty loss. In this way, the construction of the well-posed uncertainty loss unit is completed, and the cognitive uncertainty loss unit and the well-posed uncertainty loss unit are connected in series to form an uncertainty unit.

[0042] S5: An encoder unit is formed by an image feature fusion unit, a decoder unit is formed by the image feature fusion unit and an uncertainty unit, the encoder unit is connected to the decoder unit, the image to be parsed is input into the encoder unit, and a target image is output from the decoder unit.

[0043] Furthermore, the purpose of this stage is to construct an encoder unit and a decoder unit, and connect the encoder unit to the decoder unit, so as to output a target image from the decoder unit. Specifically, in step S5, the number of the image feature fusion units is determined and the image feature fusion units are connected in series to obtain an encoder unit, the image feature fusion unit is connected in sequence with the uncertainty unit to obtain a decoder sub-unit, the number of decoder sub-units is determined according to the number of the image feature fusion units, and the decoder sub-units are connected in sequence to obtain a decoder unit.

[0044] With respect to the above steps, the specific implementation of this embodiment is as follows: First, the number of image feature fusion units is determined, and then these image feature fusion units are connected in series to obtain an encoder unit. Then, the image feature fusion unit and the uncertainty unit are connected in sequence to obtain a decoder subunit. Then, based on the number of image feature fusion units in the encoder unit, the number of image feature fusion units is set to z, and the number of decoder subunits is z+1, thereby determining the number of decoder subunits, and connecting the decoder subunits in sequence to obtain a decoder unit. Then, the encoder unit is connected in sequence with the decoder unit.

[0045] The image to be parsed is input into the first image feature fusion unit of the encoder unit, and after passing through the encoder unit and the decoder unit in sequence, the decoding output result output by the last decoder subunit in the decoder unit can be used as the target image. The image to be parsed includes HS pictures and PAN pictures, and its stage will increase by 1 each time it passes through an image feature fusion unit or uncertainty unit. When the decoder unit includes multiple decoder subunits, in the decoder subunit before the last decoder subunit, the decoding output result of the uncertainty unit is upsampled by 2 times, and then can be input into the image feature fusion unit of the next decoder subunit as the feature map information of the image feature fusion unit of the next decoder subunit.

[0046] In order to verify the effectiveness of the present invention, the decoding output results of the method provided by the present invention and other methods are compared on the public data set Pavia center. The comparison results are as follows: Figure 2 As shown in Figure 2, FPFNet, Refiner, TreeNet and DFCFN are all existing methods. Figure 2 In the figure, the more blue parts there are, the higher the resolution of the HRHS image and the less distortion it has. It can be seen that the method provided by the present invention achieves the best resolution and the least distortion. The specific experimental results are shown in Table 1: Table 1 Performance comparison of HRHS images obtained by the method provided by the present invention and other methods

[0047] Among them, CC is the correlation coefficient, SAM is the spectral angle mapping, RMSE is the root mean square error, ERGAS is the dimensionless global relative error, PSNR is the peak signal-to-noise ratio, and an upward arrow indicates that the higher the value, the better, and a downward arrow indicates that the lower the value, the better. It can be seen that the method provided by the present invention has achieved the best performance in all values.

[0048] The image fusion device based on uncertainty provided by the present invention is described below. The image fusion device based on uncertainty described below and the image fusion method based on uncertainty described above can be referred to each other.

[0049] Figure 3 The structural diagram of the uncertainty-based image fusion system is shown in the following example. Figure 3 As shown, the method for performing the uncertainty-based image fusion method as described above includes: The information activation module 100 is used to obtain the feature map information and group the feature map information in the stage information extraction unit to obtain the hyperspectral information and the panchromatic information, and perform convolution activation on the hyperspectral information and the panchromatic information respectively to obtain the hyperspectral stage information and the panchromatic stage information; Information fusion module 200: used to fuse the hyperspectral information and the panchromatic information in the information space interaction unit to obtain the group image information output, obtain the image information output through the group image information output, the hyperspectral stage information and the panchromatic stage information, and connect the information extraction unit and the information space interaction unit to form an image feature fusion unit; The cognitive uncertainty module 300 is used to convolve the image information output to obtain input features in the cognitive uncertainty loss unit, perform information shielding on the input features and calculate the variance to obtain cognitive uncertainty loss; The well-posed uncertainty module 400 is included in the well-posed uncertainty unit, estimates the well-posed uncertainty loss of the decoder to solve the image information output through the branch, obtains the decoding output result through the cognitive uncertainty loss and the well-posed uncertainty loss, and connects the cognitive uncertainty loss unit and the well-posed uncertainty loss unit; Image processing module 500: used to form an encoder unit through an image feature fusion unit, form a decoder unit through the image feature fusion unit and an uncertainty unit, connect the encoder unit with the decoder unit, input the image to be analyzed into the encoder unit and output the target image from the decoder unit.

[0050] on the other hand, Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the image fusion method based on uncertainty, and the method includes: S1: In the stage information extraction unit, feature map information is obtained and grouped to obtain hyperspectral information and panchromatic information, and convolution activation is performed on the hyperspectral information and panchromatic information respectively to obtain hyperspectral stage information and panchromatic stage information; S2: In the information space interaction unit, the hyperspectral information and the panchromatic information are fused to obtain the group image information output, and the image information output is obtained through the group image information output, the hyperspectral stage information and the panchromatic stage information. The information extraction unit and the information space interaction unit are connected to form an image feature fusion unit; S3: In the cognitive uncertainty loss unit, the image information output is convolved to obtain the input features, the input features are information-shielded and the variance is calculated to obtain the cognitive uncertainty loss; S4: in the well-posed uncertainty unit, the well-posed uncertainty loss of the image information output is solved by the branch estimation decoder, the decoding output result is obtained by the cognitive uncertainty loss and the well-posed uncertainty loss, and the cognitive uncertainty loss unit and the well-posed uncertainty loss unit are connected; S5: An encoder unit is formed by an image feature fusion unit, a decoder unit is formed by the image feature fusion unit and an uncertainty unit, the encoder unit is connected to the decoder unit, the image to be parsed is input into the encoder unit, and a target image is output from the decoder unit.

[0051] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0052] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, when the program instructions are executed by a computer, the computer can execute the uncertainty-based image fusion method provided by the above methods, the method comprising: S1: In the stage information extraction unit, feature map information is obtained and grouped to obtain hyperspectral information and panchromatic information, and convolution activation is performed on the hyperspectral information and panchromatic information respectively to obtain hyperspectral stage information and panchromatic stage information; S2: In the information space interaction unit, the hyperspectral information and the panchromatic information are fused to obtain the group image information output, and the image information output is obtained through the group image information output, the hyperspectral stage information and the panchromatic stage information. The information extraction unit and the information space interaction unit are connected to form an image feature fusion unit; S3: In the cognitive uncertainty loss unit, the image information output is convolved to obtain the input features, the input features are information-shielded and the variance is calculated to obtain the cognitive uncertainty loss; S4: in the well-posed uncertainty unit, the well-posed uncertainty loss of the image information output is solved by the branch estimation decoder, the decoding output result is obtained by the cognitive uncertainty loss and the well-posed uncertainty loss, and the cognitive uncertainty loss unit and the well-posed uncertainty loss unit are connected; S5: An encoder unit is formed by an image feature fusion unit, a decoder unit is formed by the image feature fusion unit and an uncertainty unit, the encoder unit is connected to the decoder unit, the image to be parsed is input into the encoder unit, and a target image is output from the decoder unit.

[0053] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the image fusion method based on uncertainty provided by the above methods is implemented, and the method includes: S1: In the stage information extraction unit, feature map information is obtained and grouped to obtain hyperspectral information and panchromatic information, and convolution activation is performed on the hyperspectral information and panchromatic information respectively to obtain hyperspectral stage information and panchromatic stage information; S2: In the information space interaction unit, the hyperspectral information and the panchromatic information are fused to obtain the group image information output, and the image information output is obtained through the group image information output, the hyperspectral stage information and the panchromatic stage information. The information extraction unit and the information space interaction unit are connected to form an image feature fusion unit; S3: In the cognitive uncertainty loss unit, the image information output is convolved to obtain the input features, the input features are information-shielded and the variance is calculated to obtain the cognitive uncertainty loss; S4: in the well-posed uncertainty unit, the well-posed uncertainty loss of the image information output is solved by the branch estimation decoder, the decoding output result is obtained by the cognitive uncertainty loss and the well-posed uncertainty loss, and the cognitive uncertainty loss unit and the well-posed uncertainty loss unit are connected; S5: An encoder unit is formed by an image feature fusion unit, a decoder unit is formed by the image feature fusion unit and an uncertainty unit, the encoder unit is connected to the decoder unit, the image to be parsed is input into the encoder unit, and a target image is output from the decoder unit.

[0054] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0055] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0056] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. The image fusion method based on uncertainty is characterized by: include: S1: In the stage information extraction unit, feature map information is obtained and grouped to obtain hyperspectral information and panchromatic information, and convolution activation is performed on the hyperspectral information and panchromatic information respectively to obtain hyperspectral stage information and panchromatic stage information; S2: In the information space interaction unit, the hyperspectral information and the panchromatic information are fused to obtain the group image information output, and the image information output is obtained through the group image information output, the hyperspectral stage information and the panchromatic stage information. The information extraction unit and the information space interaction unit are connected to form an image feature fusion unit; S3: In the cognitive uncertainty loss unit, the image information output is convolved to obtain the input features, the input features are information-shielded and the variance is calculated to obtain the cognitive uncertainty loss; S4: in the well-posed uncertainty unit, the well-posed uncertainty loss of the image information output is solved by the branch estimation decoder, the decoding output result is obtained by the cognitive uncertainty loss and the well-posed uncertainty loss, and the cognitive uncertainty loss unit and the well-posed uncertainty loss unit are connected; S5: An encoder unit is formed by an image feature fusion unit, a decoder unit is formed by the image feature fusion unit and an uncertainty unit, the encoder unit is connected to the decoder unit, the image to be parsed is input into the encoder unit, and a target image is output from the decoder unit.

2. The image fusion method based on uncertainty according to claim 1, characterized in that: Step S1 specifically includes: S11: constructing a stage information extraction unit, in which characteristic graph information is acquired, spectral channel dimensions of the characteristic graph information are extracted, and the characteristic graph information is grouped according to the spectral channel dimensions to obtain the hyperspectral information and the panchromatic information; S12: Acquire stage fusion state information, perform convolution activation on the hyperspectral information and the panchromatic information respectively, obtain hyperspectral information stage parameters and panchromatic information stage parameters, perform stage fusion on the stage fusion state information, the hyperspectral information stage parameters and the panchromatic information stage parameters respectively, obtain the hyperspectral stage information and the panchromatic stage information, thereby completing the construction of the stage information extraction unit.

3. The image fusion method based on uncertainty according to claim 1, characterized in that: Step S2 specifically includes: S21: constructing an information space interaction unit, in which the hyperspectral information and the panchromatic information are fused through the stage fusion state information to obtain group image information output; S22: performing convolution on the hyperspectral stage information and the panchromatic stage information to obtain convolution information, performing global correlation modeling on the convolution information to obtain intermediate convolution information, performing convolution on the intermediate convolution information to obtain iterative stage fusion state information, and iterating the stage fusion state information through the iterative stage fusion state information; S23: Obtain image information output according to the stage fusion state information after iteration and the group image information output, thereby completing the construction of the information space interaction unit, and connecting the information extraction unit and the information space interaction unit in series to obtain an image feature fusion unit.

4. The image fusion method based on uncertainty according to claim 1, characterized in that: Step S3 specifically includes: S31: constructing a cognitive uncertainty loss unit, in which the image information output is upsampled, and the number of channels of the upsampled image information output is adjusted by convolution to obtain input features; S32: Obtain loss image information by performing random information shielding on the input features, calculate the mean of the loss image information, and calculate the variance of the loss image information by the mean to obtain the cognitive uncertainty loss, thereby completing the construction of the cognitive uncertainty loss unit.

5. The image fusion method based on uncertainty according to claim 1, characterized in that: In step S4, after obtaining the well-posed uncertainty loss, the spectral attention and the spatial attention are respectively calculated using the cognitive uncertainty loss, a cognitive correction value is calculated based on the spectral attention and the spatial attention, and the image information output is uncertainty guided using the cognitive correction value and the well-posed uncertainty loss to obtain the decoding output result.

6. The image fusion method based on uncertainty according to claim 1, characterized in that: In step S5, the number of the image feature fusion units is determined and the image feature fusion units are connected in series to obtain an encoder unit, the image feature fusion unit and the uncertainty unit are connected in sequence to obtain a decoder sub-unit, the number of decoder sub-units is determined according to the number of the image feature fusion units, and the decoder sub-units are connected in sequence to obtain a decoder unit.

7. An uncertainty-based image fusion system, used to execute the uncertainty-based image fusion method according to any one of claims 1 to 6, characterized in that: include: Information activation module: used for obtaining feature map information and grouping the feature map information in the stage information extraction unit to obtain hyperspectral information and panchromatic information, and performing convolution activation on the hyperspectral information and panchromatic information respectively to obtain hyperspectral stage information and panchromatic stage information; Information fusion module: used to fuse the hyperspectral information and the panchromatic information in the information space interaction unit to obtain the group image information output, obtain the image information output through the group image information output, the hyperspectral stage information and the panchromatic stage information, and connect the information extraction unit and the information space interaction unit to form an image feature fusion unit; Epistemic uncertainty module: used to convolve the image information output to obtain input features in the epistemic uncertainty loss unit, perform information shielding on the input features and calculate the variance to obtain epistemic uncertainty loss; A well-posed uncertainty module: included in the well-posed uncertainty unit, the well-posed uncertainty loss of the decoder for solving the image information output is estimated through branches, the decoding output result is obtained through the cognitive uncertainty loss and the well-posed uncertainty loss, and the cognitive uncertainty loss unit is connected with the well-posed uncertainty loss unit; Image processing module: used to form an encoder unit through an image feature fusion unit, form a decoder unit through the image feature fusion unit and the uncertainty unit, connect the encoder unit with the decoder unit, input the image to be analyzed into the encoder unit, and output the target image from the decoder unit.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the uncertainty-based image fusion method according to any one of claims 1 to 6 are implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the uncertainty-based image fusion method according to any one of claims 1 to 6 are implemented.

10. A computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, characterized in that: When the program instructions are executed by a computer, the computer can execute the steps of the uncertainty-based image fusion method according to any one of claims 1 to 6.

Images