Uncertainty-Based Image Fusion Method, System, Device, Product and Medium

Abstract

The present invention relates to the field of computer image processing technology, and provides an image fusion method, system, device, product and medium based on uncertainty, including obtaining hyperspectral information and panchromatic information, performing convolution activation, and constructing a stage information extraction unit; obtaining group image information output, obtaining image information output, constructing an information space interaction unit, connecting the information extraction unit and the information space interaction unit to form an image feature fusion unit; performing information masking and calculating variance to obtain an epistemic uncertainty loss, and constructing an epistemic uncertainty loss unit; solving the well-posed uncertainty loss of the image information output to obtain a decoded output result, constructing a well-posed uncertainty loss unit, connecting the epistemic uncertainty loss unit and the well-posed uncertainty loss unit to form an uncertainty unit; connecting the encoder unit and the decoder unit to output a target image. The present invention can output a high-quality target image.

Description

Technical Field

[0001] The present invention relates to the technical field of computer image processing, and particularly to an image fusion method, system, device, product and medium based on uncertainty. Background Art

[0002] HS (Hyperspectral) images are obtained by sampling hundreds of continuous narrow spectral bands using a spectral imaging system, and can provide rich spectral information, which is often used for the research of spectral difference characteristics of materials. However, due to the physical design limitations of the optical imaging system, the imaging results need to be considered as a trade-off between spectral resolution and spatial resolution. A single spectral imaging system cannot obtain HRHS (High Resolution Hyperspectral) images, and its imaging results often have low spatial resolution, which limits the application of HS images in the fields of mineral detection, ecosystem monitoring, agricultural detection, etc. The PAN (panchromatic) imaging system outputs only a single-band imaging PAN image, which has high spatial resolution but low spectral resolution. The hyperspectral panchromatic sharpening method aims to make full use of the spectral information of HS images and the spatial information of PAN images to generate HRHS images. Due to the high demand for HRHS images in the market, this technology has received extensive attention at present, and it shows good performance in many downstream tasks of remote sensing images such as environmental monitoring, target recognition and classification.

[0003] Traditional hyperspectral panchromatic sharpening methods can be further divided into methods based on component substitution, methods based on multi-resolution analysis, methods based on Bayesian, and methods based on models. Although traditional methods can improve the spatial resolution of HS images, due to inappropriate prior knowledge modeling, unknown sensor characteristics, mismatched prior assumptions, and the fact that the features manually constructed from the dictionary do not match the reality, the quality of the fused images will seriously decline, and traditional convolutional neural network-based methods ignore the sequence-based data structure attributes of hyperspectral images, resulting in information loss. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the related art. For this purpose, the present invention provides an image fusion method, system, device, product and medium based on uncertainty, so as to achieve the output of high-quality HRHS images.

[0005] The present invention provides an image fusion method based on uncertainty, including:

[0006] S1: In the stage information extraction unit, obtain the feature map information and group the feature map information to obtain hyperspectral information and panchromatic information. Perform convolution activation on the hyperspectral information and the panchromatic information respectively to obtain hyperspectral stage information and panchromatic stage information;

[0007] S2: In the information space interaction unit, fuse the hyperspectral information and the panchromatic information to obtain the group image information output. Obtain the image information output through the group image information output, the hyperspectral stage information, and the panchromatic stage information. Connect the information extraction unit and the information space interaction unit to form an image feature fusion unit;

[0008] S3: In the epistemic uncertainty loss unit, perform convolution on the image information output to obtain the input features. Mask the input features and calculate the variance to obtain the epistemic uncertainty loss;

[0009] S4: In the aleatoric uncertainty unit, solve the aleatoric uncertainty loss of the image information output through the branch estimation decoder. Obtain the decoded output result through the epistemic uncertainty loss and the aleatoric uncertainty loss. Connect the epistemic uncertainty loss unit and the aleatoric uncertainty loss unit;

[0010] S5: Construct an encoder unit through the image feature fusion unit, construct a decoder unit through the image feature fusion unit and the uncertainty unit. Connect the encoder unit and the decoder unit. Input the image to be parsed into the encoder unit and output the target image from the decoder unit.

[0011] According to the uncertainty-based image fusion method provided by the present invention, step S1 specifically includes:

[0012] S11: Construct a stage information extraction unit. In the stage information extraction unit, obtain the feature map information, extract the spectral channel dimension of the feature map information, and group the feature map information according to the spectral channel dimension to obtain the hyperspectral information and the panchromatic information;

[0013] S12: Obtain the stage fusion state information. Perform convolution activation on the hyperspectral information and the panchromatic information respectively to obtain hyperspectral information stage parameters and panchromatic information stage parameters. Perform stage fusion through the stage fusion state information, the hyperspectral information stage parameters, and the panchromatic information stage parameters respectively to obtain the hyperspectral stage information and the panchromatic stage information, thereby completing the construction of the stage information extraction unit.

[0014] According to the uncertainty-based image fusion method provided by the present invention, step S2 specifically includes:

[0015] S21: Construct an information space interaction unit. In the information space interaction unit, fuse the hyperspectral information and the panchromatic information through the stage fusion state information to obtain and output group image information;

[0016] S22: Perform splicing convolution on the hyperspectral stage information and the panchromatic stage information to obtain splicing convolution information. Perform global correlation modeling on the splicing convolution information to obtain intermediate splicing convolution information. Perform convolution on the intermediate splicing convolution information to obtain iterative stage fusion state information, and iterate the stage fusion state information through the iterative stage fusion state information;

[0017] S23: Obtain and output image information according to the iterated stage fusion state information and the group image information output, thereby completing the construction of the information space interaction unit. Connect the information extraction unit and the information space interaction unit in series to obtain an image feature fusion unit.

[0018] According to the uncertainty-based image fusion method provided by the present invention, step S3 specifically includes:

[0019] S31: Construct a cognitive uncertainty loss unit. In the cognitive uncertainty loss unit, perform upsampling on the image information output, and adjust the number of channels of the upsampled image information output through convolution to obtain input features;

[0020] S32: Obtain loss image information by randomly masking information of the input features, calculate the mean value of the loss image information, and calculate the variance of the loss image information through the mean value to obtain the cognitive uncertainty loss, thereby completing the construction of the cognitive uncertainty loss unit.

[0021] According to the uncertainty-based image fusion method provided by the present invention, in step S4, after obtaining the well-posed uncertainty loss, calculate spectral attention and spatial attention respectively through the cognitive uncertainty loss, calculate a cognitive correction value according to the spectral attention and the spatial attention, and perform uncertainty guidance on the image information output through the cognitive correction value and the well-posed uncertainty loss to obtain the decoded output result.

[0022] According to the uncertainty-based image fusion method provided by the present invention, in step S5, determine the number of the image feature fusion units and connect the image feature fusion units in series to obtain an encoder unit. Connect the image feature fusion unit and the uncertainty unit in sequence to obtain a decoder subunit. Determine the number of decoder subunits according to the number of the image feature fusion units, and connect the decoder subunits in sequence to obtain a decoder unit.

[0023] The present invention also provides an image fusion system based on uncertainty, including:

[0024] An information activation module: used to obtain feature map information in the stage information extraction unit, group the feature map information to obtain hyperspectral information and panchromatic information, and perform convolutional activation on the hyperspectral information and the panchromatic information respectively to obtain hyperspectral stage information and panchromatic stage information;

[0025] An information fusion module: used to fuse the hyperspectral information and the panchromatic information in the information space interaction unit to obtain group image information output, obtain image information output through the group image information output, the hyperspectral stage information and the panchromatic stage information, and connect the information extraction unit and the information space interaction unit to form an image feature fusion unit;

[0026] A cognitive uncertainty module: used to perform convolution on the image information output in the cognitive uncertainty loss unit to obtain input features, perform information masking on the input features and calculate the variance to obtain the cognitive uncertainty loss;

[0027] A well-posed uncertainty module: including in the well-posed uncertainty unit, resolving the well-posed uncertainty loss of the image information output through a branch estimation decoder, obtaining a decoded output result through the cognitive uncertainty loss and the well-posed uncertainty loss, and connecting the cognitive uncertainty loss unit and the well-posed uncertainty loss unit;

[0028] An image processing module: used to form an encoder unit through the image feature fusion unit, form a decoder unit through the image feature fusion unit and the uncertainty unit, connect the encoder unit and the decoder unit, input the image to be analyzed into the encoder unit and output the target image from the decoder unit.

[0029] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the steps of the image fusion method based on uncertainty as described in any one of the above are implemented.

[0030] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the image fusion method based on uncertainty as described in any one of the above are implemented.

[0031] The present invention also provides a computer program product. The computer program product includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the steps of the image fusion method based on uncertainty as described in any one of the above.

[0032] One or more of the above technical solutions in the embodiments of the present invention have at least one of the following technical effects:

[0033] The method, system, device, product and medium for image fusion based on uncertainty provided by the present invention process hyperspectral information and panchromatic information respectively, and guide the uncertainty of the image information output through cognitive uncertainty and well-posed uncertainty, so as to accurately generate each part in the HRHS image and obtain an HRHS image with higher resolution and less distortion.

[0034] Additional aspects and advantages of the present invention will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0036] Figure 1 is a schematic flowchart of the method for image fusion based on uncertainty provided by the present invention.

[0037] Figure 2 is a comparison diagram of experimental results of the method for image fusion based on uncertainty provided by the present invention.

[0038] Figure 3 is a schematic structural diagram of the system for image fusion based on uncertainty provided by the present invention.

[0039] Figure 4 is a schematic structural diagram of the device for image fusion based on uncertainty provided by the present invention.

[0040] Reference numerals:

[0041] 100, information activation module; 200, information fusion module; 300, cognitive uncertainty module; 400, well-posed uncertainty module; 500, image processing module; 810, processor; 820, communication interface; 830, memory; 840, communication bus. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0042] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention. The following embodiments are used to illustrate the present invention, but shall not be used to limit the scope of the present invention.

[0043] In the description of the embodiments of the present invention, it should be noted that the terms "first", "second", and "third" are only used for descriptive purposes and shall not be construed as indicating or implying relative importance.

[0044] In the description of the embodiments of the present invention, it should be noted that unless otherwise clearly specified and limited, the terms "connected" and "coupled" should be understood in a broad sense. For example, it may be a fixed connection, a detachable connection, or an integral connection; it may be a mechanical connection or an electrical connection; it may be directly connected or indirectly connected through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms in the embodiments of the present invention can be understood according to specific circumstances.

[0045] In the embodiments of the present invention, unless otherwise clearly specified and limited, the first feature being "on" or "under" the second feature may be that the first and second features are in direct contact, or the first and second features are indirectly in contact through an intermediate medium. Moreover, the first feature being "above", "over", and "on top of" the second feature may be that the first feature is directly above or obliquely above the second feature, or merely indicates that the first feature has a higher horizontal height than the second feature. The first feature being "under", "below", and "beneath" the second feature may be that the first feature is directly below or obliquely below the second feature, or merely indicates that the first feature has a lower horizontal height than the second feature.

[0046] In the description of this specification, the description with reference to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the embodiments of the present invention. In this specification, the schematic descriptions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. In addition, without contradiction, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0047] The following is combined with Figures 1 to 4 to describe the implementation scheme of the present invention:

[0048] Figure 1 It is a schematic flowchart of the image fusion method based on uncertainty provided by the present invention. First, hyperspectral information and panchromatic information are obtained in the stage information extraction unit and subjected to convolution activation; then the hyperspectral information and the panchromatic information are fused to obtain the group image information output, and an image feature fusion unit is constructed; subsequently, the cognitive uncertainty loss is obtained through the image information output in the cognitive uncertainty loss unit; then the decoding output result is obtained through the cognitive uncertainty loss and the well-posed uncertainty loss, and an uncertainty unit is constructed; finally, the encoder unit is connected to the decoder unit, the image to be analyzed is input into the encoder unit and the target image is output from the decoder unit.

[0049] The present invention provides an image fusion method based on uncertainty, including:

[0050] S1: In the stage information extraction unit, the feature map information is obtained and grouped to obtain hyperspectral information and panchromatic information, and the hyperspectral information and the panchromatic information are respectively subjected to convolution activation to obtain hyperspectral stage information and panchromatic stage information;

[0051] Furthermore, the purpose of this stage is to group and perform convolution activation on the feature map information, so as to obtain hyperspectral stage information and panchromatic stage information, in order to construct the stage information extraction unit. Specifically, step S1 specifically includes:

[0052] S11: Construct the stage information extraction unit. In the stage information extraction unit, the feature map information is obtained, the spectral channel dimension of the feature map information is extracted, and the feature map information is grouped according to the spectral channel dimension to obtain the hyperspectral information and the panchromatic information;

[0053] S12: Obtain the stage fusion state information, respectively perform convolution activation on the hyperspectral information and the panchromatic information to obtain hyperspectral information stage parameters and panchromatic information stage parameters, and perform stage fusion through the stage fusion state information, the hyperspectral information stage parameters and the panchromatic information stage parameters respectively to obtain the hyperspectral stage information and the panchromatic stage information, thereby completing the construction of the stage information extraction unit.

[0054] For the above steps, the specific implementation manner in this embodiment is as follows:

[0055] First, it is necessary to construct the stage information extraction unit. In the stage information extraction unit, first, it is necessary to obtain the feature map information, and the feature map information includes the HS feature map of the i-th stage and the PAN feature map of the i-th stage , and when i is 0, the HS feature map and the PAN feature map are the images to be parsed, that is, the original input HS image and PAN image. Subsequently, the spectral channel dimension of the feature map information is extracted from the feature map information, that is, the number of channels of the feature map information. Then, the feature map information is grouped according to the spectral channel dimension to obtain the hyperspectral information group and the panchromatic information group:

[0056]

[0057]

[0058] Among them, slicing() is an operation to group the content in the parentheses, T is the number of groups of hyperspectral information and panchromatic information obtained after grouping, is the hyperspectral information of the t-th group, is the panchromatic information of the t-th group.

[0059] Subsequently, the stage fusion state information of the (t - 1)-th group is obtained . Here, when t is 1, the value of the stage fusion state information is a fixed value determined according to experience. The t-th group of hyperspectral information is convolved and activated to obtain the hyperspectral information stage parameters including the first hyperspectral information stage parameter and the second hyperspectral information stage parameter of the t-th group:

[0060]

[0061]

[0062] Among them, sigmoid() represents the sigmoid activation function, represents a 3×3 convolution operation on the content in the parentheses, and Tanh() represents the tanh activation function.

[0063] Similarly, the t-th group of panchromatic information is convolved and activated to obtain the panchromatic information stage parameters including the first panchromatic information stage parameter and the second panchromatic information stage parameter of the t-th group:

[0064]

[0065]

[0066] Finally, the stage fusion can be performed respectively through the stage fusion state information, the hyperspectral information stage parameters, and the panchromatic information stage parameters to obtain the t-th group of hyperspectral stage information and the t-th group of panchromatic stage information :

[0067]

[0068]

[0069] Among them, is pixel-by-pixel multiplication, thus completing the construction of the stage information extraction unit.

[0070] S2: In the information space interaction unit, fuse the hyperspectral information and the panchromatic information to obtain the group image information output. Obtain the image information output through the group image information output, the hyperspectral stage information, and the panchromatic stage information. Connect the information extraction unit and the information space interaction unit to form the image feature fusion unit;

[0071] Furthermore, the purpose of this stage is to perform stage fusion to obtain the image information output, thus completing the construction of the information space interaction unit, and connecting the information extraction unit and the information space interaction unit to form the image feature fusion unit. Specifically, step S2 specifically includes:

[0072] S21: Construct the information space interaction unit. In the information space interaction unit, fuse the hyperspectral information and the panchromatic information through the stage fusion state information to obtain the group image information output;

[0073] S22: Perform splicing convolution on the hyperspectral stage information and the panchromatic stage information to obtain the spliced convolution information. Perform global correlation modeling on the spliced convolution information to obtain the intermediate spliced convolution information. Perform convolution on the intermediate spliced convolution information to obtain the iterative stage fusion state information. Iterate the stage fusion state information through the iterative stage fusion state information;

[0074] S23: Obtain the image information output according to the iterated stage fusion state information and the group image information output, thus completing the construction of the information space interaction unit. Connect the information extraction unit and the information space interaction unit in series to obtain the image feature fusion unit.

[0075] For the above steps, the specific implementation manners in this embodiment are as follows:

[0076] First, construct the information interaction unit. In the information space interaction unit, fuse the hyperspectral information and the panchromatic information through the stage fusion state information to obtain the group image information output of the t-th group :

[0077]

[0078]

[0079] Among them, Reset the parameters for the t-th group.

[0080] Subsequently, the hyperspectral stage information and the panchromatic stage information are concatenated and convolved to obtain the concatenated convolution information for the t-th group. :

[0081]

[0082] Among them, Cat() concatenates the content within the parentheses. Then, global correlation modeling is performed on the concatenated convolution information to obtain the intermediate concatenated convolution information. Then, the intermediate concatenated convolution information is convolved to obtain the fusion state information of the t-th group in the iterative stage. :

[0083]

[0084] The fusion state information in the iterative stage is the fusion state information when the next group of hyperspectral information and panchromatic information are fused in the stage. Therefore, the iteration of the fusion state information in the stage can be completed through the fusion state information in the iterative stage, and the image information output can be obtained according to the fused state information in the iterated stage. Subsequently, steps S11 to S22 are repeated to obtain the group image information output of all hyperspectral information and panchromatic information, and then the image information output of the i-th stage can be obtained. :

[0085]

[0086] In this way, the construction of the information space interaction unit is completed. By sequentially connecting the information extraction unit and the information space interaction unit, the image feature fusion unit can be obtained. When the image feature fusion unit is connected to other image feature fusion units, the image information output of the i-th stage obtained by the image feature fusion unit is downsampled by a factor of 2 to obtain the HS feature map and PAN feature map of the i+1-th stage and input them into the next image feature fusion unit.

[0087] S3: In the cognitive uncertainty loss unit, the image information output is convolved to obtain the input features, and the input features are information masked and the variance is calculated to obtain the cognitive uncertainty loss.

[0088] Furthermore, the purpose of this stage is to perform operations such as convolution and information masking on the image information output to obtain the cognitive uncertainty loss, thereby constructing the cognitive uncertainty loss unit. Specifically, step S3 specifically includes:

[0089] S31: Construct the cognitive uncertainty loss unit. In the cognitive uncertainty loss unit, the image information output is upsampled, and the number of channels of the upsampled image information output is adjusted through convolution to obtain the input features.

[0090] S32: Obtain loss image information by performing random information masking on the input features, calculate the mean of the loss image information, and calculate the variance of the loss image information through the mean to obtain the cognitive uncertainty loss, thereby completing the construction of the cognitive uncertainty loss unit.

[0091] For the above steps, the specific implementation in this embodiment is as follows:

[0092] First, construct a cognitive uncertainty loss unit. In the cognitive uncertainty loss unit, first perform 2x upsampling on the image information output, and perform 1x1 convolution on the upsampled image information output to adjust the number of channels of the upsampled image information output to be the same as a set predetermined value, thereby obtaining the input features of the i-th stage. Subsequently, perform V samplings on the input features. During each sampling, perform random information masking on the input features, that is, randomly mask 2% of the channels and 2% of the pixel positions of the input features to simulate the cognitive uncertainty loss. After performing convolution and activation on the input features after random information masking, a set of loss image information can be obtained;

[0093]

[0094] Among them, is the image information output after upsampling, is to perform 1x1 convolution on the content in the parentheses, is to perform sampling and random information masking on the content in the parentheses, is the v-th loss image information.

[0095] Then calculate the mean of all loss image information, that is, the mean of the i-th stage :

[0096]

[0097] Among them, mean() is to take the mean of the content in the parentheses. Finally, calculate the variance of the loss image information according to the mean to obtain the cognitive uncertainty loss of the i-th stage :

[0098]

[0099] Among them, var() is to take the variance of the content in the parentheses, and the mean needs to be used in the process of calculating the variance. This completes the construction of the cognitive uncertainty loss unit.

[0100] S4: In the well-posed uncertainty unit, the well-posed uncertainty loss output by the image information is resolved by the branch estimation decoder. The decoding output result is obtained through the cognitive uncertainty loss and the well-posed uncertainty loss, and the cognitive uncertainty loss unit and the well-posed uncertainty loss unit are connected.

[0101] Furthermore, the purpose of this stage is to calculate the well-posed uncertainty loss, obtain the decoding output result through the cognitive uncertainty loss and the well-posed uncertainty loss, and finally form the uncertainty unit. Specifically, in step S4, after obtaining the well-posed uncertainty loss, the spectral attention and the spatial attention are respectively calculated through the cognitive uncertainty loss. The cognitive correction value is calculated according to the spectral attention and the spatial attention. The uncertainty guidance is performed on the image information output through the cognitive correction value and the well-posed uncertainty loss to obtain the decoding output result.

[0102] For the above steps, the specific implementation in this embodiment is as follows:

[0103] First, the well-posed uncertainty loss of the i-th stage of the image information output is calculated by the branch decoder ;

[0104]

[0105] After calculating the well-posed uncertainty loss, the spectral attention of the i-th stage is calculated through the cognitive uncertainty loss and the spatial attention of the i-th stage :

[0106]

[0107]

[0108] Among them, GMP() represents performing global maximum pooling operation on the content in the brackets, MLP() represents using a multi-layer perceptron function on the content in the brackets, GAP() represents performing global average pooling operation on the content in the brackets, represents performing a 7×7 convolution on the content in the brackets.

[0109] Then, the cognitive correction value of the i-th stage can be calculated according to the spectral attention and the spatial attention :

[0110]

[0111] Then, the uncertainty guidance is performed on the upsampled image information output through the cognitive correction value and the well-posed uncertainty loss to obtain the decoding output result of the i-th stage :

[0112]

[0113] Among them, UP() represents upsampling the content in the parentheses by 2 times, and UGM() represents guiding the uncertainty of the output of the upsampled image information through the cognitive correction value and the well-posed uncertainty loss. In this way, the construction of the well-posed uncertainty loss unit is completed, and the cognitive uncertainty loss unit and the well-posed uncertainty loss unit are connected in series in sequence to form the uncertainty unit.

[0114] S5: An encoder unit is formed through the image feature fusion unit, a decoder unit is formed through the image feature fusion unit and the uncertainty unit, the encoder unit is connected to the decoder unit, and the image to be parsed is input into the encoder unit and the target image is output from the decoder unit.

[0115] Furthermore, the purpose of this stage is to form the encoder unit and the decoder unit, and connect the encoder unit to the decoder unit, so as to output the target image from the decoder unit. Specifically, in step S5, the number of the image feature fusion units is determined and the image feature fusion units are connected in series to obtain the encoder unit, the image feature fusion unit and the uncertainty unit are connected in sequence to obtain the decoder subunit, the number of the decoder subunits is determined according to the number of the image feature fusion units, and the decoder subunits are connected in sequence to obtain the decoder unit.

[0116] For the above steps, the specific implementation manner of this embodiment is as follows:

[0117] First, determine the number of the image feature fusion units, then connect these image feature fusion units in series in sequence to obtain the encoder unit. Then connect the image feature fusion unit and the uncertainty unit in sequence to obtain the decoder subunit. Subsequently, based on the number of the image feature fusion units in the encoder unit, assume the number of the image feature fusion units is z, then the number of the decoder subunits is z + 1, so as to determine the number of the decoder subunits, and connect the decoder subunits in sequence to obtain the decoder unit. Then connect the encoder unit and the decoder unit in sequence.

[0118] The image to be parsed is input into the first image feature fusion unit of the encoder unit, and after passing through the encoder unit and the decoder unit in sequence, the decoding output result output by the last decoder subunit in the decoder unit can be used as the target image. The image to be parsed includes an HS picture and a PAN picture. Each time it passes through an image feature fusion unit or an uncertainty unit, its stage will increase by 1. When there are multiple decoder subunits in the decoder unit, in the decoder subunits before the last decoder subunit, after the decoding output result of the uncertainty unit is upsampled by a factor of 2, it can be input into the image feature fusion unit of the next decoder subunit as the feature map information of the image feature fusion unit of the next decoder subunit.

[0119] To verify the effectiveness of the present invention, on the public dataset Pavia center, the decoding output results of the method provided by the present invention and other methods are compared. The comparison results are as Figure 2 shown, where FPFNet, Refiner, TreeNet, and DFCFN are all existing methods. In Figure 2 , the more blue parts there are, the higher the resolution and the less distortion of the HRHS image. It can be seen that the method provided by the present invention has achieved the best resolution and the least distortion. The specific experimental results are compared as shown in Table 1:

[0120] Table 1 Performance comparison table of HRHS images obtained by the method provided by the present invention and other methods

[0121]

[0122] Among them, CC is the correlation coefficient, SAM is the spectral angle mapping, RMSE is the root mean square error, ERGAS is the dimensionless global relative error, PSNR is the peak signal-to-noise ratio. The upward arrow indicates that the higher the value, the better, and the downward arrow indicates that the lower the value, the better. It can be seen that the method provided by the present invention has achieved the best performance in each value.

[0123] Next, the uncertainty-based image fusion device provided by the present invention will be described. The uncertainty-based image fusion device described below can be mutually referred to the uncertainty-based image fusion method described above.

[0124] Figure 3 The structural schematic diagram of the uncertainty-based image fusion system is exemplified, as Figure 3 shown, which is used to execute the uncertainty-based image fusion method as described above, including:

[0125] Information activation module 100: It is used to obtain feature map information in the stage information extraction unit, group the feature map information to obtain hyperspectral information and panchromatic information, and perform convolutional activation on the hyperspectral information and panchromatic information respectively to obtain hyperspectral stage information and panchromatic stage information;

[0126] Information fusion module 200: It is used to fuse the hyperspectral information and panchromatic information in the information space interaction unit to obtain group image information output, obtain image information output through the group image information output, hyperspectral stage information and panchromatic stage information, and connect the information extraction unit and the information space interaction unit to form an image feature fusion unit;

[0127] Cognitive uncertainty module 300: It is used to perform convolution on the image information output in the cognitive uncertainty loss unit to obtain input features, perform information masking on the input features and calculate the variance to obtain the cognitive uncertainty loss;

[0128] Well-posed uncertainty module 400: It includes calculating the well-posed uncertainty loss of the image information output through the branch estimation decoder in the well-posed uncertainty unit, obtaining the decoding output result through the cognitive uncertainty loss and the well-posed uncertainty loss, and connecting the cognitive uncertainty loss unit and the well-posed uncertainty loss unit;

[0129] Image processing module 500: It is used to form an encoder unit through the image feature fusion unit, form a decoder unit through the image feature fusion unit and the uncertainty unit, connect the encoder unit and the decoder unit, input the image to be parsed into the encoder unit and output the target image from the decoder unit.

[0130] On the other hand, Figure 4 An example of the physical structure diagram of an electronic device is shown in Figure 4 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840. Among them, the processor 810, the communication interface 820, and the memory 830 complete communication with each other through the communication bus 840. The processor 810 can call the logical instructions in the memory 830 to execute the image fusion method based on uncertainty, and this method includes:

[0131] S1: In the stage information extraction unit, obtain feature map information, group the feature map information to obtain hyperspectral information and panchromatic information, and perform convolutional activation on the hyperspectral information and panchromatic information respectively to obtain hyperspectral stage information and panchromatic stage information;

[0132] S2: In the information space interaction unit, the hyperspectral information and the panchromatic information are fused to obtain the group image information output. The image information output is obtained through the group image information output, the hyperspectral stage information, and the panchromatic stage information. The information extraction unit and the information space interaction unit are connected to form an image feature fusion unit;

[0133] S3: In the cognitive uncertainty loss unit, the image information output is convolved to obtain the input features. The input features are information masked and the variance is calculated to obtain the cognitive uncertainty loss;

[0134] S4: In the well-posed uncertainty unit, the well-posed uncertainty loss of the image information output is solved by the branch estimation decoder. The decoding output result is obtained through the cognitive uncertainty loss and the well-posed uncertainty loss. The cognitive uncertainty loss unit and the well-posed uncertainty loss unit are connected;

[0135] S5: The encoder unit is formed by the image feature fusion unit, and the decoder unit is formed by the image feature fusion unit and the uncertainty unit. The encoder unit is connected to the decoder unit. The image to be analyzed is input into the encoder unit and the target image is output from the decoder unit.

[0136] In addition, when the logic instructions in the above-mentioned memory 830 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0137] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the uncertainty-based image fusion method provided by the above-mentioned various methods. The method includes:

[0138] S1: In the stage information extraction unit, obtain the feature map information, group the feature map information to obtain hyperspectral information and panchromatic information, perform convolutional activation on the hyperspectral information and the panchromatic information respectively to obtain hyperspectral stage information and panchromatic stage information;

[0139] S2: In the information space interaction unit, fuse the hyperspectral information and the panchromatic information to obtain the group image information output, obtain the image information output through the group image information output, the hyperspectral stage information and the panchromatic stage information, connect the information extraction unit and the information space interaction unit to form an image feature fusion unit;

[0140] S3: In the epistemic uncertainty loss unit, perform convolution on the image information output to obtain the input feature, perform information masking on the input feature and calculate the variance to obtain the epistemic uncertainty loss;

[0141] S4: In the aleatoric uncertainty unit, solve the aleatoric uncertainty loss of the image information output through the branch estimation decoder, obtain the decoding output result through the epistemic uncertainty loss and the aleatoric uncertainty loss, and connect the epistemic uncertainty loss unit and the aleatoric uncertainty loss unit;

[0142] S5: Form an encoder unit through the image feature fusion unit, form a decoder unit through the image feature fusion unit and the uncertainty unit, connect the encoder unit and the decoder unit, input the image to be parsed into the encoder unit and output the target image from the decoder unit.

[0143] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it is implemented to execute the uncertainty-based image fusion method provided by the above methods, and the method includes:

[0144] S1: In the stage information extraction unit, obtain the feature map information, group the feature map information to obtain hyperspectral information and panchromatic information, perform convolutional activation on the hyperspectral information and the panchromatic information respectively to obtain hyperspectral stage information and panchromatic stage information;

[0145] S2: In the information space interaction unit, fuse the hyperspectral information and the panchromatic information to obtain the group image information output, obtain the image information output through the group image information output, the hyperspectral stage information and the panchromatic stage information, connect the information extraction unit and the information space interaction unit to form an image feature fusion unit;

[0146] S3: In the epistemic uncertainty loss unit, perform convolution on the image information output to obtain the input feature, perform information masking on the input feature and calculate the variance to obtain the epistemic uncertainty loss;

[0147] S4: In the well - posed uncertainty unit, the well - posed uncertainty loss output by the image information is resolved by the branch estimation decoder. The decoded output result is obtained through the cognitive uncertainty loss and the well - posed uncertainty loss, and the cognitive uncertainty loss unit and the well - posed uncertainty loss unit are connected.

[0148] S5: The encoder unit is formed by the image feature fusion unit, and the decoder unit is formed by the image feature fusion unit and the uncertainty unit. The encoder unit is connected to the decoder unit, and the image to be analyzed is input into the encoder unit and the target image is output from the decoder unit.

[0149] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative work.

[0150] Through the description of the above - mentioned implementation manners, those skilled in the art can clearly understand that each implementation manner can be realized by means of software plus a necessary general - purpose hardware platform, and of course, it can also be realized by hardware. Based on such an understanding, the above - mentioned technical solution, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer - readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An uncertainty-based image fusion method, characterized in that Including: S1: In the stage information extraction unit, obtain the feature map information and group the feature map information to obtain hyperspectral information and panchromatic information. Perform convolution activation on the hyperspectral information and the panchromatic information respectively to obtain hyperspectral stage information and panchromatic stage information; S2: In the information space interaction unit, fuse the hyperspectral information and the panchromatic information to obtain the group image information output. Obtain the image information output through the group image information output, the hyperspectral stage information, and the panchromatic stage information. Connect the information extraction unit and the information space interaction unit to form an image feature fusion unit; S3: In the cognitive uncertainty loss unit, perform convolution on the image information output to obtain the input feature, perform information masking on the input feature and calculate the variance to obtain the cognitive uncertainty loss; S4: In the well-posed uncertainty unit, the well-posed uncertainty loss of the image information output by the branch estimation decoder is solved, and the decoding output result is obtained through the cognitive uncertainty loss and the well-posed uncertainty loss, and the cognitive uncertainty loss unit and the well-posed uncertainty loss unit are connected; among them, the well-posed uncertainty loss in the i-th stage is; Among them, is the output of the upsampled image information, is to perform a 1×1 convolution on the content in the parentheses, is to sample and randomly mask the content in the parentheses, sigmoid() represents the sigmoid activation function, and Cat() represents concatenating the content in the parentheses; S5: Construct an encoder unit through the image feature fusion unit, construct a decoder unit through the image feature fusion unit and the uncertainty unit, connect the encoder unit and the decoder unit, input the image to be parsed into the encoder unit and output the target image from the decoder unit.

2. The method for image fusion based on uncertainty according to claim 1, wherein Step S1 specifically includes: S11: Construct a stage information extraction unit. In the stage information extraction unit, obtain the feature map information, extract the spectral channel dimension of the feature map information, and group the feature map information according to the spectral channel dimension to obtain the hyperspectral information and the panchromatic information; S12: Obtain the stage fusion state information, perform convolution activation on the hyperspectral information and the panchromatic information respectively to obtain the hyperspectral information stage parameter and the panchromatic information stage parameter. Perform stage fusion through the stage fusion state information and the panchromatic information stage parameter to obtain the panchromatic stage information, and perform stage fusion through the stage fusion state information and the hyperspectral information stage parameter to obtain the hyperspectral stage information, thereby completing the construction of the stage information extraction unit.

3. The image fusion method based on uncertainty according to claim 1, wherein Step S2 specifically includes: S21: Construct an information space interaction unit. In the information space interaction unit, fuse the hyperspectral information and the panchromatic information through the stage fusion state information to obtain the group image information output; S22: Perform splicing convolution on the hyperspectral stage information and the panchromatic stage information to obtain the splicing convolution information. Perform global correlation modeling on the splicing convolution information to obtain the intermediate splicing convolution information. Perform convolution on the intermediate splicing convolution information to obtain the iterative stage fusion state information, and perform iteration on the stage fusion state information through the iterative stage fusion state information; S23: Obtain the image information output according to the iterated stage fusion state information and the group image information output, thereby completing the construction of the information space interaction unit. Connect the information extraction unit and the information space interaction unit in series to obtain the image feature fusion unit.

4. The method for image fusion based on uncertainty according to claim 1, wherein Step S3 specifically includes: S31: Construct a cognitive uncertainty loss unit. In the cognitive uncertainty loss unit, perform upsampling on the image information output and adjust the number of channels of the upsampled image information output through convolution to obtain the input feature; S32: Obtain loss image information by performing random information masking on the input features, calculate the mean of the loss image information, and calculate the variance of the loss image information through the mean to obtain the cognitive uncertainty loss, thereby completing the construction of the cognitive uncertainty loss unit.

5. The image fusion method based on uncertainty according to claim 1, wherein In step S4, after obtaining the well-posed uncertainty loss, the spectral attention and the spatial attention are respectively calculated through the cognitive uncertainty loss, the cognitive correction value is calculated according to the spectral attention and the spatial attention, and the uncertainty guidance is performed on the image information output through the cognitive correction value and the well-posed uncertainty loss to obtain the decoded output result, wherein the spectral attention in the i-th stage is calculated through the cognitive uncertainty loss and the spatial attention in the i-th stage : Among them, GMP( ) represents performing global maximum pooling operation on the content within the parentheses, MLP( ) represents using a multi-layer perceptron function on the content within the parentheses, and GAP( ) represents performing global average pooling operation on the content within the parentheses. represents performing a 7×7 convolution on the content within the parentheses. is the cognitive uncertainty loss of the i-th stage. Cat( ) represents concatenating the content within the parentheses, and sigmoid( ) represents the sigmoid activation function. Calculate the cognitive correction value of the i-th stage according to spectral attention and spatial attention : Next, the uncertainty guidance can be performed on the upsampled image information output through the cognitive correction value and the well-posed uncertainty loss, and the decoding output result of the i-th stage can be obtained : Among them, represents the input feature of the i-th stage, is element-wise multiplication, UP() represents upsampling the content within the parentheses by a factor of 2, and UGM() represents uncertainty guidance for the output of the upsampled image information through the cognitive correction value and the well-posed uncertainty loss, is the well-posed uncertainty loss of the i-th stage.

6. The method for image fusion based on uncertainty according to claim 1, wherein In step S5, determine the number of the image feature fusion units and connect the image feature fusion units in series to obtain an encoder unit, connect the image feature fusion units and the uncertainty unit in sequence to obtain a decoder subunit, determine the number of decoder subunits according to the number of the image feature fusion units, and connect the decoder subunits in sequence to obtain a decoder unit.

7. An uncertainty-based image fusion system for performing the uncertainty-based image fusion method according to any one of claims 1 to 6, characterized in that, Comprising: Information activation module: used to obtain feature map information and group the feature map information in the stage information extraction unit to obtain hyperspectral information and panchromatic information, and perform convolutional activation on the hyperspectral information and the panchromatic information respectively to obtain hyperspectral stage information and panchromatic stage information; Information fusion module: used to fuse the hyperspectral information and the panchromatic information in the information space interaction unit to obtain group image information output, obtain image information output through the group image information output, the hyperspectral stage information and the panchromatic stage information, and connect the information extraction unit and the information space interaction unit to form an image feature fusion unit; Cognitive uncertainty module: used to perform convolution on the image information output in the cognitive uncertainty loss unit to obtain input features, perform information masking on the input features and calculate the variance to obtain the cognitive uncertainty loss; Well-posed uncertainty module: including in the well-posed uncertainty unit, estimate the well-posed uncertainty loss of the decoded image information output by the decoder through a branch, obtain the decoded output result through the cognitive uncertainty loss and the well-posed uncertainty loss, and connect the cognitive uncertainty loss unit and the well-posed uncertainty loss unit; Image processing module: used to form an encoder unit through the image feature fusion unit, form a decoder unit through the image feature fusion unit and the uncertainty unit, connect the encoder unit and the decoder unit, input the image to be parsed into the encoder unit and output the target image from the decoder unit.

8. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the uncertainty-based image fusion method according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the uncertainty-based image fusion method according to any one of claims 1 to 6.

10. A computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, characterized in that, When the program instructions are executed by the computer, the computer can execute the steps of the uncertainty-based image fusion method according to any one of claims 1 to 6.

Images