Spectral image fusion method, apparatus, device, storage medium and product
By upsampling and enhancing hyperspectral and multispectral images through a spectral fusion network, a fused image with high spatial and spectral resolution is generated. This solves the problem of low accuracy in banknote authenticity identification in existing technologies, achieving higher identification accuracy and lower computational complexity.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- INDUSTRIAL AND COMMERCIAL BANK OF CHINA
- Filing Date
- 2023-03-22
- Publication Date
- 2026-08-04
AI Technical Summary
Existing technologies can only acquire multispectral and hyperspectral images with high spatial information, and distinguish genuine and counterfeit banknotes from two dimensions, which affects the accuracy of banknote authenticity identification.
The hyperspectral and multispectral images are upsampled, spectral information enhanced, and spatial information enhanced by a spectral fusion network to generate a target fused image with both high spatial and spectral resolution. The nonlinear relationship is simulated by deep learning technology and a multi-layer fusion and multi-directional correction strategy is adopted.
It improves the accuracy of banknote authenticity identification, reduces the computational complexity of image fusion, and reduces spatial spectral distortion.
Smart Images
Figure CN116229111B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of information security, and in particular to a spectral image fusion method, apparatus, device, storage medium, and product. Background Technology
[0002] Currently, an effective solution is to distinguish genuine and counterfeit banknotes from both spatial and spectral dimensions. However, due to limitations in hardware technology, only multispectral images with high spatial information and hyperspectral images with high spectral information can be acquired. Distinguishing genuine and counterfeit banknotes from two dimensions based on these two images affects the accuracy of banknote authenticity identification. Summary of the Invention
[0003] This application provides a spectral image fusion method, apparatus, device, storage medium, and product to solve the problem that existing technologies can only acquire multispectral images with high spatial information and hyperspectral images with high spectral information, and distinguish between genuine and counterfeit banknotes from two dimensions, which affects the accuracy of banknote authenticity identification.
[0004] In a first aspect, this application provides a spectral image fusion method, including: acquiring the original multispectral image and the original hyperspectral image of the object to be identified;
[0005] Upsample the original hyperspectral image and obtain the upsampled hyperspectral image;
[0006] The upsampled hyperspectral image and the original multispectral image are input into the trained spectral fusion network to perform spectral fusion, spectral information enhancement and spatial information enhancement operations on the upsampled hyperspectral image and the original multispectral image, and obtain the target fused image of the object to be fused output by the trained spectral fusion network.
[0007] The trained spectral fusion network includes a spectral information enhancement module and a spatial information enhancement module. The spectral information enhancement module performs spectral information enhancement based on the average value of each band of the upsampled hyperspectral image, and the spatial information enhancement module performs spatial information enhancement based on the superposition value of the bands of the original multispectral image.
[0008] Optionally, the upsampling of the original hyperspectral image includes:
[0009] A bilinear interpolation algorithm is used to upsample the original hyperspectral image so that the upsampled hyperspectral image has the same spatial dimension as the original multispectral image.
[0010] Optionally, the trained spectral fusion network further includes at least one image fusion module; the image fusion module is used to perform spectral fusion operations; the spectral fusion operations include: superimposing bands on the images input to the module, inputting the result after superimposing bands into a convolutional layer for convolution, and activating it using the ReLU activation function to generate an intermediate fused image.
[0011] Optionally, the trained spectral fusion network includes a first image fusion module and a second image fusion module; the images input to the first image fusion module include: an upsampled hyperspectral image and an original multispectral image; the first image fusion module generates a first intermediate fused image; the images input to the second image fusion module include: an upsampled hyperspectral image, an original multispectral image, and the first intermediate fused image; the second image fusion module generates a second intermediate fused image.
[0012] Optionally, the spectral information enhancement module performs spectral information enhancement operations based on the average value of each band of the upsampled hyperspectral image, including:
[0013] Global pooling is performed on the upsampled hyperspectral image to obtain the average value of each band of the upsampled hyperspectral image; the average value of each band is activated by the sigmoid activation function, the activation result of the average value of each band is multiplied by the band corresponding to the second intermediate fused image, and the multiplication result is added to the second intermediate fused image to obtain the third intermediate fused image.
[0014] Optionally, the spatial information enhancement module performs spatial information enhancement operations based on the superposition values of the original multispectral image bands, including:
[0015] Each band of the multispectral image is added together to obtain the superposition value of the original multispectral image bands; the superposition value of the bands is activated using the Sigmaod activation function, the activation result of the superposition value is multiplied with the band corresponding to the third intermediate fused image, and the multiplication result is added to the third intermediate fused image to obtain the fourth intermediate fused image.
[0016] Optionally, the trained spectral fusion network includes a third image fusion module; the images input to the third image fusion module include: an upsampled hyperspectral image, an original multispectral image, and a fourth intermediate fused image; the third image fusion module generates a target fused image.
[0017] Optionally, after obtaining the target fusion image of the object to be fused output by the trained spectral fusion network, the process includes: using a trained banknote recognition model to recognize the target fusion image to determine whether the banknote corresponding to the target fusion image is a genuine banknote.
[0018] Secondly, this application provides a spectral image fusion apparatus, comprising:
[0019] The acquisition module is used to acquire the original multispectral image and the original hyperspectral image of the object to be identified.
[0020] The processing module is used to upsample the original hyperspectral image and acquire the upsampled hyperspectral image;
[0021] The fusion module is used to input the upsampled hyperspectral image and the original multispectral image into the trained spectral fusion network to perform spectral fusion operation, spectral information enhancement operation and spatial information enhancement operation on the upsampled hyperspectral image and the original multispectral image, and to obtain the target fused image of the object to be fused output by the trained spectral fusion network.
[0022] The trained spectral fusion network includes a spectral information enhancement module and a spatial information enhancement module. The spectral information enhancement module performs spectral information enhancement based on the average value of each band of the upsampled hyperspectral image, and the spatial information enhancement module performs spatial information enhancement based on the superposition value of the bands of the original multispectral image.
[0023] Thirdly, this application provides an electronic device, including: a processor, and a memory and a transceiver communicatively connected to the processor;
[0024] The memory stores computer-executed instructions; the transceiver is used for sending and receiving data.
[0025] The processor executes computer execution instructions stored in the memory to implement the spectral image fusion method described in any of the above aspects.
[0026] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the spectral image fusion method described in any of the above aspects.
[0027] Fifthly, this application provides a computer program product, including computer execution instructions, which, when executed by a processor, implement the spectral image fusion method described in any of the above aspects.
[0028] The spectral image fusion method, apparatus, device, storage medium, and product provided in this application acquire the original multispectral image and original hyperspectral image of the object to be identified; upsample the original hyperspectral image to obtain the upsampled hyperspectral image; input the upsampled hyperspectral image and the original multispectral image into a trained spectral fusion network to perform spectral fusion, spectral information enhancement, and spatial information enhancement operations on the upsampled hyperspectral image and the original multispectral image, and obtain the target fused image of the object to be fused output by the trained spectral fusion network; the trained spectral fusion network includes a spectral information enhancement module and a spatial information enhancement module; the spectral information enhancement module performs spectral information enhancement based on the average value of each band of the upsampled hyperspectral image, and the spatial information enhancement module performs spatial information enhancement based on the superposition value of the bands of the original multispectral image. By fusing high spatial resolution multispectral image and high spectral resolution hyperspectral image, a target fused image with both high spatial resolution and high spectral resolution is generated, effectively overcoming the limitations of existing technologies, enabling the target fused image of banknotes to simultaneously possess rich spatial and spectral information, and effectively improving the accuracy of distinguishing genuine from counterfeit banknotes. In addition, a multi-layer fusion and multi-directional correction strategy is used in the spectral fusion network, which makes the image fusion more complete and effectively reduces spatial spectral distortion. Attached Figure Description
[0029] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0030] Figure 1 This is a flowchart of the spectral image fusion method provided in the embodiments of this application;
[0031] Figure 2 A flowchart of another spectral image fusion method provided in the embodiments of this application;
[0032] Figure 3 This is a schematic diagram of the spectral fusion network architecture provided in an embodiment of this application;
[0033] Figure 4 This is a schematic diagram illustrating an application scenario provided in the embodiments of this application;
[0034] Figure 5 This is a schematic diagram of the spectral image fusion device provided in the embodiments of this application;
[0035] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.
[0036] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation
[0037] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0038] The terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. In the following descriptions of embodiments, "a plurality of" means two or more, unless otherwise explicitly defined.
[0039] First, the prior art involved in this invention will be described and analyzed in detail.
[0040] Multispectral images of banknotes possess high spatial resolution, providing detailed textures and geometric features to meet spatial detection requirements. Hyperspectral images, on the other hand, offer higher spectral resolution, providing more nuanced spectral curves, thus allowing for the differentiation of genuine and counterfeit banknotes from a spectral perspective. Therefore, an effective solution involves identifying genuine and counterfeit banknotes from both spatial and spectral dimensions.
[0041] Due to limitations in hardware technology, currently only multispectral images with high spatial information and hyperspectral images with high spectral information can be acquired. The identification of genuine and counterfeit banknotes is based on two dimensions, namely multispectral images and hyperspectral images, which greatly affects the accuracy of banknote authenticity identification.
[0042] The inventors discovered in their research that fusing hyperspectral and multispectral images, utilizing the rich spatial information of the multispectral image and the rich spectral information of the hyperspectral image, generates a banknote target fusion image with both high spatial resolution and high spectral resolution. This provides richer spatial and spectral information, and identification based on the banknote target fusion image can improve the accuracy of banknote authenticity identification.
[0043] The collection, storage, use, processing, transmission, provision, and disclosure of financial data or user data involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0044] It should be noted that the spectral image fusion method, apparatus, device, storage medium, and product of this application can be used in the financial field. They can also be used in any field other than finance. The application fields of the spectral image fusion method, apparatus, device, storage medium, and product of this application are not limited.
[0045] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.
[0046] Figure 1 This is a flowchart of the spectral image fusion method provided in this application embodiment. This application embodiment addresses the problem that existing technologies can only acquire multispectral images with high spatial information and hyperspectral images with high hyperspectral information, respectively, to distinguish genuine and counterfeit banknotes from two dimensions, thus affecting the accuracy of banknote authenticity identification. A spectral image fusion method is provided in this embodiment. The method in this embodiment is applied to a spectral image fusion device, which can be located in an electronic device. The electronic device can be a digital computer of various forms, such as a laptop computer, desktop computer, workbench, personal digital assistant, server, blade server, mainframe computer, and other suitable computers.
[0047] like Figure 1 As shown, the specific steps of this method are as follows:
[0048] Step S101: Obtain the original multispectral image and the original hyperspectral image of the object to be identified.
[0049] In this embodiment of the application, the original multispectral image of the object to be identified can be obtained by a multispectral sensor, and the hyperspectral image of the same object to be identified can be obtained by a hyperspectral sensor.
[0050] Step S102: Upsample the original hyperspectral image and obtain the upsampled hyperspectral image.
[0051] In this embodiment of the application, in order to keep the spatial dimensions of the hyperspectral image and the multispectral image the same, the original hyperspectral image is upsampled to generate an upsampled hyperspectral image with the same spatial dimensions as the original multispectral image.
[0052] Step S103: Input the upsampled hyperspectral image and the original multispectral image into the trained spectral fusion network to perform spectral fusion, spectral information enhancement and spatial information enhancement operations on the upsampled hyperspectral image and the original multispectral image, and obtain the target fused image of the object to be fused output by the trained spectral fusion network.
[0053] The trained spectral fusion network includes a spectral information enhancement module and a spatial information enhancement module. The spectral information enhancement module performs spectral information enhancement based on the average value of each band of the upsampled hyperspectral image, while the spatial information enhancement module performs spatial information enhancement based on the superposition value of the bands of the original multispectral image.
[0054] For example, the trained spectral fusion network can perform spectral fusion operations on the upsampled hyperspectral image and the original multispectral image, and perform spectral information enhancement operations and spatial information enhancement operations on the spectral fusion image.
[0055] This application does not limit the number of spectral fusion operations or the order in which spectral fusion, spectral information enhancement, and spatial information enhancement operations are performed; these can be specifically set as needed. For example, spectral fusion can be performed before or after the spectral information enhancement and spatial information enhancement operations.
[0056] The method provided in this application fuses high spatial resolution multispectral images with high spectral resolution hyperspectral images to generate a target fused image that simultaneously possesses high spatial and spectral resolution. This effectively overcomes the limitations of existing technologies, enabling the target fused image of banknotes to simultaneously possess rich spatial and spectral information, thus effectively improving the accuracy of counterfeit banknote identification. Furthermore, this method, based on deep learning technology, effectively simulates the nonlinear relationship between multispectral and hyperspectral images and high-resolution hyperspectral images, reducing computational complexity. In addition, the spectral fusion network employs a multi-layer fusion and multi-directional correction strategy, resulting in more thorough image fusion and effectively reducing spatial spectral distortion.
[0057] Optionally, one way to implement step S102, which involves upsampling the original hyperspectral image and obtaining the upsampled hyperspectral image, is as follows:
[0058] A bilinear interpolation algorithm is used to upsample the original hyperspectral image so that the upsampled hyperspectral image has the same spatial dimension as the original multispectral image.
[0059] The formula for the bilinear interpolation algorithm is: Y ↑ =Bilinear(Y,r). r is the ratio of the spatial resolution of the original hyperspectral image to that of the upsampled hyperspectral image, and Bilinear is the bilinear sampling function.
[0060] In this embodiment of the application, the bilinear interpolation algorithm is used to upsample the original hyperspectral image so that the upsampled hyperspectral image has the same spatial dimension as the original multispectral image. Therefore, the trained spectral fusion network can be used to fuse the multispectral image and the hyperspectral image.
[0061] Optionally, the trained spectral fusion network may include at least one image fusion module, which performs spectral fusion operations. The spectral fusion operations include: overlaying bands onto the image input to the module; inputting the overlaid bands into a convolutional layer for convolution; and activating the layer using a ReLU activation function to generate an intermediate fused image. The ReLU activation function converts all negative values to 0.
[0062] For example, if the input images to this module are an upsampled hyperspectral image and the original multispectral image, the bands of the two images can be interleaved according to the band range of the upsampled hyperspectral image and the band range of the original multispectral image to perform band superposition.
[0063] For example, one way to perform spectral fusion is to perform a band interleaving operation on the spectral image based on the band range of the hyperspectral image and the band range of the original multispectral image, and then convolve the interleaving result and use it to generate the image after spectral fusion.
[0064] In this embodiment, the images from the input image fusion module are superimposed with bands, and the result after superimposing the bands is input into a convolutional layer for convolution. The ReLU activation function is then used to activate the image to generate an intermediate fused image. The image fusion method is simple and efficient, and the spectral fusion network is relatively concise.
[0065] The following example illustrates the specific process of generating a target fusion image of the object to be fused from the output of the trained spectral fusion network. Figure 2 A flowchart of another spectral image fusion method provided in the embodiments of this application is shown below. Figure 2 As shown, it includes the following steps:
[0066] Step S201: Perform the first spectral fusion operation using the first image fusion module, and perform the second spectral fusion operation using the second image fusion module.
[0067] In this embodiment of the application, the trained spectral fusion network includes a first image fusion module and a second image fusion module. Figure 3 This is a schematic diagram of the spectral fusion network architecture provided in the embodiments of this application, such as... Figure 3As shown, the first and second image fusion modules each include a convolutional layer and an activation function. The low-resolution hyperspectral image is an upsampled hyperspectral image, and the high-resolution multispectral image is the original multispectral image. The spectral fusion operation performed by the first and second image fusion modules is similar to the spectral fusion operation described above, and will not be repeated here.
[0068] The images input to the first image fusion module include: an upsampled hyperspectral image and the original multispectral image; the first image fusion module generates a first intermediate fused image; the images input to the second image fusion module include: an upsampled hyperspectral image, the original multispectral image and the first intermediate fused image; the second image fusion module generates a second intermediate fused image.
[0069] Specifically, the upsampled hyperspectral image and the original multispectral image are input into the first image fusion module of the trained spectral fusion network, and the first image fusion module is used to perform the first spectral fusion operation to obtain the first intermediate fused image output by the first image fusion module; the first intermediate fused image, the upsampled hyperspectral image and the original multispectral image are input into the second image fusion module, and the second image fusion module is used to perform the second spectral fusion operation to obtain the second intermediate fused image output by the second image fusion module.
[0070] In this embodiment of the application, in order to make the image fusion more complete and the information more abundant, the spectral information of the hyperspectral image and the spatial information of the multispectral image are injected into the first intermediate fused image through a second spectral fusion operation.
[0071] Step S202: The spectral information enhancement module performs spectral information enhancement operation based on the average value of each band of the upsampled hyperspectral image.
[0072] In this embodiment of the application, the second intermediate fused image is input into the spectral information enhancement module so that the spectral information enhancement module can be used to perform spectral information enhancement operation.
[0073] In this embodiment, the spectral information enhancement module performs global pooling on the upsampled hyperspectral image to obtain the average value of each band of the upsampled hyperspectral image. The average value of each band is activated using the sigmoid activation function. The activation result of the average value of each band is multiplied by the corresponding band of the second intermediate fused image, and the multiplication result is added to the second intermediate fused image to obtain the third intermediate fused image. The formula for the sigmoid activation function is:
[0074]
[0075] Here, x is the input value of the sigmoid activation function, and f(x) is the output value. Activating the mean value with the sigmoid activation function transforms the mean value into a value between 0 and 1.
[0076] In this embodiment of the application, in order to avoid severe spectral distortion in the fused image, the average value of each band of the upsampled hyperspectral image is used to correct the spectral distortion of the fused image, which can enhance the fusion of spectral information.
[0077] Specifically, such as Figure 3 As shown, global pooling is performed on the upsampled hyperspectral image to obtain the average value of each band of the upsampled hyperspectral image. The activated average value is multiplied by the corresponding band of the second intermediate fused image to obtain a set of spectral constraint values. The spectral constraint values are added to the second intermediate fused image to reduce the spectral distortion of the fused image and make the spectral information richer. Finally, it is input into a convolutional layer for convolution and activated using the ReLU activation function to generate a third intermediate fused image with less spectral distortion.
[0078] Step S203: Use the spatial information enhancement module to perform spatial information enhancement operation based on the superposition values of the original multispectral image bands.
[0079] In this embodiment of the application, the third intermediate fused image is input into the spatial information enhancement module so that the spatial information enhancement module can be used to perform spatial information enhancement operation.
[0080] In this embodiment, the spatial information enhancement module adds each band of the multispectral image to obtain the superimposed value of the original multispectral image bands. The superimposed value of the bands is activated using the Sigmaod activation function. The activation result of the superimposed value is multiplied by the corresponding band of the third intermediate fused image, and the multiplication result is added to the third intermediate fused image to obtain the fourth intermediate fused image. Activating the superimposed value of the bands using the Sigmaod activation function transforms the superimposed value of the bands into a value between 0 and 1.
[0081] In this embodiment of the application, in order to enrich the spatial information of the fused image, the spatial information of the fused image is supplemented again by using the superposition value of the bands of the multispectral image to correct the spatial distortion of the fused image, which can enhance the fusion of spatial information.
[0082] Specifically, such as Figure 3As shown, each band of the original multispectral image is summed to obtain the superimposed value of each pixel. The superimposed value is then activated using the Sigmaod activation function, constraining its value to between 0 and 1. The activated superimposed value is multiplied by the corresponding pixel of the third intermediate fused image, and then added to the third intermediate fused image. In this way, the spatial distortion of the fused image is reduced, making the spatial information richer. Finally, it is input into a convolutional layer for convolution, and activated using the ReLU activation function to generate a fourth intermediate fused image with less spatial and spectral distortion and high spatial and spectral resolution.
[0083] Step S204: Perform the third spectral fusion operation using the third image fusion module.
[0084] In this embodiment, the trained spectral fusion network may further include a third image fusion module. The third image fusion module performs spectral fusion operations in a similar manner to the spectral fusion operations described above, and will not be repeated here.
[0085] The images input to the third image fusion module include: upsampled hyperspectral image, original multispectral image, and fourth intermediate fused image; the first image fusion module generates the target fused image.
[0086] In order to enrich the spatial spectral information of the fused image, this application further supplements the spatial spectral information of the fused image.
[0087] Specifically, such as Figure 3 As shown, the upsampled hyperspectral image, the original multispectral image, and the fourth intermediate fusion image are combined in the third image fusion module. The third image fusion module is then used to perform a third spectral fusion operation to obtain the target fusion image output by the third image fusion module. This target fusion image is a high-resolution hyperspectral image of the object to be identified.
[0088] This application also provides a training method for a spectral fusion network, which can be used to iteratively optimize the spectral fusion network by calculating the loss function in the spectral fusion network.
[0089] Optionally, the loss function in the spectral fusion network can be the L2 loss function. Specifically, the training multispectral images of the training samples can be upsampled spectrally, and the training hyperspectral images of the training samples can be upsampled spatially, so that the sampled training multispectral images and training hyperspectral images have the same spatial and spectral dimensions. The sampled training multispectral images and training hyperspectral images are input into the spectral fusion network to obtain the training target fused image. The training target fused image and the upsampled training multispectral images and training hyperspectral images are compared using the L2 loss function. The values in the convolutional layers of the spectral fusion network are optimized and updated through this loss function until a preset number of iterations is reached, completing the training of the spectral fusion network and obtaining the trained spectral fusion network.
[0090] The training method of the spectral fusion network provided in this application does not require a high spatial resolution hyperspectral image as the target image in advance. The spectral fusion network can be trained using training hyperspectral images and multispectral images of the training samples, which improves the application scope of the spectral image fusion method provided in this application.
[0091] Optionally, after obtaining the target fusion image of the object to be fused output by the trained spectral fusion network in step S103, the process includes:
[0092] The trained banknote recognition model is used to identify the target fused image in order to determine whether the banknote corresponding to the target fused image is a genuine banknote.
[0093] In this embodiment of the application, after the spectral fusion network is trained, the trained spectral fusion network can be used to generate the target fusion image with high spatial information and high spectral information, and the target fusion image can be put into practical use, such as for distinguishing between genuine and counterfeit banknotes.
[0094] Figure 4 This is a schematic diagram of an application scenario provided in an embodiment of this application. The object to be integrated can be banknotes, such as... Figure 4 As shown, after acquiring the original multispectral image and the original hyperspectral image of the banknote, the electronic device 1 can execute the spectral image fusion method provided in this application to perform multispectral and hyperspectral image fusion and acquire a target fused image containing the high spatial information and hyperspectral information of the banknote. After acquiring the target fused image, the target fused image of the banknote can be input into the trained banknote recognition model to recognize the target fused image, so as to determine whether the banknote corresponding to the target fused image is a genuine banknote, which can improve the recognition accuracy of the banknote.
[0095] Figure 5 This is a schematic diagram of the spectral image fusion apparatus provided in an embodiment of this application. The spectral image fusion apparatus provided in this embodiment can execute the processing flow provided in the spectral image fusion method embodiment. Figure 5 As shown, the spectral image fusion device 50 includes: an acquisition module 501, a processing module 502, and a fusion module 503.
[0096] Specifically, the acquisition module 501 is used to acquire the original multispectral image and the original hyperspectral image of the object to be identified;
[0097] Processing module 502 is used to upsample the original hyperspectral image and acquire the upsampled hyperspectral image;
[0098] The fusion module 503 is used to input the upsampled hyperspectral image and the original multispectral image into the trained spectral fusion network to perform spectral fusion operation, spectral information enhancement operation and spatial information enhancement operation on the upsampled hyperspectral image and the original multispectral image, and to obtain the target fused image of the object to be fused output by the trained spectral fusion network.
[0099] The trained spectral fusion network includes a spectral information enhancement module and a spatial information enhancement module. The spectral information enhancement module performs spectral information enhancement based on the average value of each band of the upsampled hyperspectral image, while the spatial information enhancement module performs spatial information enhancement based on the superposition value of the bands of the original multispectral image.
[0100] The apparatus provided in this application embodiment can be specifically used to execute the method embodiment provided in Embodiment 1 above, and the specific functions will not be repeated here.
[0101] Optionally, the processing module 502 is specifically used to: upsample the original hyperspectral image using a bilinear interpolation algorithm so that the upsampled hyperspectral image has the same spatial dimension as the original multispectral image.
[0102] Optionally, the trained spectral fusion network also includes at least one image fusion module; the image fusion module is used to perform spectral fusion operations; when the fusion module 503 is used for spectral fusion operations, it includes: performing band overlay on the image input to the module, inputting the result after band overlay into a convolutional layer for convolution, and activating it using the ReLU activation function to generate an intermediate fused image.
[0103] Optionally, the trained spectral fusion network includes a first image fusion module and a second image fusion module; the images input to the first image fusion module include: upsampled hyperspectral images and original multispectral images; the first image fusion module generates a first intermediate fused image; the images input to the second image fusion module include: upsampled hyperspectral images, original multispectral images and the first intermediate fused image; the second image fusion module generates a second intermediate fused image.
[0104] Optionally, when the fusion module 503 is used for spectral information enhancement operations, it specifically includes: performing global pooling on the upsampled hyperspectral image to obtain the average value of each band of the upsampled hyperspectral image; activating the average value of each band using the sigmoid activation function; multiplying the activation result of the average value of each band with the band corresponding to the second intermediate fused image; and adding the multiplication result to the second intermediate fused image to obtain the third intermediate fused image.
[0105] Optionally, when the fusion module 503 is used for spatial information enhancement operations, it specifically includes: adding each band of the multispectral image to obtain the superposition value of the original multispectral image bands; activating the superposition value of the bands using the sigmoid activation function; multiplying the activation result of the superposition value with the band corresponding to the third intermediate fused image; and adding the multiplication result with the third intermediate fused image to obtain the fourth intermediate fused image.
[0106] Optionally, the trained spectral fusion network includes a third image fusion module; the images input to the third image fusion module include: upsampled hyperspectral image, original multispectral image and fourth intermediate fused image; the third image fusion module generates the target fused image.
[0107] Optionally, the spectral image fusion device 50 further includes: an identification module, which is used to: identify the target fused image using a trained banknote recognition model to determine whether the banknote corresponding to the target fused image is a genuine banknote.
[0108] The apparatus provided in this application embodiment can be specifically used to execute the above method embodiments, and its specific functions will not be described in detail here.
[0109] Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 6 As shown, this application also provides an electronic device 60, including: a processor 601, and a memory 602 and a transceiver 603 communicatively connected to the processor 601. The memory 602 stores computer-executable instructions; the transceiver 603 is used for sending and receiving data; the processor 601 executes the computer-executable instructions stored in the memory 602 to implement the method provided in any embodiment of this application.
[0110] Specifically, the program may include program code, which includes computer-executable instructions. Memory 602 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device. The computer-executable instructions are stored in memory 602 and configured to be executed by processor 601 to implement the method provided in any embodiment of this application. Related descriptions can be understood by referring to the relevant descriptions and effects corresponding to the steps in the accompanying drawings, and will not be elaborated upon here.
[0111] In this embodiment, the memory 602 and the processor 601 are connected via a bus. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0112] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method provided in any embodiment of this application.
[0113] This application also provides a computer program product, including computer execution instructions, which, when executed by a processor, implement the method provided in any embodiment of this application.
[0114] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.
[0115] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.
[0116] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or in a combination of hardware and software functional modules.
[0117] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to the processor or controller of a general-purpose computer, special-purpose computer, or other programmable full-path fusion device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0118] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0119] Furthermore, although the operations are described in a specific order, this should be understood as requiring that such operations be performed in the specific order shown or in sequential order, or requiring that all illustrated operations be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this application. Certain features described in the context of individual embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations.
[0120] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.
[0121] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.
Claims
1. A spectral image fusion method, characterized in that, include: Acquire the original multispectral and original hyperspectral images of the object to be identified; Upsample the original hyperspectral image and obtain the upsampled hyperspectral image; The upsampled hyperspectral image and the original multispectral image are input into the trained spectral fusion network to perform spectral fusion, spectral information enhancement and spatial information enhancement operations on the upsampled hyperspectral image and the original multispectral image, and obtain the target fused image of the object to be fused output by the trained spectral fusion network. The trained spectral fusion network includes a spectral information enhancement module and a spatial information enhancement module. The spectral information enhancement module performs spectral information enhancement based on the average value of each band of the upsampled hyperspectral image, and the spatial information enhancement module performs spatial information enhancement based on the superposition value of the bands of the original multispectral image. The trained spectral fusion network also includes at least one image fusion module; the image fusion module is used to perform spectral fusion operations. The spectral fusion operation includes: overlaying bands on the image input to the module, inputting the result after overlaying bands into a convolutional layer for convolution, and activating it using the ReLU activation function to generate an intermediate fused image; The trained spectral fusion network includes a first image fusion module, a second image fusion module, and a third image fusion module; The images input to the first image fusion module include: upsampled hyperspectral images and original multispectral images; the first image fusion module generates a first intermediate fused image; The images input to the second image fusion module include: an upsampled hyperspectral image, the original multispectral image, and a first intermediate fused image; the second image fusion module generates a second intermediate fused image; The images input to the third image fusion module include: upsampled hyperspectral image, original multispectral image, and fourth intermediate fused image; the third image fusion module generates a target fused image, and the fourth intermediate fused image is an intermediate fused image after spectral information enhancement operation and spatial information enhancement operation.
2. The method according to claim 1, characterized in that, The upsampling of the original hyperspectral image includes: A bilinear interpolation algorithm is used to upsample the original hyperspectral image so that the upsampled hyperspectral image has the same spatial dimension as the original multispectral image.
3. The method according to claim 1, characterized in that, The spectral information enhancement module performs spectral information enhancement operations based on the average value of each band of the upsampled hyperspectral image, including: Global pooling is performed on the upsampled hyperspectral image to obtain the average value of each band of the upsampled hyperspectral image; The average value of each band is activated using the sigmoid activation function. The activation result of the average value of each band is multiplied by the band corresponding to the second intermediate fused image, and the multiplication result is added to the second intermediate fused image to obtain the third intermediate fused image.
4. The method according to claim 3, characterized in that, The spatial information enhancement module performs spatial information enhancement operations based on the superposition values of the original multispectral image bands, including: Each band of the multispectral image is added together to obtain the superimposed value of the original multispectral image bands; The superimposed value of the band is activated using the Sigmaod activation function. The activation result of the superimposed value is multiplied with the band corresponding to the third intermediate fused image, and the multiplication result is added to the third intermediate fused image to obtain the fourth intermediate fused image.
5. The method according to any one of claims 1-4, characterized in that, After obtaining the target fusion image of the object to be fused output by the trained spectral fusion network, the process includes: The trained banknote recognition model is used to identify the target fused image in order to determine whether the banknote corresponding to the target fused image is a genuine banknote.
6. A spectral image fusion device, characterized in that, include: The acquisition module is used to acquire the original multispectral image and the original hyperspectral image of the object to be identified. The processing module is used to upsample the original hyperspectral image and acquire the upsampled hyperspectral image; The fusion module is used to input the upsampled hyperspectral image and the original multispectral image into the trained spectral fusion network to perform spectral fusion operation, spectral information enhancement operation and spatial information enhancement operation on the upsampled hyperspectral image and the original multispectral image, and to obtain the target fused image of the object to be fused output by the trained spectral fusion network. The trained spectral fusion network includes a spectral information enhancement module and a spatial information enhancement module. The spectral information enhancement module performs spectral information enhancement based on the average value of each band of the upsampled hyperspectral image, and the spatial information enhancement module performs spatial information enhancement based on the superposition value of the bands of the original multispectral image. The trained spectral fusion network also includes at least one image fusion module; the image fusion module is used to perform spectral fusion operations. The spectral fusion operation includes: overlaying bands on the image input to the module, inputting the result after overlaying bands into a convolutional layer for convolution, and activating it using the ReLU activation function to generate an intermediate fused image; The trained spectral fusion network includes a first image fusion module, a second image fusion module, and a third image fusion module; The images input to the first image fusion module include: upsampled hyperspectral images and original multispectral images; the first image fusion module generates a first intermediate fused image; The images input to the second image fusion module include: an upsampled hyperspectral image, the original multispectral image, and a first intermediate fused image; the second image fusion module generates a second intermediate fused image; The images input to the third image fusion module include: upsampled hyperspectral image, original multispectral image, and fourth intermediate fused image; the third image fusion module generates a target fused image, and the fourth intermediate fused image is an intermediate fused image after spectral information enhancement operation and spatial information enhancement operation.
7. An electronic device, characterized in that, include: A processor, and a memory and a transceiver communicatively connected to the processor; The memory stores computer-executed instructions; the transceiver is used for sending and receiving data. The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-5.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-5.
9. A computer program product comprising computer-executable instructions, characterized in that, When the computer execution instructions are executed by the processor, they implement the method as described in any one of claims 1-5.