A brain-inspired multi-modal image resolution fusion enhancement method
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- UNIV OF ELECTRONICS SCI & TECH OF CHINA
- Filing Date
- 2026-02-03
- Publication Date
- 2026-06-26
Smart Images
Figure CN121724848B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision and remote sensing image technology, and in particular to a brain-inspired multimodal image resolution fusion enhancement method. Background Technology
[0002] Hyperspectral imaging (HSI) technology can capture information about a target in a continuous and dense narrow spectral band, forming a "data cube" containing both spatial and spectral information. Compared to traditional RGB or multispectral (MSI) images, hyperspectral images have extremely high spectral resolution, accurately reflecting the spectral characteristics of ground objects. Therefore, they play an irreplaceable role in fields such as precision agriculture, geological exploration, environmental monitoring, medical diagnosis, and military reconnaissance. However, due to the physical principles of optical imaging systems, hyperspectral imaging often requires a trade-off between spatial resolution, spectral resolution, and temporal resolution. Typically, to obtain high spectral resolution, spatial resolution must be sacrificed, resulting in blurred spatial details in the acquired hyperspectral images, making it difficult to meet the requirements of high-precision applications. Conversely, while multispectral images have fewer spectral bands, they usually have high spatial resolution. Therefore, fusing low spatial resolution hyperspectral images with high spatial resolution multispectral images of the same scene to reconstruct a hyperspectral image with both high spatial and spectral resolution (referred to as a high spatial resolution hyperspectral image) has become an effective way to resolve this contradiction.
[0003] Traditional hyperspectral image resolution enhancement methods mainly include matrix factorization and tensor factorization. These methods typically extract and reconstruct high-resolution images by performing mathematical transformations on the image. However, existing image resolution enhancement methods have the following limitations:
[0004] 1) Lack of effective utilization of multimodal information: Traditional methods mostly process information from different modalities (such as spatial and spectral information) separately, lacking efficient cross-modal fusion within the same framework. For example, multispectral images provide high spatial resolution, while hyperspectral images have more spectral information. However, traditional methods often fail to effectively combine these two types of information, resulting in the incomplete utilization of the complementarity of spatial and spectral information during image fusion. This leads to suboptimal quality in the final fusion result, particularly in terms of efficiency in detail restoration and feature fusion.
[0005] 2) Strong scale dependence: Most current image resolution fusion enhancement methods assume a fixed scale and cannot flexibly adapt to the complex relationships between images of different resolutions. In practical applications, especially in remote sensing image processing, the magnification factor of images is usually non-integer, which makes traditional methods unable to cope with diverse scale requirements, resulting in loss of image details or poor fusion effects.
[0006] 3) Lack of biologically inspired mechanisms: Existing traditional methods mostly rely on mathematical tools such as matrix operations and tensor operations, lacking the imitation of the human visual system. When processing visual information, the human brain prioritizes overall information (such as scene overview) before focusing on local details. Existing methods ignore this layer-by-layer focusing perceptual process and fail to effectively integrate the global information of low-resolution images with the detailed information of high-resolution images. Summary of the Invention
[0007] The objective of this invention is to provide a brain-inspired multimodal image resolution fusion enhancement method. This method simulates the working principle of the human brain when processing multimodal information, achieving efficient fusion of spatial resolution and spectral information. The method uses a hidden neural representation network to adaptively process the spatial, spectral, and coordinate information of images from different modalities.
[0008] The technical solution adopted in this invention is: a brain-inspired multimodal image resolution fusion enhancement method, which includes the following steps:
[0009] Step 1: Input a low-resolution hyperspectral image of the target region. and high-resolution multispectral images ;
[0010] For images Upsampling is performed to obtain the image Images with consistent resolution ;
[0011] Step 2, input the image through the first nested kernel layer. Mapped to a fixed feature dimension The spectral modal features are obtained; then the spectral latent coding features are obtained through a spectral encoder. ;
[0012] Image With images The concatenation is performed along the channel dimension, and then mapped to a fixed feature dimension via the second nested kernel input layer. Spatial modal features are obtained; then spatial latent coding features are obtained through a spatial encoder. ;
[0013] exist Construct the corresponding image above Normalized continuous coordinate grid with high resolution , for grid Any query coordinates on Find its nearest neighbor coordinates in a low-resolution grid. And calculate coordinate features ;
[0014] Step 3: Process the obtained encoded features through a multimodal implicit fusion module. , and coordinate features Perform fusion processing and output each query coordinate. The residual feature vector, based on all query coordinates The residual feature vectors are used to obtain the fused feature map. ;
[0015] Step 4: Fuse the feature maps using the nested kernel output layer. Mapping back to image The corresponding number of spectral bands, and output with the image. High-resolution spectral residuals with consistent resolution ;
[0016] Step 5, process the image and high-resolution spectral residuals Pixel-by-pixel addition is performed to generate a reconstructed high spatial resolution hyperspectral image. .
[0017] Furthermore, in step 1, bicubic interpolation is used to interpolate the image. Perform upsampling processing.
[0018] Furthermore, the first nested core input layer has the same network structure as the second nested core input layer, and it includes a maximum preset number of bands. Nested convolution weights ,in, For a pre-defined fixed feature dimension, The kernel size is the convolution kernel size. For the real number field; based on the image of the current input. Number of spectral bands When performing nested convolution processing, the first channel of the input channel... The synapses corresponding to each channel are activated, and the effective weights are obtained through slicing. Based on this, convolution operations are performed to adaptively map the input data to a fixed feature dimension. .
[0019] Furthermore, in step 2, the network structures of the spectral encoder and the spatial encoder are the same, and the encoder structure is based on residual network or enhanced deep super-resolution network.
[0020] Furthermore, in step 2, the coordinates are queried. for:
[0021]
[0022] in, , For image Height and width, , , respectively, are the x and y coordinates of the high-resolution multispectral image, and , .
[0023] Furthermore, in step 3, the multimodal implicit fusion module employs a multilayer perceptron (MLP), and each hidden layer of the MLP is activated by a nonlinear activation function. To simulate the threshold mechanism of excitation / inhibition in biological neurons.
[0024] Furthermore, nonlinear activation functions Using the ReLU function, when the sum of the weighted signals received by the neuron is less than or equal to 0, its output is forced to 0, and it is in an inhibited state; when the sum of the weighted signals exceeds 0, the neuron is in an excited state, and its output (i.e., the sum of the weighted signals) is propagated backward.
[0025] Furthermore, based on a bucketing strategy, the first nested kernel input layer, the spectral encoder, the second nested kernel input layer, the spatial encoder, the multimodal implicit fusion module, and the nested kernel output layer are jointly trained, including:
[0026] The collected heterogeneous datasets are divided into environmental buckets according to their source, number of bands and spatial size. The samples contained in each environmental bucket have the same tensor shape, that is, the same number of bands and spatial size.
[0027] In each iteration of the training loop, a two-stage sampling is performed: First, an environment bucket is randomly selected from all environment buckets according to a uniform distribution; then, a batch of training data is drawn from the selected environment bucket; finally, joint training is performed based on the current batch of training data and the set training loss function, which includes high-resolution spectral residuals. With reconstructed high spatial resolution hyperspectral images The absolute value loss and structural similarity loss between them.
[0028] Furthermore, the training loss function is specifically set as follows:
[0029]
[0030] in, Denotes the first-order norm. Represents structural similarity loss. These are preset weighting coefficients.
[0031] The technical solution provided by this invention brings at least the following beneficial effects:
[0032] This invention proposes a multimodal fusion strategy that conforms to the laws of human visual cognition. It uses a low-resolution image to provide a global spectral overview and a high-resolution image to supplement local spatial details. Coordinate information serves as a guiding signal, and the two are aligned and fused within a unified framework. This significantly improves the local texture clarity of the reconstructed image while ensuring global structural and spectral consistency.
[0033] This invention achieves continuous resolution enhancement based on a coordinate-guided implicit fusion method. The model is no longer limited by fixed pixel grids and integer magnification, and can query any spatial location and achieve super-resolution reconstruction of hyperspectral images at any magnification (including non-integer multiples), greatly improving the flexibility and practicality of hyperspectral data in multi-scale application scenarios.
[0034] This invention enhances feature representation capabilities through a biomimetic MLP. The constructed MLP module simulates the dendritic input, cell body integration, and axonal output processes of biological neurons, and uses a nonlinear activation function to characterize the excitation / inhibition state of neurons. This enables the network to suppress background noise and highlight key features when processing complex hyperspectral data, thereby enhancing its ability to model nonlinear mapping relationships. Attached Figure Description
[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0036] Figure 1 This is a flowchart illustrating a brain-inspired multimodal image resolution fusion enhancement method according to an embodiment of the present invention;
[0037] Figure 2 This is a schematic diagram of the structure of the biomimetic neuron information processing module (simulating dendrites, cell body, and axon) used in an embodiment of the present invention;
[0038] Figure 3 This is a schematic diagram illustrating synaptic plasticity learning (joint training) under simulated environmental stimuli according to an embodiment of the present invention. Detailed Implementation
[0039] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be described in detail and completely below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Generally, the components of the embodiments of the present invention described and shown in the accompanying drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present invention.
[0040] This invention provides a brain-inspired multimodal image resolution fusion enhancement method, such as... Figure 1 The process includes the following steps:
[0041] Step 1: Data Acquisition and Preprocessing;
[0042] Simulated processing of datasets from different sensors, including cropping, downsampling, and normalization, is performed to obtain low-resolution hyperspectral images of the same target region to be fused. and high-resolution multispectral images In order to prepare for subsequent model training;
[0043] Step 2: Biomimetic variable synaptic information processing;
[0044] When the environment is simple (few spectral bands), only some synaptic connections are activated; when the environment is complex (many spectral bands), all synaptic connections are activated. Through the first kernel input layer, connections with a certain number of spectral bands are generated. Low-resolution hyperspectral images Mapped to a fixed feature dimension Thus, spectral modal characteristics are obtained. .
[0045] Low-resolution hyperspectral images Bicubic interpolation upsampling yields the image Images with consistent resolution , to image and splicing in the channel dimension As a spatial modality, this spatial modality is mapped to the feature dimensions through the second kernel input layer to obtain spatial modality features. .
[0046] spectral modal features Calculate the corresponding spectral latent coding features using a spectral encoder. Spatial modal features Calculate the corresponding spatial latent coding features using a spatial encoder. .
[0047] Representing the center position of a pixel as a coordinate point, Construct corresponding high-resolution multispectral images Normalized continuous coordinate grid with high resolution For the grid Any query coordinates on Find its nearest neighbor coordinates in a low-resolution grid. And calculate coordinate features .
[0048] Step 3: Multimodal implicit fusion based on brain-inspired mechanisms;
[0049] Constructing a multilayer perceptron (MLP) as a brain-inspired implicit fusion module (also known as a multimodal implicit fusion module): The MLP is obtained in step 2. , and The multimodal features are used as input. In the hidden layer, a threshold mechanism similar to the excitation / inhibition of biological neurons is implemented through a nonlinear activation function to suppress noise and highlight effective features. The query coordinates are then given in the output layer. The residual eigenvectors at the specified locations serve as the basis for subsequent spectral reconstruction. Thus, based on all query coordinates... The residual feature vectors are used to obtain the fused feature map, which has a fixed feature dimension. .
[0050] Step 4: Biomimetic variable synaptic information reconstruction;
[0051] It adopts the same "nested kernel + variable synapse" idea as step 2, but with the opposite direction of action, to reconstruct a low-resolution hyperspectral image from a unified feature space. The original spectral dimension corresponding to the number of spectral bands. Through a nested kernel output layer, the feature dimension obtained in step 3 is fixed. The fused feature map is mapped back to the original spectral dimension. This layer also utilizes a nested weight slicing mechanism to dynamically adjust the convolution kernel according to the required number of spectral bands in the output, outputting the final high-resolution spectral residual.
[0052] Step 5: Image reconstruction;
[0053] Compare the final high-resolution spectral residual output from step 4 with the image. The sums are then added together to obtain the final high spatial resolution hyperspectral image.
[0054] In this embodiment of the invention, when training the model parameters of the bionic neuron information processing module (first nested core input layer, second nested core input layer, spectral encoder, spatial encoder), the multimodal implicit fusion module (implemented based on a multilayer perceptron), and the bionic variable synaptic information reconstruction module (nested core output layer), synaptic plasticity simulation and joint training can be adopted to simulate the process by which the human brain continuously adjusts synaptic weight through learning. Training stops when a preset convergence condition is met (the number of training iterations reaches a preset upper limit, or the training loss function value converges). During joint training, a bucketing strategy is used, and the connection weights (i.e., network parameters) between bionic neurons are adjusted through a backpropagation algorithm, enabling the model to adapt to stimuli from different environments (different band numbers, different resolutions), achieving a similar "learning by analogy" and continuous learning ability as the human brain.
[0055] In one embodiment, the brain-inspired multimodal image resolution fusion enhancement method provided by this invention includes the following specific implementation steps:
[0056] Step 1. Construct a visual environment for biomimetic learning (simulated image dataset processing):
[0057] In order to simulate signals of different resolutions and modalities received by biological visual systems in the real world, this invention selects the publicly available CAVE (Computational and Applied Vision Laboratory) dataset, PaviaC (Pavia Centre) dataset, and WashingtonDC (Washington DC Mall) dataset as basic materials and performs fine simulation processing on them.
[0058] First, for the CAVE dataset, 22 images were randomly selected from its 32 scenes as the "visual memory bank" for training, and the remaining 10 images were used as the "new environment" for testing. For the PaviaC dataset, the top-left region was selected as the training set, and the rest was used for testing. For the WashingtonDC dataset, it was similarly divided into non-overlapping training and testing regions to ensure the objectivity of model evaluation.
[0059] Given that it is often difficult to directly obtain perfect high-resolution hyperspectral images (i.e., Ground Truth, or GT) in practical applications, this embodiment employs a simulation generation strategy that conforms to the principles of photophysics. Specifically, during the training process, a large number of overlapping image patches are cropped from selected training images to serve as standard GT images, representing "perfect visual perception." For the CAVE dataset, the patch size is set to [size missing]. For the PaviaC (102 bands) and WashingtonDC (191 bands) datasets, which have more spectral bands, the spatial dimensions are also cropped accordingly. The patch is then processed, preserving its entire spectral band. Subsequently, to generate a low-resolution hyperspectral image simulating the human eye's perception of macroscopic overview, a standard deviation of 0.5 is first used. A Gaussian kernel is used to blur the ground truth (GT) image to simulate the point spread effect of an optical system; then, a scaling factor of 4 is used to downsample the blurred image patch to obtain a spatial size of [missing value]. Low-resolution hyperspectral images.
[0060] Meanwhile, to generate high-resolution multispectral images that simulate the human eye's perception of microscopic details, it is necessary to simulate the response of wide-band sensors. For the CAVE dataset, the camera's spectral response function is introduced to synthesize RGB images; for the PaviaC and WashingtonDC datasets, spectral response functions similar to those of Landsat or remote sensing satellites are used to reduce the dimensionality of hyperspectral data and synthesize the corresponding high-resolution multispectral images. This process simulates the integral response of retinal cone cells or different sensor channels to specific bands of light.
[0061] Finally, the generated input pairs are randomly divided into training and validation sets at a ratio of 80% to 20%. Through this series of operations, a dataset containing low-resolution hyperspectral images is constructed. High-resolution multispectral images And the true value GT ( Paired datasets.
[0062] Step 2. Generate spectral latent coding features using biomimetic variable synaptic information processing. and spatial latent coding features ,as well as:
[0063] Step 2.1: To simulate the human eye's perception process from macroscopic to microscopic, it is first necessary to construct feature representations at different scales. This involves using low-resolution hyperspectral images... As a macroscopic overview input, bicubic interpolation upsampling is used to obtain an image with a macroscopic overview. Images with consistent image size (i.e., resolution) This process can be represented as:
[0064]
[0065] in, It is a bicubic interpolation upsampling function. It is a low-resolution hyperspectral image with a spatial size of And the number of channels is set to the number of spectral bands. , For the real number field; For image The size of the upsampled image is And the number of channels remains constant. Simultaneously, high-resolution multispectral images were acquired. Its space size is The number of channels is ( ).
[0066] To achieve adaptive encoding along the spectral dimension (simulating the brain's plasticity to different environmental stimuli), this step uses a nested kernel as the core component for feature extraction. The nested kernel's input layer maintains a maximum preset number of bands. Nested convolution weights ,in For fixed feature dimensions, This represents the kernel size. It also represents the number of spectral bands for the current sample. Only the first input channel Each channel corresponds to an "activated" synapse, and the effective weights are obtained through slicing. Based on this, convolution operations are performed to adaptively map the variable number of input bands to a fixed feature dimension. No network structure needs to be modified.
[0067] Will Input a spectral encoder to generate spectral latent coding features. Spectral characteristics, also known as spectral features; at the same time and By splicing them together along the channel dimension, we obtain This serves as input to the spatial encoder to generate spatial latent encoded features. Spatial features, or simply spatial features, are expressed as follows;
[0068]
[0069] In the formula, It is a spectral feature (macroscopic overview feature), and its spatial size is The channel number value is set to ; For spatial features (microscopic detail features), its spatial size is And the number of channels remains constant. . This represents a nested kernel layer at the input end, which can adjust according to the number of spectral bands of the current input. Dynamically adjust the convolution kernel weights to map inputs of arbitrary dimensions to fixed feature dimensions. . For spectral coding functions, This indicates its learnable parameters; For spatial coding functions, This indicates its learnable parameters. and Both are implemented using encoder structures based on ResNet or EDSR (Enhanced Deep Super-Resolution network).
[0070] Step 2.2: To achieve resolution enhancement at arbitrary scales (simulating the gaze-point zoom mechanism of the human eye), this invention introduces coordinate modal features. For a target size of... A normalized sampling grid is constructed from the high-resolution image. For any pixel in the target image, its corresponding normalized query coordinates... Obtained through the following transformation:
[0071]
[0072] in, , .Right now , These are the x and y coordinates of the high-resolution image. For each query coordinate... (As a visual gaze point), distance is identified in a low-resolution grid. nearest feature point Furthermore, the fine relative positional offset between the query point and its nearest feature point is calculated and used as the coordinate feature:
[0073]
[0074] Based on coordinate features Features obtained in step 2.1 and These features collectively constitute multimodal characteristics, serving as the complete input for the subsequent bionic neuron information processing module to reconstruct high spatial resolution hyperspectral images. The complete input for the bionic neuron information processing module includes: a macroscopic overview signal based on macroscopic overview features, a microscopic detail signal based on microscopic detail features, and an attention guidance signal based on coordinate features. Processing this complete input enables multimodal information processing that simulates the human cognitive pattern of "holistic to local."
[0075] Step 3. Using the constructed biomimetic neuron information processing module (i.e., multilayer perceptron MLP), perform deep nonlinear fusion on the multimodal features obtained in Step 2:
[0076] Step 3.1: Multimodal dendritic signal convergence, for each query coordinate in the target high-resolution image. (Simulated visual gaze point) To ensure the continuity of spatial perception, this embodiment selects its nearest neighbor point in the low-resolution feature grid. The following three signals are converged as "dendritic inputs" to the input layer of the MLP:
[0077] (1) Macroscopic overview signal: at point Spectral characteristics of samples taken at the location ;
[0078] (2) Microscopic detail signals: at point Spatial characteristics of sampling ;
[0079] (3) Attention-guided signals: coordinate features .
[0080] These three elements are concatenated along the channel dimension to form a complete perceptual feature vector, simulating the comprehensive reception of multi-source information by the dendrites of biological neurons.
[0081] Step 3.2: Cell body information integration and threshold activation, resulting in a concatenated multimodal feature vector. Enter the hidden layer of the MLP. The MLP is composed of... It consists of fully connected layers, and the operations in each layer simulate the integration and threshold filtering of input potentials by the neuron cell body. layer( ) output It can be represented as:
[0082]
[0083] Among them, input features Set as ,Right now ; and The first The weight matrix and bias vector of the layer are used to simulate the strength of synaptic connections; As a nonlinear activation function, this embodiment uses the ReLU function to simulate the excitation / inhibition mechanism of biological neurons, and its mathematical expression is:
[0084]
[0085] in, This represents the function's input parameters.
[0086] According to the above formula, when the sum of weighted signals received by a neuron is less than or equal to 0, its output is forced to 0, and it is in an inhibited state, thus effectively filtering out low-amplitude background noise; only when the signal strength exceeds 0 is the neuron activated and transmits signals, ensuring that key high-frequency effective features such as edges and textures flow in the network. The MLP's final output after layer processing is the query point. Fusion residual feature vector at the location ,in, This represents the fused feature map composed of all query points.
[0087] In this embodiment, the obtained spectral features, spatial features, and coordinate features are mapped by dendritic reception (the mapping parameters include corresponding weights and bias terms) to obtain cell bodies. Then, the mappings of each feature are weighted and integrated to obtain the output value of each neuron, which is then processed by a nonlinear activation function. Excitation / inhibition processing is performed (axonal conduction occurs if the current neuron's output value is greater than 1; otherwise, there is no signal), and then the signal is transmitted to the next layer, ultimately outputting the residual features of each query point, such as... Figure 2 As shown.
[0088] In step 3, the obtained macroscopic overview signal (corresponding to encoded features) is processed by the bionic neuron information processing module. ), and the coding features corresponding to microscopic detail signals ( ) and attention guidance signals (corresponding coordinate features) The coordinates of each query are then merged and output. fused residual feature vector Based on all query coordinates of Obtain the fused feature map .
[0089] Step 4. Adaptive reconstruction of spectral bands: the resulting fused feature map It has fixed feature dimensions To map it back to the original spectral dimensions (i.e., the number of spectral bands). This is input into the output nested core layer. The output nested core layer maintains a maximum preset number of bands. Nested convolution weights The fused feature map obtained in step 3 Based on the target number of output bands, effective convolution kernels are obtained by slicing along the output channel direction. This layer is based on the input image. Number of spectral bands Dynamically change the value of the feature dimension from Transform into Output and image High-resolution spectral residuals with consistent resolution :
[0090]
[0091] in, This indicates that the number of its output channels is configured to be equal to the number of spectral bands. The nested kernel operation. This step is used to ensure that the reconstructed image accurately reproduces the spectral characteristics of the original sensor.
[0092] Step 5. Residual Learning, Image Reconstruction: Introducing a mechanism similar to "detail filling" in biological vision systems, the high-resolution spectral residual image obtained in Step 4 through nested kernel layer decoding at the output end is reconstructed. With low-resolution hyperspectral images Image obtained by bicubic interpolation upsampling Perform pixel-by-pixel addition:
[0093]
[0094] in, This results in the final reconstructed high spatial resolution hyperspectral image. Through this residual learning approach, the model only needs to focus on predicting missing high-frequency textures and edge details, without having to relearn the basic color distribution, which greatly reduces the learning difficulty and improves the reconstruction accuracy.
[0095] In embodiments of the present invention, synaptic plasticity simulation and joint training are as follows: Figure 3 Specifically, it includes:
[0096] (1) In order to solve the problem of different sensor data in the number of spectral bands To address the issue of inconsistent spatial resolution preventing direct batch processing, this invention employs a "bucketing strategy." This involves combining multiple collected heterogeneous datasets... The training samples are divided into buckets based on the number of bands and spatial dimensions, ensuring that the samples within each bucket have the exact same tensor shape (i.e., the same number of bands and the same dimensions). Here, the subscript N represents the number of modes in the dataset. ~ This represents a dataset with N modalities. For example, samples (31 bands) from the CAVE dataset are categorized into the environment. The samples (102 bands) from the PaviaC dataset are categorized into the environment bucket. Samples (band 191) from the Washington DC dataset were categorized into the environment bucket. Environmental bucket , and The multiple environmental stimuli that constitute this embodiment.
[0097] (2) In each iteration of the training loop, perform two-stage sampling:
[0098] 1) First, randomly select a bucket from all available buckets according to a uniform distribution, such as the environment bucket. This simulates the process by which an organism is randomly placed in a specific visual environment.
[0099] 2) Extract a batch of data from the selected bucket. This strategy ensures that the data structure within the same batch is uniform, thus enabling efficient parallel computation; while the batch structure between different iteration steps is highly heterogeneous, forcing the model to adapt to changing environments.
[0100] After the data is input into the network, the nested kernel layers in the network are configured according to the number of spectral bands in the current batch of data. Automatically perform weight slicing operation, activating only the first few elements in the superset weight tensor. Each channel. This is equivalent to the brain dynamically calling upon the appropriate subset of neuronal synapses to participate in computation based on the complexity of the current environment.
[0101] (3) Constructing a system containing A hybrid loss function combining loss and structural similarity (SSIM) loss:
[0102]
[0103] in, Denotes the first-order norm, Represents structural similarity loss. These are the preset weighting coefficients.
[0104] The gradient is calculated using the backpropagation algorithm, and the currently activated synaptic weights (model parameters) are updated. The model's weight channels are divided into three levels based on the number of spectral bands corresponding to their weight channels: minimum number of bands, intermediate number of bands, and maximum number of bands. Three types of synapses are then established accordingly. Figure 3As shown in the diagram, the model consists of three layers: Basic synapses: These correspond to the weights with the minimum number of bands. Regardless of the training environment, these synapses are always activated and updated, thus learning the common spectral-spatial features shared by all environments. Mid-level synapses: These correspond to the weights with a medium increase in the number of bands. They are activated only when the environment complexity reaches a medium level, responsible for capturing advanced features of that type of environment. High-level synapses: These correspond to the weights with the highest increase in the number of bands. They are activated only when processing the most complex high-dimensional data. As training alternates between different environment buckets, the basic synapses are repeatedly optimized, becoming extremely robust; while the mid-level and high-level synapses learn specifically under certain stimuli. Ultimately, this joint training strategy endows the model with plasticity similar to that of a biological brain, enabling it to flexibly call upon synaptic connections of the corresponding levels based on the dimension of the input data, processing never-before-seen sensor data and arbitrary spatial scales.
[0105] To further verify the enhanced performance of the method proposed in this embodiment, four widely accepted evaluation metrics were used to quantify the fusion performance of the trained optimal all-in-one model: PSNR (Peak Signal-to-Noise Ratio), SAM (Spectral Angle Mapping), ERGAS (Earning Error Relative to Overall Dimension), and SSIM (Structural Similarity). Higher PSNR and SSIM values, and lower SAM and ERGAS values, represent better image reconstruction quality. Using the CAVE and PaviaC datasets as the main test objects, the following excellent results were obtained in a standard four-fold super-resolution fusion test: For the CAVE test set, the average PSNR of the method in this embodiment was 45.96 dB, the average SAM was 2.55, the average ERGAS was 1.72, and the average SSIM was 0.994. For the PaviaC test set (remote sensing dataset), the average PSNR of the method in this embodiment was 45.92 dB, the average SAM was 2.02, the average ERGAS was 1.07, and the average SSIM was 0.996. For the Washington DC test set with complex spectral characteristics, the method of this invention achieved an average PSNR of 48.91 dB, an average SAM of 0.76, and an average ERGAS of 1.92. Furthermore, to verify the unique "arbitrary scale" enhancement capability of the method proposed in this embodiment, tests were specifically conducted at non-integer magnifications not previously seen by the model. The results show that at a ×3.2 magnification on the CAVE dataset, the model achieved a PSNR of 45.02 dB, a SAM of 2.48, an ERGAS of 1.81, and an SSIM of 0.993; at the more challenging ×5.7 magnification, the model still maintained a PSNR of 43.49 dB, a SAM of 2.91, an ERGAS of 2.17, and an SSIM of 0.991.
[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
[0107] The above descriptions are merely some embodiments of the present invention. Those skilled in the art can make various modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the scope of protection of the present invention.
Claims
1. A brain-inspired multimodal image resolution fusion enhancement method, characterized in that, Includes the following steps: Step 1: Input a low-resolution hyperspectral image of the target region. and high-resolution multispectral images ; For images Upsampling is performed to obtain the image Images with consistent resolution ; Step 2, input the image through the first nested kernel layer. Mapped to a fixed feature dimension Spectral modal characteristics are obtained; then Spectral latent coding features are obtained through a spectral encoder. ; Image With images The concatenation is performed along the channel dimension, and then mapped to a fixed feature dimension via the second nested kernel input layer. Spatial modal features are obtained; then spatial latent coding features are obtained through a spatial encoder. ; exist Construct the corresponding image above Normalized continuous coordinate grid with high resolution , for grid Any query coordinates on Find its nearest neighbor coordinates in a low-resolution grid. And calculate coordinate features ; Step 3: Process the obtained encoded features through a multimodal implicit fusion module. , and coordinate features Perform fusion processing and output each query coordinate. The residual feature vector, based on all query coordinates The residual feature vectors are used to obtain the fused feature map. ; Step 4: Fuse the feature maps using the nested kernel output layer. Mapping back to image The corresponding number of spectral bands, and output with the image. High-resolution spectral residuals with consistent resolution ; Step 5, process the image and high-resolution spectral residuals Pixel-by-pixel addition is performed to generate a reconstructed high spatial resolution hyperspectral image. ; In step 3, the multimodal implicit fusion module employs a multilayer perceptron, and each hidden layer of the multilayer perceptron is activated by a nonlinear activation function. Simulating the threshold mechanism of excitation / inhibition in biological neurons; nonlinear activation function Using the ReLU function, when the sum of the weighted signals received by a neuron is less than or equal to 0, its output is forced to 0, indicating an inhibited state; when the sum of the weighted signals exceeds 0, the neuron is in an excited state, and its output is propagated backward. Encoding features along the channel dimension... , and coordinate features The concatenated multimodal feature vectors are then processed. Entering the hidden layer MLP of the multilayer perceptron; MLP is composed of It consists of fully connected layers, and the operation of each layer simulates the integration and thresholding of input potentials by the neuron cell body. Layer output Represented as: in, Input features Set as ,Right now ; and The first The weight matrix and bias vector of the layer simulate the strength of synaptic connections; when the sum of the weighted signals received by a neuron is less than or equal to 0, its output is forced to 0, placing it in an inhibited state to filter out low-amplitude background noise; only when the signal strength exceeds 0 is the neuron activated and transmits the signal, ensuring that key high-frequency effective features flow in the network; after The MLP's final output after layer processing is the query point. Fusion residual feature vector at the location ; When training the model parameters for the first nested kernel input layer, the second nested kernel input layer, the spectral encoder, the spatial encoder, the multimodal implicit fusion module, and the nested kernel output layer, the number of spectral bands corresponding to the model's weight channels is divided into three levels: minimum number of bands, medium number of bands, and maximum number of bands. Based on this, three types of synapses are set: Basic synapses: The weighted portion corresponding to the minimum number of bands; regardless of the training environment, these synapses are always activated and updated to learn the common spectral-spatial features shared by all environments. Mid-level synapses: corresponding to the weighted portion of the medium-level band number increment; they are only activated when the environmental complexity reaches a medium level, and are responsible for capturing the advanced features of this type of environment; High-level synapses: the weighted portion corresponding to the highest band number increment; activated only when processing the most complex high-dimensional data; The collected heterogeneous datasets are divided into environmental buckets based on the number of bands and spatial size. Each environmental bucket contains samples with the same number of bands and spatial size. In each iteration of the training loop, a two-stage sampling is performed: an environment bucket is randomly selected from all environment buckets according to a uniform distribution; a batch of training data is drawn from the selected environment bucket. Joint training is performed based on the current batch of training data and the set training loss function; this training loss function includes high-resolution spectral residuals. With reconstructed high spatial resolution hyperspectral images The absolute value loss and structural similarity loss between them.
2. The brain-inspired multimodal image resolution fusion enhancement method as described in claim 1, characterized in that, In step 1, bicubic interpolation is used to process the image. Perform upsampling processing.
3. The brain-inspired multimodal image resolution fusion enhancement method as described in claim 1, characterized in that, The first nested core input layer has the same network structure as the second nested core input layer, and it includes a maximum preset number of bands. Nested convolution weights ,in, The kernel size is the convolution kernel size. For the real number field; based on the image of the current input. Number of spectral bands When performing nested convolution processing, the first channel of the input channel The synapses corresponding to each channel are activated, and the effective weights are obtained through slicing. Based on this, convolution operations are performed to adaptively map the input data to a fixed feature dimension. .
4. The brain-inspired multimodal image resolution fusion enhancement method as described in claim 1, characterized in that, In step 2, the network structures of the spectral encoder and the spatial encoder are the same, and the encoder structure is based on residual network or enhanced deep super-resolution network.
5. The brain-inspired multimodal image resolution fusion enhancement method as described in claim 1, characterized in that, In step 2, query coordinates for: in, , For image Height and width, , , respectively, are the x and y coordinates of the high-resolution multispectral image, and , .
6. The brain-inspired multimodal image resolution fusion enhancement method as described in claim 1, characterized in that, The training loss function is specifically set as follows: in, Denotes the first-order norm. Represents structural similarity loss. These are preset weighting coefficients.