Image lossless fusion method and system based on depth feature extraction

Abstract

The invention relates to the technical field of image optimization, and discloses an image lossless fusion method and system based on depth feature extraction, and the method comprises the following steps: collecting multi-modal input data from different sensors, carrying out the preprocessing of each modal in the multi-modal input data, and carrying out the preprocessing of each modal in the multi-modal input data; the multi-modal input data comprises a visible light image, an infrared image and a depth map, a multi-layer pulse neural network is used for simulating pulse emission characteristics of biological neurons, time sequence feature extraction is performed on the multi-modal input data, and feature representations of different levels are captured. According to the method, deep feature extraction is carried out through the SNN algorithm to carry out image fusion, the problems of feature mutual exclusion and modal conflict when infrared and visible light images are processed through a traditional method can be effectively solved, a concrete image can be converted into an abstract feature, and then the abstract feature is converted into an end-to-end conversion process of the fused image. And the real-time and fidelity requirements in a dynamic scene are met.

Description

Technical Field

[0001] The present invention relates to the field of image optimization, and more specifically, to a method and system for lossless image fusion based on deep feature extraction. Background Art

[0002] With the rapid development of computer vision and deep learning, image fusion technology has been widely used in many fields, such as medical imaging, remote sensing, security monitoring, and autonomous driving.

[0003] Traditional image fusion methods usually rely on manually designed feature extraction and fusion strategies, which are difficult to adapt to complex and changing dynamic scenes, especially when the image types are infrared images and visible light images. The mutual exclusivity of the temperature sensing features of infrared images and the texture features of visible light causes a trust crisis caused by environmental interference due to sensor modal conflicts, making it impossible for the fused image to quickly reach the optimal state. Summary of the invention

[0004] The present invention provides a method and system for lossless image fusion based on deep feature extraction, which solves the technical problems in related technologies.

[0005] The present invention provides a method for lossless image fusion based on deep feature extraction, comprising the following steps:

[0006] S100, input data preparation: collecting multimodal input data from different sensors and preprocessing each modality in the multimodal input data, where the multimodal input data includes a visible light image, an infrared image, and a depth map;

[0007] S200, Bio-Inspired Feature Extraction: Use a multi-layer spiking neural network to simulate the spiking characteristics of biological neurons, extract temporal features from multi-modal input data, and capture feature representations at different levels;

[0008] S300, dynamic weight adjustment mechanism: introduces an adaptive weight adjustment mechanism to dynamically adjust the weight according to the performance of the sensor modality in a specific environment;

[0009] S400, modal conflict detection: implementing a modal conflict detection algorithm to identify conflicts between different sensor modalities, and using the algorithm to analyze the consistency of modal outputs to determine whether there is a conflict;

[0010] S500, feature fusion: Apply an improved fusion strategy. During the fusion process, dynamic weight allocation and confidence-weighted fusion are used to maximize the impact of important modalities on the final result.

[0011] S600, output generation: decoding the compressed features into preliminary image data, reconstructing the image using the pulse timing information, and outputting the final image after performing detail enhancement and color correction.

[0012] Furthermore, the multi-layer spiking neuron network includes an L1 layer and an L2 layer, wherein the L1 layer is used for retinal bipolar cell simulation and the L2 layer is used for ganglion cell simulation.

[0013] Furthermore, in the L1 layer, the membrane potential equation is as follows:

[0014] ;

[0015] in Represents the pulse characteristic output, represents the short-term memory constant, =10ms, represents the edge detection convolution kernel, , represents the membrane potential of neurons in layer L1, represents the rate of change of membrane potential in layer L1 over time, Represents the convolution operation;

[0016] Pulse emission conditions:

[0017] ;

[0018] in represents the high sensitivity threshold, , represents the pulse output of L1 layer at time t, represents the membrane potential of L1 layer at time t;

[0019] In the L2 layer, the dynamic synaptic equation is as follows:

[0020] ;

[0021] in represents the synaptic decay constant, =30ms, where represents the impulse response strength, =0.8, represents the synaptic current of layer L2, Indicates the number of pulses received, represents the timestamp of the kth pulse, Indicates the current time;

[0022] The feature output is as follows:

[0023] ;

[0024] in represents the feature output of the L2 layer, represents the synaptic current of L2 layer at time t.

[0025] Furthermore, in S200, when extracting time series features from multimodal input data, STDP weight adjustment is required, and the weight update rule is as follows:

[0026] ;

[0027] ;

[0028] in ,in represents the learning rate, =20ms, where represents a time-dependent window, where Indicates the pulse timing difference, represents the weight change, represents the pulse output of the front neuron i, represents the pulse output of neuron j, represents the spike time of the front neuron i, represents the spike time of the later neuron j;

[0029] Constraints:

[0030] ;

[0031] in Indicates the upper limit of weight, =2.5, represents the connection weight between neurons i and j, Represents the operation of restricting weights to within a specified range.

[0032] Furthermore, in S200, the pulse density gating mechanism is used, and the pulse density is calculated as follows:

[0033] ;

[0034] ;

[0035] in represents the adjustment coefficient, =1.5, represents the base density, =0.6, represents the Sigmoid function, represents the time window length, =8, represents the total number of neurons, , represents the dynamic fusion weight, represents the pulse output of the nth neuron in the L1 layer at time t, represents the pulse density of L1 layer;

[0036] ;

[0037] in represents the fused features, Represents the feature output of the L1 layer; Represents the feature output of the L2 layer;

[0038] Pooling by downsampling spikes:

[0039] ;

[0040] in represents the pyramid level, ,in represents the scale attenuation coefficient, , represents the features after downsampling at the lth layer, represents the maximum pooling operation, Indicates the pooling kernel size;

[0041] Concatenate by upsampling features:

[0042] ;

[0043] in Indicates the channel splicing operation, represents the feature pyramid output, represents the upsampling operation, Indicates the upsampling scale factor.

[0044] Furthermore, in S300, the following steps are specifically included:

[0045] S310, sensor output confidence monitoring: calculating the confidence of each sensor modal output;

[0046] S320, initial weight allocation: assigning an initial weight to each modality based on historical data of sensor performance;

[0047] S330, dynamic weight update: updating weights in real time based on sensor output confidence;

[0048] S340, Bio-inspired Feedback Regulation Mechanism: Introducing conflict detection based feedback regulation to optimize weight oscillation problem.

[0049] Furthermore, in S400, the calculation formula of the modal conflict detection algorithm is as follows:

[0050] Conflict determination conditions:

[0051] ;

[0052] in represents the weight difference threshold, =0.4, is the weight of the jth neuron at time t, is the weight of the ith neuron at time t, Indicates the conflict determination result;

[0053] Construct a pulse gating function:

[0054] ;

[0055] in represents the pulse density of the ith neuron, where represents the pulse density of the jth neuron, Indicates the zero-proof constant, Represents the conflict adjustment coefficient.

[0056] Furthermore, in S500, the calculation formula for dynamic weight allocation is as follows:

[0057] ;

[0058]

[0059] in represents the modified dynamic weight, represents the total number of neurons;

[0060] The calculation formula of confidence weighted fusion is as follows:

[0061] ;

[0062] ;

[0063] in represents the enhancement function of the feature, represents the learnable convolution kernel, , represents the bias term, , represents the fused features, where represents attention weighting;

[0064] ;

[0065] in represents the affine transformation network, represents the parameters learned in STDP, represents the pulse characteristic output of the i-th neuron, , Represents the feature output of the ith neuron after alignment.

[0066] Furthermore, in S600, the calculation formula of the coding network is as follows:

[0067] ;

[0068] in represents the deconvolution kernel, ,in =3 corresponds to RGB channels, represents the decoding bias term, , represents the ReLU function, represents the decoded features, represents the compressed features of the final output, Represents the convolution operation;

[0069] The calculation formula for image reconstruction is as follows:

[0070] ;

[0071] ;

[0072] in represents the time decay function, Indicates the decay time. =25ms, represents the reconstruction time step, =15, represents the reconstructed image;

[0073] The calculation formula for detail enhancement is as follows:

[0074] ;

[0075] in represents the Laplace sharpening kernel, Indicates increased strength, =0.3, represents the enhanced image;

[0076] The color correction formula is as follows:

[0077] ;

[0078] in represents the color correction matrix, , represents the image after color correction;

[0079] The final image is calculated as follows:

[0080] ;

[0081] in Represents the final output image.

[0082] The present invention also proposes a system for lossless image fusion based on deep feature extraction, which executes the steps in the aforementioned method for lossless image fusion based on deep feature extraction, including:

[0083] Multimodal acquisition and preprocessing module: collects multimodal input data from different sensors and preprocesses each mode in the multimodal input data;

[0084] Biological pulse feature coding module: hierarchical pulse neuron network realizes retinal bionic coding, extracts multi-scale spatiotemporal feature pyramid, and constructs cross-layer feature association of pulse density gating;

[0085] Dynamic weight optimization module: introduces an adaptive weight adjustment mechanism to dynamically adjust the weight according to the performance of the sensor modality in a specific environment;

[0086] Modal conflict detection module: implements the modal conflict detection algorithm to identify conflicts between different sensor modalities and uses the algorithm to analyze the consistency of modal outputs to determine whether there is a conflict;

[0087] Cross-modal fusion decision module: Applying an improved fusion strategy, during the fusion process, using dynamic weight allocation and confidence-weighted fusion to maximize the impact of important modalities on the final result;

[0088] Pulse inverse mapping reconstruction module: decodes the compressed features into preliminary image data, reconstructs the image using pulse timing information, and outputs the final image after detail enhancement and color correction.

[0089] The beneficial effects of the present invention are:

[0090] The present invention uses the SNN algorithm to perform deep feature extraction and image fusion, which can effectively solve the feature mutual exclusion and modal conflict problems faced by traditional methods when processing infrared and visible light images. It can convert concrete images into abstract features, and then convert abstract features into fused images through an end-to-end conversion process, which meets the real-time and fidelity requirements in dynamic scenes, improves the quality and reliability of fused images, and ensures that the best effect is quickly achieved in dynamic and complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] Figure 1 It is a flow chart of a lossless image fusion method based on deep feature extraction proposed by the present invention;

[0092] Figure 2 It is a structural block diagram of an image lossless fusion system based on deep feature extraction proposed by the present invention;

[0093] Figure 3 It is the key data flow verification table in the example proposed by the present invention.

[0094] In the figure: 101, multimodal acquisition and preprocessing module; 102, biological pulse feature encoding module; 103, dynamic weight optimization module; 104, modal conflict detection module; 105, cross-modal fusion decision module; 106, pulse inverse mapping reconstruction module. DETAILED DESCRIPTION

[0095] The subject matter described herein will now be discussed with reference to example embodiments. It should be understood that the discussion of these embodiments is only to enable those skilled in the art to better understand and implement the subject matter described herein, and the functions and arrangements of the elements discussed may be changed without departing from the scope of protection of the contents of this specification. Each example may omit, replace or add various processes or components as needed. In addition, the features described relative to some examples may also be combined in other examples.

[0096] like Figure 1 As shown, a lossless image fusion method based on deep feature extraction includes the following steps:

[0097] S100, input data preparation: collect multimodal input data from different sensors (including visible light images, infrared images, depth maps, etc.).

[0098] Preprocessing is also required for each modality, including normalization, denoising, and feature extraction.

[0099] In one embodiment of the present invention, the following steps are specifically included:

[0100] S110, Multimodal Data Acquisition: Input Visible Light Image , infrared image , depth map ;

[0101] Perform time synchronization error on the above input image ,in is the minimum sensor frame rate, and the spatial resolution is unified as ;

[0102] S120, Mode Specific Normalization

[0103] Visible light image:

[0104] Execution formula:

[0105] ;

[0106] in and Respectively represent channels The mean and standard deviation of and Represent the horizontal and vertical coordinates of the image respectively, represents the normalized visible light image;

[0107] Output range: ;

[0108] Infrared image:

[0109] Dynamic Range Compression:

[0110] ;

[0111] in Indicates the infrared sensor range, where , and Respectively represent the maximum and minimum values ​​of the infrared sensor range, Indicates location Infrared image pixel value at ;

[0112] Depth map:

[0113] Distance Normalization:

[0114] ;

[0115] in Indicates the maximum detection distance, represents the scaling factor, , Represents the normalized depth value, Represents the depth value at position (x, y);

[0116] S130, noise suppression processing

[0117] Visible light image:

[0118] Gaussian-bilateral joint filtering:

[0119] ;

[0120] in represents the Gaussian kernel, , represents bilateral filtering, where the spatial domain , represents the weight coefficient, represents the denoised visible light image;

[0121] Infrared image:

[0122] Non-local means denoising:

[0123] ;

[0124] ;

[0125] in represents the denoised infrared image, Indicates location The infrared image pixel value at Indicates A 7×7 image patch centered at represents the smoothing parameter, , represents the normalization constant, ensuring that the weights sum to 1, represents the set of pixels within the search window, Indicates location The weight of the pixel at ;

[0126] S140, pulse feature pre-extraction

[0127] Pulse Coded Network (PCN):

[0128] Membrane potential equation:

[0129] ;

[0130] in represents the membrane time constant, , represents the encoded convolution kernel, , represents the neuronal membrane potential, represents the rate of change of membrane potential over time, represents the denoised input image;

[0131] Pulse trigger conditions:

[0132] ;

[0133] in Indicates the pulse trigger threshold, =0.8, represents the pulse output at time t, represents the membrane potential at time t;

[0134] Output features:

[0135] ;

[0136] in represents the time decay factor, , represents the time step, =10, Indicates pulse characteristic output;

[0137] S200, Bio-Inspired Feature Extraction: Feature Extraction Using Spiking Neural Networks (SNN).

[0138] Among them, SNN can simulate the pulse emission characteristics of biological neurons, extract timing features, and design multi-layer pulse neuron networks to capture feature representations at different levels.

[0139] In one embodiment of the present invention, the following steps are specifically included:

[0140] S210, layered pulse signature coding

[0141] Input data: preprocessed pulse characteristics ;

[0142] Level division:

[0143] L1 layer (retinal bipolar cell simulation):

[0144] Membrane potential equation:

[0145] ;

[0146] in represents the short-term memory constant, =10ms, represents the edge detection convolution kernel, , represents the membrane potential of neurons in layer L1, represents the rate of change of membrane potential in layer L1 over time, Represents the convolution operation;

[0147] Pulse emission conditions:

[0148] ;

[0149] in represents the high sensitivity threshold, , represents the pulse output of L1 layer at time t, represents the membrane potential of L1 layer at time t;

[0150] Layer L2 (ganglion cell simulation):

[0151] Dynamic synapse equation:

[0152] ;

[0153] in represents the synaptic decay constant, =30ms, where represents the impulse response strength, =0.8, represents the synaptic current of layer L2, Indicates the number of pulses received, represents the timestamp of the kth pulse, Indicates the current time;

[0154] Feature output:

[0155] ;

[0156] in represents the feature output of the L2 layer, represents the synaptic current of L2 layer at time t;

[0157] S220, timing-dependent feature enhancement:

[0158] Through STDP weight adjustment, the weight update rule is as follows:

[0159] ;

[0160] ;

[0161] in ,in represents the learning rate, =20ms, where represents a time-dependent window, where Indicates the pulse timing difference, represents the weight change, represents the pulse output of the front neuron i, represents the pulse output of neuron j, represents the spike time of the front neuron i, represents the spike time of the later neuron j;

[0162] Constraints:

[0163] ;

[0164] in Indicates the upper limit of weight, =2.5, represents the connection weight between neurons i and j, Represents an operation that limits weights to a specified range;

[0165] S230, cross-level feature fusion:

[0166] Using the pulse density gating mechanism, the pulse density is calculated as follows:

[0167] ;

[0168] ;

[0169] in represents the adjustment coefficient, =1.5, represents the base density, =0.6, represents the Sigmoid function, represents the time window length, =8, represents the total number of neurons, , represents the dynamic fusion weight, represents the pulse output of the nth neuron in the L1 layer at time t, represents the pulse density of L1 layer;

[0170] The final fusion output:

[0171] ;

[0172] in represents the cross-level fusion features, , Represents the feature output of the L1 layer; Represents the feature output of the L2 layer;

[0173] S240, multi-scale feature pyramid construction:

[0174] Downsampling Pulse Pooling:

[0175] ;

[0176] in represents the pyramid level, ,in represents the scale attenuation coefficient, , represents the features after downsampling at the lth layer, represents the maximum pooling operation, Indicates the pooling kernel size;

[0177] Upsampling feature concatenation:

[0178] ;

[0179] in Indicates the channel splicing operation, represents the feature pyramid output, , represents the upsampling operation, Indicates the upsampling scale factor;

[0180] S300, dynamic weight adjustment mechanism: introduces an adaptive weight adjustment mechanism to dynamically adjust the weight according to the performance of the sensor modality in a specific environment;

[0181] The confidence of the sensor output is monitored, and the weights are updated in real time based on the relative importance of the modes. Bio-inspired mechanisms such as feedback regulation based on conflict detection are used to optimize the weight oscillation problem.

[0182] In one embodiment of the present invention, the following steps are specifically included:

[0183] S310, sensor output confidence monitoring: calculating the confidence of each sensor modal output;

[0184] The calculation formula is as follows:

[0185] ;

[0186] in, is the confidence of the i-th sensor mode, is the output score of the i-th mode, is the output score of the j-th mode, is the total number of modes.

[0187] S320, initial weight allocation: assigning an initial weight to each modality based on historical data of sensor performance;

[0188] The calculation formula is as follows:

[0189] ;

[0190] in, is the initial weight of the i-th sensor, is the confidence of the i-th sensor mode, is the confidence of the kth sensor mode, is the total number of modes;

[0191] S330, dynamic weight update: updating weights in real time based on sensor output confidence;

[0192] The calculation formula is as follows:

[0193] ;

[0194] in, is the weight of the ith sensor at time t, is the learning rate, controlling the magnitude of the update, is the weight of the i-th sensor at time t-1;

[0195] S340, Bio-inspired Feedback Regulation Mechanism: Introducing conflict detection-based feedback regulation to optimize weight oscillation problem;

[0196] The calculation formula is as follows:

[0197] ;

[0198] ;

[0199] in is the conflict measure of the i-th mode, is the intensity coefficient of feedback regulation.

[0200] S400, modal conflict detection: implementing a modal conflict detection algorithm to identify conflicts between different sensor modalities;

[0201] Use statistical methods or deep learning algorithms to analyze the consistency of modal outputs and determine if there is a conflict.

[0202] In one embodiment of the present invention, the following steps are specifically included:

[0203] S410, feature space consistency measure

[0204] Input data: Cross-level fusion features from S230 ;

[0205] Define the inter-modality feature similarity matrix ,in:

[0206] ;

[0207] ;

[0208] ;

[0209] in and represents the pulse features extracted from different modes, represents the pulse characteristic output of the i-th mode, represents the pulse feature output of the jth mode, N represents the total number of modes (for example, N=3 corresponds to RGB / IR / Depth), Represents the feature similarity between the i-th modality and the j-th modality;

[0210] S420, pulse timing synchronization analysis

[0211] Input data: Pulse train from S210

[0212] Synchronicity indicators:

[0213] ;

[0214] in and Respectively represent the pulse time of mode a and b in the i-th and j-th neurons, =15ms, represents the synchronization tolerance window, Indicates the time window length;

[0215] S430, dynamic confidence conflict detection

[0216] Input parameters: Dynamic weight from S330 ;

[0217] Conflict determination conditions:

[0218] ;

[0219] in represents the weight difference threshold, =0.4, is the weight of the jth neuron at time t, Indicates the conflict determination result;

[0220] S440, conflict resolution based on pulse competition suppression:

[0221] Construct a pulse gating function:

[0222] ;

[0223] in represents the pulse density of the ith neuron, where represents the pulse density of the jth neuron, Indicates the zero-proof constant, represents the conflict adjustment coefficient;

[0224] S500, Feature Fusion: Apply improved fusion strategies such as weighted averaging, feature concatenation, or neural network-based fusion methods.

[0225] During the fusion process, dynamic weights and modal confidence are used for weighting to ensure that the impact of important modes on the final result is maximized.

[0226] In one embodiment of the present invention, the following steps are specifically included:

[0227] S510, dynamic weight allocation

[0228] Input parameters: Dynamic weight of S330 , conflict adjustment coefficient of S440 ;

[0229] Weight correction formula:

[0230] ;

[0231] in represents the modified dynamic weight;

[0232] Constraints:

[0233] ;

[0234] in represents the total number of neurons;

[0235] S520, spatiotemporal feature alignment: input S240 multi-scale feature pyramid and pulse characteristics of S140 ;

[0236] Alignment transformation:

[0237] ;

[0238] in represents the affine transformation network, represents the parameters learned by STDP, represents the pulse characteristic output of the i-th neuron, , Represents the feature output of the i-th neuron after alignment;

[0239] S530, confidence weighted fusion:

[0240] Fusion formula:

[0241] ;

[0242] ;

[0243] in represents the enhancement function of the feature, represents the learnable convolution kernel, , represents the fusion bias term, , Represents the fused features;

[0244] S540, pulse feedback optimization:

[0245] Input signal: S210 pulse emission sequence

[0246] Optimization objective function:

[0247] ;

[0248] in and denote the first and second balance coefficients respectively, =0.8, =0.2, represents the real fusion target of the acquisition, represents the gradient operator, Represents a multi-scale network;

[0249] S550, multimodal feature compression

[0250] Dimensionality reduction operation:

[0251] ;

[0252] in represents the compressed matrix, , =128, output dimension compression ratio , represents the compressed features of the final output, , Represents a flattening operation. represents the dimension reduction weight parameter;

[0253] S600, output generation: decoding the compressed features into preliminary image data, reconstructing the image using the pulse timing information, and outputting the final image after performing detail enhancement and color correction;

[0254] In one embodiment of the present invention, the following steps are specifically included:

[0255] S610, pulse feature decoding: input S550 compression feature To the decoding network;

[0256] The calculation formula of the decoding network is as follows:

[0257]

[0258] in represents the deconvolution kernel, ,in =3 corresponds to RGB channels, represents the decoding bias term, , represents the ReLU function, represents the decoded features, Represents the convolution operation;

[0259] S620, performing image reconstruction by pulse inverse mapping;

[0260] The calculation formula for image reconstruction is as follows:

[0261] ;

[0262] ;

[0263] in represents the time decay function, Indicates the decay time. =25ms, represents the reconstruction time step, =15, represents the reconstructed image;

[0264] S630, image post-processing: performing detail enhancement and color correction on the reconstructed image;

[0265] Detail Enhancement:

[0266] ;

[0267] in represents the Laplace sharpening kernel, Indicates increased strength, =0.3, represents the enhanced image, where represents attention weighting;

[0268] Color Correction:

[0269] ;

[0270] in represents the color correction matrix, , represents the image after color correction;

[0271] S640, outputting a final image: the output format of the final image is a lossless PNG image;

[0272] ;

[0273] in Represents the final output image. The output format of the image is lossless PNG image (H×W×3, 8 bits / channel). represents the Sigmoid function, Represents the operation of restricting weights to within a specified range.

[0274] The final output image of the above output also needs quality verification:

[0275] ;

[0276] in Indicates brightness consistency, , Indicates contrast preservation, , and are the squares of the standard deviations of the pixel values ​​x and y in the image, respectively. and Respectively represent the mean of the pixel value x in the image and the pixel value y in the image, represents the covariance.

[0277] Through the above image lossless fusion method, the following SF-Net full process example is given: multimodal fusion of night vision driving scenes;

[0278] Step 1, input data preparation:

[0279] Input visible light image IRGB (1280×720 pixels, 30fps), infrared image IIR (640×480 pixels, 25fps), lidar point cloud D (100,000 points / frame);

[0280] Preprocessing:

[0281] Visible light normalization:

[0282] =( −[125.3,116.2,110.1]) / [63.0,61.5,65.2];

[0283] Infrared dynamic compression: =( −500) / 3500;

[0284] Depth map generation: =Point Cloud Depth(D,50m), where Point Cloud Depth represents point cloud extraction;

[0285] Step 2, bio-inspired feature extraction:

[0286] Pulse coding:

[0287] L1 layer output pulse sequence ;

[0288] Membrane potential parameters: =8ms, =1.5;

[0289] Feature Pyramid:

[0290] Multi-scale features ;

[0291] Step 3, dynamic weight adjustment:

[0292] Initial weights: =[0.45(RGB),0.35(IR),0.20(D)];

[0293] After dynamic update: =[0.62,0.28,0.10] (infrared sensor fails temporarily);

[0294] Step 4: Modal conflict detection:

[0295] Feature Similarity: =0.32 (significant difference in low light);

[0296] Conflict determination: ∣=0.34> =0.3, it is determined that a conflict exists;

[0297] Step 5, feature fusion:

[0298] Modified weight: =[0.58,0.32,0.10] (pulse competitive inhibition modulation);

[0299] Fusion output: ;

[0300] Step 6, output generation: night vision fusion effect comparison (left: visible light input, middle: infrared input, right: fusion result);

[0301] Quality indicator: SSIM=0.927 (benchmark value 0.913).

[0302] In one embodiment of the present invention, the result of the key data flow verification in the above example is as follows: Figure 3 As shown in the figure, this example successfully achieved pedestrian recognition at a distance of 200 meters in a real vehicle test (missed detection rate < 0.1%), verifying the effectiveness of the process.

[0303] like Figure 2 As shown, the present invention also proposes an image lossless fusion system based on deep feature extraction, comprising the following modules:

[0304] Multimodal acquisition and preprocessing module 101: collects multimodal input data from different sensors and preprocesses each mode in the multimodal input data;

[0305] Biological pulse feature coding module 102: hierarchical pulse neuron network realizes retinal bionic coding, extracts multi-scale spatiotemporal feature pyramid, and constructs pulse density gated cross-layer feature association;

[0306] Dynamic weight optimization module 103: introduces an adaptive weight adjustment mechanism to dynamically adjust the weight according to the performance of the sensor modality in a specific environment;

[0307] Modal conflict detection module 104: implements a modal conflict detection algorithm to identify conflicts between different sensor modalities, and uses the algorithm to analyze the consistency of modal outputs to determine whether there is a conflict;

[0308] Cross-modal fusion decision module 105: applying an improved fusion strategy, using dynamic weight allocation and confidence-weighted fusion during the fusion process to maximize the impact of important modalities on the final result;

[0309] The pulse inverse mapping reconstruction module 106 decodes the compressed features into preliminary image data, reconstructs the image using the pulse timing information, performs detail enhancement and color correction, and then outputs the final image.

[0310] An embodiment of the present invention further provides a storage medium storing non-temporary computer-readable instructions for executing one or more steps in the aforementioned image lossless fusion method based on deep feature extraction.

[0311] The computer program may be stored / distributed on suitable media, such as optical storage media or solid-state media supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems.

[0312] The embodiments of the present invention are described above, but the present invention is not limited to the above-mentioned specific implementation modes. The above-mentioned specific implementation modes are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms, all of which are within the protection of the present invention.

Claims

1. A lossless image fusion method based on deep feature extraction, characterized in that: The following steps are involved: S100, input data preparation: collecting multimodal input data from different sensors and preprocessing each modality in the multimodal input data, where the multimodal input data includes a visible light image, an infrared image, and a depth map; S200, Bio-Inspired Feature Extraction: Use a multi-layer spiking neural network to simulate the spiking characteristics of biological neurons, extract temporal features from multi-modal input data, and capture feature representations at different levels; S300, dynamic weight adjustment mechanism: introduces an adaptive weight adjustment mechanism to dynamically adjust the weight according to the performance of the sensor modality in a specific environment; S400, modal conflict detection: implementing a modal conflict detection algorithm to identify conflicts between different sensor modalities, and using the algorithm to analyze the consistency of modal outputs to determine whether there is a conflict; S500, feature fusion: Apply an improved fusion strategy. During the fusion process, dynamic weight allocation and confidence-weighted fusion are used to maximize the impact of important modalities on the final result. S600, output generation: decoding the compressed features into preliminary image data, reconstructing the image using the pulse timing information, and outputting the final image after performing detail enhancement and color correction.

2. The image lossless fusion method based on deep feature extraction according to claim 1 is characterized in that: The multi-layer spiking neuron network includes an L1 layer and an L2 layer, wherein the L1 layer is used for simulating retinal bipolar cells and the L2 layer is used for simulating ganglion cells.

3. The image lossless fusion method based on deep feature extraction according to claim 2 is characterized in that: In the L1 layer, the membrane potential equation is as follows: ； in Represents the pulse characteristic output, represents the short-term memory constant, =10ms, represents the edge detection convolution kernel, , represents the membrane potential of neurons in layer L1, represents the rate of change of membrane potential in layer L1 over time, Represents the convolution operation; Pulse emission conditions: ； in represents the high sensitivity threshold, , represents the pulse output of L1 layer at time t, represents the membrane potential of L1 layer at time t; In the L2 layer, the dynamic synaptic equation is as follows: ； in represents the synaptic decay constant, =30ms, where represents the impulse response strength, =0.8, represents the synaptic current of layer L2, Indicates the number of pulses received, represents the timestamp of the kth pulse, Indicates the current time; The feature output is as follows: ； in represents the feature output of the L2 layer, represents the synaptic current of L2 layer at time t.

4. The image lossless fusion method based on deep feature extraction according to claim 3 is characterized in that: In S200, when extracting time series features from multimodal input data, STDP weight adjustment is required, and the weight update rule is as follows: ； ； in ,in represents the learning rate, =20ms, where represents a time-dependent window, where Indicates the pulse timing difference, represents the weight change, represents the pulse output of the front neuron i, represents the pulse output of neuron j, represents the spike time of the front neuron i, represents the spike time of the later neuron j; Constraints: ； in Indicates the upper limit of weight, =2.5, represents the connection weight between neurons i and j, Represents the operation of restricting weights to within a specified range.

5. The image lossless fusion method based on deep feature extraction according to claim 4 is characterized in that: In S200, the pulse density gating mechanism is used, and the pulse density is calculated as follows: ； ； in represents the adjustment coefficient, =1.5, represents the base density, =0.6, represents the Sigmoid function, represents the time window length, =8, represents the total number of neurons, , represents the dynamic fusion weight, represents the pulse output of the nth neuron in the L1 layer at time t, represents the pulse density of L1 layer; ； in represents the fused features, Represents the feature output of the L1 layer; Represents the feature output of the L2 layer; By downsampling pulse pooling: ； in represents the pyramid level, ,in represents the scale attenuation coefficient, , represents the features after downsampling at the lth layer, represents the maximum pooling operation, Indicates the pooling kernel size; Concatenate by upsampling features: ； in Indicates the channel splicing operation, represents the feature pyramid output, represents the upsampling operation, Indicates the upsampling scale factor.

6. The image lossless fusion method based on deep feature extraction according to claim 5 is characterized in that: In S300, the following steps are specifically included: S310, sensor output confidence monitoring: calculating the confidence of each sensor modal output; S320, initial weight allocation: assigning an initial weight to each modality based on historical data of sensor performance; S330, dynamic weight update: updating weights in real time based on sensor output confidence; S340, Bio-inspired Feedback Regulation Mechanism: Introducing conflict detection based feedback regulation to optimize weight oscillation problem.

7. The image lossless fusion method based on deep feature extraction according to claim 6 is characterized in that: In S400, the calculation formula of the modal conflict detection algorithm is as follows: Conflict determination conditions: ； in represents the weight difference threshold, =0.4, is the weight of the jth neuron at time t, is the weight of the ith neuron at time t, Indicates the conflict determination result; Construct a pulse gating function: ； in represents the pulse density of the ith neuron, where represents the pulse density of the jth neuron, Indicates the zero-proof constant, Represents the conflict adjustment coefficient.

8. The image lossless fusion method based on deep feature extraction according to claim 7 is characterized in that: In S500, the calculation formula for dynamic weight allocation is as follows: ； ； in represents the modified dynamic weight, represents the total number of neurons; The calculation formula of confidence weighted fusion is as follows: ； ； in represents the enhancement function of the feature, represents the learnable convolution kernel, , represents the bias term, , represents the fused features, where represents attention weighting; ； in represents the affine transformation network, represents the parameters learned in STDP, represents the pulse characteristic output of the i-th neuron, , Represents the feature output of the ith neuron after alignment.

9. The image lossless fusion method based on deep feature extraction according to claim 8, characterized in that: In S600, the calculation formula of the coding network is as follows: ； in represents the deconvolution kernel, ,in =3 corresponds to RGB channels, represents the decoding bias term, , represents the ReLU function, represents the decoded features, represents the compressed features of the final output, Represents the convolution operation; The calculation formula for image reconstruction is as follows: ； ； in represents the time decay function, Indicates the decay time. =25ms, represents the reconstruction time step, =15, represents the reconstructed image; The calculation formula for detail enhancement is as follows: ； in represents the Laplace sharpening kernel, Indicates increased strength, =0.3, represents the enhanced image; The color correction formula is as follows: ； in represents the color correction matrix, , represents the image after color correction; The final image is calculated as follows: ； in Represents the final output image.

10. A lossless image fusion system based on deep feature extraction, characterized in that: Executing the steps in the method for lossless image fusion based on deep feature extraction as described in any one of claims 1 to 9, comprising: Multimodal acquisition and preprocessing module (101): collects multimodal input data from different sensors and preprocesses each mode in the multimodal input data; Biological pulse feature coding module (102): hierarchical pulse neuron network realizes retinal bionic coding, extracts multi-scale spatiotemporal feature pyramid, and constructs pulse density-gated cross-layer feature association; Dynamic weight optimization module (103): introduces an adaptive weight adjustment mechanism to dynamically adjust the weight according to the performance of the sensor modality in a specific environment; Modal conflict detection module (104): implements a modal conflict detection algorithm to identify conflicts between different sensor modalities, and uses the algorithm to analyze the consistency of modal outputs to determine whether there is a conflict; Cross-modal fusion decision module (105): applies an improved fusion strategy. During the fusion process, dynamic weight allocation and confidence-weighted fusion are used to maximize the impact of important modalities on the final result. Pulse inverse mapping reconstruction module (106): decodes the compressed features into preliminary image data, reconstructs the image using pulse timing information, performs detail enhancement and color correction, and then outputs the final image.

Images