Inter-frame image auxiliary information generation method based on pulse diffusion model

Abstract

A pulse diffusion model-based inter-frame image auxiliary information generation method belongs to the field of image data processing, and comprises the following steps of: 1, generating preliminary auxiliary information based on the similarity of inter-frame images; 2, the initial auxiliary information is coded through SNN, the activity mode of biological neurons is simulated, and the time sequence processing capacity of the auxiliary information is optimized; 3, gradually injecting noise into the auxiliary information coded by the pulse neural network by adopting a diffusion model; 4, recovering high-quality auxiliary information from the auxiliary information containing the noise through a reverse process of the diffusion model, gradually eliminating the noise and generating optimized auxiliary information; and 5, combining the optimized auxiliary information with the local features of the video blocks, and applying a deconvolution image processing technology to complete efficient transmission and decoding reconstruction of compressed video data. According to the method, more accurate auxiliary information is generated by simulating the diffusion process of the video signal in time and space dimensions, the video compression efficiency is improved, and the transmission performance is optimized.

Description

Technical Field

[0001] The invention belongs to the field of image data processing, and in particular relates to a method for generating auxiliary information of inter-frame images based on a pulse diffusion model. Background Art

[0002] Image data processing technology has been widely used in the generation of inter-frame image auxiliary information, especially in the fields of video surveillance, video conferencing and streaming services, which significantly improves the transmission efficiency and quality of video data. In video surveillance systems, distributed image processing ensures that remote cameras can transmit high-quality image data in real time by reducing bandwidth requirements. In video conferencing, this technology optimizes the use of network resources and ensures the stability and smoothness of multi-party communication. In streaming services, distributed image processing can maintain video quality and reduce video freezes and buffering under unstable network conditions.

[0003] In the process of generating auxiliary information between frames, auxiliary information generation is a key link in image processing. Auxiliary information generation refers to the creation and transmission of data describing the relationship between the parts of the image during image processing and encoding to help the decoder reconstruct the complete video. Typically, image data is divided into multiple coding units, and auxiliary information is generated using prediction techniques. Auxiliary information can include motion vectors, residuals, or other prediction errors to describe the dependencies between coding units. This information plays a vital role in the decoding stage, optimizing encoding and transmission efficiency by reducing data redundancy, improving compression ratio, and ensuring image reconstruction quality during decoding.

[0004] Auxiliary information generation has wide applicability in image processing and can adapt to different network environments and data loss conditions to ensure the effective use of transmission bandwidth and the stability of image quality. Existing auxiliary information generation methods are mainly divided into two categories: prediction error coding and reference frame difference coding. Prediction error coding generates auxiliary information for accurate recovery of lost data by modeling the prediction error of video frames. Its advantage is that it can efficiently compress data and maintain high reconstruction quality, but there may be problems of high computational complexity when dealing with complex dynamic scenes. Reference frame difference coding generates auxiliary information by comparing the difference between the current frame and the reference frame. It is suitable for real-time transmission and decoding scenarios and can stably transmit data under different network conditions, but it may cause error propagation problems in the case of multi-frame dependence. Summary of the invention

[0005] The purpose of the present invention is to propose a method for generating auxiliary information of inter-frame images based on a pulse diffusion model, which generates more accurate auxiliary information by simulating the diffusion process of video signals in time and space dimensions, aiming to improve video compression efficiency, optimize transmission performance, and enhance the robustness of video decoding in complex environments.

[0006] The technical solution adopted by the present invention is: The method for generating auxiliary information of inter-frame images based on a pulse diffusion model comprises the following steps: Step 1: Initial auxiliary information generation: Based on the similarity of images between frames, traditional compression algorithms are used to generate preliminary auxiliary information as the basis for subsequent optimization processing; this initial step can quickly generate preliminary predictions and reduce the computational burden of subsequent processing.

[0007] Step 2: Pulse neural network encoding: The initial auxiliary information is encoded through a spiking neural network (SNN), simulating the activity pattern of biological neurons and optimizing the timing processing capability of the auxiliary information, thereby improving the accuracy of data representation and the temporal and spatial correlation; Step 3: Noise injection based on diffusion model: A diffusion model is used to gradually inject noise into the auxiliary information after the pulse neural network is encoded, simulating random disturbances in the real environment, enhancing the robustness of the generated auxiliary information in complex scenarios, and ensuring that it can cope with external interference during transmission or decoding. Step 4: Optimize auxiliary information decoding: Through the reverse process of the diffusion model, high-quality auxiliary information is recovered from the noisy auxiliary information, and the noise is gradually eliminated to generate optimized auxiliary information; The optimized auxiliary information is reversely decoded through a spiking neural network to restore it to an intermediate result with physical meaning, ensuring the stability and accuracy of the decoding process and providing reliable auxiliary information for subsequent video frame reconstruction. Step 5: Reconstruct video frame generation: Combining the optimized auxiliary information with the local features of the video block, deconvolution image processing technology is applied to reconstruct the final video frame and complete the efficient transmission and decoding reconstruction of the compressed video data.

[0008] Furthermore, in step 1, the initial auxiliary information is generated by a linear or nonlinear interpolation method according to the structure and motion characteristics of the current video frame sequence, which is specifically expressed as: , in, Represents the auxiliary information of initialization, represents the difference function, and Represents the previous and next frames respectively.

[0009] Furthermore, in step 2, a pulse emission model is specifically used to simulate the activity of biological neurons and optimize the representation of auxiliary information, which is specifically expressed as: , In the formula, is the potential of neuron i at time t, is the membrane time constant, is the reversal potential, is the input current; In SNN, the input current is the auxiliary information for initialization The directly related ones are expressed as follows: , in, α and β It is a parameter that adjusts the influence of auxiliary information.

[0010] Furthermore, in step 3, the auxiliary information encoded by the SNN is input into the diffusion model, and noise is gradually injected to simulate the process of random disturbance, generating auxiliary information that is closer to the real distribution, which is specifically expressed as: , In the formula, represents the diffusion coefficient, represents Gaussian noise, Indicates the auxiliary information status at the current moment, Considered as SNN neuron potential The function output of ; at each time step t, the state update of the neuron membrane potential in the SNN network will affect the final auxiliary information, mapping the neuron output (such as pulse firing rate) to the auxiliary information , forming the following relationship: , in, is the neuronal potential A nonlinear transformation of , perhaps based on spike rate or some other measure related to the neuron's state.

[0011] Furthermore, in step 4, the process of generating optimized auxiliary information is expressed as: , in, Through the reverse process, it is obtained, which is specifically expressed as: , In the formula, represents the auxiliary information generation function after SNN optimization, SI opt Auxiliary information for final optimization; represents the noise estimation function learned through the neural network; The network architecture of the generating function includes: 501: Input layer: receiving input auxiliary information , and encode it into a pulse signal; 502: Encoding layer: converts the input signal into a pulse train through the LIF neuron model; 503: Spike neuron layer: simulates the pulse emission mechanism according to the input pulse sequence; 504: Synaptic weight layer: adjust the synaptic weight through the STDP algorithm; 505: Membrane potential accumulation layer: The membrane potential of the neuron is accumulated and updated according to the received pulses, and a pulse is emitted when it exceeds the threshold; 506: Output layer: Generates the final optimized auxiliary information by decoding the pulse sequence into a continuous signal SI opt .

[0012] This architecture simulates the working mechanism of biological neural networks to generate auxiliary information for video compression. The process is more efficient and intelligent.

[0013] Furthermore, in step 5, the process of reconstructing the final video frame is expressed as: , In the formula, represents the reconstructed video frame after decoding, represents a reconstruction function; the network architecture of the reconstruction function includes: 601: Input layer: receiving input optimized auxiliary information SI opt , the input dimension is H × W × C ; 602: Convolutional layer: Multi-layer convolutional layer from SI opt Extract low-level spatial features and gradually go deeper to extract higher-level features; 603: Pooling layer: Use maximum pooling or average pooling for downsampling to reduce the size and computational complexity of the feature map; 604: Upsampling layer: restore the feature map to the original resolution through deconvolution or bilinear interpolation to reconstruct the details of the image; 605: Skip connection: Fusion of low-level features with high-level features to preserve details; 606: Output layer: The last layer generates the final reconstructed video frame F rec , used for video frame recovery.

[0014] These steps utilize a combination of spiking neural networks and diffusion models to improve the quality and efficiency of distributed video compression by dynamically adjusting and optimizing auxiliary information.

[0015] Beneficial effects of the present invention: The present invention generates auxiliary information by simulating the spatiotemporal diffusion behavior of video signals. This method combines pulse neural networks and diffusion processes to effectively capture the complex temporal and spatial features in video signals, thereby generating efficient and robust auxiliary information to ensure high image reconstruction quality under high compression rate conditions. Compared with traditional auxiliary information generation methods, the method based on the pulse diffusion model has higher compression efficiency and can effectively improve data processing capabilities in complex dynamic scenes. Specifically, it is manifested in: (1) Auxiliary information generation mechanism of pulse neural network combined with diffusion model: By combining the pulse neural network with the diffusion model, the firing mechanism of biological neurons and the random noise injection process are simulated, thereby improving the accuracy and effectiveness of auxiliary information generation and significantly improving video compression performance.

[0016] (2) Adaptive weight update mechanism based on synaptic plasticity: Introducing the principle of synaptic plasticity and using the pulse timing-dependent synaptic plasticity (STDP) algorithm, the connection weights between neurons are dynamically adjusted to achieve adaptive auxiliary information generation and optimization, thereby improving data restoration and network adaptability during video compression.

[0017] (3) Adaptive optimization mechanism of auxiliary information: Dynamically adjust the parameters and diffusion process of the pulse neural network according to the content and characteristics of the video frame to ensure high video reconstruction quality under different compression conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 yes F SNN Network architecture diagram.

[0019] Figure 2 yes F recon Network architecture diagram. DETAILED DESCRIPTION

[0020] The present invention will be described in further detail below with reference to the accompanying drawings and embodiments.

[0021] The method for generating auxiliary information between frames of images based on the pulse diffusion model utilizes the event-driven computing characteristics of the pulse neural network and generates efficient auxiliary information by imitating the firing mechanism of brain neurons, including the following steps: Step 1: Initial auxiliary information generation: Based on the similarity of images between frames, traditional compression algorithms are used to generate preliminary auxiliary information as the basis for subsequent optimization processing; this initial step can quickly generate preliminary predictions and reduce the computational burden of subsequent processing.

[0022] The initial auxiliary information is generated by a linear or nonlinear interpolation method based on the structure and motion characteristics of the current video frame sequence, and is specifically expressed as: , in, Represents the auxiliary information of initialization, represents the difference function. In this embodiment, high-order polynomial interpolation is used. and Represents the previous and next frames respectively.

[0023] Step 2: Pulse neural network encoding: The initial auxiliary information is encoded through a spiking neural network (SNN), simulating the activity pattern of biological neurons and optimizing the timing processing capability of the auxiliary information, thereby improving the accuracy of data representation and the temporal and spatial correlation; Specifically, the pulse emission model is used to simulate the activity of biological neurons and optimize the representation of auxiliary information, which is specifically expressed as: , In the formula, is the potential of neuron i at time t, is the membrane time constant, is the reversal potential, is the input current; In SNN, the input current is the auxiliary information for initialization The directly related ones are expressed as follows: , in, α and β It is a parameter that adjusts the influence of auxiliary information.

[0024] Step 3: Noise injection based on diffusion model: A diffusion model is used to gradually inject noise into the auxiliary information after the pulse neural network is encoded, simulating random disturbances in the real environment, enhancing the robustness of the generated auxiliary information in complex scenarios, and ensuring that it can cope with external interference during transmission or decoding. The auxiliary information encoded by SNN is input into the diffusion model, and noise is gradually injected to simulate the process of random disturbance to generate auxiliary information that is closer to the real distribution. It is specifically expressed as: , In the formula, represents the diffusion coefficient, represents Gaussian noise, Indicates the auxiliary information status at the current moment, Considered as SNN neuron potential The function output of ; at each time step t, the state update of the neuron membrane potential in the SNN network will affect the final auxiliary information, mapping the neuron output (such as pulse firing rate) to the auxiliary information , forming the following relationship: , in, is the neuronal potential A nonlinear transformation of , perhaps based on spike rate or some other measure related to the neuron's state.

[0025] Step 4: Optimize auxiliary information decoding: Through the reverse process of the diffusion model, high-quality auxiliary information is recovered from the noisy auxiliary information, and the noise is gradually eliminated to generate optimized auxiliary information; The optimized auxiliary information is reversely decoded through a spiking neural network to restore it to an intermediate result with physical meaning, ensuring the stability and accuracy of the decoding process and providing reliable auxiliary information for subsequent video frame reconstruction. The process of generating optimized auxiliary information is expressed as: , in, Through the reverse process, it is obtained, which is specifically expressed as: , In the formula, represents the auxiliary information generation function after SNN optimization, SI opt Auxiliary information for final optimization; represents the noise estimation function learned through the neural network; The network architecture of the generating function is as follows Figure 1 As shown, including: 1) Input layer: receiving input auxiliary information x t−1 , and encode it into a pulse signal; 2) Encoding layer: The input signal is converted into a pulse train through the LIF (Leaky Integrate-and-Fire) neuron model; 3) Spiking neuron layer: simulates the pulse emission mechanism according to the input pulse sequence; 4) Synaptic weight layer: adjust the synaptic weight through the STDP (Spike-Timing Dependent Plasticity) algorithm; 5) Membrane potential accumulation layer: The membrane potential of the neuron is accumulated and updated according to the received pulses, and a pulse is emitted when it exceeds the threshold; 6) Output layer: Generates the final optimized auxiliary information by decoding the pulse sequence into a continuous signal SI opt .

[0026] This architecture simulates the working mechanism of biological neural networks to generate auxiliary information for video compression. The process is more efficient and intelligent.

[0027] Step 5: Reconstruct video frame generation: Combining the optimized auxiliary information with the local features of the video block, deconvolution image processing technology is applied to reconstruct the final video frame and complete the efficient transmission and decoding reconstruction of the compressed video data.

[0028] The process of reconstructing the final video frame is expressed as: , In the formula, represents the reconstructed video frame after decoding, Represents a reconstruction function; the network architecture of the reconstruction function is as follows Figure 2 As shown, including: 1) Input layer: receiving input optimized auxiliary information SI opt , the input dimension is H × W × C ; 2) Convolutional layer: Multiple convolutional layers from SI opt Extract low-level spatial features and gradually go deeper to extract higher-level features; 3) Pooling layer: Use maximum pooling or average pooling for downsampling to reduce the size and computational complexity of the feature map; 4) Upsampling layer: restore the feature map to the original resolution through deconvolution or bilinear interpolation to reconstruct the details of the image; 5) Skip connection: Fusion of low-level features with high-level features, preserving details, similar to the U-Net architecture; 6) Output layer: The last layer generates the final reconstructed video frame F rec , used for video frame recovery.

[0029] These steps utilize a combination of spiking neural networks and diffusion models to improve the quality and efficiency of distributed video compression by dynamically adjusting and optimizing auxiliary information.

[0030] Inter-frame image auxiliary information generation is an innovative image processing technology suitable for inter-frame information generation in video compression. This technology does not rely on traditional video encoders to perform complex computational processing on each frame of an image, but instead completes the compression of image data through the collaborative work of the encoder and decoder. This method divides the video encoding task into two parts: the encoder is responsible for generating auxiliary information, and the decoder decodes this auxiliary information in conjunction with the reference frame to reconstruct the video frame. The core idea is to introduce redundant information and computational distribution, thereby reducing the computational complexity of the encoder and improving the flexibility and adaptability of the system. This technology is particularly suitable for devices with limited computing power and scenarios with poor network conditions.

[0031] Spiking Diffusion Models (SDM) combine spiking neural networks with diffusion processes to simulate the propagation and processing of signals in complex environments. Spiking neural networks (SNNs) can capture instantaneous changes in data by simulating the instantaneous discharge characteristics of biological neurons, while the diffusion process is used to describe the propagation characteristics of signals in time and space. This modeling method can efficiently process high-dimensional video data and generate image auxiliary information with high temporal and spatial correlation, thereby improving video compression efficiency.

Claims

1. A method for generating auxiliary information of inter-frame images based on a pulse diffusion model, characterized in that: The following steps are involved: Step 1: Initial auxiliary information generation: Based on the similarity of inter-frame images, the traditional compression algorithm is used to generate preliminary auxiliary information as the basis for subsequent optimization processing; Step 2: Pulse neural network encoding: Encode the initial auxiliary information through a pulse neural network, simulate the activity pattern of biological neurons, optimize the timing processing capability of auxiliary information, and thus improve the accuracy of data representation and spatiotemporal correlation; Step 3: Noise injection based on diffusion model: Use the diffusion model to gradually inject noise into the auxiliary information after the pulse neural network is encoded, simulate the random disturbance in the real environment, enhance the robustness of the generated auxiliary information in complex scenarios, and ensure that it can cope with external interference during the transmission or decoding process; Step 4: Optimize auxiliary information decoding: Recover high-quality auxiliary information from noisy auxiliary information through the reverse process of the diffusion model, gradually eliminate the noise and generate optimized auxiliary information; reversely decode the optimized auxiliary information through the pulse neural network to restore it to an intermediate result with physical meaning; Step 5: Reconstruct video frame generation: Combine the optimized auxiliary information with the local features of the video block, apply deconvolution image processing technology, reconstruct the final video frame, and complete the efficient transmission and decoding reconstruction of compressed video data.

2. The method for generating auxiliary information of inter-frame images based on a pulse diffusion model according to claim 1, characterized in that: In step 1, the initial auxiliary information is generated by using a linear or nonlinear interpolation method based on the structure and motion characteristics of the current video frame sequence, which is specifically expressed as: ， in, Represents the auxiliary information of initialization, represents the difference function, and Represents the previous and next frames respectively.

3. The method for generating auxiliary information of inter-frame images based on a pulse diffusion model according to claim 1, characterized in that: In step 2, a pulse emission model is specifically used to simulate the activity of biological neurons and optimize the representation of auxiliary information, which is specifically expressed as: ， In the formula, is the potential of neuron i at time t, is the membrane time constant, is the reversal potential, is the input current; In SNN, the input current is the auxiliary information for initialization The directly related ones are expressed as follows: , in, α and β It is a parameter that adjusts the influence of auxiliary information.

4. The method for generating auxiliary information of inter-frame images based on a pulse diffusion model according to claim 1, characterized in that: In step 3, the auxiliary information encoded by SNN is input into the diffusion model, and noise is gradually injected to simulate the process of random disturbance to generate auxiliary information that is closer to the real distribution, which is specifically expressed as: ， In the formula, represents the diffusion coefficient, represents Gaussian noise, and Respectively represent the auxiliary information of the next moment and the current moment, Considered as SNN neuron potential The function output of ; at each time step t, the state update of the neuron membrane potential in the SNN network will affect the final auxiliary information, mapping the output of the neuron to the auxiliary information , forming the following relationship: ， in, is the neuronal potential Nonlinear transformation.

5. The method for generating auxiliary information of inter-frame images based on a pulse diffusion model according to claim 1, characterized in that: In step 4, the process of generating optimized auxiliary information is expressed as: ， in, Through the reverse process, it is obtained, which is specifically expressed as: ， In the formula, represents the auxiliary information generation function after SNN optimization, SI opt Auxiliary information for final optimization; represents the noise estimation function learned through the neural network; The network architecture of the generating function includes: 501: Input layer, receiving input auxiliary information , and encode it into a pulse signal; 502: encoding layer, converting the input signal into a pulse train through the LIF neuron model; 503: Spike neuron layer, simulating the pulse emission mechanism according to the input pulse sequence; 504: synaptic weight layer, adjusting synaptic weights through STDP algorithm; 505: membrane potential accumulation layer, the membrane potential of the neuron is accumulated and updated according to the received pulses, and a pulse is emitted when it exceeds the threshold; 506: Output layer, which generates the final optimized auxiliary information by decoding the pulse sequence into a continuous signal SI opt .

6. The method for generating auxiliary information of inter-frame images based on a pulse diffusion model according to claim 1, characterized in that: In step five, the process of reconstructing the final video frame is expressed as: ， In the formula, represents the reconstructed video frame after decoding, represents a reconstruction function; the network architecture of the reconstruction function includes: 601: Input layer, receiving input optimization auxiliary information SI opt , the input dimension is H × W × C ; 602: Convolutional layer, multi-layer convolutional layer from SI opt Extract low-level spatial features and gradually go deeper to extract higher-level features; 603: Pooling layer, using maximum pooling or average pooling for downsampling to reduce the size and computational complexity of the feature map; 604: Upsampling layer, which restores the feature map to the original resolution through deconvolution or bilinear interpolation and reconstructs the details of the image; 605: Skip connection, which fuses low-level features with high-level features and preserves details; 606: Output layer, the last layer generates the final reconstructed video frame F rec , used for video frame recovery.

Images