Event camera image processing method, training method, system, device and medium

Abstract

The embodiments of the present application provide an image processing method, training method, system, device and medium for an event camera, which belongs to the field of image processing technology. The method preprocesses the event stream to obtain an event stream in the form of a voxel grid, and then aggregates information based on the event stream in the form of a voxel grid to obtain a first reconstructed image. The first reconstructed image is superimposed with the denoised event stream in the form of a voxel grid and the first reconstructed image to obtain a second reconstructed image, so that the edge information of the second reconstructed image is more prominent, and then a noise image is generated. The noise image and the second reconstructed image are input into a pre-trained diffusion model, and the second reconstructed image is used as a condition of the diffusion model. The noise image is iteratively denoised through the denoising network of the diffusion model to generate a repaired image. The generated repaired image has high image quality and clearer edge information.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, training method, system, device and medium for an event camera. Background Art

[0002] An event camera is a new type of asynchronous sensor. Unlike traditional cameras that capture images at a fixed frequency, its operation is inspired by biology, independently measuring changes in brightness at each pixel (called events). By measuring only brightness changes, event cameras typically record edge information of moving objects, discarding redundant visual information and significantly reducing bandwidth. As a result, event cameras offer advantages such as high dynamic range (140dB, HDR), high temporal resolution (measured in microseconds), and low power consumption, making them suitable for some HDR and high-speed motion scenarios.

[0003] The event stream contains highly compressed visual information, sufficient to reconstruct images or videos. However, due to the accumulated noise and motion blur in the event stream, the images reconstructed by the image reconstruction methods in related technologies are often of low quality and have unclear edge information. Summary of the Invention

[0004] The main purpose of the embodiments of the present application is to propose an image processing method, system, device and medium for an event camera, which can generate a repaired image based on an event stream, improve the quality of the repaired image, and make the edge information of the repaired image clearer.

[0005] To achieve the above-mentioned objectives, a first aspect of an embodiment of the present application provides an image processing method for an event camera, the method comprising:

[0006] Acquire an initial event stream, and pre-process the initial event stream to obtain the event stream in a voxel grid form;

[0007] performing information aggregation on the event stream in the form of a voxel grid to obtain a first reconstructed image;

[0008] Performing denoising on the initial event stream to obtain a denoised event stream, and preprocessing the denoised event stream to obtain the denoised event stream in a voxel grid form;

[0009] superimposing the denoised event stream in voxel grid form with the first reconstructed image to obtain a second reconstructed image;

[0010] A noisy image is generated, and the noisy image and the second reconstructed image are input into a pre-trained diffusion model. The second reconstructed image is used as a condition of the diffusion model, and the noisy image is iteratively denoised by a denoising network of the diffusion model to generate a repaired image.

[0011] In some embodiments, aggregating information on the event stream in the form of a voxel grid to obtain a first reconstructed image includes:

[0012] Performing circular convolution on the event stream in the form of a voxel grid through a circular convolution module to obtain a multi-scale feature map;

[0013] Performing information aggregation on the multi-scale feature map through an information aggregation module to obtain an information aggregation feature result;

[0014] Upsampling the information aggregation image through a multi-dimensional upsampling module to obtain a first upsampled image, and upsampling the multi-scale feature map through the multi-dimensional upsampling module to obtain a second upsampled image;

[0015] The first up-sampled image and the second up-sampled image are combined to obtain the first reconstructed image.

[0016] In some embodiments, performing denoising on the initial event stream to obtain the denoised event stream includes:

[0017] Generate a preliminary movement trajectory of the initial event flow on a first coordinate plane; the first coordinate plane is constructed based on the initial event flow;

[0018] Projecting the event stream onto a preset second coordinate plane according to the preliminary movement trajectory to obtain a projected event stream;

[0019] Calculating the total number of projection events of the projection event stream on the first coordinate plane;

[0020] calculating a variance of the total number of projection events, and determining a predicted movement trajectory of the object in the event stream based on the variance and the preliminary movement trajectory;

[0021] Projecting the event stream onto a second coordinate plane according to the predicted movement trajectory to obtain a target projected event stream;

[0022] The number of projection events in each pixel of the first coordinate plane is determined according to the target projection event stream. If the number of projection events is less than an adaptive threshold, the events in the pixel are deleted to obtain the denoised event stream.

[0023] In some embodiments, using the second reconstructed image as a condition of the diffusion model, denoising the noisy image through a denoising network of the diffusion model to generate a repaired image includes:

[0024] Using the denoising network, concatenate the second reconstructed image and the noisy image in a channel dimension to generate a predicted noise vector;

[0025] Iterative denoising is performed according to an iterative formula and the predicted noise vector until the number of iterations reaches a preset number to obtain the repaired image; the iterative formula is:

[0026]

[0027] Among them, g t Characterize the noise image at time t, g t-1 Characterize the noise image at time t-1, 0<α t <1,α t is a hyperparameter, f θ represents the denoising network, r represents the second reconstructed image, γ t Characterize the noise parameter at time t, t~{0,…,T}, T represents the preset number, ∈ t is the predicted noise vector at time t.

[0028] In some embodiments, the diffusion model is trained by the following steps:

[0029] Obtaining an initial event stream training sample, and obtaining a real image training sample corresponding to the initial event stream training sample;

[0030] Preprocessing the initial event stream training samples to obtain the event stream training samples in a voxel grid form;

[0031] performing information aggregation on the event stream training samples in a voxel grid format to obtain a first reconstructed training image;

[0032] Performing denoising processing on the initial event stream training samples to obtain denoised event stream training samples, and preprocessing the denoised event stream training samples to obtain the denoised event stream training samples in the form of a voxel grid;

[0033] superimposing the denoised event stream training samples in a voxel grid format with the first reconstructed training image to obtain a second reconstructed training image;

[0034] Noising the real image training sample to generate a noisy training image;

[0035] Inputting the noisy training image and the second reconstructed training image into a diffusion model, and performing channel-wise splicing on the noisy training image and the second reconstructed training image using a denoising network of the diffusion model to generate predicted noise for the noisy training image;

[0036] Calculating a loss value based on the predicted noise and the second reconstructed training image;

[0037] The diffusion model is optimized according to the loss value to obtain the trained diffusion model.

[0038] In some embodiments, calculating the loss value based on the predicted noise and the second reconstructed training image includes:

[0039] The loss value is calculated according to the objective function, and the objective function is:

[0040]

[0041]

[0042] Wherein, ∈ represents the prediction noise, ∈~N(0,I), I represents the unit matrix; f θ represents the denoising network, γ represents the noise parameter, γ ~ p(γ); p = 1, represents the L1 norm; g0 represents the real image training sample, r represents the second reconstructed training image, E represents the mathematical expectation, and g represents the real image training sample set.

[0043] To achieve the above objectives, a second aspect of the embodiments of the present application provides a method for training a diffusion model, including:

[0044] Obtaining an initial event stream training sample, and obtaining a real image training sample corresponding to the initial event stream training sample;

[0045] Preprocessing the initial event stream training samples to obtain the event stream training samples in a voxel grid form;

[0046] performing information aggregation on the event stream training samples in a voxel grid format to obtain a first reconstructed training image;

[0047] Performing denoising processing on the initial event stream training samples to obtain denoised event stream training samples, and preprocessing the denoised event stream training samples to obtain the denoised event stream training samples in the form of a voxel grid;

[0048] superimposing the denoised event stream training samples in a voxel grid format with the first reconstructed training image to obtain a second reconstructed training image;

[0049] Noising the real image training sample to generate a noisy training image;

[0050] Inputting the noisy training image and the second reconstructed training image into a diffusion model, and performing channel-wise splicing on the noisy training image and the second reconstructed training image using a denoising network of the diffusion model to generate predicted noise for the noisy training image;

[0051] Calculating a loss value based on the predicted noise and the second reconstructed training image;

[0052] The diffusion model is optimized according to the loss value to obtain the trained diffusion model.

[0053] To achieve the above objectives, a third aspect of the embodiments of the present application provides an event camera image processing system based on a diffusion model, the system comprising:

[0054] an acquisition module, configured to acquire an initial event stream, and preprocess the initial event stream to obtain the event stream in a voxel grid form;

[0055] a reconstruction module, configured to aggregate information on the event stream in the form of a voxel grid to obtain a first reconstructed image;

[0056] An event stream denoising module, configured to perform denoising on the initial event stream to obtain a denoised event stream, and pre-process the denoised event stream to obtain the denoised event stream in a voxel grid form;

[0057] a superposition module, configured to superimpose the denoised event stream in a voxel grid format with the first reconstructed image to obtain a second reconstructed image;

[0058] An image generation module is configured to generate a noisy image, input the noisy image and the second reconstructed image into a pre-trained diffusion model, use the second reconstructed image as a condition of the diffusion model, and iteratively denoise the noisy image through a denoising network of the diffusion model to generate a repaired image.

[0059] To achieve the above-mentioned purpose, the fourth aspect of the embodiments of the present application proposes an electronic device, which includes a memory and a processor, the memory storing a computer program, and the processor implementing the image processing method of the event camera described in the first aspect of the embodiment when executing the computer program.

[0060] To achieve the above-mentioned purpose, the fifth aspect of the embodiments of the present application proposes a storage medium, which is a computer-readable storage medium and stores a computer program. When the computer program is executed by a processor, it implements the image processing method of the event camera described in the first aspect of the embodiment.

[0061] The image processing method, training method, system, device and medium of the event camera proposed in the embodiments of the present application, and the image processing method of the event camera can be applied in an event camera image processing system based on a diffusion model. By executing the image processing method of the event camera, the initial event stream is preprocessed to obtain an event stream in the form of a voxel grid, and then the event stream in the form of a voxel grid is aggregated to reconstruct a first reconstructed image, and then the denoised event stream in the form of a voxel grid is superimposed on the first reconstructed image to obtain a second reconstructed image, so that the edge information of the second reconstructed image is more prominent, and then a noise image is generated, and the noise image and the second reconstructed image are input into a pre-trained diffusion model, and the second reconstructed image is used as a condition of the diffusion model. The noise image is iteratively denoised through the denoising network of the diffusion model to generate a repaired image. Since the second reconstructed image with more prominent edge information is used as a condition of the diffusion model in the iterative denoising process, the generated repaired image has a higher image quality, and the edge information of the repaired image is clearer. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 This is an optional flowchart of the image processing method of the event camera provided in an embodiment of the present application;

[0063] Figure 2 yes Figure 1 Flowchart of step S102 in FIG.

[0064] Figure 3 yes Figure 1 Flow chart of step S103 in FIG.

[0065] Figure 4 yes Figure 1 Flow chart of step S105 in FIG.

[0066] Figure 5 It is a flowchart of the training steps of the diffusion model;

[0067] Figure 6 is a schematic diagram of the processing process of the denoising network during training in some embodiments of the present application;

[0068] Figure 7 A framework diagram of an event camera image processing system based on a diffusion model according to some embodiments of the present application;

[0069] Figure 8 The image processing method of the event camera of the embodiment of the present application is used to process the data on the HQF dataset, and a comparative schematic diagram of images of each step in the processing process is obtained;

[0070] Figure 9A schematic diagram showing the comparison of images of each step in the processing process is obtained by processing the data on the IJRR dataset using the image processing method of the event camera according to an embodiment of the present application;

[0071] Figure 10 A schematic diagram showing the comparison of images of each step in the processing process is obtained by processing the data on the MVSEC dataset using the image processing method of the event camera according to the embodiment of the present application;

[0072] Figure 11 Schematic diagram of the functional modules of the event camera image processing system based on the diffusion model provided in an embodiment of the present application;

[0073] Figure 12 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0075] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0076] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0077] First, let’s analyze some of the terms used in this application:

[0078] Artificial intelligence (AI) is a new technical discipline that studies and develops theories, methods, technologies, and application systems for simulating, extending, and disseminating human intelligence. A branch of computer science, AI seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thinking. It also encompasses the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and disseminate human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0079] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0080] An event-based camera is a new type of bio-inspired visual sensor, sometimes also called a dynamic vision sensor (DVS) or DAVIS (Dynamic and Active-Pixel Vision Sensor). Compared to standard cameras that capture images at a fixed frame rate, it offers low latency, high dynamic range, low power consumption, and high temporal resolution. When an object moves, a standard camera loses some data due to the sampling frequency, while the data collected by an event camera is relatively more complete. Furthermore, when the target object is in relative motion with the camera, images captured with a standard camera are prone to motion blur due to factors such as exposure time. Data captured with an event camera does not suffer from this problem.

[0081] Event cameras are a new type of asynchronous sensor. Unlike traditional cameras that capture images at a fixed frequency, their operation is biologically inspired, independently measuring brightness changes (called events) at each pixel. By measuring only brightness changes, event cameras typically record edge information of moving objects, discarding redundant visual information and significantly reducing bandwidth. Consequently, event cameras offer advantages such as high dynamic range (140dB, HDR), high temporal resolution (in microseconds), and low power consumption, making them suitable for some HDR and high-speed motion scenarios. However, due to the sparse and asynchronous nature of the event stream, it can only describe scene changes, not the absolute intensity, making many traditional computer vision algorithms incapable of direct application. A common approach is to reconstruct the event stream into intensity frames, which are highly beneficial for downstream tasks such as object detection, tracking, and SLAM. In theory, the event stream contains highly compressed visual information, sufficient to provide the necessary information for reconstructing images or videos. However, due to the accumulated noise and motion blur in the event stream, existing image reconstruction methods often produce images with artifacts, unclear edges, and a lack of perceptual realism. If the event stream is denoised first, event information will be lost, thus affecting the reconstruction effect. If the reconstructed image is to be denoised, real image frames are required as auxiliary training, which is not feasible for some pure event streams. Based on this, the embodiments of the present application provide an image processing method, training method, system, device and medium for an event camera, which can generate a repaired image based on the event stream, improve the quality of the repaired image, and make the edge information of the repaired image clearer.

[0082] The image processing method, training method, system, device and medium of the event camera provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the image processing method of the event camera in the embodiments of the present application is described.

[0083] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and diffuse human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0084] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0085] The image processing method for an event camera provided in an embodiment of the present application relates to the field of artificial intelligence technology. The image processing method for an event camera provided in an embodiment of the present application can be applied in a terminal, can be applied in a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the image processing method for an event camera, etc., but is not limited to the above forms.

[0086] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0087] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first. For example, when obtaining user stored data and the user's cached data access request, the user's permission or consent will be obtained first; when obtaining resource data, the present application embodiment will first obtain the user's permission or consent. Moreover, the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the present application embodiment needs to obtain the user's sensitive personal information, it will obtain the user's separate permission or consent through a pop-up window or jump to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the present application embodiment to operate normally will be obtained.

[0088] Reference Figure 1 , Figure 1 This is an optional flowchart of the image processing method of the event camera provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S105.

[0089] Step S101, obtaining an initial event stream, and preprocessing the event stream to obtain an event stream in the form of a voxel grid;

[0090] For example, an event stream is captured by an event camera, and the event stream includes N events, N events ε n The resulting time span is: Δt = t N-1 -t0, where t N-1 is the end time of the event stream, and t0 is the start time of the event stream.

[0091] For example, an event stream can be captured by an event camera and stored in an event database. The event stream can be obtained from the event database, or from a publicly available real-world event camera dataset (HighQuality Frames, HQF). There are many ways to obtain event streams from an event database. For example, a custom application can be used to access the event database to obtain a newly added molecular dataset, or data newly stored in the drug molecule database can be directly copied as a newly added molecular dataset. However, no matter which method is used, the quality and security of the data must be ensured.

[0092] For example, when preprocessing the event stream to obtain an event stream in the form of a voxel grid, the entire event stream is evenly superimposed into 5 consecutive voxel grids. The preprocessing formula is as follows:

[0093]

[0094] Where B is the number of voxel grids, B = 5, p i = ±1 represents the polarity of the event stream, t represents the timestamp of the event, V j∈[0,B-1] Represents the event stream in the form of a grid of voxels of the jth voxel.

[0095] Step S102 , performing information aggregation on the event stream in the form of a voxel grid to obtain a first reconstructed image;

[0096] For example, a voxel grid-like event stream is input into a Transformer-based reconstruction network for information aggregation, resulting in a first reconstructed image with noise and artifacts. The Transformer is a deep learning model based on the self-attention mechanism, used for natural language processing tasks such as machine translation and language modeling. The Transformer model consists of two parts: an encoder and a decoder. The encoder maps the input sequence into a high-dimensional space, while the decoder decodes the encoder output vector into a target sequence. Both the encoder and decoder consist of multiple identical layers, each of which includes a self-attention sublayer and a feedforward neural network sublayer. The self-attention mechanism is the core of the Transformer. In the self-attention sublayer, each input vector is evaluated for similarity with all other input vectors to obtain a weight vector representing its importance to the other vectors. All input vectors are then weighted and summed according to the weights to obtain a new vector representation. This allows each input vector to leverage information from other vectors, thereby improving model performance.

[0097] Step S103, performing denoising processing on the initial event stream to obtain a denoised event stream, and pre-processing the denoised event stream to obtain a denoised event stream in the form of a voxel grid;

[0098] Exemplarily, the event stream is denoised based on a contrast maximization framework to obtain a denoised event stream, which is then preprocessed using a preprocessing formula to obtain a denoised event stream in the form of a voxel grid. The denoised event stream in the form of a voxel grid can represent the edge information of objects in the first reconstructed image.

[0099] Step S104, superimposing the denoised event stream in the form of a voxel grid with the first reconstructed image to obtain a second reconstructed image;

[0100] Since the denoised event stream in the form of a voxel grid can represent the edge information of objects in the first reconstructed image, when the denoised event stream in the form of a voxel grid is superimposed on the first reconstructed image to obtain a second reconstructed image, the edge information of the second reconstructed image is more prominent than that of the first reconstructed image.

[0101] Step S105: Generate a noise image, input the noise image and the second reconstructed image into a pre-trained diffusion model, use the second reconstructed image as a condition of the diffusion model, iteratively denoise the noise image through the denoising network of the diffusion model, and generate a repaired image.

[0102] Exemplarily, Gaussian noise may be randomly generated as a noise image, and then the noise image and the second reconstructed image are input into a pre-trained diffusion model to generate a repaired image.

[0103] In steps S101 to S105 of the embodiment of the present application, an image processing method for an event camera is executed to preprocess the initial event stream to obtain an event stream in the form of a voxel grid. Information aggregation is then performed on the event stream in the form of a voxel grid to reconstruct a first reconstructed image with artifacts and noise. The denoised event stream in the form of a voxel grid is then superimposed on the first reconstructed image to obtain a second reconstructed image. Because the denoised event stream in the form of a voxel grid can represent the edge information of objects in the first reconstructed image, the edge information of the second reconstructed image is more prominent than that of the first reconstructed image. A noisy image is then generated, and the noisy image and the second reconstructed image are input into a pre-trained diffusion model. The second reconstructed image is used as a condition for the diffusion model, and the noisy image is iteratively denoised using the denoising network of the diffusion model to generate a restored image. Because the second reconstructed image with more prominent edge information is used as a condition for the diffusion model during the iterative denoising process, the restored image has higher image quality and clearer edge information.

[0104] It is understood that in step S102, the event stream in the form of a voxel grid is input into the reconstruction network based on Transformer to perform image reconstruction and obtain a first reconstructed image. The reconstruction network based on Transformer includes a circular convolution module, an information aggregation module and a multi-dimensional upsampling module. Figure 2 , Figure 2 yes Figure 1 Flowchart of step S102 in FIG. Step S102 may include but is not limited to steps S201 to S204.

[0105] Step S201, performing circular convolution on the event stream in the form of a voxel grid through a circular convolution module to obtain a multi-scale feature map;

[0106] For example, the event stream in the form of a voxel grid is represented as V∈R B×H×W , input the voxel V into the circular convolution module Where C0 = 32, H and W are the height and width of voxel V, respectively, and B is the number of voxel grids in voxel V. The circular convolution module has three layers, and the number of output channels C0 in each layer is twice that of the previous layer. In the circular convolution module, the ConvLSTM layer uses the previous state to enhance the temporal stability of the reconstructed image. The processing of the circular convolution module can be expressed as:

[0107]

[0108] Among them, l∈{0,1,2,3} represents the lth layer of the circular convolution module, and It represents the network of layer l, and each layer of the network adopts the ConvLSTM structure. It represents the state of the lth layer at time t. Through the circular convolution module, a multi-scale feature map is obtained, which is expressed as:

[0109] {f l RCB |l∈{0,1,2,3}},

[0110] Among them, f l RCB Representing multi-scale feature maps, l∈{0,1,2,3} represents the lth layer of the circular convolution module.

[0111] Step S202: performing information aggregation on the multi-scale feature map through an information aggregation module to obtain an information aggregation feature result;

[0112] For example, in the information aggregation module, the multi-scale feature map is serialized and divided into 8 small blocks. These small blocks are then expanded into a one-dimensional vector Then by linear projection f l proj and position code e i ∈R D , D = 256 maps this vector to the latent space T l,i The mapping formula is as follows:

[0113]

[0114] Among them, T l,i express The representation of the vector in the latent space. The information aggregation module can effectively extract the context information of the event stream. The output of the information aggregation module is a two-dimensional matrix, which is expressed as:

[0115]

[0116] Step S203: upsample the information aggregation image through a multi-dimensional upsampling module to obtain a first upsampled image, and upsample the multi-scale feature map through a multi-dimensional upsampling module to obtain a second upsampled image.

[0117] Step S204: combining the first up-sampled image and the second up-sampled image to obtain a first reconstructed image.

[0118] Exemplarily, the multi-dimensional upsampling module is composed of three stacked upsampling modules and connected to a simple convolutional layer. The information aggregation image is upsampled through the multi-dimensional upsampling module to obtain a first upsampled image. The multi-scale feature map is upsampled through the multi-dimensional upsampling module to obtain a second upsampled image. The first upsampled image and the second upsampled image are then combined to obtain a first reconstructed image, which is specifically expressed as:

[0119]

[0120] Among them, f l MLU Characterize the first reconstructed image, Characterize the first upsampled image, f TPA Characterize the second upsampled image.

[0121] See also Figure 3 , Figure 3 1 is a flow chart of step S103 of an embodiment of the present application. In some embodiments, step S103 may include steps S301 to S306:

[0122] Step S301, generating a preliminary movement trajectory of the event stream on a first coordinate plane; the first coordinate plane is constructed based on the initial event stream;

[0123] Step S302: projecting the event stream onto a preset second coordinate plane according to the preliminary movement trajectory to obtain a projected event stream;

[0124] Step S303, calculating the total number of projection events of the projection event stream on the first coordinate plane;

[0125] Step S304, calculating the variance of the total number of projection events, and determining the predicted movement trajectory of the object in the event stream based on the variance and the preliminary movement trajectory;

[0126] Step S305: projecting the event stream onto the second coordinate plane according to the predicted movement trajectory to obtain a target projected event stream;

[0127] It is worth noting that if the event stream is directly projected into the image frame, artifacts will appear. Therefore, it is necessary to predict the movement trajectory of objects in the event stream. When the event stream is projected into the image frame according to the correct trajectory at one end, artifacts can be reduced.

[0128] For example, when denoising the event stream, a preliminary moving trajectory of the event stream on the first coordinate plane is randomly generated θ=(v x , v y ), through the projection formula, each event e of the event stream i Projected onto the second coordinate plane, we get the projection event stream, where each projection event in the projection event stream is ei ′, the projection formula is:

[0129] (x′ i ,y′ i )=(x i ,y i )-(t i -t ref )θ,

[0130] Among them, t ref is the reference time, taking the timestamp t of the last event in the event stream N-1 As the reference time, x′ i and y i is e′ i The spatial position in the second coordinate plane, e i Represents the event before the projection, e′ i Represents the projection event after projection, e′ i The polarity p′ i With the original event e i Polarity p i The same. Then the total number of projection events in the projection event stream on the first coordinate plane is calculated using the total number of projection events calculation formula. The total number of projection events calculation formula is:

[0131]

[0132] Where δ represents the Dirac function, H(x, y; θ) represents the total number of projection events in the projection event flow on the first coordinate plane. x represents the abscissa in the first coordinate system, y represents the ordinate in the second coordinate system, and x′ i Represents the horizontal coordinate of the i-th projection event in the second coordinate system, y′ i Representing the vertical coordinate of the i-th projection event in the second coordinate system, by calculating the variance of H f(θ) = var(H(x, y; θ)), the predicted movement trajectory θ of the object in the event stream can be estimated * , where θ * =argmax θ F(θ). It should be noted that the event stream can be obtained by shooting the object through an event camera. *Then, the predicted trajectory is used as the preliminary trajectory. Based on the predicted trajectory, the event stream is projected onto the second coordinate plane. Steps S301 to S305 are repeatedly performed based on the Nelder-Mead algorithm to obtain multiple predicted trajectories, thereby determining multiple variances. The maximum value among the multiple variances is determined based on the contrast function, and the maximum value is used as the target variance. The predicted trajectory corresponding to the target variance is then determined. Based on the predicted trajectory corresponding to the target variance, the event stream is projected onto the second coordinate plane, and the resulting projected event stream is used as the target projected event stream.

[0133] It is worth noting that when the variance f(θ) is maximum, the calculated predicted movement trajectory is the correct movement trajectory of the object.

[0134] It is worth noting that the first coordinate system is a coordinate system constructed based on the event stream, and the second coordinate system is a coordinate system customized by those skilled in the art.

[0135] Step S306 , determining the number of projection events in each pixel of the first coordinate plane according to the target projection event stream, and if the number of projection events is less than an adaptive threshold, deleting the events in the pixel to obtain a denoised event stream.

[0136] After determining the target projected event stream, the number of projected events in each pixel in the first coordinate plane is determined. If the number of projected events is less than the adaptive threshold, it means that the event in the pixel is noise, so the event in the pixel is deleted; if the number of projected events is greater than or equal to the adaptive threshold, the event in the pixel is retained, thereby obtaining a denoised event stream.

[0137] Reference Figure 4 , Figure 4 yes Figure 1 In some embodiments, step S105 may include but is not limited to steps S401 to S402.

[0138] Step S401: Using a denoising network, the second reconstructed image and the noise image are spliced ​​in the channel dimension to generate a predicted noise vector;

[0139] In the diffusion model, U-Net is used as the denoising network. The noise image and the second reconstructed image are input into the denoising network. The second reconstructed image and the noise image are spliced ​​in the channel dimension to obtain the predicted noise vector ∈. Specifically, the noise image is pure noise. The denoising process in the denoising network is a Markov inverse process. From the pure noise g T Starting from, using the denoising network f θ Proceeding in the opposite direction of forward diffusion, it is expressed as:

[0140]

[0141] p(g T )=N(g T |0,i)

[0142]

[0143] Where I is the identity matrix, p θ () and p() represent the Markov reverse process, T represents the preset number of times, N represents the normal distribution, r represents the second reconstructed image, g t Represents the noise image at time t, g t-1 Characterize the noise image at time t-1. Given g t In the case of , g0 can be estimated:

[0144]

[0145] will be estimated Substitute q(g t-1 |g0,g t ), we get:

[0146]

[0147] Among them, μ θ (r,g t ,γ t ) is g t-1 The mean of .

[0148] Step S402, iterative denoising is performed according to the iterative formula and the predicted noise vector until the number of iterations reaches a preset number to obtain a restored image; the iterative formula is:

[0149]

[0150] Among them, g t Represents the noise image at time t, g t-1 Represents the noise image at time t-1, 0<α t <1,α t is a hyperparameter, f θ represents the denoising network, r represents the second reconstructed image, γ t Represents the noise parameters at time t, t~{0,…,T}, T represents the preset number, ∈ t is the predicted noise vector at time t. Iterate T times according to the iterative formula, and the final g0 is the repaired image.

[0151] It's worth noting that the U-Net network primarily consists of an encoder, a decoder, and skip connections. The U-Net network employs an encoder-decoder architecture. The compression channel is an encoder, extracting image features layer by layer, while the diffusion channel is a decoder, restoring image positional information. Each hidden layer in the U-Net network has a high feature dimension, which facilitates the model's learning of diverse and comprehensive features. During the downsampling process in the U-Net, five convolutional layers with 3×3 kernels are constructed, with output channels of 64, 128, 256, 512, and 512, respectively. These layers also include self-attention layers and residual connection modules. In the U-Net, the upsampling and downsampling processes correspond, resulting in an output that is the same size as the original image.

[0152] Reference Figure 5 In one embodiment, the diffusion model is trained by the following steps:

[0153] Step S501, obtaining an initial event stream training sample, and obtaining a real image training sample corresponding to the initial event stream training sample;

[0154] Exemplarily, randomly selected photos from the COCO dataset are simulated using the event camera simulator ESIM to obtain simulated event streams and corresponding real image frames. The simulated event streams are used as event stream training samples, and the real image frames are used as corresponding real image training samples. Multiple event stream training samples constitute an event stream training sample set, and multiple real image training samples constitute a real image training sample set.

[0155] Step S502 , preprocessing the initial event stream training samples to obtain event stream training samples in the form of voxel grids;

[0156] For example, in order to apply the deep learning-based diffusion model to the event stream of the event camera, the sparse asynchronous event stream training samples are converted into a fixed-size tensor representation. The event stacking method with a fixed time period is adopted, and the time length is the shooting interval between two real images. Specifically, the event stream training samples include N events, N events ε n The resulting time span is: Δt = t N-1 -t0, where t N-1 is the end time of the event stream training sample, and t0 is the start time of the event stream training sample. The entire event stream is evenly superimposed on 5 consecutive voxel grids. The preprocessing formula is as follows:

[0157]

[0158] Where B is the number of voxel grids, B = 5, p1 = ± 1 represents the polarity of the event stream training sample, t represents the timestamp of the event, V j∈[0,B-1] Represents the jth event stream training sample in the form of a voxel grid.

[0159] Step S503 , performing information aggregation on the event stream training samples in the form of voxel grids to obtain a first reconstructed training image;

[0160] Step S504, performing denoising processing on the initial event stream training samples to obtain denoised event stream training samples, and pre-processing the denoised event stream training samples to obtain denoised event stream training samples in the form of voxel grids;

[0161] Step S505 , superimposing the denoised event stream training sample in the form of a voxel grid with the first reconstructed training image to obtain a second reconstructed training image;

[0162] In one embodiment, the sampling time is divided into five time periods, each of which corresponds to a voxel grid, namely B0, ..., B n , n = 5, B voxel grids are averaged, specifically:

[0163]

[0164] Among them, B=5, The second reconstructed training image is superimposed with the first reconstructed training image to obtain a second reconstructed training image.

[0165] Step S506, performing noise addition on the real image training sample to generate a noise training image;

[0166] For example, refer to Figure 6 , Figure 6 This is a schematic diagram of the processing process of the denoising network during training in some embodiments of the present application. Through the forward Markov process q, Gaussian noise is gradually added to the real image training samples g0~p(g|r) to obtain pure noise g t , the specific process is expressed as:

[0167]

[0168]

[0169] Among them, 0<α t <1 is a hyperparameter, p represents the distribution, g represents the real image training sample set, and g0 represents a sample in the real image training sample set, which ensures that the variance of the generated random variable is bounded. It can be directly represented by g given g0. t The distribution of , without the need for intermediate processes, is expressed as:

[0170]

[0171] in q() characterizes the Markov process. t ), we can get g through algebraic operation. t-1 The posterior distribution of :

[0172] q(g t-1 |g0,g t )=N(g t-1 |μ,σ 2 I)

[0173]

[0174]

[0175] Among them, during the training process, G t Represents the noise training image at time T, that is, pure noise; G t-1 Represents the noise training image at time t-1, 0<α t <1,α t is a hyperparameter, f θ represents the denoising network, r represents the second reconstructed training image, γ t Represents the noise parameters at time t, t~{0,…,T}, T represents the preset number, ∈ t is the predicted noise vector at time t. 2 represents the variance of Gaussian noise, and μ represents the mean of Gaussian noise.

[0176] Step S507: Input the noisy training image and the second reconstructed training image into the diffusion model, and perform channel-wise concatenation of the noisy training image and the second reconstructed training image using the denoising network of the diffusion model to generate predicted noise for the noisy training image.

[0177] Step S508, calculating a loss value based on the predicted noise and the second reconstructed training image;

[0178] For example, the loss value is calculated according to the objective function, and the objective function is:

[0179]

[0180]

[0181] Among them, ∈ represents the prediction noise, ∈~N(0,I), I represents the unit matrix; f θrepresents the denoising network, γ represents the noise parameter, γ~p(γ); p=1, represents the L1 norm; g0 represents the real image training sample, r represents the second reconstructed training image, E represents the mathematical expectation, and g represents the real image training sample set.

[0182] It is worth noting that during training, we first uniformly sample a time step t~{0,…,T}, and then sample γ~U(γ t-1 ,γ t ), T = 2000, U is uniform distribution.

[0183] Step S509: Optimize the diffusion model according to the loss value to obtain a trained diffusion model.

[0184] Exemplarily, the optimizer used to optimize the network parameters of the diffusion model is the Adam optimizer, the learning rate is set to 0.0001, and the dropout rate is set to 0.2 to prevent overfitting.

[0185] It should be noted that the diffusion model mainly includes a denoising network, Markov operations, and inverse Markov operations. Therefore, when performing training optimization, the denoising network is mainly trained and optimized.

[0186] Reference Figure 7 , Figure 7 This is a framework diagram of an event camera image processing system based on a diffusion model for some embodiments of the present application. In this framework, an event stream is first acquired through an event camera, and the event stream is preprocessed to obtain an event stream in the form of a voxel grid. The event stream in the form of a voxel grid is then input into a transformer-based reconstruction network to obtain a first reconstructed image with noise and artifacts. The event stream is then denoised based on a contrast maximization framework to obtain a denoised event stream, and the denoised event stream is preprocessed to obtain a denoised event stream in the form of a voxel grid. The denoised event stream in the form of a voxel grid is superimposed on the first reconstructed image to obtain a second reconstructed image, so that the edge information of the second reconstructed image is more prominent. A noise image is then randomly generated, and the noise image and the second reconstructed image are input into a pre-trained diffusion model to generate a repaired image. Figure 7 Stage 1 is equivalent to this application Figure 1 Step S102 in this application; Phase 2 is equivalent to Figure 1 Step S105 in Figure 7 The denoising process is equivalent to this application Figure 1 Step S103.

[0187] Reference Figures 8 to 10 , Figure 8In order to process the data on the HQF dataset using the image processing method of the event camera of the embodiment of the present application, a comparative schematic diagram of the images of each step in the processing process is obtained. Specifically, Figure 8 In the figure, the images corresponding to the "event stream" are the 5 groups of event streams on the HQF dataset; the images corresponding to the "first reconstructed image" are the images obtained by processing the 5 groups of events through steps S101 to S102; the images corresponding to the "repaired image" are the repaired images obtained by processing each group of event streams and the first reconstructed image through steps S103 to S105; the images corresponding to the "real image" are the real images corresponding to the event streams. First, obtain the 5 groups of event streams on the HQF dataset, obtain the real images corresponding to the 5 groups of event streams, execute steps 101 to S102 for each component stream, obtain the corresponding first reconstructed image, and then execute steps S103 to S105 to obtain the corresponding repaired image. Figure 8 It can be seen that the five restored images corresponding to the five groups of event streams are of higher quality and clearer than the corresponding first reconstructed image, and each restored image is almost consistent with the real image.

[0188] Figure 9 In order to process the data on the IJRR (The International Journal of Robotics Research) dataset using the image processing method of the event camera in the embodiment of the present application, a comparative schematic diagram of the images of each step in the processing process is obtained. Specifically, Figure 9 In the figure, the images corresponding to the "event stream" are the 5 groups of event streams on the IJRR dataset; the images corresponding to the "first reconstructed image" are the images obtained by processing the 5 groups of events through steps S101 to S102; the images corresponding to the "repaired image" are the repaired images obtained by processing each group of event streams and the first reconstructed image through steps S103 to S105; the images corresponding to the "real image" are the real images corresponding to the event streams. First, obtain the 5 groups of event streams on the IJRR dataset, obtain the real images corresponding to the 5 groups of event streams, execute steps 101 to S102 for each component stream, obtain the corresponding first reconstructed image, and then execute steps S103 to S105 to obtain the corresponding repaired image. Figure 9 It can be seen that the five restored images corresponding to the five groups of event streams are of higher quality and clearer than the corresponding first reconstructed image, and each restored image is almost consistent with the real image.

[0189] Figure 10In order to process the data on the MVSEC (Multi Vehicle Stereo Event Camera) dataset using the image processing method of the event camera of the embodiment of the present application, a comparative schematic diagram of the images of each step in the processing process is obtained. Specifically, Figure 10 In the figure, the images corresponding to the "event stream" are the five groups of event streams on the MVSEC dataset; the images corresponding to the "first reconstructed image" are the images obtained by processing the five groups of events through steps S101 to S102; the images corresponding to the "repaired image" are the repaired images obtained by processing each group of event streams and the first reconstructed image through steps S103 to S105; the images corresponding to the "real image" are the real images corresponding to the event streams. First, obtain the five groups of event streams on the MVSEC dataset, obtain the real images corresponding to the five groups of event streams, execute steps 101 to S102 for each component stream, obtain the corresponding first reconstructed image, and then execute steps S103 to S105 to obtain the corresponding repaired image. Figure 10 It can be seen that the five restored images corresponding to the five groups of event streams are of higher quality and clearer than the corresponding first reconstructed image, and each restored image is almost consistent with the real image.

[0190] It is understood that the embodiment of the present application also provides a method for training a diffusion model, including but not limited to the following steps:

[0191] Obtaining an initial event stream training sample and obtaining a real image training sample corresponding to the initial event stream training sample;

[0192] Preprocessing the initial event stream training samples to obtain event stream training samples in the form of voxel grids;

[0193] Aggregating information on event stream training samples in the form of voxel grids to obtain a first reconstructed training image;

[0194] Denoising the initial event stream training samples to obtain denoised event stream training samples, and pre-processing the denoised event stream training samples to obtain denoised event stream training samples in the form of voxel grids;

[0195] Superimposing the denoised event stream training sample in the form of a voxel grid with the first reconstructed training image to obtain a second reconstructed training image;

[0196] Noise the real image training samples to generate noisy training images;

[0197] The noisy training image and the second reconstructed training image are input into the diffusion model, and the noisy training image and the second reconstructed training image are spliced ​​in the channel dimension through the denoising network of the diffusion model to generate predicted noise of the noisy training image;

[0198] The loss value is calculated based on the predicted noise and the second reconstructed training image;

[0199] The diffusion model is optimized according to the loss value to obtain a trained diffusion model.

[0200] It's worth noting that the Adam optimizer was used to optimize the network parameters of the diffusion model, with a learning rate of 0.0001 and a dropout rate of 0.2 to prevent overfitting. It's important to note that the diffusion model primarily includes a denoising network, Markov operations, and inverse Markov operations. Therefore, during training optimization, the denoising network is the primary focus.

[0201] See also Figure 11 The present application also provides an event camera image processing system based on a diffusion model, which can implement the above-mentioned event camera image processing method. The event camera image processing system based on a diffusion model includes:

[0202] An acquisition module 111 is used to acquire an initial event stream and pre-process the initial event stream to obtain an event stream in a voxel grid form;

[0203] A reconstruction module 112 is configured to aggregate information on the event stream in the form of a voxel grid to obtain a first reconstructed image;

[0204] An event stream denoising module 113 is configured to denoise the initial event stream to obtain a denoised event stream, and pre-process the denoised event stream to obtain a denoised event stream in the form of a voxel grid;

[0205] an overlay module 114 for overlaying the denoised event stream in the form of a voxel grid with the first reconstructed image to obtain a second reconstructed image;

[0206] The image generation module 115 is used to generate a noisy image, input the noisy image and the second reconstructed image into a pre-trained diffusion model, use the second reconstructed image as a condition of the diffusion model, iteratively denoise the noisy image through the denoising network of the diffusion model, and generate a repaired image.

[0207] The specific implementation of this diffusion model-based event camera image processing system is essentially the same as the specific embodiment of the event camera image processing method described above and will not be further described here. Provided that the requirements of the embodiments of this application are met, the diffusion model-based event camera image processing system may further include other functional modules to implement the event camera image processing method described in the above embodiments.

[0208] The present application also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the image processing method for the event camera. The electronic device can be any smart terminal, such as a tablet computer or an in-vehicle computer.

[0209] See also Figure 12 , Figure 12 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0210] The processor 1201 can be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0211] The memory 1202 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1202 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1202 and is called by the processor 1201 to execute the image processing method of the event camera in the embodiments of this application.

[0212] Input / output interface 1203, used to implement information input and output;

[0213] Communication interface 1204, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0214] Bus 1205 , which transmits information between various components of the device (e.g., processor 1201 , memory 1202 , input / output interface 1203 , and communication interface 1204 );

[0215] The processor 1201 , the memory 1202 , the input / output interface 1203 and the communication interface 1204 are connected to each other in communication within the device via the bus 1205 .

[0216] An embodiment of the present application further provides a computer-readable storage medium storing a computer program, which implements the above-mentioned image processing method for an event camera when executed by a processor.

[0217] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0218] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0219] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0220] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0221] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0222] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0223] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0224] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of the above units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0225] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0226] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0227] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0228] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. An image processing method for an event camera, characterized in that: The method comprises: Acquire an initial event stream, and pre-process the initial event stream to obtain the event stream in a voxel grid form; performing information aggregation on the event stream in the form of a voxel grid to obtain a first reconstructed image; Performing denoising on the initial event stream to obtain a denoised event stream, and preprocessing the denoised event stream to obtain the denoised event stream in a voxel grid form; superimposing the denoised event stream in voxel grid form with the first reconstructed image to obtain a second reconstructed image; A noisy image is generated, and the noisy image and the second reconstructed image are input into a pre-trained diffusion model. The second reconstructed image is used as a condition of the diffusion model, and the noisy image is iteratively denoised by a denoising network of the diffusion model to generate a repaired image.

2. The image processing method of an event camera according to claim 1, characterized in that: The step of aggregating information on the event stream in the form of a voxel grid to obtain a first reconstructed image includes: Performing circular convolution on the event stream in the form of a voxel grid through a circular convolution module to obtain a multi-scale feature map; Performing information aggregation on the multi-scale feature map through an information aggregation module to obtain an information aggregation feature result; Upsampling the information aggregation feature result through a multi-dimensional upsampling module to obtain a first upsampled image, and upsampling the multi-scale feature map through the multi-dimensional upsampling module to obtain a second upsampled image; The first up-sampled image and the second up-sampled image are combined to obtain the first reconstructed image.

3. The image processing method of an event camera according to claim 1, characterized in that: The denoising process is performed on the initial event stream to obtain a denoised event stream, comprising: Generate a preliminary movement trajectory of the initial event flow on a first coordinate plane; the first coordinate plane is constructed based on the initial event flow; Projecting the event stream onto a preset second coordinate plane according to the preliminary movement trajectory to obtain a projected event stream; Calculating the total number of projection events of the projection event stream on the first coordinate plane; calculating a variance of the total number of projection events, and determining a predicted movement trajectory of the object in the event stream based on the variance and the preliminary movement trajectory; Projecting the event stream onto a second coordinate plane according to the predicted movement trajectory to obtain a target projected event stream; The number of projection events in each pixel of the first coordinate plane is determined according to the target projection event stream. If the number of projection events is less than an adaptive threshold, the events in the pixel are deleted to obtain the denoised event stream.

4. The image processing method of an event camera according to claim 1, characterized in that: The step of using the second reconstructed image as a condition of the diffusion model and denoising the noisy image through a denoising network of the diffusion model to generate a repaired image includes: Using the denoising network, concatenate the second reconstructed image and the noisy image in a channel dimension to generate a predicted noise vector; Iterative denoising is performed according to an iterative formula and the predicted noise vector until the number of iterations reaches a preset number to obtain the repaired image; the iterative formula is: in, Characterization The noise image at time t, Characterization The noise image at time -1, , is a hyperparameter, Characterize the denoising network, characterizing the second reconstructed image, Characterization The noise parameters at time t, , Characterizing the preset number of times, for The predicted noise vector at time .

5. The image processing method of an event camera according to claim 1, characterized in that: The diffusion model is trained by the following steps: Obtaining an initial event stream training sample, and obtaining a real image training sample corresponding to the initial event stream training sample; Preprocessing the initial event stream training samples to obtain the event stream training samples in a voxel grid form; performing information aggregation on the event stream training samples in a voxel grid format to obtain a first reconstructed training image; Performing denoising processing on the initial event stream training samples to obtain denoised event stream training samples, and preprocessing the denoised event stream training samples to obtain the denoised event stream training samples in the form of a voxel grid; superimposing the denoised event stream training samples in a voxel grid format with the first reconstructed training image to obtain a second reconstructed training image; Noising the real image training sample to generate a noisy training image; Inputting the noisy training image and the second reconstructed training image into a diffusion model, and performing channel-wise splicing on the noisy training image and the second reconstructed training image using a denoising network of the diffusion model to generate predicted noise for the noisy training image; Calculating a loss value based on the predicted noise and the second reconstructed training image; The diffusion model is optimized according to the loss value to obtain the trained diffusion model.

6. The image processing method of an event camera according to claim 5, characterized in that: The calculating the loss value according to the predicted noise and the second reconstructed training image includes: The loss value is calculated according to the objective function, and the objective function is: in, Characterize the prediction noise, , Represents the identity matrix; Characterize the denoising network, Characterize the noise parameters, ; p= 1, represents the L1 norm; Representing real image training samples, characterizing the second reconstructed training image, Representing the mathematical expectation, Characterize the real image training sample set.

7. A method for training a diffusion model, characterized in that: include: Obtaining an initial event stream training sample, and obtaining a real image training sample corresponding to the initial event stream training sample; Preprocessing the initial event stream training samples to obtain the event stream training samples in a voxel grid form; performing information aggregation on the event stream training samples in a voxel grid format to obtain a first reconstructed training image; Performing denoising processing on the initial event stream training samples to obtain denoised event stream training samples, and preprocessing the denoised event stream training samples to obtain the denoised event stream training samples in the form of a voxel grid; superimposing the denoised event stream training samples in a voxel grid format with the first reconstructed training image to obtain a second reconstructed training image; Noising the real image training sample to generate a noisy training image; Inputting the noisy training image and the second reconstructed training image into a diffusion model, and performing channel-wise splicing on the noisy training image and the second reconstructed training image using a denoising network of the diffusion model to generate predicted noise for the noisy training image; Calculating a loss value based on the predicted noise and the second reconstructed training image; The diffusion model is optimized according to the loss value to obtain the trained diffusion model.

8. An event camera image processing system based on a diffusion model, characterized in that: The system comprises: an acquisition module, configured to acquire an initial event stream, and preprocess the initial event stream to obtain the event stream in a voxel grid form; a reconstruction module, configured to aggregate information on the event stream in a voxel grid format to obtain a first reconstructed image; An event stream denoising module, configured to perform denoising on the initial event stream to obtain a denoised event stream, and pre-process the denoised event stream to obtain the denoised event stream in a voxel grid form; a superposition module, configured to superimpose the denoised event stream in a voxel grid format with the first reconstructed image to obtain a second reconstructed image; An image generation module is configured to generate a noisy image, input the noisy image and the second reconstructed image into a pre-trained diffusion model, use the second reconstructed image as a condition of the diffusion model, and iteratively denoise the noisy image through a denoising network of the diffusion model to generate a repaired image.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the image processing method of the event camera according to any one of claims 1 to 6, or the training method of the diffusion model according to claim 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the image processing method of the event camera according to any one of claims 1 to 6 or the training method of the diffusion model according to claim 7 is implemented.

Images