Image processing method and device and electronic equipment

Abstract

The invention discloses an image processing method and device and electronic equipment, and belongs to the technical field of images. The image processing method comprises the steps of inputting an RGB image and corresponding event camera data into a feature extraction module of an image processing model, performing feature extraction on the RGB image and the event camera data, and outputting image feature data and brightness change feature data; inputting the image feature data and the brightness change feature data into a cross-modal channel attention mechanism module, performing feature fusion processing on the image feature data and the brightness change feature data, and outputting fused feature data; inputting the image feature data and the fused feature data into a coding and decoding module, performing feature splicing extraction on the image feature data and the fused feature data, and outputting spliced feature data; and inputting the RGB image and the splicing feature data into a fusion module, and performing fusion processing on the splicing feature data and the RGB image to obtain a deblurred image of the RGB image.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to an image processing method, device and electronic equipment. Background Art

[0002] Images often suffer from motion blur due to camera shake or object motion during exposure. Given a blurred image, the goal of deblurring is to recover a sharp image with clear edges and texture details from the blurred image.

[0003] Image restoration is the inverse process of image degradation. Most of the existing deblurring methods are based on Bayesian inference. Through statistical theory, image data is modeled and different priors are artificially designed to constrain the solution space of the problem. In real scenes, due to the lack of relevant information about the blur kernel, the blur form of the image is often globally inconsistent, and it is also disturbed by other external factors such as noise and illumination, which makes the image deblurring problem more complicated, and the unknown variables are far more than the known variables. In this regard, a deep network is introduced to estimate the blur kernel, and deconvolution operations or deep networks are used to restore the final clear image. However, due to the need to independently estimate the blur kernel, it is relatively complicated for model training and the construction of blur kernel datasets, and it is extremely inconvenient to use in actual tasks. When the blur kernel estimation is not accurate enough, the error will be passed to the subsequent deconvolution step, which will affect the final restoration effect. Summary of the invention

[0004] The purpose of the embodiment of the present application is to provide an image processing method, which can extract the features of RGB images and event camera data through an image processing model, and combine the cross-modal channel attention mechanism and codec to perform feature fusion on the features of RGB images and event camera data, so as to obtain a mapping relationship from blurred images to clear images, and complete deblurring and restoration. In addition, since event camera data can have the characteristics of high temporal resolution, high dynamic range and high pixel bandwidth that RGB image data does not have. Therefore, event camera data can supplement the image features of RGB images, so that by fusing the features of RGB images and event camera data, richer image features can be obtained to restore blurred images, thereby achieving the purpose of optimizing the restoration effect of blurred images.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising:

[0006] Inputting the RGB image and the corresponding event camera data into a feature extraction module of an image processing model, performing feature extraction on the RGB image and the event camera data through the feature extraction module, and outputting image feature data of the RGB image and brightness change feature data of the event camera data;

[0007] Inputting the image feature data and the brightness change feature data into the cross-modal channel attention mechanism module of the image processing model, performing feature fusion processing on the image feature data and the brightness change feature data through the cross-modal channel attention mechanism module, and outputting fused feature data;

[0008] Inputting the image feature data and the fused feature data into the encoding and decoding module of the image processing model, performing feature splicing extraction on the image feature data and the fused feature data through the encoding and decoding module, and outputting splicing feature data;

[0009] The RGB image and the splicing feature data are input into a fusion module of the image processing model, and the splicing feature data and the RGB image are fused through the fusion module to obtain a deblurred image of the RGB image.

[0010] In a second aspect, an embodiment of the present application provides an image processing device, the device comprising:

[0011] An extraction processing module, used for inputting the RGB image and the corresponding event camera data into a feature extraction module of an image processing model, performing feature extraction on the RGB image and the event camera data through the feature extraction module, and outputting image feature data of the RGB image and brightness change feature data of the event camera data;

[0012] A first fusion processing module, configured to input the image feature data and the brightness change feature data into a cross-modal channel attention mechanism module of the image processing model, perform feature fusion processing on the image feature data and the brightness change feature data through the cross-modal channel attention mechanism module, and output fusion feature data;

[0013] A splicing extraction processing module, used for inputting the image feature data and the fused feature data into the encoding and decoding module of the image processing model, performing feature splicing extraction on the image feature data and the fused feature data through the encoding and decoding module, and outputting splicing feature data;

[0014] The second fusion processing module is used to input the RGB image and the splicing feature data into the fusion module of the image processing model, and perform fusion processing on the splicing feature data and the RGB image through the fusion module to obtain a deblurred image of the RGB image.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps described in the first aspect are implemented.

[0016] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0017] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0018] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0019] In an embodiment of the present application, after the image feature data of the RGB image and the brightness change feature data of the event camera data are extracted by the feature extraction module of the image processing model, the image feature data and the brightness change feature data are fused to obtain fused feature data by using the cross-modal channel attention mechanism module of the image processing model, and then the image feature data and the fused feature data are subjected to feature splicing extraction to obtain splicing feature data in combination with the encoding and decoding module of the image processing model, so as to obtain the deblurred image of the RGB image by fusing the splicing feature data and the RGB image through the fusion module of the image processing model. In this technical solution, the image processing model can extract the features of the RGB image and the event camera data, and combine the cross-modal channel attention mechanism and encoding and decoding to perform feature fusion on the features of the RGB image and the event camera data, thereby obtaining a mapping relationship from the blurred image to the clear image, and completing the deblurring restoration. And because the event camera data can have the characteristics of high temporal resolution, high dynamic range and high pixel bandwidth that the RGB image data does not have. Therefore, the event camera data can supplement the image features of the RGB image, so that by fusing the features of the RGB image and the event camera data, more abundant image features can be obtained to restore the blurred image and optimize the restoration effect of the blurred image. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flowchart of an image processing method provided by some embodiments of the present application;

[0021] Figure 2 is a schematic diagram of the structure of an image processing model provided by some embodiments of the present application;

[0022] Figure 3 is a schematic diagram of the structure of a cross-modal channel attention mechanism module provided by some embodiments of the present application;

[0023] Figure 4is a schematic diagram illustrating an image exposure process provided by some embodiments of the present application;

[0024] Figure 5 is a schematic diagram illustrating the correlation between RGB images and event data provided by some embodiments of the present application;

[0025] Figure 6 is a schematic diagram of the structure of a channel attention mechanism module provided in some embodiments of the present application;

[0026] Figure 7 is a block diagram of an image processing device provided by some embodiments of the present application;

[0027] Figure 8 is a schematic diagram of the hardware structure of an electronic device provided in some embodiments of the present application;

[0028] Fig. 9 It is a schematic diagram of the hardware structure of an electronic device provided in some embodiments of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings of the embodiments of the present application to clearly describe the technical solutions of the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present application belong to the scope of protection of this application.

[0030] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally a class, and the number of objects is not limited. For example, the target object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0031] The image processing method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0032] For ease of understanding, the concepts involved in the technical solution of this application are first explained below.

[0033] Event camera data: refers to the data collected by the event camera.

[0034] Event Camera: Also known as Event Camera Sensor (EVS), it is a neuromorphic sensor. Event cameras can be used to asynchronously detect brightness changes of each pixel. When the logarithmic change of the potential irradiance exceeds the preset threshold, an output event is triggered. This event can be simply understood as "the change in pixel brightness". Event cameras have characteristics that traditional image red (R), green (G), and blue (B) cameras do not have: high temporal resolution, very high dynamic range, low power consumption, and high pixel bandwidth, making them very suitable for image deblurring.

[0035] For a typical event camera, each event can be represented as (x, y, t, p). (x, y) represents the pixel coordinate position corresponding to the output event triggered at time t, and p represents the polarity. The definition is as follows:

[0036]

[0037] In formula (1), I represents the brightness value, u represents (x, y), c(u) represents the preset threshold value of the position corresponding to (x, y), 1 represents an increase in brightness, and -1 represents a decrease in brightness.

[0038] The blurred image can be described as the integral of the light source illumination I(t) over time [0, t], see formula (2):

[0039]

[0040] In formula (2), B represents the blurred image, ∈ = ∈(u), represents the image noise in the imaging pipeline in the camera, T represents the exposure time, and f and t both represent the current moment.

[0041] Combining formula (1) and formula (2), the formula of event-based double integral model (EDI) is as follows:

[0042]

[0043] In formula (3), S represents a clear image, B represents a blurred image, and E(t) represents the event expression of EVS in the time interval [fT / 2, f+T / 2]. See formula (4):

[0044]

[0045] It can be seen that according to the blurred image B and the event expression E(t) in the corresponding time interval, the image can be deblurred to obtain a clear image S. Therefore, from formula (3), formula (5) can be obtained:

[0046]

[0047] In formula (5), represents a clear image of the prediction, Represents a blurred image. In this way, based on formula (5), the RGB image and its corresponding event camera data can be used as input to learn the mapping relationship from blurred image to clear image and realize image deblurring and restoration. Among them, the RGB image is the blurred image Event camera data is also called EVS data, which is the event expression E(t).

[0048] It should be noted that the image processing method provided in the embodiments of the present application can be executed by electronic devices such as mobile phones, tablet computers, laptop computers, PDAs, and vehicle-mounted electronic devices. In some embodiments of the present application, the image processing method provided in the embodiments of the present application is described by taking the electronic device as the execution subject to execute the image processing method as an example.

[0049] The image processing method provided in the embodiment of the present application can be applied to the scene of restoring a blurred image with missing details and unclear edges into a clear image with rich details and clear edges, wherein a specific application scenario is to restore a motion blurred image, which refers to a blurred image caused by the movement of the camera or the photographed object. For example, when a camera shoots a moving object, a motion blurred image may be obtained, and the motion blurred image has obvious traces of the object moving and dragging. By deblurring the motion blurred image, an image of the object with clear moving motion is obtained. Another specific application scenario is to restore a focused blurred image. A focused blurred image is usually caused by the failure to correctly focus on the photographed object during shooting. In a focused blurred image, the photographed object usually has unclear edges. For example, if the camera fails to accurately focus on the flower when shooting the flower, the image obtained may have a problem of blurred flower boundaries. By deblurring the image, a flower image with clear flower boundaries can be obtained.

[0050] Please refer to Figure 1 , which shows a flowchart of an image processing method provided by an embodiment of the present application.

[0051] like Figure 1 As shown, the image processing method includes:

[0052] Step 110: Input the RGB image and the corresponding event camera data into the feature extraction module of the image processing model, perform feature extraction on the RGB image and the event camera data through the feature extraction module, and output image feature data of the RGB image and brightness change feature data of the event camera data.

[0053] In the embodiment of the present application, the image processing model is a data module that implements image deblurring based on RGB images and event camera data. The image processing model is also called a deblurring neural network model, an event-assisted deblurring (EventAidedDeblur, EAD) network (Net) model (EADNet). Figure 2 As shown, the image processing model includes a feature extraction module, a cross-modal channel attention mechanism module 22, a codec module 23 and a fusion module 24.

[0054] The feature extraction module in the image processing model is used to extract features from the RGB image and the event camera data, respectively, and output image feature data of the RGB image and brightness change feature data of the event camera data. Optionally, the image feature data of the RGB image includes, but is not limited to, at least one of edge feature data, line feature data, texture feature data, and corner feature data. The brightness change feature data of the event camera data may include, but is not limited to, brightness change feature data of each pixel row in the RGB image during the exposure period.

[0055] Step 120: Input the image feature data and the brightness change feature data into the cross-modal channel attention mechanism module of the image processing model, perform feature fusion processing on the image feature data and the brightness change feature data through the cross-modal channel attention mechanism module, and output the fused feature data.

[0056] In an embodiment of the present application, the cross-modal channel attention mechanism module of the image processing model, also known as the Event-Based Cross-modal Attention Fusion (EBCAF) module, is used to perform feature fusion processing on the image feature data and the brightness change feature data, and output fused feature data. Optionally, the feature fusion processing can be by splicing, weighting or combining the image feature data and the brightness change feature data to output more expressive fused feature data, which can be a feature vector.

[0057] Step 130: input the image feature data and the fused feature data into the encoding and decoding module of the image processing model, perform feature splicing extraction on the image feature data and the fused feature data through the encoding and decoding module, and output splicing feature data.

[0058] In an embodiment of the present application, the encoding and decoding module of the image processing model is used to perform feature splicing extraction on the image feature data and the fused feature data, and output the spliced ​​feature data. Among them, the feature splicing extraction process can also be understood as a specific feature fusion process, which splices at least two data of the input process to obtain spliced ​​data, and then performs data feature extraction on the spliced ​​data to output feature data of the spliced ​​data, that is, spliced ​​feature data. Optionally, in the feature splicing extraction process, the at least two data of the input process can be spliced ​​in the depth dimension.

[0059] Step 140: Input the RGB image and the splicing feature data into a fusion module of the image processing model. The fusion module performs fusion processing on the splicing feature data and the RGB image to obtain a deblurred image of the RGB image.

[0060] In the embodiment of the present application, the fusion module of the image processing model is used to perform fusion processing on the spliced ​​feature data and the RGB image to obtain a deblurred image of the RGB image. Optionally, the fusion processing may be a weighted fusion processing on the spliced ​​feature data and the RGB image to obtain a deblurred image of the RGB image. Alternatively, the fusion processing may be an addition processing on the spliced ​​feature data and the RGB image to obtain a deblurred image of the RGB image.

[0061] In an embodiment of the present application, after the image feature data of the RGB image and the brightness change feature data of the event camera data are extracted by the feature extraction module of the image processing model, the image feature data and the brightness change feature data are fused to obtain fused feature data by using the cross-modal channel attention mechanism module of the image processing model, and then the image feature data and the fused feature data are subjected to feature splicing extraction in combination with the codec module of the image processing model to obtain splicing feature data, so as to obtain the deblurred image of the RGB image by fusing the splicing feature data and the RGB image through the fusion module of the image processing model. In this technical solution, the image processing model can extract the features of the RGB image and the event camera data, and combine the cross-modal channel attention mechanism and codec to perform feature fusion on the features of the RGB image and the event camera data, thereby obtaining a mapping relationship from a blurred image to a clear image, and completing deblurring restoration. And because the event camera data can have the characteristics of high temporal resolution, high dynamic range and high pixel bandwidth that the RGB image data does not have. Therefore, the event camera data can supplement the image features of the RGB image, so that by fusing the features of the RGB image and the event camera data, richer image features can be obtained to restore the blurred image, thereby improving the restoration effect of the blurred image.

[0062] In the process of the image processing method of another embodiment of the present application, as Figure 2As shown, the cross-modal channel attention mechanism module 22 may include: a first cross-modal attention submodule (i.e., a first EBCAF). Accordingly, fusing feature data includes: directly fusing feature data. Based on this, step 120 performs feature fusion processing on the image feature data and the brightness change feature data through the cross-modal channel attention mechanism module, and the process of outputting the fused feature data may include:

[0063] The image feature data and the brightness change feature data are input into the first cross-modal attention submodule, and the directly fused feature data is output. The directly fused feature data is the data after the image feature data and the brightness change feature data are directly fused. Optionally, the first cross-modal attention submodule can be used to fuse the image feature data and the brightness change feature data by splicing, weighting or combination, so as to output more expressive fused feature data, so as to facilitate the subsequent use of the richly expressed fused feature data for blurred image restoration and improve the restoration effect of blurred images.

[0064] In an optional implementation, the electronic device inputs the image feature data and the brightness change feature data into the first cross-modal attention submodule, and the process of outputting the directly fused feature data may include the following steps A1201 to A1207.

[0065] In step A1201, the image feature data is subjected to normalization processing, convolution processing, and dimensionality adjustment processing in sequence to obtain a first query vector.

[0066] Among them, through the first cross-modal attention submodule, the image feature data can be normalized to convert the image feature data into scalar data to obtain normalized data. Then, the normalized data is convolved to extract features from the normalized data to obtain convolution feature data. After that, the convolution feature data is dimensionally adjusted to convert each dimensional data in the convolution feature data into feature data in a preset dimensional format to obtain a first query vector (query) Q.

[0067] For example, Figure 3As shown, the first cross-modal attention submodule includes a normalization (Norm) layer, a 1x1 convolution (Convolution, Conv) layer and a dimension adjustment module. The electronic device can sequentially pass through the normalization layer 301, the 1x1 convolution layer 302 and the dimension adjustment module of the first cross-modal attention submodule to perform normalization processing, convolution processing, and dimension adjustment processing on the image feature data Ima in turn to obtain feature data in a preset dimensional format. Furthermore, the feature data in the preset dimensional format and the preset query weight matrix WQ are linearly changed to obtain a first query vector Q. Among them, the dimensional adjustment processing includes shape shaping (Reshape, R) processing and position exchange (Transpose, T) processing. The shape shaping processing is used to change the shape of the convolution feature data, but does not change the total amount of convolution feature data. The position exchange processing is used to change the dimension of the convolution feature data. In addition, the preset dimensional format can be cx HW or HW xc. cx represents the number of channels, H represents the height of the feature data (i.e., the feature data graph), and W represents the width of the feature data.

[0068] In step A1202, the brightness change feature data is subjected to normalization processing, convolution processing, and dimensionality adjustment processing in sequence to obtain a first key vector and a first value.

[0069] Among them, through the first cross-modal attention submodule, the brightness change feature data can be normalized to convert the brightness change feature data into scalar data to obtain normalized data. Then, the normalized data is convolved to extract features from the normalized data to obtain convolution feature data. After that, the convolution feature data is dimensionally adjusted to convert each dimensional data in the convolution feature data into feature data in a preset dimensional format to obtain a first key vector (key) K and a first value (value) V.

[0070] For example, please continue to refer to Figure 3 , the first cross-modal attention submodule also includes a normalization layer, a 1x1 convolution layer and a dimension adjustment module. The electronic device can sequentially pass through the normalization layer 303, the 1x1 convolution layer 304 and the dimension adjustment module of the first cross-modal attention submodule to sequentially perform normalization processing, convolution processing, and dimension adjustment processing on the brightness change feature data Eve to obtain feature data in a preset dimensional format. Furthermore, a linear change processing is performed on the feature data in the preset dimensional format and the preset key matrix WK to obtain a first key vector K, and a linear change processing is performed on the feature data in the preset dimensional format and the preset value matrix WV to obtain a first value V. Among them, the dimensional adjustment processing includes shape shaping processing. The preset dimensional format can be cx HW or HWx c.

[0071] In step A1203, first attention map data is output based on the dot product of the first query vector and the first key vector.

[0072] Among them, through the first cross-modal attention submodule, the dot product of the first query vector Q and the first key vector K is calculated, and the first attention map data C×C is output. The first attention map data C×C is used to represent the correlation between the first query vector Q and the first key vector K.

[0073] In step A1204, first attention feature map data is output based on the product of the first attention map and the first value.

[0074] Among them, the product of the first attention map C×C and the first value V is calculated by the first cross-modal attention submodule, and the first attention feature map data is output. Optionally, the first attention map C×C and the first value V can be normalized and convolved by a normalized exponential function (softmax function) in sequence to obtain the first attention feature map data.

[0075] In step A1205, the first image feature data and the first attention feature map data are added to obtain first spliced ​​map data.

[0076] Optionally, the first cross-modal attention submodule can be used to add the feature data of each corresponding channel in the first image feature data and the first attention feature map data to obtain the first spliced ​​map data.

[0077] In step A1206, the first spliced ​​graph data is input into a feedforward neural network, and the first network processed graph data is output.

[0078] Among them, the feedforward neural network (FFN) is used to perform nonlinear mapping on the first spliced ​​image data to enhance the characteristics of the first spliced ​​image data, while avoiding gradient vanishing, accelerating the convergence speed of the first cross-modal attention submodule, and thereby improving the convergence speed of the image processing model.

[0079] Optionally, the feedforward neural network may include a first fully connected layer, a Gaussian Error Linear Units (GELU) layer, and a second fully connected layer. Among them, the Gaussian Error Linear Unit layer is essentially an activation function with a strong nonlinear mapping construction capability. Therefore, the feedforward neural network can be used to perform nonlinear mapping processing on the input data through the Gaussian Error Linear Unit layer to achieve the purpose of enhancing the characteristics of the input data, effectively avoiding the disappearance of the gradient, and improving the convergence speed of the model.

[0080] In step A1207, the first network processed graph data and the first spliced ​​graph data are added to output direct fusion feature data.

[0081] Optionally, the feature data of each corresponding channel in the first network processed graph data and the first spliced ​​graph data can be added through the first cross-modal attention submodule to obtain directly fused feature data.

[0082] In the embodiment of the present application, a specific processing flow of a first cross-modal attention submodule is proposed. In the processing flow, the first query vector of the image feature data of the RGB image and the first key vector and the first value of the brightness change feature data of the event camera data are extracted by using the self-attention and cross-attention mechanisms, so as to construct the direct fusion feature data of the RGB image and the event camera data through a feedforward neural network by using the first query vector, the first key vector and the first value, so as to realize global cross-modal feature interaction and information fusion, so that the first cross-modal attention submodule can have a stronger ability to capture global features, expand the spatial range of feature extraction, and effectively improve the feature fusion effect of the event camera data and the RGB image.

[0083] In the process of the image processing method of another embodiment of the present application, as Figure 2 As shown, the encoding and decoding module 23 may include: a first encoding module 231, a first decoding module 232, a second encoding module 233 and a second decoding module 234. Based on this, step 130 performs feature splicing extraction on the image feature data and the fusion feature data through the encoding and decoding module, and the process of outputting the spliced ​​feature data may include:

[0084] In step 1301, image feature data is input into a first encoding module, and image encoding data is output.

[0085] Optionally, the first encoding module may perform downsampling processing on the image feature data to compress the size of the image feature data and reduce the amount of subsequent data calculations, and then perform feature extraction processing on the downsampled image feature data to obtain and output image encoding data.

[0086] For example, Figure 2 As shown, the first encoding module 231 includes a first downsampling layer and a first residual convolution layer. The image feature data is sequentially input into the first downsampling layer and the first residual convolution layer, so that the image feature data is sequentially downsampled through the first downsampling layer, and the feature extraction process is performed on the downsampled image feature data through the first residual convolution layer, and the image encoding data is output.

[0087] In step 1302, the image encoding data is input into the second encoding module, and the re-encoded image data is output.

[0088] Optionally, the image coding data can be downsampled by the second coding module to further compress the size of the image coding data and reduce the amount of subsequent data calculation. Then, feature extraction is performed on the downsampled image coding data to obtain and output re-encoded image data. In some embodiments, the first coding module and the second coding module can have the same sampling size for downsampling their input data. Of course, the first coding module and the second coding module can also have different sampling sizes for downsampling their input data, which can be set according to actual needs.

[0089] For example, Figure 2 As shown, the second encoding module 233 includes a second downsampling layer and a second residual convolution layer. The image encoding data is sequentially input into the second downsampling layer and the second residual convolution layer, so that the image encoding data is sequentially downsampled through the second downsampling layer, and the feature extraction process is performed on the image encoding data after the downsampling process through the second residual convolution layer, and the re-encoded image data is output.

[0090] In step 1303, the re-encoded image data and the image encoding data are input into a second decoding module, and basic splicing feature data is output.

[0091] Among them, the basic splicing feature is the feature data obtained by performing feature extraction processing on the image data after decoding the re-encoded image data and the image coding data. Optionally, the re-encoded image data can be up-sampled by the second decoding module to restore the size of the re-encoded image data to the size of the image coding data. Then, the re-encoded image data and the image coding data after the up-sampling process can be spliced, and the feature extraction process can be performed on the spliced ​​data to obtain and output the basic splicing feature data. Among them, in some embodiments, the data size of the re-encoded image data and the image coding data is the same. Therefore, the re-encoded image data and the image coding data can be spliced ​​in the depth dimension.

[0092] For example, Figure 2 As shown, the second decoding module 234 includes a second upsampling layer and a fourth residual convolution layer, and the fourth residual convolution layer is jump-connected to the first residual convolution layer of the first encoding process, so as to receive the image encoding data output by the first residual convolution layer. The re-encoded image data is sequentially input into the second upsampling layer and the fourth residual convolution layer, so as to sequentially perform upsampling processing on the re-encoded image data through the second upsampling layer, perform data splicing on the re-encoded image data and the image encoding data after the upsampling processing through the fourth residual convolution layer, perform feature extraction processing on the spliced ​​data, and output basic splicing feature data.

[0093] In step 1304, the image feature data, the direct fusion feature data and the decoded image data are input into a first decoding module, and the spliced ​​feature data is output.

[0094] The spliced ​​feature data is feature data obtained by performing feature extraction processing on the decoded image data, the directly fused feature data, and the image feature data after splicing.

[0095] Optionally, the decoded image data may be upsampled by the first decoding module to restore the size of the decoded image data to the size of the image feature data. Then, the decoded image data, image feature data and directly fused feature data after upsampling may be spliced, and feature extraction may be performed on the spliced ​​data to obtain and output spliced ​​feature data. In some embodiments, the sizes of the image feature data, fused feature data and decoded image data are the same. Therefore, the image feature data, fused feature data and decoded image data may be spliced ​​in the depth dimension.

[0096] For example, Figure 2 As shown, the first decoding module 232 includes a first upsampling layer and a third residual convolution layer, and the third residual convolution layer is jump-connected to the feature extraction module to receive the image feature data of the RGB image output by the feature extraction module. The decoded image data is sequentially input into the first upsampling layer and the third residual convolution layer, so that the decoded image data is sequentially upsampled through the first upsampling layer, and the decoded image data, image feature data and direct fusion feature data after upsampling are spliced ​​through the third residual convolution layer, and feature extraction is performed on the spliced ​​data to output spliced ​​feature data.

[0097] In an embodiment of the present application, the encoding module of the image processing model includes a first encoding module, a first decoding module, a second encoding module, and a second decoding module to perform two-stage encoding and decoding. In this way, the image processing model can be an encoder-decoder structure based on the UNet architecture. For the main encoding and decoding path under this structure, it consists of two downsampling layers and two upsampling layers to achieve two-stage encoding and decoding. Among them, the two downsampling layers can correspond to the first downsampling layer and the second downsampling layer in the image processing model respectively; the two upsampling layers can correspond to the first upsampling layer and the second upsampling layer in the image processing model respectively. In addition, under the UNet architecture, the added residual convolution layer can be called a residual connection network (Residual Net, Res) convolution block, which is used to perform residual convolution operations on input data. The residual convolution layer can be composed of two 3×3 convolution layers and a 1×1 convolution layer directly connected to the leaky ReLU function. The first residual convolution layer, the second residual convolution layer, the third residual convolution layer, and the fourth residual convolution layer in the image processing model can correspond to the residual convolution layer.

[0098] In an embodiment of the present application, the encoding module can implement a two-stage encoding and decoding process of the input image feature data through a first encoding module, a first decoding module, a second encoding module, and a second decoding module. In addition, the encoding module can be combined with a cross-modal channel attention mechanism so that the second decoding module can obtain the direct fusion feature data output by the first cross-modal attention submodule after fusing the feature data of the RGB image and the event camera data, and re-extract the fusion features of the direct fusion feature data, the image feature data, and the decoded image data to obtain richer image features, which is convenient for subsequent processing to obtain an effective mapping relationship from a blurred image to a clear image, thereby improving the restoration effect of the blurred image.

[0099] In the process of the image processing method of another embodiment of the present application, as Figure 2 As shown, the image processing model may further include a third encoding module 35. The cross-modal channel attention mechanism module 22 includes: a second cross-modal attention submodule (i.e., a second EBCAF). Accordingly, the fused feature data also includes: compressed fused feature data. Based on this, the image processing method further includes: inputting the brightness change feature data into the third encoding module and outputting the brightness feature encoding data.

[0100] Optionally, the brightness change feature data can be downsampled by a third encoding module to compress the size of the brightness change feature data and reduce the amount of subsequent data calculation. Then, feature extraction is performed on the downsampled brightness change feature data to obtain and output brightness feature encoded data. The encoding size of the third encoding module is the same as that of the first encoding module. Therefore, the downsampling granularity of the third encoding module is the same as that of the first encoding module, so that the compression ratio of the brightness feature encoded data and the image feature data is the same.

[0101] For example, Figure 2 As shown, the third encoding module 35 includes a third downsampling layer and a fifth residual convolution layer. The brightness change feature data is sequentially input into the third downsampling layer and the fifth residual convolution layer, so that the brightness change feature data is sequentially downsampled through the third downsampling layer, and the brightness change feature data after the downsampling is subjected to feature extraction through the fifth residual convolution layer, and the brightness feature encoding data is output.

[0102] Correspondingly, optionally, step 120 performs feature fusion processing on the image feature data and the brightness change feature data through a cross-modal channel attention mechanism module, and the process of outputting the fused feature data may also include:

[0103] The brightness feature encoding data and the image encoding data are input into the second cross-modal attention submodule, and the compressed fused feature data is output. Optionally, the second cross-modal attention submodule can be used to fuse the feature data of the image feature data and the brightness change feature data after the first level encoding by splicing, weighting or combination. In this way, the feature data of the image feature data and the brightness change feature data are directly fused in combination with the first cross-modal attention submodule. The cross-modal attention module in the image processing model of the present application can realize the fusion of feature data of multi-level RGB images and event camera data, so as to obtain multi-level fused feature data with rich expressions, so as to facilitate the subsequent use of multi-level fused feature data with rich expressions to restore blurred images and improve the restoration effect of blurred images.

[0104] Further correspondingly and optionally, the process of inputting the re-encoded image data and the image encoding data into the second decoding module and outputting the basic splicing feature data in step 1303 may include:

[0105] The re-encoded image data, the compressed fusion feature data and the image coding data are input into the second decoding module, and the basic splicing feature data is output. The basic splicing feature is the feature data obtained by performing feature extraction processing on the image data after decoding the re-encoded image data, the image coding data and the compressed fusion feature data.

[0106] Optionally, the re-encoded image data may be up-sampled by the second decoding module to restore the size of the re-encoded image data to the size of the image encoding data. Then, the re-encoded image data after the up-sampling process, the compressed fusion feature data and the image encoding data may be spliced, and feature extraction may be performed on the spliced ​​data to obtain and output basic splicing feature data.

[0107] For example, Figure 2 As shown, the re-encoded image data is sequentially input into the second up-sampling layer and the fourth residual convolution layer, so as to sequentially perform up-sampling processing on the re-encoded image data through the second up-sampling layer, and the re-encoded image data, the compressed fusion feature data and the image coding data after the up-sampling processing are spliced ​​through the fourth residual convolution layer, and feature extraction processing is performed on the spliced ​​data to output basic splicing feature data.

[0108] In an optional implementation, the electronic device inputs the brightness feature encoding data and the image encoding data into the second cross-modal attention sub-module, and the process of outputting the compressed fusion feature data may include the following steps B1201 to B1207.

[0109] In step B1201, the brightness feature encoding data is subjected to normalization processing, convolution processing, and dimensionality adjustment processing in sequence to obtain a second query vector.

[0110] In step B1202, the image coding data is subjected to normalization processing, convolution processing, and dimension adjustment processing in sequence to obtain a second key vector and a second value;

[0111] In step B1203, based on the dot product of the second query vector and the second key vector, output second attention map data;

[0112] In step B1204, based on the product of the second attention map and the second value, output the second attention feature map data;

[0113] In step B1205, the second image feature data and the second attention feature map data are added to obtain second spliced ​​map data;

[0114] In step B1206, the second spliced ​​graph data is input into a feedforward neural network, and the second network processed graph data is output;

[0115] In step B1207, the second network processed graph data and the second spliced ​​graph data are added to output compressed fusion feature data.

[0116] It should be noted that the second cross-modal attention submodule processes its input data in the same way as the first cross-modal attention submodule processes its input data. That is, the explanation and implementation of steps B1201 to B1207 can refer to the explanation and implementation of the aforementioned steps A1201 to A1207, and the embodiments of the present application will not be repeated here. Similarly, a specific processing flow of a second cross-modal attention submodule is proposed in the embodiments of the present application. In this processing flow, the second query vector of the image coding data after the first-level encoding of the image feature data of the RGB image, and the second key vector and second value of the brightness feature coding data after the first-level encoding of the brightness change feature data of the event camera data are extracted by using the self-attention and cross-attention mechanisms, so as to construct the compressed fusion feature data of the RGB image and the event camera data through a feedforward neural network by using the second query vector, the second key vector and the second value, so as to realize the global cross-modal feature interaction and information fusion of the data level after the first-level encoding, and enrich the fusion data of the event camera data and the RGB image.

[0117] In an embodiment of the present application, the encoding module can implement a two-stage encoding and decoding process of the input image feature data through a first encoding module, a first decoding module, a second encoding module, and a second decoding module. In addition, the encoding module can be combined with a cross-modal channel attention mechanism so that the first decoding module can obtain the feature data of the second cross-modal attention submodule fused image feature data and brightness change feature data after first-level encoding, and output direct fusion feature data to re-encode image data, compress fusion feature data, and image encoding data to input the fusion feature of the second decoding module for re-extraction.

[0118] In addition, the second decoding module can also obtain the direct fusion feature data output by the first cross-modal attention submodule after fusing the feature data of the RGB image and the event camera data, and re-extract the fusion features of the direct fusion feature data, the image feature data, and the decoded image data. In this way, the combination of the encoding and decoding and the cross-modal attention module can realize the fusion of the feature data of the multi-level RGB image and the event camera data, so as to obtain the multi-level fusion feature data with rich expression, so that the subsequent use of the multi-level fusion feature data can be processed to obtain a more effective mapping relationship from the blurred image to the clear image, thereby improving the restoration effect of the blurred image.

[0119] In the process of the image processing method of another embodiment of the present application, before step 110, the image processing method further includes:

[0120] Step C1: Obtain the exposure period of each pixel row of the RGB image.

[0121] After the exposure time for shooting the RGB image is determined, the exposure period of each pixel row (i.e., exposure row) of the RGB image can be calculated according to the rolling shutter characteristic of the camera. For reference, for each pixel row, three processes need to be performed: reset, light exposure, and reading.

[0122] Among them, Figure 4 As shown, T r (time reset) 401 represents the reset time of the pixel row, which refers to the time it takes for each pixel row to empty the potential well before formally sensing light. This time is generally in the nanosecond (ns) level. For example, the reset time T of the pixel row r It can be 0.95~1.05ns. e (time exposure) 402 represents the exposure time of the pixel row, which affects the number of electrons in the potential well, thereby affecting the brightness of the pixel, and is usually expressed in milliseconds (ms). e It can be 10 to 80 ms. ro(time read out) 403 represents the read out time of the pixel row, which refers to the time it takes for each pixel row to count the electrons in the potential well and convert them into voltage signals after being exposed to light. The read out time is calculated by multiplying the number of pixels in a row by the number of clocks required to read a pixel, and then dividing it by the system clock. This time is generally in microseconds (us). For example, the read out time T of a pixel row is ro It can be 0.5~2us.

[0123] like Figure 5 As shown, starting from the first exposure row Line0, each time a row of exposure is completed, the corresponding RGB image is read, and finally the RGB image 501 of the entire image is obtained. Specifically, the start time of exposure of the first exposure row is the exposure start time of the entire image, and the exposure start times of two adjacent rows are separated by a reading time T. ro . Set the index of the exposure line 0-N, the row identifier of the exposure line is Line0-LineN, N is a positive integer. For example, the index of the first exposure line is 0, and the exposure line is defined as Line0; the index of the second exposure line is 1, and the exposure line is defined as Line1, and so on.

[0124] For any exposure row, its exposure start time is expressed as The exposure end time is expressed as Exposure start time satisfy:

[0125]

[0126] In formula (8), Indicates the exposure start time of any exposure row, t 00 It indicates the exposure start time of the whole image, and L indicates the index of the exposure row.

[0127] And, the exposure ends at satisfy:

[0128]

[0129] In formula (9), represents the end time of exposure of any exposure row. Thus, the exposure period of the i-th pixel row in the RGB image is [t 00 +Li×T ro , t 00 +Li×T ro +T e ], 0<i≤N+1.

[0130] For example, the exposure start time of the exposure period for the second pixel row is =1+1×1.2×10-6. Exposure end time is: 1+1×1.2×10-6+910-3.

[0131] Step C2: Collect event data of the event camera during the exposure period of each pixel row. The event data during the exposure period of a single pixel row is: the brightness polarity statistics of each pixel in the pixel row during the exposure period.

[0132] In the image deblurring process provided by the present application, event expression plays an important role. The data stream output by the event camera sensor is asynchronous and spatially sparse, and cannot be directly applied to traditional image algorithms and subsequent feature extraction, and event expression is required in advance. Among them, the processing and expression of the data stream output by the event camera sensor is mainly divided into independent event expression and batch event expression, typically timestamp event polarity accumulation expression, 3D voxel grid (3D Voxelgrid) expression, etc.

[0133] Among them, 3D Voxel grid expression is to express a specific time period (such as t0-t A-1 ) all events in {(x i ,y i ,t i ,p i )} i＝0,1,…,A-1 , converted into a three-dimensional voxel grid. The length, width and height of the grid correspond to the event pixel coordinate position x, y and the discretized timestamp t respectively. The number of time slices is B, and the index is from 0 to B-1. First, the timestamp t is normalized, referring to formula (6):

[0134]

[0135] Further, referring to formula (7), the time t can be calculated B The event expression of time t B Corresponding event data. Time t B The contribution of the corresponding pixel coordinate position (x, y) in the event expression, and the contribution weight is related to the time interval.

[0136]

[0137] Among them, formula (7) completely covers the time information and spatial information, and can be directly used as the input of the deep learning network. However, the fixed time interval length may affect the size of the voxel grid data, so it is difficult to balance the computational cost and output effect.

[0138] In some embodiments of the present application, a new event expression method is proposed to generate event camera data. For reference, the characteristics of the RGB image sensor's row-by-row exposure and the low-latency pixel-by-pixel sampling exposure of the event camera sensor can be used. After determining the exposure time, the exposure time period corresponding to each pixel row is obtained according to the characteristics of the row-by-row exposure, and then the exposure time period is aligned with the timestamp position of the event camera data, so that the event data corresponding to each exposure row can be obtained more accurately, thereby avoiding the introduction of event camera data caused by pixel changes unrelated to the exposure row. In comparison, the 3D Voxel grid expression only involves issues related to event camera data. The new event expression method fully associates the correlation between the RGB image and the event camera data, such as the correlation between the sampling timestamps of the two data, the correlation between the movement of each pixel corresponding to the spatial position and the event camera data, etc., effectively improving the accuracy of information expression of the event camera data. .

[0139] Specifically, optionally, since each exposure row corresponds to a light-sensitive time T e They are all the same, except that there is a delay in the start of exposure. Therefore, when expressing an event, the event camera can determine the exposure period of the corresponding pixel row according to the row direction of the image resolution (i.e., according to the pixel row scanning direction). The difference compared to the event expression of the three-dimensional voxel grid is to adjust formula (6) to formula (10):

[0140]

[0141] Among them, for each pixel row, get the exposure period Each event data frame collected by the event camera. A single event data frame can be understood as an event (x) collected by the event camera at a collection time (i.e., the aforementioned timestamp). i ,y i ,t i ,p i ). According to formula (10), for the event camera in the exposure period The acquisition time ti of multiple event data frames collected in the system is normalized to obtain the normalized value of each acquisition time ti. Update each event dataframe's events to events

[0142] For each pixel in the pixel row, t B The value ranges from 0 to B-1, and the coordinate data (x, y) of the pixel and each event in each event data frame are recorded. Substitute into formula (7) to calculate the exposure period The brightness polarity statistics of the pixel in each event data frame collected by the event camera are given in t B B V(x, y, t B ) value. Then, traverse each pixel of the pixel row to obtain the brightness polarity statistics of each pixel of the pixel row to obtain the event data within the exposure period of the pixel row. The event data within the exposure period of a single pixel row is the event data of the pixel row. The event camera data is generated according to the event data of each pixel row. The size of the event camera data is H×W×B, where H is the number of pixel rows, W is the number of pixels in a single row, B is the number of time slices, and x is the x coordinate of the pixel, and y is the y coordinate of the pixel.

[0143] For reference, the event camera sensor is a high time resolution sensor, usually 9000fps, and correspondingly, the interval sampling time of the event camera sensor is 0.1ms. The event camera sensor samples multiple event data frames according to its interval sampling time. Each event data frame includes the polarity of the pixel that triggers the event in the RGB image. For each pixel row, the electronic device obtains multiple event data frames collected by the event camera sensor during the exposure period of the pixel row, extracts the polarity of each pixel in the pixel row from the multiple event data frames, obtains the brightness polarity statistics of each pixel in the pixel row, obtains the event data within the exposure period of the pixel row, and obtains the event data of the pixel row.

[0144] like Figure 5 As shown, the event camera sensor samples multiple event data frames according to its interval sampling time. For the first pixel row Line0, multiple event data frames 504 collected during the exposure period of the pixel row Line0 can be read, and the brightness polarity statistics of each pixel in the pixel row can be extracted from the multiple event data frames to obtain the event data of the first pixel row Line0. Similarly, the event data of each pixel row can be obtained. Among them, Figure 5 The middle dotted line 502 to the dotted line 503 indicate the exposure period of the first pixel row Line0.

[0145] Step C3: Generate event camera data corresponding to the RGB image. The event camera data includes event data within the exposure period of each pixel row.

[0146] In the embodiment of the present application, the electronic device generates event camera data corresponding to the RGB image based on the event data of each pixel row.

[0147] Optionally, the electronic device may also utilize a hybrid sensor having the functions of an RGB image sensor and an event camera sensor to collect RGB images and event camera data. The present application also proposes a method that can assist in improving the efficiency of event expression, by setting a trigger condition in the hybrid sensor hardware, that is, automatically providing an exposure start time when each row of pixels is exposed. and the end time of exposure The statistical time range used to trigger the internal event expression of the control. It can be seen that implementing the event expression in hardware can reduce and save the asynchronous data transmission bandwidth.

[0148] In an embodiment of the present application, according to the time when the event data of the event camera is adopted, the event data of the event camera is divided according to the exposure period of each pixel row, and the event data collected by the event camera in the exposure period of a single row of pixel rows is obtained, and then the event data of the single pixel row is obtained, so as to obtain the event data of each pixel row for deblurring processing by the image processing model. In the process of obtaining the event camera data corresponding to the RGB image, by dividing the event data of the event camera according to the exposure period of each pixel row, the correlation between the sampling time of the RGB image and the event camera data is fully utilized. In addition, by obtaining the event data of the single pixel row from the event data collected by the event camera in the exposure period of a single row of pixel rows, the correlation between the spatial position of the RGB image and the event camera data is fully utilized. Thereby, the event camera data provided in the embodiment of the present application can more accurately express the actual brightness change of each row of pixels, and is more in line with the physical mechanism of motion blur.

[0149] Furthermore, based on the corresponding correlation between the sampling time and spatial position of the RGB image and the event camera data, and by utilizing the row-by-row exposure characteristics during RGB image sampling and accurately locating the time period corresponding to the exposure row on the spatial resolution, the sampling range of each frame of event data on the time axis in the sampling data of the matching event camera can be obtained more accurately, effectively avoiding the introduction of motion event information in a time irrelevant to the exposure interval of a specific row, so as to avoid the event data of a single row of pixels being erroneously counted.

[0150] In the process of the image processing method of another embodiment of the present application, as Figure 2 As shown, the feature extraction module includes a first feature extraction module 211 and a second feature extraction module 212. Based on this, step 110 extracts features from the RGB image and the event camera data through the feature extraction module, and the process of outputting the image feature data of the RGB image and the brightness change feature data of the event camera data may include:

[0151] In step 1101, the RGB image is sequentially input into the convolution layer, the channel attention mechanism (CAM) layer and the residual convolution layer in the first feature extraction module, and the image feature data is output.

[0152] Among them, the RGB image can be subjected to feature extraction processing through a convolution layer to obtain first feature data. Furthermore, the feature data of each channel in the first feature data is enhanced or suppressed through a channel attention mechanism layer to obtain adjusted first feature data. Afterwards, the adjusted first feature data is subjected to feature extraction processing again through a residual convolution layer to obtain image feature data, thereby completing shallow feature extraction of the RGB image. Optionally, the image feature data of the RGB image includes, but is not limited to, at least one of edge feature data, line feature data, texture feature data, and corner feature data.

[0153] Optionally, the RGB image is sequentially input into the convolution layer, the channel attention mechanism layer, and the residual convolution layer in the first feature extraction module, and the process of outputting the image feature data may include:

[0154] In step 1911, the RGB image is input into the convolution layer, and the first convolution feature map is output.

[0155] Optionally, the RGB image is input into the convolution layer to perform feature extraction on the RGB image through the convolution layer, and a first convolution feature map is output. Each convolution kernel in the convolution layer can obtain feature data of a channel after performing a convolution operation on the input data to form a first convolution feature map output. The first convolution feature map can be a feature map of size H×W×c. H represents height, W represents width, and c represents the number of channels, that is, the number of convolution kernels.

[0156] For example, the convolution layer can be a 3×3 convolution layer. Each convolution kernel in the 3×3 convolution layer can obtain feature data of a channel after performing a convolution operation on the input data to form the first convolution feature map output. It should be noted that the following steps 11012 to 11017 are all data processing processes of the channel attention mechanism layer, and the processing process can correspond to reference Figure 6 The processing flow of the channel attention mechanism layer shown in Figure 2. The input data of the channel attention mechanism layer is the first convolution feature map output by the convolution layer.

[0157] In step 11012, the first convolutional feature map is subjected to global maximum pooling downsampling processing to obtain a first significant feature map.

[0158] Alternatively, if Figure 6As shown, the first convolution feature map can be subjected to global maximum pooling (MaxPool) downsampling processing through the channel attention mechanism layer to obtain a first significant feature map. The global maximum pooling downsampling processing is used to extract the maximum value of the feature data of each channel in the first convolution feature map to obtain the first significant feature map. The first significant feature map can be a feature map of size 1×1×c, which is used to highlight the most significant feature information in the first convolution feature map.

[0159] In step 11013, the first convolution feature map is subjected to global average pooling downsampling processing to obtain a first mean feature map.

[0160] Alternatively, if Figure 6 As shown, the first convolution feature map can be subjected to global average pooling (AvgPool) downsampling processing through the channel attention mechanism layer to obtain a first mean feature map. The global average pooling downsampling processing is used to extract the average value of the feature data of each channel in the first convolution feature map to obtain the first mean feature map. The first mean feature map can be a feature map of size 1×1×c, each element of which represents the average activation value on the corresponding channel, which is used to highlight the overall feature information in the first convolution feature map.

[0161] In step 11014, the first significant feature map and the first mean feature map are input into a multi-layer perceptron, and a first integrated feature map corresponding to the first significant feature map and a second integrated feature map corresponding to the first mean feature map are output.

[0162] Among them, the multi-layer perceptron (MLP) is also called the fully connected layer. Optionally, Figure 6 As shown, the first significant feature map and the first mean feature map can be processed by the fully connected layer of the channel attention mechanism layer through nonlinear processing, feature extraction processing, etc., to output the first integrated feature map corresponding to the first significant feature map and the second integrated feature map corresponding to the first mean feature map. The first integrated feature map and the second integrated feature map are both feature maps of size 1×1×c.

[0163] In step 11015, the first integrated feature map and the second integrated feature map are added to obtain a first comprehensive feature map.

[0164] Alternatively, if Figure 6 As shown, the feature data of the corresponding channels in the first integrated feature map and the second integrated feature map can be added to obtain the first comprehensive feature map. The first comprehensive feature map can effectively fuse the features of the first integrated feature map and the second integrated feature map to enhance important features and weaken noise.

[0165] In step 11016, the first comprehensive feature map is normalized to obtain the first channel attention weight matrix.

[0166] Alternatively, if Figure 6 As shown, the first comprehensive feature map can be normalized by a normalization function to suppress the feature data of each channel in the first comprehensive feature map in the interval [0,1] to obtain a first channel attention weight matrix. The data of each channel in the first channel attention weight matrix is ​​the weight of the feature data of each channel in the first convolution feature map, which is used to enhance or suppress the feature data of each channel in the first convolution feature map. In some embodiments, the normalization function can be a sigmoid activation function.

[0167] In step 11017, the product of the first channel attention weight matrix and the first convolution feature map is calculated, and the first adjusted feature map is output.

[0168] like Figure 6 As shown, in this step, the product of the first channel attention weight matrix and the first convolution feature map is calculated, and the first adjusted feature map is output.

[0169] In step 11018, the first adjusted feature map is input into the residual convolution layer, and image feature data is output.

[0170] Optionally, the first adjusted feature map is input into the residual convolution layer to perform feature extraction on the first adjusted feature map through the residual convolution layer, output image feature data, and complete the extraction of shallow features.

[0171] In the implementation of the present application, the first feature extraction module can model the importance of the feature data of each channel in the first convolution feature map output by the convolution layer through the channel attention mechanism layer, and then enhance or suppress the feature data of different channels according to the importance, so as to improve the feature extraction accuracy of the first feature extraction module for RGB images.

[0172] In step 1102, the event camera data is sequentially input into the convolution layer, the channel attention mechanism layer and the residual convolution layer in the second feature extraction module, and the brightness change feature data is output.

[0173] Among them, the event camera data can be processed for feature extraction through the convolution layer to obtain the second feature data. Then, the feature data of each channel in the second feature data is enhanced or suppressed through the channel attention mechanism layer to obtain the adjusted second feature data. After that, the adjusted second feature data is subjected to feature extraction again through the residual convolution layer to obtain the brightness change feature data, thereby completing the shallow feature extraction of the brightness change feature data.

[0174] Optionally, the process of sequentially inputting the event camera data into the convolution layer, the channel attention mechanism layer, and the residual convolution layer in the second feature extraction module and outputting the brightness change feature data may include:

[0175] In step 2911, the event camera data is input into the convolution layer and a second convolution feature map is output.

[0176] In step 21012, the second convolutional feature map is subjected to global maximum pooling downsampling processing to obtain a second significant feature map.

[0177] In step 21013, the second convolution feature map is subjected to global average pooling downsampling processing to obtain a second mean feature map.

[0178] In step 21014, the second significant feature map and the second mean feature map are input into a multi-layer perceptron, and a third integrated feature map corresponding to the second significant feature map and a fourth integrated feature map corresponding to the second mean feature map are output.

[0179] In step 21015, the third integrated feature map and the fourth integrated feature map are added to obtain a second comprehensive feature map.

[0180] In step 21016, the second comprehensive feature map is normalized to obtain a second channel attention weight matrix.

[0181] In step 21017, the product of the second channel attention weight matrix and the second convolution feature map is calculated, and the second adjusted feature map is output.

[0182] In step 21018, the second adjusted feature map is input into the residual convolution layer, and the brightness change feature data is output.

[0183] It should be noted that the channel attention mechanism layer in the second feature extraction module processes its input data in the same way as the channel attention mechanism layer in the first feature extraction module processes its input data. That is, the explanation and implementation of steps 2911 to 21018 can refer to the explanation and implementation of the aforementioned steps 1911 to 11018, and the embodiments of the present application are not repeated here. Similarly, in the implementation of the present application, the second feature extraction module can model the importance of the feature data of each channel in the second convolution feature map output by the convolution layer through the channel attention mechanism layer, and then enhance or suppress the feature data of different channels according to the importance, so as to improve the feature extraction accuracy of the event camera data by the second feature extraction module.

[0184] The image processing model provided in the embodiments of the present application needs to be trained and evaluated on a simulation data set and a real data set in advance, so as to test the model's recovery of moving targets under low light sources in the laboratory under the training parameters, so that the model has a good deblurring effect. In the training process, the predicted output results of the model can be compared with the real data, and the various parameters in the model can be iteratively obtained by minimizing the loss function through the error loss function combined with the gradient descent algorithm. Furthermore, the trained model is deployed on the terminal, and the blurred image and event camera data are input into the model to deblur the blurred image.

[0185] In summary, in an embodiment of the present application, after the image feature data of the RGB image and the brightness change feature data of the event camera data are extracted by the feature extraction module of the image processing model, the image feature data and the brightness change feature data are fused to obtain fused feature data by using the cross-modal channel attention mechanism module of the image processing model, and then the image feature data and the fused feature data are subjected to feature splicing extraction in combination with the encoding and decoding module of the image processing model to obtain splicing feature data, so as to obtain a deblurred image of the RGB image by fusing the splicing feature data and the RGB image through the fusion module of the image processing model. In this technical solution, the image processing model can extract the features of the RGB image and the event camera data, and combine the cross-modal channel attention mechanism and encoding and decoding to perform feature fusion on the features of the RGB image and the event camera data, thereby obtaining a mapping relationship from a blurred image to a clear image, and completing deblurring restoration. And because the event camera data can have the characteristics of high temporal resolution, high dynamic range and high pixel bandwidth that the RGB image data does not have. Therefore, event camera data can supplement the image features of RGB images, so that by fusing the features of RGB images and event camera data, richer image features can be obtained to restore blurred images, thereby improving the restoration effect of blurred images. In addition, a hardware control design in the EVS-RGB hybrid sensor is proposed, which can choose to deploy the event expression implementation in hardware, reduce the asynchronous transmission bandwidth of event data, and reduce power consumption.

[0186] The image processing method provided in the embodiment of the present application can be executed by an image processing device. In the embodiment of the present application, an image processing device executing the image processing method is taken as an example to illustrate the image processing device provided in the embodiment of the present application.

[0187] Please refer to Figure 7 , which shows a block diagram of an image processing device provided by an embodiment of the present application. Figure 7 As shown, the image processing device 800 includes: an extraction processing module 701 , a first fusion processing module 702 , a splicing extraction processing module 703 and a second fusion processing module 704 .

[0188] An extraction processing module 701 is used to input the RGB image and the corresponding event camera data into a feature extraction module of an image processing model, perform feature extraction on the RGB image and the event camera data through the feature extraction module, and output image feature data of the RGB image and brightness change feature data of the event camera data;

[0189] A first fusion processing module 702 is used to input the image feature data and the brightness change feature data into the cross-modal channel attention mechanism module of the image processing model, perform feature fusion processing on the image feature data and the brightness change feature data through the cross-modal channel attention mechanism module, and output fusion feature data;

[0190] The splicing extraction processing module 703 is used to input the image feature data and the fusion feature data into the encoding and decoding module of the image processing model, perform feature splicing extraction on the image feature data and the fusion feature data through the encoding and decoding module, and output the splicing feature data;

[0191] The second fusion processing module 704 is used to input the RGB image and the splicing feature data into the fusion module of the image processing model, and perform fusion processing on the splicing feature data and the RGB image through the fusion module to obtain a deblurred image of the RGB image.

[0192] Optionally, the cross-modal channel attention mechanism module includes: a first cross-modal attention sub-module, and the fused feature data includes: directly fused feature data; the first fusion processing module 702 is also used to: input the image feature data and the brightness change feature data into the first cross-modal attention sub-module, and output the directly fused feature data.

[0193] Optionally, the encoding and decoding module includes: a first encoding module, a first decoding module, a second encoding module and a second decoding module; the splicing extraction processing module 703 is further used to:

[0194] Inputting the image feature data into a first encoding module and outputting image encoding data;

[0195] Inputting the image encoding data into the second encoding module and outputting the re-encoded image data;

[0196] Input the re-encoded image data and the image encoding data into a second decoding module, and output basic splicing feature data, wherein the basic splicing feature is feature data obtained by performing feature extraction processing on the splicing of the image data after decoding the re-encoded image data and the image encoding data;

[0197] The image feature data, the direct fusion feature data and the decoded image data are input into the first decoding module, and the spliced ​​feature data is output. The spliced ​​feature data is the feature data obtained by performing feature extraction processing on the decoded image data and the direct fusion feature data and the image feature data after splicing.

[0198] Optionally, the image processing model further includes: a third encoding module; the cross-modal channel attention mechanism module includes: a second cross-modal attention submodule, and the fused feature data includes: compressed fused feature data; the image processing device 800 further includes: an encoding processing module, which is used to input the brightness change feature data into the third encoding module and output brightness feature encoding data, and the encoding size of the third encoding module is the same as that of the first encoding module;

[0199] The first fusion processing module 702 is further used to: input the brightness feature encoding data and the image encoding data into the second cross-modal attention submodule, and output compressed fusion feature data;

[0200] The splicing extraction processing module 703 is also used to: input the re-encoded image data, the compressed fusion feature data and the image coding data into the second decoding module, and output the basic splicing feature data; wherein the basic splicing feature is the feature data obtained by performing feature extraction processing on the image data after decoding the re-encoded image data and the image coding data and the compressed fusion feature data.

[0201] Optionally, the first fusion processing module 702 is further configured to:

[0202] The image feature data is subjected to normalization processing, convolution processing, and dimension adjustment processing in sequence to obtain a first query vector;

[0203] The brightness change feature data is subjected to normalization processing, convolution processing, and dimension adjustment processing in sequence to obtain a first key vector and a first value;

[0204] Outputting first attention map data based on the dot product of the first query vector and the first key vector;

[0205] Outputting first attention feature map data based on the product of the first attention map and the first value;

[0206] Adding the first image feature data and the first attention feature map data to obtain first spliced ​​map data;

[0207] Inputting the first spliced ​​graph data into a feedforward neural network and outputting the first network processed graph data;

[0208] The first network processed graph data and the first spliced ​​graph data are added together to output direct fusion feature data.

[0209] Optionally, the first fusion processing module 702 is further configured to:

[0210] The brightness feature encoding data is subjected to normalization processing, convolution processing, and dimension adjustment processing in sequence to obtain a second query vector;

[0211] The image coding data is subjected to normalization processing, convolution processing, and dimension adjustment processing in sequence to obtain a second key vector and a second value;

[0212] Outputting second attention map data based on the dot product of the second query vector and the second key vector;

[0213] Outputting second attention feature map data based on the product of the second attention map and the second value;

[0214] Adding the second image feature data and the second attention feature map data to obtain second spliced ​​map data;

[0215] Inputting the second spliced ​​graph data into a feedforward neural network and outputting the second network processed graph data;

[0216] The second network processed graph data and the second spliced ​​graph data are added together to output compressed fused feature data.

[0217] Optionally, the image processing device 800 further includes:

[0218] An acquisition module, used for acquiring an exposure period of each pixel row of an RGB image;

[0219] An acquisition module, used for acquiring event data of a pixel row during an exposure period of each pixel row of the event camera, wherein the event data of a single pixel row is brightness polarity statistics of each pixel in the pixel row during the exposure period;

[0220] The generation module is used to generate event camera data corresponding to the RGB image, and the event camera data includes event data of each pixel row.

[0221] Optionally, the feature extraction module includes a first feature extraction module and a second feature extraction module; the extraction processing module 701 is further used for:

[0222] Input the RGB image into the convolution layer, channel attention mechanism layer and residual convolution layer in the first feature extraction module in sequence, and output image feature data;

[0223] The event camera data is sequentially input into the convolution layer, channel attention mechanism layer and residual convolution layer in the second feature extraction module to output brightness change feature data.

[0224] Optionally, the extraction processing module 701 is further used for:

[0225] Input the RGB image into the convolution layer and output the first convolution feature map;

[0226] Performing global maximum pooling downsampling processing on the first convolution feature map to obtain a first significant feature map;

[0227] Performing global average pooling downsampling processing on the first convolution feature map to obtain a first mean feature map;

[0228] Inputting the first significant feature map and the first mean feature map into a multi-layer perceptron, and outputting a first integrated feature map corresponding to the first significant feature map and a second integrated feature map corresponding to the first mean feature map;

[0229] Adding the first integrated feature map and the second integrated feature map to obtain a first comprehensive feature map;

[0230] Normalize the first comprehensive feature map to obtain the first channel attention weight matrix;

[0231] Calculate the product of the first channel attention weight matrix and the first convolution feature map, and output the image feature data.

[0232] Optionally, the extraction processing module 701 is further used for:

[0233] Input the event camera data into the convolution layer and output the second convolution feature map;

[0234] Performing global maximum pooling downsampling processing on the second convolutional feature map to obtain a second significant feature map;

[0235] Performing global average pooling downsampling processing on the second convolution feature map to obtain a second mean feature map;

[0236] Inputting the second significant feature map and the second mean feature map into a multilayer perceptron, and outputting a third integrated feature map corresponding to the second significant feature map and a fourth integrated feature map corresponding to the second mean feature map;

[0237] Adding the third integrated feature map and the fourth integrated feature map to obtain a second comprehensive feature map;

[0238] Normalize the second comprehensive feature map to obtain the second channel attention weight matrix;

[0239] Calculate the product of the second channel attention weight matrix and the second convolution feature map, and output the brightness change feature data.

[0240] In an embodiment of the present application, after the image feature data of the RGB image and the brightness change feature data of the event camera data are extracted by the feature extraction module of the image processing model, the image feature data and the brightness change feature data are fused to obtain fused feature data by using the cross-modal channel attention mechanism module of the image processing model, and then the image feature data and the fused feature data are subjected to feature splicing extraction in combination with the codec module of the image processing model to obtain splicing feature data, so as to obtain the deblurred image of the RGB image by fusing the splicing feature data and the RGB image through the fusion module of the image processing model. In this technical solution, the image processing model can extract the features of the RGB image and the event camera data, and combine the cross-modal channel attention mechanism and codec to perform feature fusion on the features of the RGB image and the event camera data, thereby obtaining a mapping relationship from a blurred image to a clear image, and completing deblurring restoration. And because the event camera data can have the characteristics of high temporal resolution, high dynamic range and high pixel bandwidth that the RGB image data does not have. Therefore, the event camera data can supplement the image features of the RGB image, so that by fusing the features of the RGB image and the event camera data, richer image features can be obtained to restore the blurred image, thereby improving the restoration effect of the blurred image.

[0241] The device in the embodiment of the present application can be a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or it can be other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a car electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmentedreality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0242] The device of the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0243] The device provided in the embodiment of the present application can implement each process implemented in the above method embodiment and achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0244] Alternatively, if Figure 8 As shown, an embodiment of the present application also provides an electronic device 800, including a processor 801, a memory 802, and a program or instruction stored in the memory 802 and executable on the processor 801. When the program or instruction is executed by the processor 801, each step of any of the above-mentioned image processing method embodiments is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0245] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0246] Fig. 9 The hardware structure diagram of an electronic device for implementing the embodiment of the present application is shown in FIG. The electronic device 900 includes, but is not limited to, a radio frequency unit 901, a network module 902, an audio output unit 903, an input unit 904, a sensor 905, a display unit 906, a user input unit 907, an interface unit 908, a memory 909, a processor 910, a camera 911, and other components.

[0247] Those skilled in the art will appreciate that the electronic device 900 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 910 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig. 9 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0248] Processor 910, configured to input the RGB image and the corresponding event camera data into a feature extraction module of an image processing model, perform feature extraction on the RGB image and the event camera data through the feature extraction module, and output image feature data of the RGB image and brightness change feature data of the event camera data;

[0249] Inputting the image feature data and the brightness change feature data into the cross-modal channel attention mechanism module of the image processing model, performing feature fusion processing on the image feature data and the brightness change feature data through the cross-modal channel attention mechanism module, and outputting fused feature data;

[0250] Inputting the image feature data and the fused feature data into the encoding and decoding module of the image processing model, performing feature splicing extraction on the image feature data and the fused feature data through the encoding and decoding module, and outputting splicing feature data;

[0251] The RGB image and the splicing feature data are input into a fusion module of the image processing model, and the splicing feature data and the RGB image are fused through the fusion module to obtain a deblurred image of the RGB image.

[0252] In an embodiment of the present application, after the image feature data of the RGB image and the brightness change feature data of the event camera data are extracted by the feature extraction module of the image processing model, the image feature data and the brightness change feature data are fused to obtain fused feature data by using the cross-modal channel attention mechanism module of the image processing model, and then the image feature data and the fused feature data are subjected to feature splicing extraction in combination with the codec module of the image processing model to obtain splicing feature data, so as to obtain the deblurred image of the RGB image by fusing the splicing feature data and the RGB image through the fusion module of the image processing model. In this technical solution, the image processing model can extract the features of the RGB image and the event camera data, and combine the cross-modal channel attention mechanism and codec to perform feature fusion on the features of the RGB image and the event camera data, thereby obtaining a mapping relationship from a blurred image to a clear image, and completing deblurring restoration. And because the event camera data can have the characteristics of high temporal resolution, high dynamic range and high pixel bandwidth that the RGB image data does not have. Therefore, the event camera data can supplement the image features of the RGB image, so that by fusing the features of the RGB image and the event camera data, richer image features can be obtained to restore the blurred image, thereby improving the restoration effect of the blurred image.

[0253] Optionally, the cross-modal channel attention mechanism module includes: a first cross-modal attention sub-module, and the fused feature data includes: directly fused feature data; the processor 910 is also used to: input the image feature data and the brightness change feature data into the first cross-modal attention sub-module, and output the directly fused feature data.

[0254] Optionally, the encoding and decoding module includes: a first encoding module, a first decoding module, a second encoding module, and a second decoding module; the processor 910 is further configured to:

[0255] Input the image feature data into the first encoding module, and output image encoding data;

[0256] inputting the image encoding data into the second encoding module and outputting re-encoded image data;

[0257] Input the re-encoded image data and the image encoding data into the second decoding module, and output basic splicing feature data, wherein the basic splicing feature is feature data obtained by performing feature extraction processing on the splicing of the image data after decoding the re-encoded image data and the image encoding data;

[0258] The image feature data, the direct fusion feature data and the decoded image data are input into the first decoding module, and spliced ​​feature data is output, wherein the spliced ​​feature data is feature data obtained by performing feature extraction processing on the image data after decoding the decoded image data and the direct fusion feature data and the image feature data.

[0259] Optionally, the image processing model further includes: a third encoding module; the cross-modal channel attention mechanism module includes: a second cross-modal attention submodule, and the fused feature data includes: compressed fused feature data; the processor 910 is further used to: input the brightness change feature data into the third encoding module, and output brightness feature encoding data, and the encoding size of the third encoding module is the same as that of the first encoding module;

[0260] The processor 910 is further configured to: input the brightness feature encoding data and the image encoding data into the second cross-modal attention submodule, and output the compressed fusion feature data;

[0261] Processor 910 is also used to: input the re-encoded image data, the compressed fusion feature data and the image coding data into the second decoding module, and output basic splicing feature data; wherein the basic splicing feature is feature data obtained by performing feature extraction processing on the image data after decoding the re-encoded image data and the image coding data and the compressed fusion feature data.

[0262] Optionally, the processor 910 is further configured to:

[0263] The image feature data is sequentially subjected to normalization processing, convolution processing, and dimension adjustment processing to obtain a first query vector;

[0264] The brightness change feature data is subjected to normalization processing, convolution processing, and dimension adjustment processing in sequence to obtain a first key vector and a first value;

[0265] Outputting first attention map data based on a dot product of the first query vector and the first key vector;

[0266] Outputting first attention feature map data based on the product of the first attention map and the first value;

[0267] Adding the first image feature data and the first attention feature map data to obtain first spliced ​​map data;

[0268] Inputting the first spliced ​​graph data into a feedforward neural network and outputting first network processed graph data;

[0269] The first network processed graph data and the first spliced ​​graph data are added together to output direct fusion feature data.

[0270] Optionally, the processor 910 is further configured to:

[0271] The brightness feature encoding data is sequentially subjected to normalization processing, convolution processing, and dimension adjustment processing to obtain a second query vector;

[0272] The image coding data is subjected to normalization processing, convolution processing, and dimension adjustment processing in sequence to obtain a second key vector and a second value;

[0273] Outputting second attention map data based on the dot product of the second query vector and the second key vector;

[0274] Outputting second attention feature map data based on the product of the second attention map and the second value;

[0275] Adding the second image feature data and the second attention feature map data to obtain second spliced ​​map data;

[0276] Inputting the second spliced ​​graph data into a feedforward neural network and outputting second network processed graph data;

[0277] The second network processed graph data and the second spliced ​​graph data are added together to output compressed fusion feature data.

[0278] Optionally, the processor 910 is further configured to:

[0279] Obtaining an exposure period of each pixel row of the RGB image;

[0280] Collecting event data of the pixel row during the exposure period of each pixel row by the event camera, wherein the event data of a single pixel row is brightness polarity statistical data of each pixel in the pixel row during the exposure period;

[0281] The event camera data corresponding to the RGB image is generated, wherein the event camera data includes the event data of each pixel row.

[0282] Optionally, the feature extraction module includes a first feature extraction module and a second feature extraction module; the processor 910 is further configured to:

[0283] Input the RGB image into the convolution layer, the channel attention mechanism layer and the residual convolution layer in the first feature extraction module in sequence, and output the image feature data;

[0284] The event camera data is sequentially input into the convolution layer, the channel attention mechanism layer and the residual convolution layer in the second feature extraction module, and the brightness change feature data is output.

[0285] Optionally, the processor 910 is further configured to:

[0286] Input the RGB image into the convolution layer, and output a first convolution feature map;

[0287] Performing global maximum pooling downsampling processing on the first convolutional feature map to obtain a first salient feature map;

[0288] Performing global average pooling downsampling processing on the first convolution feature map to obtain a first mean feature map;

[0289] Inputting the first significant feature map and the first mean feature map into a multi-layer perceptron, and outputting a first integrated feature map corresponding to the first significant feature map and a second integrated feature map corresponding to the first mean feature map;

[0290] Adding the first integrated feature map and the second integrated feature map to obtain a first comprehensive feature map;

[0291] Normalizing the first comprehensive feature map to obtain a first channel attention weight matrix;

[0292] Calculate the product of the first channel attention weight matrix and the first convolution feature map, and output a first adjusted feature map;

[0293] The first adjusted feature map is input into the residual convolution layer, and the image feature data is output.

[0294] Optionally, the processor 910 is further configured to:

[0295] Inputting the event camera data into the convolution layer, and outputting a second convolution feature map;

[0296] Performing global maximum pooling downsampling processing on the second convolutional feature map to obtain a second salient feature map;

[0297] Performing global average pooling downsampling processing on the second convolution feature map to obtain a second mean feature map;

[0298] Inputting the second significant feature map and the second mean feature map into a multilayer perceptron, and outputting a third integrated feature map corresponding to the second significant feature map and a fourth integrated feature map corresponding to the second mean feature map;

[0299] Adding the third integrated feature map and the fourth integrated feature map to obtain a second comprehensive feature map;

[0300] Normalizing the second comprehensive feature map to obtain a second channel attention weight matrix;

[0301] Calculate the product of the second channel attention weight matrix and the second convolution feature map, and output a second adjusted feature map;

[0302] The second adjusted feature map is input into the residual convolution layer, and the brightness change feature data is output.

[0303] In summary, in an embodiment of the present application, after the image feature data of the RGB image and the brightness change feature data of the event camera data are extracted by the feature extraction module of the image processing model, the image feature data and the brightness change feature data are fused to obtain fused feature data by using the cross-modal channel attention mechanism module of the image processing model, and then the image feature data and the fused feature data are subjected to feature splicing extraction in combination with the encoding and decoding module of the image processing model to obtain splicing feature data, so as to obtain a deblurred image of the RGB image by fusing the splicing feature data and the RGB image through the fusion module of the image processing model. In this technical solution, the image processing model can extract the features of the RGB image and the event camera data, and combine the cross-modal channel attention mechanism and encoding and decoding to perform feature fusion on the features of the RGB image and the event camera data, thereby obtaining a mapping relationship from a blurred image to a clear image, and completing deblurring restoration. And because the event camera data can have the characteristics of high temporal resolution, high dynamic range and high pixel bandwidth that the RGB image data does not have. Therefore, the event camera data can supplement the image features of the RGB image, so that by fusing the features of the RGB image and the event camera data, richer image features can be obtained to restore the blurred image, thereby improving the restoration effect of the blurred image.

[0304] It should be understood that in the embodiment of the present application, the input unit 904 may include a graphics processor (Graphics Processing Unit, GPU) 9041 and a microphone 9042, and the graphics processor 9041 processes the image data of the static picture or video image obtained by the image capture device (such as a camera) in the video image capture mode or the image capture mode. The display unit 906 may include a display panel 9061, and the display panel 9061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 907 includes a touch panel 9071 and at least one of other input devices 9072. The touch panel 9071 is also called a touch screen. The touch panel 9071 may include two parts: a touch detection device and a touch controller. Other input devices 9072 may include, but are not limited to, a physical keyboard, a function key (such as a volume control button, a switch button, etc.), a trackball, a mouse, and an action rod, which will not be repeated here. The memory 909 can be used to store software programs and various data, including but not limited to cameras and operating systems. The processor 910 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, user pages, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 910.

[0305] The memory 909 can be used to store software programs and various data. The memory 909 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 909 may include a volatile memory or a non-volatile memory, or the memory 909 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 909 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0306] The processor 910 may include one or more processing units; optionally, the processor 910 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 910.

[0307] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned image processing method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0308] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0309] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0310] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0311] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0312] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0313] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0314] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Inputting the RGB image and the corresponding event camera data into a feature extraction module of an image processing model, performing feature extraction on the RGB image and the event camera data through the feature extraction module, and outputting image feature data of the RGB image and brightness change feature data of the event camera data; Inputting the image feature data and the brightness change feature data into the cross-modal channel attention mechanism module of the image processing model, performing feature fusion processing on the image feature data and the brightness change feature data through the cross-modal channel attention mechanism module, and outputting fused feature data; Inputting the image feature data and the fused feature data into the encoding and decoding module of the image processing model, performing feature splicing extraction on the image feature data and the fused feature data through the encoding and decoding module, and outputting splicing feature data; The RGB image and the splicing feature data are input into a fusion module of the image processing model, and the splicing feature data and the RGB image are fused through the fusion module to obtain a deblurred image of the RGB image.

2. The method according to claim 1, characterized in that The cross-modal channel attention mechanism module includes: a first cross-modal attention submodule, and the fused feature data includes: directly fused feature data; The step of performing feature fusion processing on the image feature data and the brightness change feature data through the cross-modal channel attention mechanism module and outputting fused feature data includes: The image feature data and the brightness change feature data are input into the first cross-modal attention submodule, and direct fusion feature data is output.

3. The method according to claim 2, characterized in that The encoding and decoding module includes: a first encoding module, a first decoding module, a second encoding module and a second decoding module; The step of performing feature splicing extraction on the image feature data and the fusion feature data through the encoding and decoding module and outputting splicing feature data comprises: Input the image feature data into the first encoding module, and output image encoding data; inputting the image encoding data into the second encoding module and outputting re-encoded image data; Input the re-encoded image data and the image encoding data into the second decoding module, and output basic splicing feature data, wherein the basic splicing feature is feature data obtained by performing feature extraction processing on the splicing of the image data after decoding the re-encoded image data and the image encoding data; The image feature data, the direct fusion feature data and the decoded image data are input into the first decoding module, and spliced ​​feature data is output, wherein the spliced ​​feature data is feature data obtained by performing feature extraction processing on the image data after decoding the decoded image data and the direct fusion feature data and the image feature data.

4. The method according to claim 3, characterized in that The image processing model further includes: a third encoding module; the cross-modal channel attention mechanism module includes: a second cross-modal attention submodule, and the fused feature data includes: compressed fused feature data; The method further comprises: Input the brightness change characteristic data into the third encoding module, and output brightness characteristic encoding data, wherein the encoding size of the third encoding module is the same as that of the first encoding module; The method further comprises: performing feature fusion processing on the image feature data and the brightness change feature data through the cross-modal channel attention mechanism module, and outputting fused feature data. Input the brightness feature encoding data and the image encoding data into the second cross-modal attention submodule, and output the compressed fusion feature data; The step of inputting the re-encoded image data and the image encoding data into the second decoding module and outputting basic splicing feature data comprises: Input the re-encoded image data, the compressed fusion feature data and the image encoding data into the second decoding module, and output basic splicing feature data; The basic splicing feature is feature data obtained by performing feature extraction processing on the image data after decoding the re-encoded image data, the image encoding data, and the compressed fusion feature data.

5. The method according to claim 2, characterized in that: The step of inputting the image feature data and the brightness change feature data into the first cross-modal attention submodule and outputting direct fusion feature data comprises: The image feature data is sequentially subjected to normalization processing, convolution processing, and dimension adjustment processing to obtain a first query vector; The brightness change feature data is subjected to normalization processing, convolution processing, and dimension adjustment processing in sequence to obtain a first key vector and a first value; Outputting first attention map data based on a dot product of the first query vector and the first key vector; Outputting first attention feature map data based on the product of the first attention map and the first value; Adding the first image feature data and the first attention feature map data to obtain first spliced ​​map data; Inputting the first spliced ​​graph data into a feedforward neural network and outputting first network processed graph data; The first network processed graph data and the first spliced ​​graph data are added together to output direct fusion feature data.

6. The method according to claim 4, characterized in that The step of inputting the brightness feature encoding data and the image encoding data into the second cross-modal attention submodule and outputting the compressed fusion feature data comprises: The brightness feature encoding data is sequentially subjected to normalization processing, convolution processing, and dimension adjustment processing to obtain a second query vector; The image coding data is subjected to normalization processing, convolution processing, and dimension adjustment processing in sequence to obtain a second key vector and a second value; Outputting second attention map data based on the dot product of the second query vector and the second key vector; Outputting second attention feature map data based on the product of the second attention map and the second value; Adding the second image feature data and the second attention feature map data to obtain second spliced ​​map data; Inputting the second spliced ​​graph data into a feedforward neural network and outputting second network processed graph data; The second network processed graph data and the second spliced ​​graph data are added together to output compressed fusion feature data.

7. The method according to claim 1, characterized in that Before inputting the RGB image and the corresponding event camera data into the image processing model, the method further includes: Obtaining an exposure period of each pixel row of the RGB image; Collecting event data of the event camera during an exposure period of each pixel row, wherein the event data during the exposure period of a single pixel row is brightness polarity statistical data of each pixel in the pixel row during the exposure period; The event camera data corresponding to the RGB image is generated, wherein the event camera data includes the event data within an exposure period of each pixel row.

8. The method according to claim 1, characterized in that The feature extraction module includes a first feature extraction module and a second feature extraction module; The feature extraction module is used to extract features from the RGB image and the event camera data, and outputs image feature data of the RGB image and brightness change feature data of the event camera data, including: Input the RGB image into the convolution layer, the channel attention mechanism layer and the residual convolution layer in the first feature extraction module in sequence, and output the image feature data; The event camera data is sequentially input into the convolution layer, the channel attention mechanism layer and the residual convolution layer in the second feature extraction module, and the brightness change feature data is output.

9. The method according to claim 8, characterized in that The step of sequentially inputting the RGB image into the convolution layer, the channel attention mechanism layer, and the residual convolution layer in the first feature extraction module, and outputting the image feature data, comprises: Input the RGB image into the convolution layer, and output a first convolution feature map; Performing global maximum pooling downsampling processing on the first convolutional feature map to obtain a first salient feature map; Performing global average pooling downsampling processing on the first convolution feature map to obtain a first mean feature map; Inputting the first significant feature map and the first mean feature map into a multi-layer perceptron, and outputting a first integrated feature map corresponding to the first significant feature map and a second integrated feature map corresponding to the first mean feature map; Adding the first integrated feature map and the second integrated feature map to obtain a first comprehensive feature map; Normalizing the first comprehensive feature map to obtain a first channel attention weight matrix; Calculate the product of the first channel attention weight matrix and the first convolution feature map, and output a first adjusted feature map; The first adjusted feature map is input into the residual convolution layer, and the image feature data is output.

10. The method according to claim 8, characterized in that The step of sequentially inputting the event camera data into the convolution layer, the channel attention mechanism layer, and the residual convolution layer in the second feature extraction module, and outputting the brightness change feature data, comprises: Inputting the event camera data into the convolution layer, and outputting a second convolution feature map; Performing global maximum pooling downsampling processing on the second convolutional feature map to obtain a second salient feature map; Performing global average pooling downsampling processing on the second convolution feature map to obtain a second mean feature map; Inputting the second significant feature map and the second mean feature map into a multilayer perceptron, and outputting a third integrated feature map corresponding to the second significant feature map and a fourth integrated feature map corresponding to the second mean feature map; Adding the third integrated feature map and the fourth integrated feature map to obtain a second comprehensive feature map; Normalizing the second comprehensive feature map to obtain a second channel attention weight matrix; Calculate the product of the second channel attention weight matrix and the second convolution feature map, and output a second adjusted feature map; The second adjusted feature map is input into the residual convolution layer, and the brightness change feature data is output.

11. An image processing device, characterized in that: The device comprises: An extraction processing module, used for inputting the RGB image and the corresponding event camera data into a feature extraction module of an image processing model, performing feature extraction on the RGB image and the event camera data through the feature extraction module, and outputting image feature data of the RGB image and brightness change feature data of the event camera data; A first fusion processing module, configured to input the image feature data and the brightness change feature data into a cross-modal channel attention mechanism module of the image processing model, perform feature fusion processing on the image feature data and the brightness change feature data through the cross-modal channel attention mechanism module, and output fusion feature data; A splicing extraction processing module, used for inputting the image feature data and the fused feature data into the encoding and decoding module of the image processing model, performing feature splicing extraction on the image feature data and the fused feature data through the encoding and decoding module, and outputting splicing feature data; The second fusion processing module is used to input the RGB image and the splicing feature data into the fusion module of the image processing model, and perform fusion processing on the splicing feature data and the RGB image through the fusion module to obtain a deblurred image of the RGB image.

12. The device according to claim 11, characterized in that The cross-modal channel attention mechanism module includes: a first cross-modal attention sub-module, and the fused feature data includes: directly fused feature data; the first fusion processing module is also used to: input the image feature data and the brightness change feature data into the first cross-modal attention sub-module, and output the directly fused feature data.

13. The device according to claim 12, characterized in that The encoding and decoding module includes: a first encoding module, a first decoding module, a second encoding module and a second decoding module; the splicing extraction processing module is also used for: Input the image feature data into the first encoding module, and output image encoding data; inputting the image encoding data into the second encoding module and outputting re-encoded image data; Input the re-encoded image data and the image encoding data into the second decoding module, and output basic splicing feature data, wherein the basic splicing feature is feature data obtained by performing feature extraction processing on the splicing of the image data after decoding the re-encoded image data and the image encoding data; The image feature data, the direct fusion feature data and the decoded image data are input into the first decoding module, and spliced ​​feature data is output, wherein the spliced ​​feature data is feature data obtained by performing feature extraction processing on the image data after decoding the decoded image data and the direct fusion feature data and the image feature data.

14. The device according to claim 13, characterized in that The image processing model further includes: a third encoding module; the cross-modal channel attention mechanism module includes: a second cross-modal attention submodule, the fused feature data includes: compressed fused feature data; the device further includes: A coding processing module, used for inputting the brightness change characteristic data into the third coding module and outputting brightness characteristic coding data, wherein the coding size of the third coding module is the same as that of the first coding module; The first fusion processing module is further used to: input the brightness feature encoding data and the image encoding data into the second cross-modal attention submodule, and output the compressed fusion feature data; The splicing extraction processing module is further used to: input the re-encoded image data, the compressed fusion feature data and the image encoding data into the second decoding module, and output basic splicing feature data; The basic splicing feature is feature data obtained by performing feature extraction processing on the image data after decoding the re-encoded image data, the image encoding data, and the compressed fusion feature data.

15. The device according to claim 12, characterized in that The first fusion processing module is further used for: The image feature data is sequentially subjected to normalization processing, convolution processing, and dimension adjustment processing to obtain a first query vector; The brightness change feature data is subjected to normalization processing, convolution processing, and dimension adjustment processing in sequence to obtain a first key vector and a first value; Outputting first attention map data based on a dot product of the first query vector and the first key vector; Outputting first attention feature map data based on the product of the first attention map and the first value; Adding the first image feature data and the first attention feature map data to obtain first spliced ​​map data; Inputting the first spliced ​​graph data into a feedforward neural network and outputting first network processed graph data; The first network processed graph data and the first spliced ​​graph data are added together to output direct fusion feature data.

16. The device according to claim 14, characterized in that The first fusion processing module is further used for: The brightness feature encoding data is sequentially subjected to normalization processing, convolution processing, and dimension adjustment processing to obtain a second query vector; The image coding data is subjected to normalization processing, convolution processing, and dimension adjustment processing in sequence to obtain a second key vector and a second value; Outputting second attention map data based on the dot product of the second query vector and the second key vector; Outputting second attention feature map data based on the product of the second attention map and the second value; Adding the second image feature data and the second attention feature map data to obtain second spliced ​​map data; Inputting the second spliced ​​graph data into a feedforward neural network and outputting second network processed graph data; The second network processed graph data and the second spliced ​​graph data are added together to output compressed fusion feature data.

17. The device according to claim 11, characterized in that The device also includes: An acquisition module, used to acquire an exposure period of each pixel row of the RGB image; An acquisition module, used for acquiring event data of the event camera in an exposure period of each pixel row, wherein the event data in the exposure period of a single pixel row is brightness polarity statistical data of each pixel in the pixel row in the exposure period; A generating module is used to generate the event camera data corresponding to the RGB image, wherein the event camera data includes the event data within an exposure period of each pixel row.

18. The device according to claim 11, characterized in that The feature extraction module includes a first feature extraction module and a second feature extraction module; the extraction processing module is further used to: Input the RGB image into the convolution layer, the channel attention mechanism layer and the residual convolution layer in the first feature extraction module in sequence, and output the image feature data; The event camera data is sequentially input into the convolution layer, the channel attention mechanism layer and the residual convolution layer in the second feature extraction module, and the brightness change feature data is output.

19. The device according to claim 18, characterized in that The extraction processing module is also used for: Input the RGB image into the convolution layer, and output a first convolution feature map; Performing global maximum pooling downsampling processing on the first convolutional feature map to obtain a first salient feature map; Performing global average pooling downsampling processing on the first convolution feature map to obtain a first mean feature map; Inputting the first significant feature map and the first mean feature map into a multi-layer perceptron, and outputting a first integrated feature map corresponding to the first significant feature map and a second integrated feature map corresponding to the first mean feature map; Adding the first integrated feature map and the second integrated feature map to obtain a first comprehensive feature map; Normalizing the first comprehensive feature map to obtain a first channel attention weight matrix; Calculate the product of the first channel attention weight matrix and the first convolution feature map, and output the image feature data.

20. The device according to claim 18, characterized in that The extraction processing module is also used for: Inputting the event camera data into the convolution layer, and outputting a second convolution feature map; Performing global maximum pooling downsampling processing on the second convolutional feature map to obtain a second salient feature map; Performing global average pooling downsampling processing on the second convolution feature map to obtain a second mean feature map; Inputting the second significant feature map and the second mean feature map into a multilayer perceptron, and outputting a third integrated feature map corresponding to the second significant feature map and a fourth integrated feature map corresponding to the second mean feature map; Adding the third integrated feature map and the fourth integrated feature map to obtain a second comprehensive feature map; Normalizing the second comprehensive feature map to obtain a second channel attention weight matrix; Calculate the product of the second channel attention weight matrix and the second convolution feature map, and output the brightness change feature data.

21. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the image processing method according to any one of claims 1 to 10 are implemented.

Images