An image processing method and related apparatus

Abstract

An image processing method applicable to the field of artificial intelligence is disclosed. The method includes: acquiring a first feature representation of a blurred image and a second feature representation of event data acquired by an event camera; the first and second feature representations have the same size; based on the first feature representation of the blurred image and the second feature representation of the event data, obtaining a first scene flow corresponding to the blurred image and a second scene flow corresponding to the event data through a scene flow prediction network; performing an affine transformation on the first feature representation based on the first scene flow to obtain a third feature representation; performing an affine transformation on the second feature representation based on the second scene flow to obtain a fourth feature representation; and using the third and fourth feature representations to deblur the blurred image. This application achieves refined alignment of blurred image features and event information features through multi-scale bidirectional scene flow alignment, thereby improving the deblurring effect of blurred images.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to an image processing method and related apparatus. Background Technology

[0002] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0003] Motion blur typically occurs in scenes with significant motion within an exposure time, especially on lightweight mobile devices in low-light conditions, such as smartphones and automotive cameras. While motion blur causes undesirable image degradation, making visual content difficult to interpret, motion-blurred images also encode rich information about the relative motion between the camera and the observed scene. Therefore, recovering (reconstructing) a sequence of sharp frames from a single motion-blurred image (photo-sequencing) helps in understanding scene dynamics and has wide applications in image reconstruction, autonomous driving, and video surveillance. A motion-blurred image can be viewed as the average of high-resolution frames within an exposure time. Because averaging disrupts the temporal order of frames, recovering a set of sharp frame sequences from a single motion-blurred image is highly inappropriate; that is, the sequence to be recovered is not unique, and sequences of different high-resolution frames may form the same motion-blurred image.

[0004] To address the non-uniqueness of sequences to be recovered, event cameras are introduced. Event cameras can provide temporal inter-frame changes to guide sequence recovery. Event cameras are bio-inspired, event-driven, time-based neuromorphic vision sensors that perceive the world on principles drastically different from traditional cameras. They measure brightness changes asynchronously, triggering an event once the change exceeds a threshold. Event cameras abandon the concepts of exposure time and frames found in traditional intensity cameras, enabling them to capture almost continuous motion in frameless mode (microsecond temporal resolution), thus avoiding problems like blur. Using event cameras will be extremely helpful in recovering sharp frames from blurry images.

[0005] Image deblurring using optical flow based on event information is based on the core idea of ​​calculating optical flow from event information, applying this optical flow to the blurred image using an affine transformation (warp), and combining it with various losses to achieve image deblurring at any point within the exposure time. However, since event information and the blurred image belong to two different modalities, there is currently no metric to characterize the inconsistency between the two scenes. The optical flow is not precise and suffers from pixel-level misalignment issues. Summary of the Invention

[0006] This application provides an image processing method that, through the alignment of multi-scale bidirectional scene flows, can achieve fine alignment between blurred image features and event information features, thereby obtaining accurate scene flow information. This solves the problem of inadequate consideration of pixel-level alignment in existing event-based deblurring methods, and thus improves the deblurring effect of blurred images.

[0007] In a first aspect, embodiments of this application provide an image processing method, comprising: acquiring a first feature representation of a blurred image and a second feature representation of event data acquired by an event camera; the first feature representation and the second feature representation have the same size; based on the first feature representation of the blurred image and the second feature representation of the event data, obtaining a first scene flow corresponding to the blurred image and a second scene flow corresponding to the event data through a scene flow prediction network, wherein the first scene flow and the first feature representation have the same size, each pixel feature in the first scene flow indicates motion information from a pixel feature at a corresponding pixel position in the first feature representation to a pixel feature at a corresponding pixel position in the second feature representation, the second scene flow and the second feature representation have the same size, each pixel feature in the second scene flow indicates motion information from a pixel feature at a corresponding pixel position in the second feature representation to a pixel feature at a corresponding pixel position in the first feature representation; performing an affine transformation (warp) on the first feature representation based on the first scene flow to obtain a third feature representation; performing an affine transformation on the second feature representation based on the second scene flow to obtain a fourth feature representation; the third feature representation and the fourth feature representation are used to deblur the blurred image.

[0008] Here, the “size” of the feature representation can be understood as the width and height of the feature representation.

[0009] Here, pixel features can refer to a point in space (x, y), which may contain multiple channels.

[0010] By aligning multi-scale bidirectional scene flows, fine-grained alignment of blurred image features and event information features can be achieved, resulting in accurate scene flow information. This solves the problem of inadequate consideration of pixel-level alignment in existing event-based deblurring methods, thereby improving the deblurring effect of blurred images.

[0011] In one possible implementation, the blurred image and the event data are acquired within the same time period for the same scene.

[0012] In one possible implementation, the scene flow prediction network may include a first encoding module, a second encoding module, a fusion module, a first decoding module, and a second decoding module; the step of obtaining the first scene flow corresponding to the blurred image and the second scene flow corresponding to the event data through the scene flow prediction network based on the first feature representation of the blurred image and the second feature representation of the event data may include:

[0013] Based on the first feature representation, a first encoding result is obtained through the first encoding module; based on the second feature representation, a second encoding result is obtained through the second encoding module; based on the first encoding result and the second encoding result, a fusion result is obtained through the fusion module; based on the fusion result, a first scene stream corresponding to the blurred image and a second scene stream corresponding to the event data are obtained through the first decoding module and the second decoding module, respectively.

[0014] In one possible implementation, the fusion module is used to achieve a first fusion of the first encoding result and the second encoding result based on an attention mechanism.

[0015] In this embodiment of the application, the first scene stream can represent the alignment relationship between blurred image features and event information features, and the second scene stream can represent the alignment relationship between event feature information and blurred image.

[0016] In this case, the feature representation of the blurred image and the feature representation of the event data are not information of the same modality. If the feature representation of the blurred image and the feature representation of the event data are directly fused, the fusion result will be inaccurate. In this embodiment, the feature representation of the blurred image and the feature representation of the event data are first encoded by two different encoding modules to convert them into data similar to the same modality, and the encoding results are fused to obtain an accurate fusion result.

[0017] It should be noted that the scene flow in this embodiment is similar to optical flow, and the information of each pixel position is a directional vector.

[0018] This application enables pixel-level alignment between blurred image features and event data features through scene flow prediction.

[0019] For images captured by RGB cameras, since they are obtained by fusing multiple frames captured within the same exposure time, the fused image may lose some information compared to event data (event data captured by event cameras includes multiple frames of event data captured within the same exposure time). For example, when capturing an image of a scene, if an object in the scene is occluded within the exposure time, this object will be invalid in the blurred RGB image, while its information will be valid in the event data. Similarly, since event data is only identified when the brightness change at a pixel location exceeds a threshold, some image data may be invalid. Therefore, directly using information from invalid (or occluded) areas in the image or time data will result in artifacts caused by degraded image quality.

[0020] In one possible implementation, for occluded regions in a blurred image, a second occluded region can be identified based on the fourth feature representation and the first feature representation (e.g., a lightweight network (such as consecutive convolutions and residuals) can be used to determine the occluded region), wherein the image data of the second occluded region in the blurred image is valid in the second occluded region of the event data; the feature representations in the first feature representation excluding the second occluded region and the feature representation of the second occluded region in the fourth feature representation are then fused to obtain a second fused feature representation. In other words, some features of the blurred image are occluded, and information from the event data can be used to replace these occluded features, thereby obtaining a more accurate feature representation.

[0021] In one possible implementation, the second occlusion region can be represented by a second mask, which has the same size as the fourth feature representation. Each pixel in the second mask is used to indicate whether the pixel feature at the corresponding position in the first feature representation is valid in the blurred image. For example, 0 and 1 can be used in the second mask to indicate whether the pixel feature at the corresponding position is valid in the event data, such as 0 indicating invalid and 1 indicating valid.

[0022] In one possible implementation, for occluded regions in the event data, a first occluded region can be determined based on the third feature representation and the second feature representation (e.g., a lightweight network (such as consecutive convolutions and residuals) can be used to determine the occluded region), wherein the image data of the first occluded region in the event data is valid in the first occluded region of the blurred image; the feature representations in the second feature representation excluding the first occluded region and the feature representation of the first occluded region in the third feature representation are then fused in a second way to obtain a first fused feature representation. In other words, some features of the event information are occluded, and information from the blurred image can be used to replace these occluded features, thereby obtaining a more accurate feature representation.

[0023] In one possible implementation, the first occlusion region is represented by a first mask, which has the same size as the third feature representation. Each pixel in the first mask is used to indicate whether the pixel feature at the corresponding position in the third feature representation is valid in the event data. For example, 0 and 1 can be used in the first mask to indicate whether the pixel feature at the corresponding position is valid in the event data, such as 0 indicating invalid and 1 indicating valid.

[0024] In one possible implementation, the second fusion is an addition operation of corresponding pixel positions.

[0025] By using the above method and setting explicit occlusion perception feature fusion, it is possible to process the occluded areas in a blurred image, thereby reducing the artifact problem caused by the occluded areas.

[0026] In one possible implementation, the method further includes: processing the feature representation of the blurred image and the feature representation of the event data through N concatenated feature nesting blocks to obtain a processing result for deblurring; wherein each of the feature nesting blocks is used to perform the image processing method as described in the first aspect, the first feature nesting block is used to obtain the feature representation extracted from the blurred image and the event data by the feature extraction network, the nth feature nesting block is used to obtain the feature representation output by the (n-1)th feature nesting block, and n is less than N.

[0027] In one possible implementation, the feature representation output by the Nth nested feature block is fused with the feature representation extracted from the blurred image by the feature extraction network to obtain residual information, which is then fused with the blurred image to achieve deblurring of the blurred image.

[0028] Secondly, this application provides an image processing apparatus, the apparatus comprising:

[0029] The acquisition module is used to acquire a first feature representation of the blurred image and a second feature representation of the event data acquired by the event camera; the first feature representation and the second feature representation have the same size;

[0030] A scene flow prediction module is used to obtain a first scene flow corresponding to the blurred image and a second scene flow corresponding to the event data through a scene flow prediction network based on a first feature representation of the blurred image and a second feature representation of the event data. The first scene flow and the first feature representation have the same size. Each pixel feature in the first scene flow indicates motion information from the pixel feature at the corresponding pixel position in the first feature representation to the pixel feature at the corresponding pixel position in the second feature representation. The second scene flow and the second feature representation have the same size. Each pixel feature in the second scene flow indicates motion information from the pixel feature at the corresponding pixel position in the second feature representation to the pixel feature at the corresponding pixel position in the first feature representation.

[0031] The affine transformation module is used to perform an affine transformation on the first feature representation based on the first scene flow to obtain the third feature representation;

[0032] Based on the second scene flow, an affine transformation is performed on the second feature representation to obtain a fourth feature representation; the third feature representation and the fourth feature representation are used to deblur the blurred image.

[0033] By aligning multi-scale bidirectional scene flows, fine-grained alignment of blurred image features and event information features can be achieved, resulting in accurate scene flow information. This solves the problem of inadequate consideration of pixel-level alignment in existing event-based deblurring methods, thereby improving the deblurring effect of blurred images.

[0034] In one possible implementation, the blurred image and the event data are acquired within the same time period for the same scene.

[0035] In one possible implementation, the scene flow prediction network includes a first encoding module, a second encoding module, a fusion module, a first decoding module, and a second decoding module;

[0036] The scene flow prediction module is specifically used for:

[0037] Based on the first feature representation, a first encoding result is obtained through the first encoding module;

[0038] Based on the second feature representation, the second encoding result is obtained through the second encoding module;

[0039] Based on the first encoding result and the second encoding result, a fusion result is obtained through the fusion module;

[0040] Based on the fusion result, the first scene stream corresponding to the blurred image and the second scene stream corresponding to the event data are obtained through the first decoding module and the second decoding module, respectively.

[0041] In this case, the feature representation of the blurred image and the feature representation of the event data are not information of the same modality. If the feature representation of the blurred image and the feature representation of the event data are directly fused, the fusion result will be inaccurate. In this embodiment, the feature representation of the blurred image and the feature representation of the event data are first encoded by two different encoding modules to convert them into data similar to the same modality, and the encoding results are fused to obtain an accurate fusion result.

[0042] In one possible implementation, the device further includes:

[0043] An occlusion region identification module is used to identify a second occlusion region based on the fourth feature representation and the first feature representation, wherein image data of the second occlusion region in the blurred image is valid in the second occlusion region of the event data;

[0044] The feature representations in the first feature representation excluding the second occluded region and the feature representations in the fourth feature representation of the second occluded region are fused together to obtain a second fused feature representation.

[0045] In one possible implementation, the second occlusion region is represented by a second mask, which has the same size as the fourth feature representation, and each pixel in the second mask is used to indicate whether the pixel feature at the corresponding position in the first feature representation is valid in the blurred image.

[0046] In one possible implementation, the device further includes:

[0047] An occlusion region identification module is used to determine a first occlusion region based on the third feature representation and the second feature representation, wherein the image data of the first occlusion region in the event data is valid in the first occlusion region of the blurred image;

[0048] The feature representations in the second feature representation excluding the first occluded region and the feature representations in the third feature representation of the first occluded region are fused together to obtain the first fused feature representation.

[0049] In one possible implementation, the first occlusion region is represented by a first mask, the first mask and the third feature representation having the same size, and each pixel in the first mask is used to indicate whether the pixel feature at the corresponding position in the third feature representation is valid in the event data.

[0050] By using the above method and setting explicit occlusion perception feature fusion, it is possible to process the occluded areas in a blurred image, thereby reducing the artifact problem caused by the occluded areas.

[0051] In one possible implementation, the second fusion is an addition operation of corresponding pixel positions.

[0052] In one possible implementation, the apparatus further includes: a feature nesting module, configured to process the feature representation of a blurred image and the feature representation of event data through N serially connected feature nesting blocks to obtain a processing result for deblurring; wherein each of the feature nesting blocks is configured to perform an image processing method as described in the first aspect, the first feature nesting block is configured to obtain the feature representation extracted from the blurred image and the event data by a feature extraction network, the nth feature nesting block is configured to obtain the feature representation output by the (n-1)th feature nesting block, where n is less than N.

[0053] In one possible implementation, the feature representation output by the Nth nested feature block is fused with the feature representation extracted from the blurred image by the feature extraction network to obtain residual information, which is then fused with the blurred image to achieve deblurring of the blurred image.

[0054] Thirdly, embodiments of this application provide an image processing apparatus, which may include a memory, a processor, and a bus system, wherein the memory is used to store a program, and the processor is used to execute the program in the memory to perform any of the optional methods described in the first aspect above.

[0055] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the methods described in the first aspect and any optional method.

[0056] Fifthly, embodiments of this application provide a computer program product, including code, which, when executed, is used to implement the first aspect and any optional method described above.

[0057] Sixthly, this application provides a chip system including a processor for supporting an execution device or training device in implementing the functions involved in the foregoing aspects, such as transmitting or processing data involved in the foregoing methods; or, information. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for the execution device or training device. This chip system may be composed of chips or may include chips and other discrete devices. Attached Figure Description

[0058] Figure 1 A structural diagram illustrating the main framework of artificial intelligence;

[0059] Figure 2 A schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0060] Figure 3 A schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0061] Figure 4 A schematic diagram of a convolutional neural network provided in an embodiment of this application;

[0062] Figure 5 A schematic diagram of a convolutional neural network provided in an embodiment of this application;

[0063] Figure 6 A schematic diagram of the structure of a system provided in this application embodiment;

[0064] Figure 7 A schematic diagram of a chip structure provided in an embodiment of this application;

[0065] Figure 8 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0066] Figure 9 This is a flowchart illustrating an image processing method.

[0067] Figure 10 This is a flowchart illustrating an image processing method.

[0068] Figure 11 This is a flowchart illustrating an image processing method.

[0069] Figure 12 This is a flowchart illustrating an image processing method.

[0070] Figure 13 This is a schematic diagram illustrating the effect of an image processing method provided in an embodiment of this application;

[0071] Figure 14 This is a schematic diagram illustrating the effect of an image processing method provided in an embodiment of this application;

[0072] Figure 15 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application;

[0073] Figure 16 A schematic diagram of an execution device provided in an embodiment of this application;

[0074] Figure 17 This is a schematic diagram of a training device provided in an embodiment of this application. Detailed Implementation

[0075] The embodiments of the present invention will now be described with reference to the accompanying drawings. The terminology used in the embodiments section is for illustrative purposes only and is not intended to limit the scope of the invention.

[0076] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0077] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0078] First, the overall workflow of the artificial intelligence system is described; please refer to [link / reference]. Figure 1 , Figure 1 The diagram illustrates a structural framework for artificial intelligence (AI). The framework is further elaborated below along two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that AI brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed through technological means) to the industrial ecosystem of the system.

[0079] (1) Infrastructure

[0080] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0081] (2) Data

[0082] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0083] (3) Data processing

[0084] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0085] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0086] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0087] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0088] (4) General ability

[0089] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0090] (5) Smart Products and Industry Applications

[0091] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.

[0092] The image processing method in this embodiment can be applied to intelligent vehicles with assisted driving and autonomous driving capabilities, as well as to areas in computer vision fields such as smart cities and smart terminals that require image enhancement (e.g., image denoising). The following sections will describe these applications in conjunction with... Figure 2 and Figure 3 This section provides a brief introduction to video streaming and video surveillance scenarios.

[0093] Video streaming scenarios:

[0094] For example, when playing videos on a client device using a smart terminal (such as a mobile phone, car, robot, tablet, desktop computer, smartwatch, virtual reality (VR), augmented reality (AR) devices, etc.), to reduce the bandwidth requirements of the video stream, the server can transmit a downsampled, lower-resolution, low-quality video stream to the client over the network. The client can then enhance the images in this low-quality video stream. For example, it can perform super-resolution or noise reduction operations on the images in the video, ultimately presenting a high-quality image to the user.

[0095] Video surveillance scenarios:

[0096] In the security field, due to limitations such as the installation location of surveillance cameras and limited storage space, the image quality of some video surveillance is poor, which affects the accuracy of human or recognition algorithms in identifying targets. Therefore, the image processing method provided in this application can be used to convert low-quality video surveillance videos into high-quality high-definition videos, thereby effectively restoring a large amount of detail in the surveillance images and providing more effective and richer information for subsequent target recognition tasks.

[0097] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts such as neural networks involved in the embodiments of this application will be introduced below.

[0098] (1) Neural Network

[0099] A neural network can be composed of neural units, which can be defined as a computational unit that takes xs (i.e., input data) and an intercept of 1 as input. The output of this computational unit can be:

[0100]

[0101] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer, and the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0102] (2) A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer refers to the layer of neurons in a CNN that performs convolutional processing on the input signal. In a convolutional layer of a CNN, a neuron can be connected to only some of the neurons in its neighboring layers. A convolutional layer typically contains several feature planes, each composed of rectangularly arranged neural units. Neural units on the same feature plane share weights, which are the convolutional kernel. Shared weights can be understood as the way features are extracted being independent of their location. The convolutional kernel can be formalized as a matrix of random size, and during the training process of the CNN, the kernel can learn reasonable weights. Furthermore, the direct benefit of shared weights is reducing the connections between layers in the CNN, while also reducing the risk of overfitting.

[0103] CNN is a very common type of neural network. Below, we will combine... Figure 4 This section focuses on a detailed explanation of the structure of CNNs. As mentioned in the basic concept introduction above, a Convolutional Neural Network (CNN) is a deep neural network with a convolutional structure. It is a deep learning architecture, which refers to learning at multiple levels of abstraction through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network, in which each neuron can respond to the input image.

[0104] like Figure 4As shown, the convolutional neural network (CNN) 200 may include an input layer 210, a convolutional / pooling layer 220 (where the pooling layer is optional), and a fully connected layer 230.

[0105] Convolutional / pooling layers 220:

[0106] Convolutional layers:

[0107] like Figure 4 The convolutional / pooling layer 220 shown may include layers as in Examples 221-226. For instance, in one implementation, layer 221 is a convolutional layer, layer 222 is a pooling layer, layer 223 is a convolutional layer, layer 224 is a pooling layer, layer 225 is a convolutional layer, and layer 226 is a pooling layer; in another implementation, layers 221 and 222 are convolutional layers, layer 223 is a pooling layer, layers 224 and 225 are convolutional layers, and layer 226 is a pooling layer. That is, the output of the convolutional layer can be used as the input to a subsequent pooling layer, or as the input to another convolutional layer to continue the convolution operation.

[0108] The following section will use convolutional layer 221 as an example to introduce the internal working principle of a convolutional layer.

[0109] Convolutional layer 221 can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter to extract specific information from the input image matrix. Essentially, a convolution operator can be a weight matrix, which is usually predefined. During the convolution operation, the weight matrix typically processes the input image pixel by pixel (or two pixels by two pixels, depending on the stride) along the horizontal direction, thus extracting specific features from the image. The size of the weight matrix should be related to the image size. It's important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during convolution, the weight matrix extends to the entire depth of the input image. Therefore, convolution with a single weight matrix produces a single-depth convolutional output. However, in most cases, a single weight matrix is ​​not used; instead, multiple weight matrices of the same size (rows × columns) are applied—multiple identical matrices. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image; this dimension can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features from an image. For example, one weight matrix can be used to extract image edge information, another weight matrix can be used to extract specific colors of the image, and yet another weight matrix can be used to blur unwanted noise in the image. These multiple weight matrices have the same size (rows × columns), and the feature maps extracted by these multiple weight matrices of the same size also have the same size. The extracted feature maps of the same size are then merged to form the output of the convolution operation.

[0110] The weight values ​​in these weight matrices need to be obtained through extensive training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, thereby enabling the convolutional neural network 200 to make correct predictions.

[0111] When a convolutional neural network 200 has multiple convolutional layers, the initial convolutional layers (e.g., 221) tend to extract more general features, which can also be called low-level features. As the depth of the convolutional neural network 200 increases, the features extracted by later convolutional layers (e.g., 226) become more and more complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem to be solved.

[0112] Pooling layer:

[0113] Because it is often necessary to reduce the number of training parameters, pooling layers are often introduced periodically after convolutional layers, such as... Figure 4Layers 221-226 in example 220 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In image processing, the sole purpose of pooling layers is to reduce the spatial size of the image. Pooling layers can include average pooling and / or max pooling operators to sample the input image to obtain a smaller image size. The average pooling operator calculates the average value of pixel values ​​within a specific range as the result of average pooling. The max pooling operator takes the pixel with the largest value within a specific range as the result of max pooling. Furthermore, just as the size of the weight matrix in a convolutional layer should be related to the image size, the operators in a pooling layer should also be related to the image size. The size of the output image after processing by the pooling layer can be smaller than the size of the input image of the pooling layer. Each pixel in the output image of the pooling layer represents the average or maximum value of the corresponding sub-region of the input image of the pooling layer.

[0114] Fully connected layer 230:

[0115] After processing by the convolutional / pooling layers 220, the convolutional neural network 200 is still insufficient to output the required information. As mentioned earlier, the convolutional / pooling layers 220 only extract features and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), the convolutional neural network 200 needs to utilize fully connected layers 230 to generate one or a set of outputs representing the required number of classes. Therefore, the fully connected layers 230 can include multiple hidden layers (such as...). Figure 4 As shown in 231, 232 to 23n), the parameters contained in these multi-layer hidden layers can be pre-trained based on relevant training data for specific task types. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.

[0116] After the multiple hidden layers in the fully connected layer 230, the final layer of the entire convolutional neural network 200 is the output layer 240. This output layer 240 has a loss function similar to the classification cross-entropy, specifically used to calculate the prediction error. Once the entire convolutional neural network 200 has propagated forward (e.g., ... Figure 4 Propagation from 210 to 240 degrees is considered forward propagation, while backward propagation (e.g.) is completed. Figure 4 The propagation from 240 to 210 (backpropagation) will begin to update the weight values ​​and biases of the layers mentioned above, in order to reduce the loss of the convolutional neural network 200 and the error between the output of the convolutional neural network 200 through the output layer and the ideal result.

[0117] It should be noted that, as Figure 4The convolutional neural network 200 shown is merely an example of a convolutional neural network. In specific applications, convolutional neural networks can also exist in the form of other network models, for example, including only... Figure 4 As shown in the network structure, for example, the convolutional neural network used in the embodiments of this application may only include an input layer 210, a convolutional / pooling layer 220, and an output layer 240.

[0118] It should be noted that, as Figure 4 The convolutional neural network 100 shown is merely an example of a convolutional neural network. In specific applications, convolutional neural networks can also exist in the form of other network models, such as... Figure 5 The multiple convolutional / pooling layers shown are implemented in parallel, and the extracted features are all input into the fully connected layer 230 for processing.

[0119] (3) Deep Neural Networks

[0120] Deep Neural Networks (DNNs), also known as multilayer neural networks, can be understood as neural networks with many hidden layers, though there's no specific metric for "many." DNNs can be categorized into three layers based on their position: input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. All layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer. Although DNNs appear complex, the operation of each layer is actually quite simple, resembling a linear relationship as follows: in, It is the input vector. It is the output vector. α is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number of these parameters is therefore quite large. The definitions of these parameters in a DNN are as follows: Taking the coefficient W as an example: Assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W resides, while the subscript corresponds to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the k-th neuron in layer L-1 to the j-th neuron in layer L are defined as follows: It's important to note that the input layer does not have a W parameter. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by the vectors W from many layers).

[0121] (4) Super-resolution

[0122] Super-resolution (SR) is an image enhancement technique that, given one or a set of low-resolution images, restores high-frequency details by learning prior knowledge of the images, leveraging the similarity between images, and complementing information from multiple frames, thereby generating a higher-resolution target image. In applications, super-resolution can be categorized into single-frame image super-resolution and video super-resolution, depending on the number of input images. Super-resolution has significant applications in high-definition television, surveillance equipment, satellite imagery, and medical imaging.

[0123] (5) Noise Reduction

[0124] Images are often affected by imaging equipment and the external environment during digitization and transmission, resulting in noise in the images. The process of reducing noise in images is called image denoising, and sometimes it can also be called image noise reduction.

[0125] (6) Image features

[0126] Image features mainly include color features, texture features, shape features, and spatial relationship features.

[0127] Color features are global features that describe the surface properties of objects within an image or image region. Generally, color features are pixel-based, meaning all pixels belonging to the image or image region contribute to the overall picture. Because color is insensitive to changes in the orientation and size of the image or image region, color features cannot effectively capture the local features of objects within an image.

[0128] Texture features are also a type of global feature, describing the surface properties of objects within an image or image region. However, since texture is only a characteristic of an object's surface and cannot fully reflect the object's essential attributes, high-level image content cannot be obtained solely using texture features. Unlike color features, texture features are not pixel-based; they require statistical calculations within a region containing multiple pixels.

[0129] Shape features can be represented in two ways: contour features and region features. Contour features of an image mainly target the outer boundary of an object, while region features of an image relate to the entire shape region.

[0130] Spatial relationship features refer to the spatial position or relative orientation of multiple targets segmented from an image. These relationships can be categorized into connectivity / adjacency, overlap / intersection, and containment / enclosure relationships. Spatial position information is typically divided into two categories: relative spatial position information and absolute spatial position information. The former emphasizes the relative relationship between targets, such as up / down and left / right relationships, while the latter emphasizes the distance and orientation between targets.

[0131] It should be noted that the image features listed above are just some examples of the features that an image may have. Images may also have other features, such as higher-level features: semantic features, which will not be discussed here.

[0132] (7) Image / Video Enhancement

[0133] Image / video enhancement refers to actions performed on images / videos to improve image quality. Examples of enhancement processes include super-resolution, noise reduction, sharpening, and depixelation.

[0134] (8) Peak signal-to-noise ratio (PSNR)

[0135] Peak signal-to-noise ratio (PSNR) is an engineering term representing the ratio of the maximum possible power of a signal to the power of destructive noise that affects the accuracy of its representation. It is frequently used as a measure of signal reconstruction quality in fields such as image processing, and is typically defined simply by mean square error. Generally, the higher the PSNR, the smaller the gap between the representation and the true value.

[0136] (9) Structural Similarity (SSIM)

[0137] SSIM is a metric that measures the similarity between two images, ranging from 0 to 1. When two images are exactly the same, the SSIM value is equal to 1.

[0138] (10) Receptive Field

[0139] In the field of deep neural networks within computer vision, the receptive field is a term used to describe the size of the receptive field of neurons at different locations within a neural network. A larger receptive field value indicates a wider range of the original image that the neuron can access, meaning that the neuron may contain more global and semantically higher-level features; conversely, a smaller value indicates that the features it contains are more local and detailed. The receptive field value can be used to roughly determine the level of abstraction of each layer.

[0140] (11) Event cameras

[0141] It is a bio-inspired visual sensor that works in a completely different way from standard cameras. Instead of outputting intensity image frames at a constant rate, the event camera outputs only information related to local pixel-level brightness changes.

[0142] (12) Optical flow

[0143] Optical flow represents the velocity and direction of motion of each pixel in two adjacent frames. It is the instantaneous velocity of a moving object in space on the observation imaging plane. It's a method that uses the temporal changes of pixels in an image sequence and the correlation between adjacent frames to find the correspondence between the previous and current frames, thereby calculating the motion information of objects between adjacent frames. Generally, optical flow is caused by the movement of foreground objects in the scene, camera movement, or both.

[0144] (13) Scene Flow

[0145] Similar to optical flow, but not strictly optical flow. Optical flow characterizes the instantaneous velocity of a pixel and is generally derived from features of the same modality, such as adjacent image frames or images from different RGB cameras. In the embodiments of this application, it can characterize the control positional relationship between two different modal information: event information and image information (e.g., grayscale information), represented using scene flow.

[0146] (14) Warp operation

[0147] Operations that are typically performed in conjunction with a stream (such as optical flow, and scene flow in this application) represent an affine transformation of an image relative to a stream (such as optical flow, scene flow), such as rotation, translation, scaling, etc.

[0148] (15) Loss Function

[0149] In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss.

[0150] (16) Backpropagation algorithm

[0151] Backpropagation (BP) can be used during training to correct the parameters in the initial model, thereby reducing the model's error loss. Specifically, forward propagation of the input signal to the output generates error loss; this error loss information is then propagated back to update the parameters in the initial model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining optimal model parameters, such as the weight matrix.

[0152] The following is combined with Figure 6 The system architecture provided in the embodiments of this application will be described in detail. Figure 6 This is a schematic diagram of a system architecture provided for an embodiment of this application. For example... Figure 6 As shown, the system architecture 500 includes an execution device 510, a training device 520, a database 530, a client device 540, a data storage system 550, and a data acquisition device 560.

[0153] The execution device 510 includes a calculation module 511, an I / O interface 512, a preprocessing module 513, and a preprocessing module 514. The calculation module 511 may include a target model / rule 501, while the preprocessing modules 513 and 514 are optional.

[0154] Data acquisition device 560 is used to collect training data. The video samples can be low-quality images, while the supervision images are high-quality images corresponding to the image samples acquired before model training. For example, the image samples can be low-resolution images, and the supervision images can be high-resolution images; alternatively, the image samples can be videos containing fog or noise, and the supervision images can be images with the fog or noise removed. After collecting the training data, data acquisition device 560 stores this training data in database 530, and training device 520 trains the target model / rule 501 based on the training data maintained in database 530.

[0155] The aforementioned target model / rule 501 (e.g., the model including the scene flow prediction network in this embodiment) can be used to achieve the image denoising task; that is, by inputting the image to be processed into the target model / rule 501, the denoised image can be obtained. It should be noted that in practical applications, the training data maintained in the database 530 may not all come from the data acquisition device 560; it may also be received from other devices. Furthermore, it should be noted that the training device 520 may not necessarily train the target model / rule 501 entirely based on the training data maintained in the database 530; it may also obtain training data from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0156] The target model / rule 501 trained using training device 520 can be applied to different systems or devices, such as... Figure 6 The execution device 510 shown can be a terminal, such as a mobile phone terminal, tablet computer, laptop computer, augmented reality (AR) / virtual reality (VR) device, vehicle terminal, etc., or it can be a server or cloud, etc. Figure 6 In the execution device 510, an input / output (I / O) interface 512 is configured for data interaction with external devices. Users can input data to the I / O interface 512 through the client device 540.

[0157] Preprocessing modules 513 and 514 are used to preprocess the input data received from the I / O interface 512. It should be understood that preprocessing modules 513 and 514 may be absent, or only one preprocessing module may be used. When preprocessing modules 513 and 514 are absent, the calculation module 511 can be used directly to process the input data.

[0158] During the preprocessing of input data by the execution device 510, or during the calculation module 511 of the execution device 510 performing calculations and other related processes, the execution device 510 can call data, code, etc. in the data storage system 550 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 550.

[0159] Finally, the I / O interface 512 presents the processing result, such as the denoised image obtained after processing, to the client device 540, thereby providing it to the user.

[0160] It is worth noting that the training device 520 can generate corresponding target models / rules 501 based on different training data for different targets or tasks. The corresponding target models / rules 501 can be used to achieve image denoising tasks, thereby providing users with the desired results.

[0161] exist Figure 6 In the illustrated scenario, the user can manually provide input data, which can be done through the interface provided by I / O interface 512. Alternatively, the client device 540 can automatically send input data to I / O interface 512. If user authorization is required for the client device 540 to automatically send input data, the user can set the corresponding permissions in the client device 540. The user can view the output results of the execution device 510 on the client device 540, which can be presented in various forms such as display, sound, or animation. The client device 540 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 512 as shown in the figure, and storing them as new sample data in database 530. Alternatively, data can be collected directly from the I / O interface 512 without going through the client device 540, using the input data and output results of the input I / O interface 512 as shown in the figure, and storing them as new sample data in database 530.

[0162] It is worth noting that, Figure 6 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 6 In this context, the data storage system 550 is an external memory relative to the execution device 510. In other cases, the data storage system 550 may also be placed within the execution device 510.

[0163] The following describes a chip hardware structure provided by an embodiment of this application.

[0164] Figure 7This application provides a chip hardware structure diagram according to an embodiment of the present application. The chip includes a neural network processor 700. This chip can be configured as follows: Figure 6 The execution device 510 shown is used to perform the calculations of the calculation module 511. This chip can also be placed in, for example... Figure 6 The training device 520 shown is used to complete the training work of the training device 520 and output the target model / rule 501. For example... Figure 6 The algorithms for each layer in the model shown can all be implemented in, for example... Figure 7 This is achieved in the chip shown.

[0165] The neural processing unit (NPU) 700 is mounted as a coprocessor on the host central processing unit (host CPU), and tasks are assigned by the host CPU. The core of the NPU is the arithmetic circuit 703, and the controller 704 controls the arithmetic circuit 703 to retrieve data from the memory (weight memory 702 or input memory 701) and perform calculations.

[0166] In some implementations, the arithmetic circuit 703 internally includes multiple process engines (PEs). In some implementations, the arithmetic circuit 703 is a two-dimensional pulsating array. The arithmetic circuit 703 can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 703 is a general-purpose matrix processor.

[0167] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit 703 retrieves the corresponding data for matrix B from the weight memory 702 and caches it in each PE (Engineer Component) of the arithmetic circuit 703. The arithmetic circuit 703 retrieves the data for matrix A from the input memory 701 and performs matrix operations with matrix B. The partial or final result of the obtained matrix is ​​stored in the accumulator 708.

[0168] The vector computation unit 707 can further process the output of the arithmetic circuit 703, such as vector multiplication, vector addition, exponentiation, logarithmic operations, size comparisons, etc. For example, the vector computation unit 707 can be used for network computation in non-convolutional / non-FC layers of neural networks, such as pooling, batch normalization, local response normalization, etc.

[0169] In some implementations, the vector computation unit 707 can store the processed output vector in the unified memory 706. For example, the vector computation unit 707 can apply a nonlinear function to the output of the arithmetic circuit 703, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit 707 generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as an activation input to the arithmetic circuit 703, for example, for use in subsequent layers of a neural network.

[0170] The unified memory 706 is used to store input data and output data.

[0171] The weight data is directly transferred from the external memory to the input memory 701 and / or the unified memory 706 through the direct memory access controller (DMAC) 705, the weight data in the external memory is stored in the weight memory 702, and the data in the unified memory 706 is stored in the external memory.

[0172] The bus interface unit (BIU) 710 is used to enable interaction between the main CPU, DMAC and instruction fetch memory 709 via a bus.

[0173] The instruction fetch buffer 709, which is connected to the controller 704, is used to store the instructions used by the controller 704.

[0174] The controller 704 is used to call the instructions cached in the instruction fetch memory 709 to control the operation of the arithmetic accelerator.

[0175] Generally, the unified memory 706, input memory 701, weighted memory 702, and instruction fetch memory 709 are all on-chip memories, while the external memory is memory outside the NPU. This external memory can be double data rate synchronous dynamic random access memory (DDR SDRAM), high bandwidth memory (HBM), or other readable and writable memory.

[0176] Motion blur typically occurs in scenes with significant motion within an exposure time, especially on lightweight mobile devices in low-light conditions, such as smartphones and automotive cameras. While motion blur causes undesirable image degradation, making visual content difficult to interpret, motion-blurred images also encode rich information about the relative motion between the camera and the observed scene. Therefore, recovering (reconstructing) a sequence of sharp frames from a single motion-blurred image (photo-sequencing) helps in understanding scene dynamics and has wide applications in image reconstruction, autonomous driving, and video surveillance. A motion-blurred image can be viewed as the average of high-resolution frames within an exposure time. Because averaging disrupts the temporal order of frames, recovering a set of sharp frame sequences from a single motion-blurred image is highly inappropriate; that is, the sequence to be recovered is not unique, and sequences of different high-resolution frames may form the same motion-blurred image.

[0177] To address the non-uniqueness of sequences to be recovered, event cameras are introduced. Event cameras can provide temporal inter-frame changes to guide sequence recovery. Event cameras are bio-inspired, event-driven, time-based neuromorphic vision sensors that perceive the world on principles drastically different from traditional cameras. They measure brightness changes asynchronously, triggering an event once the change exceeds a threshold. Event cameras abandon the concepts of exposure time and frames found in traditional intensity cameras, enabling them to capture almost continuous motion in frameless mode (microsecond temporal resolution), thus avoiding problems like blur. Using event cameras will be extremely helpful in recovering sharp frames from blurry images.

[0178] Image deblurring using optical flow based on event information is based on the core idea of ​​calculating optical flow from event information, applying this optical flow to the blurred image using an affine transformation (warp), and combining it with various losses to achieve image deblurring at any point within the exposure time. However, since event information and the blurred image belong to two different modalities, there is currently no metric to characterize the inconsistency between the two scenes. The optical flow is not precise and suffers from pixel-level misalignment issues.

[0179] To address the aforementioned issues, this application provides an image processing method that can be either a feedforward process for model training or an inference process.

[0180] Reference Figure 8 , Figure 8 This is an example illustration of an image processing method provided in this application. Figure 8 As shown, an image processing method provided in this application embodiment includes:

[0181] 801. Obtain a first feature representation of the blurred image and a second feature representation of the event data acquired by the event camera; the first feature representation and the second feature representation have the same size.

[0182] In this embodiment of the application, the execution subject of step 801 can be a terminal device, which can be a portable mobile device, such as, but not limited to, mobile or portable computing devices (such as smartphones), personal computers, server computers, handheld devices (such as tablets) or laptop devices, multiprocessor systems, game consoles or controllers, microprocessor-based systems, set-top boxes, programmable consumer electronics, mobile phones, mobile computing and / or communication devices with wearable or accessory form factors (such as watches, glasses, headphones or earphones), network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0183] In this embodiment of the application, the execution subject of step 801 can be a cloud-side server. The server can receive the blurred image sent from the terminal device and the event data collected by the event camera, and then the server can obtain the blurred image and the event data collected by the event camera.

[0184] In one possible implementation, the blurred image and the event data are acquired within the same time period for the same scene. For example, the blurred image can be an image captured by an RGB camera on the terminal device, and the event data can be an image captured by an event camera on the terminal device for the same scene.

[0185] The blurred image can be the average of multiple frames (images obtained within the exposure time), and the event data can include event points within the time period corresponding to the blurred image. In other words, the blurred image can be obtained by averaging multiple existing consecutive frames to obtain a single frame of synthesized blurred image.

[0186] The time period corresponding to the aforementioned blurred image can be determined by the time periods corresponding to the existing consecutive high-definition images. This time period can be the exposure time of the camera or webcam during the actual shooting. In other words, within the exposure time period, the subject's movement causes blurring, resulting in a blurred image frame. This blurred image frame corresponds to a sequence of image frames. For example, assuming that the blurred image B1 is obtained by averaging 6 consecutive images between times T0 and T1, then the time period corresponding to the blurred image B1 is T0-T1.

[0187] Event data can include multiple time points, and event points can also be called events. The most basic principle of an event camera is that when the cumulative brightness change of a pixel reaches a trigger condition (the change reaches a certain level), an event point is output. Therefore, an event point can be understood as an expression of an event: at what time (timestamp), which pixel (pixel coordinates), the brightness increased or decreased (brightness change) occurred.

[0188] In one possible implementation, the blurred image can be a grayscale image with dimensions H*W, where H represents the height of the image and W represents the width of the image. Image features F of the blurred image can be extracted using a feature extraction network (e.g., multiple convolutional layers). B (e.g., the first feature representation in the embodiments of this application).

[0189] In one possible implementation, event data (or event information) can be processed by a feature extraction network (e.g., multiple convolutional layers) to extract event features F. E (For example, the second feature representation in the embodiments of this application). It should be noted that the spatial resolution of the event information at a certain moment can be the same as that of the blurred image, which is H*W; however, the input here is all the event information within the exposure time of the blurred image, which contains M channels, so the event information input is H*W*M, where M represents the number of event information.

[0190] The first and second feature representations obtained through the feature extraction network can be feature representations of the same size. Here, the "size" of the feature representation can be understood as the width and height of the feature representation.

[0191] It should be understood that the feature representation of the blurred image and the feature representation of the event data are processed through N concatenated nested feature blocks. The first feature representation and the second feature representation can be the feature representation of one of the N concatenated nested feature blocks. If the first feature representation and the second feature representation can be the feature input to the first nested feature block, then the first feature representation and the second feature representation can be the feature representation extracted from the blurred image and the event data by the feature extraction network. If the first feature representation and the second feature representation can be the feature input to the nth (n is greater than 1) nested feature block, then the first feature representation and the second feature representation can be the feature representation output by the (n-1)th nested feature block.

[0192] 802. Based on the first feature representation of the blurred image and the second feature representation of the event data, a scene flow prediction network is used to obtain a first scene flow corresponding to the blurred image and a second scene flow corresponding to the event data. The first scene flow and the first feature representation have the same size. Each pixel feature in the first scene flow indicates motion information from the pixel feature at the corresponding pixel position in the first feature representation to the pixel feature at the corresponding pixel position in the second feature representation. The second scene flow and the second feature representation have the same size. Each pixel feature in the second scene flow indicates motion information from the pixel feature at the corresponding pixel position in the second feature representation to the pixel feature at the corresponding pixel position in the first feature representation.

[0193] In one possible implementation, the scene flow prediction network can be a network included in a feature nesting block as described above, and the first feature representation of the blurred image and the second feature representation of the event data can be input into the scene flow prediction network.

[0194] In one possible implementation, the first feature representation of the blurred image and the second feature representation of the event data can be input into a scene flow prediction network to obtain a first scene flow corresponding to the blurred image and a second scene flow corresponding to the event data.

[0195] The scene flow prediction network in this application embodiment will be introduced next:

[0196] In one possible implementation, the scene flow prediction network may include a first encoding module, a second encoding module, a fusion module, a first decoding module, and a second decoding module; the step of obtaining a first scene flow corresponding to the blurred image and a second scene flow corresponding to the event data through the scene flow prediction network based on a first feature representation of the blurred image and a second feature representation of the event data may include: obtaining a first encoding result through the first encoding module based on the first feature representation; obtaining a second encoding result through the second encoding module based on the second feature representation; obtaining a fusion result through the fusion module based on the first encoding result and the second encoding result; and obtaining the first scene flow corresponding to the blurred image and the second scene flow corresponding to the event data through the first decoding module and the second decoding module, respectively, based on the fusion result.

[0197] In one possible implementation, the fusion module is used to achieve a first fusion of the first encoding result and the second encoding result based on an attention mechanism.

[0198] Among them, the scene flow prediction network can also be called a multi-scale bidirectional scene flow network (e.g. Figure 9The multi-scale bidirectional scene flow prediction network shown in Figure 2i10 can be described as a "two-input-two-output" network. The specific structure can be found in [the figure]. Figure 10 This is an illustration. For input... (first feature representation) and (Second feature representation) First, features can be extracted using independent encoder networks (e.g., a first encoder module and a second encoder module, where the first encoder module processes the first feature representation and the second encoder module processes the second feature representation). Then, a fusion module is used to fuse the inputs of blurred image features and event information features (e.g., fusion is achieved based on an attention module, which generates attention features used to fuse blurred image features and event data features). The fused features can then be used by independent decoder networks (e.g., a first decoder module and a second decoder module, where the first decoder module generates the first scene flow and the second decoder module generates the second scene flow) to generate corresponding scene flows. (First Scene Flow) and (Second scene flow)

[0199] In this embodiment of the application, the first scene stream can represent the alignment relationship between blurred image features and event information features, and the second scene stream can represent the alignment relationship between event feature information and blurred image.

[0200] In this case, the feature representation of the blurred image and the feature representation of the event data are not information of the same modality. If the feature representation of the blurred image and the feature representation of the event data are directly fused, the fusion result will be inaccurate. In this embodiment, the feature representation of the blurred image and the feature representation of the event data are first encoded by two different encoding modules to convert them into data similar to the same modality, and the encoding results are fused to obtain an accurate fusion result.

[0201] In one possible implementation, the first scene stream and the first feature representation have the same size (e.g., the width and height of the feature representation), each pixel feature in the first scene stream indicates motion information from the pixel feature at the corresponding pixel position in the first feature representation to the pixel feature at the corresponding pixel position in the second feature representation, the second scene stream and the second feature representation have the same size, and each pixel feature in the second scene stream indicates motion information from the pixel feature at the corresponding pixel position in the second feature representation to the pixel feature at the corresponding pixel position in the first feature representation.

[0202] Here, pixel features can refer to a point in space (x, y), which may contain multiple channels.

[0203] The motion information can be represented as a two-dimensional instantaneous velocity field, where the two-dimensional velocity vector is the projection of the three-dimensional velocity vector of visible points in the scene onto the imaging surface.

[0204] It should be noted that the scene flow in this embodiment is similar to optical flow, and the information of each pixel position is a directional vector.

[0205] It should be noted that this patent does not impose restrictions on the specific construction of the encoder, attention, and decoder. Optionally, the encoder module (such as the first and second encoder modules described above) can be a series of convolutions and downsampling; the fusion module can be a spatial attention structure or a channel attention structure; and the decoder module (such as the first and second decoder modules described above) can be a series of upsampling and convolutions.

[0206] Scene flow prediction can achieve pixel-level alignment between blurred image features and event data features.

[0207] 803. Based on the first scene flow, perform an affine transformation (warp) on the first feature representation to obtain the third feature representation.

[0208] 804. Based on the second scene flow, perform an affine transformation on the second feature representation to obtain a fourth feature representation; the third feature representation and the fourth feature representation are used to deblur the blurred image.

[0209] like Figure 9 The diagram shown is an internal implementation diagram of multi-scale bidirectional scene flow alignment, which specifically includes multi-scale bidirectional scene flow prediction 2i10, blurred image feature warp operation 2i11, and event information feature warp operation 2i12.

[0210] After obtaining two scene streams (the first scene stream and the second scene stream), a cross-warp operation can be performed with the corresponding features to obtain the warped features (the third feature representation and the fourth feature representation). For example, it can be as shown in Equation 1, where warp(*) is a traditional pixel-to-pixel spatial warp operation, which is a non-learnable operator.

[0211]

[0212]

[0213] In this embodiment, pixel-level alignment between blurred image features and event features can be achieved through multi-scale bidirectional scene flow alignment. This network structure, combined with attention structure and warp operation, can extract fine information at different granular levels from event information, which is beneficial for extracting clear texture structures and facilitating the deblurring of blurred images.

[0214] For images captured by RGB cameras, since they are obtained by fusing multiple frames captured within the same exposure time, the fused image may lose some information compared to event data (event data captured by event cameras includes multiple frames of event data captured within the same exposure time). For example, when capturing an image of a scene, if an object in the scene is occluded within the exposure time, this object will be invalid in the blurred RGB image, while its information will be valid in the event data. Similarly, since event data is only identified when the brightness change at a pixel location exceeds a threshold, some image data may be invalid. Therefore, directly using information from invalid (or occluded) areas in the image or time data will result in artifacts caused by degraded image quality.

[0215] In one possible implementation, for occluded regions in a blurred image, a second occluded region can be identified based on the fourth feature representation and the first feature representation (e.g., a lightweight network (such as consecutive convolutions and residuals) can be used to determine the occluded region), wherein the image data of the second occluded region in the blurred image is valid in the second occluded region of the event data; the feature representations in the first feature representation excluding the second occluded region and the feature representation of the second occluded region in the fourth feature representation are then fused to obtain a second fused feature representation. In other words, some features of the blurred image are occluded, and information from the event data can be used to replace these occluded features, thereby obtaining a more accurate feature representation.

[0216] In one possible implementation, the second occlusion region can be represented by a second mask, which has the same size as the fourth feature representation. Each pixel in the second mask is used to indicate whether the pixel feature at the corresponding position in the first feature representation is valid in the blurred image. For example, 0 and 1 can be used in the second mask to indicate whether the pixel feature at the corresponding position is valid in the event data, such as 0 indicating invalid and 1 indicating valid.

[0217] In one possible implementation, for occluded regions in the event data, a first occluded region can be determined based on the third feature representation and the second feature representation (e.g., a lightweight network (such as consecutive convolutions and residuals) can be used to determine the occluded region), wherein the image data of the first occluded region in the event data is valid in the first occluded region of the blurred image; the feature representations in the second feature representation excluding the first occluded region and the feature representation of the first occluded region in the third feature representation are then fused in a second way to obtain a first fused feature representation. In other words, some features of the event information are occluded, and information from the blurred image can be used to replace these occluded features, thereby obtaining a more accurate feature representation.

[0218] In one possible implementation, the first occlusion region is represented by a first mask, which has the same size as the third feature representation. Each pixel in the first mask is used to indicate whether the pixel feature at the corresponding position in the third feature representation is valid in the event data. For example, 0 and 1 can be used in the first mask to indicate whether the pixel feature at the corresponding position is valid in the event data, such as 0 indicating invalid and 1 indicating valid.

[0219] In one possible implementation, the second fusion is an addition operation of corresponding pixel positions.

[0220] By using the above method and setting explicit occlusion perception feature fusion, it is possible to process the occluded areas in a blurred image, thereby reducing the artifact problem caused by the occluded areas.

[0221] In one possible implementation, the feature representations of the blurred image and the event data can be processed by N concatenated nested feature blocks to obtain a processing result for deblurring. Each nested feature block is used to execute the image processing method described above. The first nested feature block is used to obtain the feature representations extracted from the blurred image and the event data by the feature extraction network, and the nth nested feature block is used to obtain the feature representation output by the (n-1)th nested feature block, where n is less than N.

[0222] In one possible implementation, the feature representation output by the Nth nested feature block is fused with the feature representation extracted from the blurred image by the feature extraction network to obtain residual information, which is then fused with the blurred image to achieve deblurring of the blurred image.

[0223] In one possible implementation, refer to Figure 9For each scene flow-guided dual-feature nested block 2i0, it contains two symmetrical occlusion-aware feature fusions (event information feature occlusion-aware feature fusion 2i2, and blurred image feature occlusion-aware feature fusion 2i3). Here, we will take blurred feature occlusion-aware feature fusion 2i3 as an example for detailed explanation. Its internal structure is as follows: Figure 11 It includes occlusion area mask generation 2i30, occlusion area feature generation 2i31, and feature fusion 2i32.

[0224] For example, occlusion mask generation 2i30: receives features generated after a warp operation on blurred image features. (referred to as warp features), event information features (Referring to self-features) are used as input, and a lightweight network (such as consecutive convolutions and residuals) is used to adaptively generate an occlusion region mask M of the same resolution. B (Optionally, the mask can use one-hot encoding, and its value can only be 0 or 1), representing the correlation between the blurred image features and event information features only obtained by warping. When mask = 0, it represents the features of that region after warping (such as features generated after the blurred image feature warping operation). In the case of occlusion, the original self-features (such as event information features) are tended to be used. In any case, when mask=1, it indicates that the features of the region are occluded in the original self-features, and warp features are preferred.

[0225] Among them, the occlusion region feature generation 2i31: After obtaining the occlusion mask, it can be multiplied with the warp feature to obtain the occlusion-processed feature (generated by module 2i3). The dual module 2i2 generates as ), which represents the visibility characteristics after the occlusion effect is processed.

[0226] Among them, feature fusion 2i32: features received after occlusion processing and original self-characteristics As input, the final fused features are generated through operations such as concatenation and convolution. The event information features are incorporated with blurred image features after occlusion processing.

[0227] In one possible implementation, the feature representation output by the Nth nested feature block is fused with the feature representation extracted from the blurred image by the feature extraction network to obtain residual information, which is then fused with the blurred image to achieve deblurring of the blurred image.

[0228] Reference Figure 12 Blurred image features can be received via scene-flow-guided double-feature nesting (200). B and event characteristics F E As input, after N rounds of scene-flow-guided double-feature nested block processing, nested blurred image features are generated. and event characteristics The blurred image features are obtained by global feature fusion after nesting 300. Event characteristics and the original fuzzy feature F B As input, the fused hybrid features F are generated through operations such as convolution (or addition, or concatenation). mix The mixed feature F is received through a summation operation 400. mix Using the original blurred image B as input, an addition operation is performed to generate the final clear result O after deblurring.

[0229] More specifically, refer to Figure 12 During the training phase, training can be performed on a given paired dataset (input = [B, E], output = O) using relevant loss functions (in this embodiment, loss functions such as MSE loss and perception loss can be used) to obtain trainable parameters such as blurred image feature extraction 100, event information feature extraction 101, scene flow-guided double feature nesting 200, and global feature fusion 300.

[0230] During the inference phase, a given input image input = [B, E] can be received. After processes such as blurred image feature extraction 100, event information feature extraction 101, scene flow-guided double feature nesting 200, global feature fusion 300, and summation operation 400, the final enhanced result O is generated.

[0231] The specific process is as follows:

[0232] Blurry image feature extraction 100: For a given input blurred image B (generally a grayscale image with dimensions H*W, where H represents the image height and W represents the image width), blurred image features F are extracted through multiple convolutional layers. B .

[0233] Event information feature extraction 101: For a given input event information E, event features F are extracted through multiple convolutional layers. E It should be noted that the spatial resolution of the event information at a certain moment is the same as that of the blurred image, which is H*W; however, the input here is all the event information within the exposure time of the blurred image, containing N channels, so the event information input is H*W*N, where N represents the number of event information.

[0234] Scene-flow guided double-feature nesting 200, receiving blurred image features F B and event characteristics F E As input, after N rounds of scene-flow-guided double-feature nested block processing, nested blurred image features are generated. and event characteristics The processing of the i-th scene flow-guided dual feature embedding can be expressed as follows: Formula 2:

[0235]

[0236] For the i-th scene flow-guided double-feature nested processing, the input is the blurred image feature information after the previous processing. Event Information Characteristics After processing including multi-scale bidirectional scene flow alignment 2i1, event information feature occlusion perception feature fusion 2i2, and blurred image information occlusion perception feature fusion 2i3, blurred image feature information is generated. Event Information Characteristics

[0237] The multi-scale scene flow alignment 2i1 includes multi-scale bidirectional scene flow prediction 2i10, blurred image feature warp operation 2i11, and event information feature warp operation 2i12. For detailed operations, please refer to the multi-scale dual scene flow alignment 2i1 described above.

[0238] In addition, in the event information feature occlusion perception feature fusion 2i2, blurred image features are received. Event information characteristics after warp First, the mask is processed by a lightweight convolutional occlusion mask generation module 2i20 to generate a one-hot encoded mask M with the same resolution as the input features. E (A code of 0 indicates that the region's features are occluded in the event information, favoring the use of blurred image features; conversely, a code of 1 indicates that the region's features are occluded in the blurred image, favoring the use of event information features); then warp the event information features. With occlusion mask M E Performing a dot product operation yields the occlusion-corrected features. Final blurred image features Features after occlusion processing Channel fusion and convolution operations are performed to generate blurred image features after fusion.

[0239] Similarly, in the fusion of occlusion perception features in blurred image information 2i3, the received event information features Blurred image features after warp After processing by modules such as occlusion area mask generation 2i30, occlusion area feature generation 2i31, and feature fusion 2i32, the fused event information features are generated.

[0240] By repeatedly performing the above process and undergoing N scene flow-guided double-feature nesting processes, the blurred image features are finally generated. and event information characteristics

[0241] Global feature fusion 300, receiving nested blurred image features Event characteristics and the original fuzzy feature F B As input, the fused hybrid features F are generated through operations such as convolution (or addition, or concatenation). mix .

[0242] Summation operation 400, receiving mixed feature F mix Using the original blurred image B as input, an addition operation is performed to generate the final clear result O after deblurring.

[0243] Tested on open-source datasets, this patented method demonstrates better PSNR / SSIM (higher is better) compared to existing methods. Specifically, the single-frame PSNR is improved by 2.9 dB compared to the existing state-of-the-art PSNR, and the multi-frame deblurring is improved by 2.7 dB, as detailed in Table 1.

[0244] Table 1

[0245]

[0246] In addition, refer to Figure 13 Compared with several benchmark results, the method of this patent has clearer and sharper results, which are closer to GT, and there are no artifacts.

[0247] Furthermore, ablation experiments show that the dual-feature nesting, multi-scale bidirectional scene flow prediction, and occlusion-aware feature fusion described in this invention all have positive effects. Among them, the multi-scale bidirectional scene flow prediction has the most significant effect, improving PSNR by more than 2 dB.

[0248] Table 2 presents the ablation experiment results. In Table 2, D indicates the absence of the dual-feature nesting in the multi-scale bidirectional scene flow prediction described in this invention; MSE represents the multi-scale scene flow prediction described in this invention; and OAFF represents occlusion-aware feature fusion.

[0249] Table 2

[0250]

[0251] Furthermore, referring to Figure 14 Visualize the scene flow and occlusion-aware masks. Figure 14 (a) in the image is a blurred image, and you can see that... Figure 14 In (b) of the text, some scenes with significant changes are in the flow. Figure 14 In (c), the mask is used for good annotation, representing the occluded areas. Through additional processing of the occluded areas, the final generated result... Figure 14 (d) in the formula can handle occlusion well and produce clear results.

[0252] This application provides an image processing method, comprising: acquiring a first feature representation of a blurred image and a second feature representation of event data acquired by an event camera; the first feature representation and the second feature representation have the same size; based on the first feature representation of the blurred image and the second feature representation of the event data, obtaining a first scene flow corresponding to the blurred image and a second scene flow corresponding to the event data through a scene flow prediction network, wherein the first scene flow and the first feature representation have the same size, each pixel feature in the first scene flow indicates motion information from a pixel feature at a corresponding pixel position in the first feature representation to a pixel feature at a corresponding pixel position in the second feature representation, the second scene flow and the second feature representation have the same size, each pixel feature in the second scene flow indicates motion information from a pixel feature at a corresponding pixel position in the second feature representation to a pixel feature at a corresponding pixel position in the first feature representation; performing an affine transformation (warp) on the first feature representation based on the first scene flow to obtain a third feature representation; performing an affine transformation on the second feature representation based on the second scene flow to obtain a fourth feature representation; the third feature representation and the fourth feature representation are used to deblur the blurred image. By aligning multi-scale bidirectional scene flows, fine-grained alignment of blurred image features and event information features can be achieved, resulting in accurate scene flow information. This solves the problem of inadequate consideration of pixel-level alignment in existing event-based deblurring methods.

[0253] Furthermore, embodiments of this application provide an image processing system, which may include a user device and a data processing device. The user device includes smart terminals such as mobile phones, personal computers, or information processing centers. The user device is the initiator of image processing, acting as the initiator of image enhancement requests, typically initiated by the user through the user device.

[0254] The aforementioned data processing equipment can be devices or servers with data processing capabilities, such as cloud servers, network servers, application servers, and management servers. The data processing equipment receives image enhancement requests from smart terminals through an interactive interface, and then performs image processing methods such as machine learning, deep learning, search, reasoning, and decision-making through a storage device and a data processing processor. The storage device can be a general term, including local storage and a database storing historical data. The database can be located on the data processing equipment or on other network servers.

[0255] The user equipment can receive user instructions. For example, the user equipment can acquire an image input / selected by the user and then send a request to the data processing device, causing the data processing device to perform image enhancement processing applications (such as image super-resolution reconstruction, image denoising, image dehazing, image deblurring, and image contrast enhancement) on the image acquired by the user equipment, thereby obtaining the corresponding processing result for the image. For instance, the user equipment can acquire an image input by the user and then send an image denoising request to the data processing device, causing the data processing device to perform image denoising on the image, thereby obtaining the denoised image.

[0256] The data processing device can execute the image processing method of the embodiments of this application.

[0257] Optionally, the user equipment can directly function as a data processing device, capable of directly acquiring input from the user and processing it directly through its own hardware. The user equipment can receive user instructions; for example, it can acquire an image selected by the user and then perform image processing applications (such as image super-resolution reconstruction, image denoising, image dehazing, image deblurring, and image contrast enhancement) on that image to obtain the corresponding processing result. In this case, the user equipment itself can execute the image processing method of this application embodiment.

[0258] The following describes an image processing apparatus provided in the embodiments of this application from the perspective of the apparatus itself, referring to... Figure 15 , Figure 15 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application, such as... Figure 15As shown, an image processing apparatus 1500 provided in this application embodiment includes:

[0259] The acquisition module 1501 is used to acquire a first feature representation of a blurred image and a second feature representation of event data acquired by an event camera; the first feature representation and the second feature representation have the same size.

[0260] The specific description of the acquisition module 1501 can be found in the description of step 801 in the above embodiments, and will not be repeated here.

[0261] The scene flow prediction module 1502 is used to obtain a first scene flow corresponding to the blurred image and a second scene flow corresponding to the event data through a scene flow prediction network based on the first feature representation of the blurred image and the second feature representation of the event data. The first scene flow and the first feature representation have the same size. Each pixel feature in the first scene flow indicates motion information from the pixel feature at the corresponding pixel position in the first feature representation to the pixel feature at the corresponding pixel position in the second feature representation. The second scene flow and the second feature representation have the same size. Each pixel feature in the second scene flow indicates motion information from the pixel feature at the corresponding pixel position in the second feature representation to the pixel feature at the corresponding pixel position in the first feature representation.

[0262] The specific description of the scene flow prediction module 1502 can be found in the description of step 802 in the above embodiment, and will not be repeated here.

[0263] The affine transformation module 1503 is used to perform an affine transformation on the first feature representation based on the first scene flow to obtain a third feature representation;

[0264] Based on the second scene flow, an affine transformation is performed on the second feature representation to obtain a fourth feature representation; the third feature representation and the fourth feature representation are used to deblur the blurred image.

[0265] The specific description of the affine transformation module 1503 can be found in the descriptions of steps 803 and 804 in the above embodiments, and will not be repeated here.

[0266] In one possible implementation, the blurred image and the event data are acquired within the same time period for the same scene.

[0267] In one possible implementation, the scene flow prediction network includes a first encoding module, a second encoding module, a fusion module, a first decoding module, and a second decoding module;

[0268] The scene flow prediction module is specifically used for:

[0269] Based on the first feature representation, a first encoding result is obtained through the first encoding module;

[0270] Based on the second feature representation, the second encoding result is obtained through the second encoding module;

[0271] Based on the first encoding result and the second encoding result, a fusion result is obtained through the fusion module;

[0272] Based on the fusion result, the first scene stream corresponding to the blurred image and the second scene stream corresponding to the event data are obtained through the first decoding module and the second decoding module, respectively.

[0273] In this case, the feature representation of the blurred image and the feature representation of the event data are not information of the same modality. If the feature representation of the blurred image and the feature representation of the event data are directly fused, the fusion result will be inaccurate. In this embodiment, the feature representation of the blurred image and the feature representation of the event data are first encoded by two different encoding modules to convert them into data similar to the same modality, and the encoding results are fused to obtain an accurate fusion result.

[0274] In one possible implementation, the device further includes:

[0275] An occlusion region identification module is used to identify a second occlusion region based on the fourth feature representation and the first feature representation, wherein image data of the second occlusion region in the blurred image is valid in the second occlusion region of the event data;

[0276] The feature representations in the first feature representation excluding the second occluded region and the feature representations in the fourth feature representation of the second occluded region are fused together to obtain a second fused feature representation.

[0277] In one possible implementation, the second occlusion region is represented by a second mask, which has the same size as the fourth feature representation, and each pixel in the second mask is used to indicate whether the pixel feature at the corresponding position in the first feature representation is valid in the blurred image.

[0278] In one possible implementation, the device further includes:

[0279] An occlusion region identification module is used to determine a first occlusion region based on the third feature representation and the second feature representation, wherein the image data of the first occlusion region in the event data is valid in the first occlusion region of the blurred image;

[0280] The feature representations in the second feature representation excluding the first occluded region and the feature representations in the third feature representation of the first occluded region are fused together to obtain the first fused feature representation.

[0281] In one possible implementation, the first occlusion region is represented by a first mask, the first mask and the third feature representation having the same size, and each pixel in the first mask is used to indicate whether the pixel feature at the corresponding position in the third feature representation is valid in the event data.

[0282] By using the above method and setting explicit occlusion perception feature fusion, it is possible to process the occluded areas in a blurred image, thereby reducing the artifact problem caused by the occluded areas.

[0283] In one possible implementation, the second fusion is an addition operation of corresponding pixel positions.

[0284] In one possible implementation, the apparatus further includes: a feature nesting module, configured to process the feature representation of the blurred image and the feature representation of the event data through N serially connected feature nesting blocks to obtain a processing result for deblurring; wherein each of the feature nesting blocks is configured to execute the image processing method described above, the first feature nesting block is configured to obtain the feature representation extracted from the blurred image and the event data by the feature extraction network, the nth feature nesting block is configured to obtain the feature representation output by the (n-1)th feature nesting block, where n is less than N.

[0285] In one possible implementation, the feature representation output by the Nth nested feature block is fused with the feature representation extracted from the blurred image by the feature extraction network to obtain residual information, which is then fused with the blurred image to achieve deblurring of the blurred image.

[0286] The following describes an execution device provided in an embodiment of this application. Please refer to [link / reference]. Figure 16 , Figure 16 This is a schematic diagram of an execution device provided in an embodiment of this application. The execution device 1600 can specifically be a mobile phone, tablet, laptop, smart wearable device, server, etc., and is not limited thereto. The execution device 1600 implements... Figure 8The image processing method in the corresponding embodiment has functionality. Specifically, the execution device 1600 includes: a receiver 1601, a transmitter 1602, a processor 1603, and a memory 1604 (wherein the number of processors 1603 in the execution device 1600 can be one or more), wherein the processor 1603 may include an application processor 16031 and a communication processor 16032. In some embodiments of this application, the receiver 1601, transmitter 1602, processor 1603, and memory 1604 can be connected via a bus or other means.

[0287] Memory 1604 may include read-only memory and random access memory, and provides instructions and data to processor 1603. A portion of memory 1604 may also include non-volatile random access memory (NVRAM). Memory 1604 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0288] Processor 1603 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together through a bus system, which may include not only the data bus, but also power buses, control buses, and status signal buses. However, for clarity, all buses are referred to as the bus system in the diagram.

[0289] The methods disclosed in the embodiments of this application can be applied to or implemented by processor 1603. Processor 1603 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by integrated logic circuits in the hardware of processor 1603 or by instructions in software form. Processor 1603 can be a general-purpose processor, digital signal processor (DSP), microprocessor or microcontroller, as well as a vision processing unit (VPU), tensor processing unit (TPU), or other processors suitable for AI computation. It may further include application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Processor 1603 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 1604. Processor 1603 reads information from memory 1604 and, in conjunction with its hardware, completes steps 801 to 804 in the above embodiments.

[0290] Receiver 1601 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the execution device. Transmitter 1602 can be used to output digital or character information through the first interface; transmitter 1602 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; transmitter 1602 may also include a display device such as a display screen.

[0291] This application also provides a training device; please refer to [link / reference]. Figure 17 , Figure 17This is a schematic diagram of a training device provided in an embodiment of this application. Specifically, the training device 1700 is implemented by one or more servers. The training device 1700 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1717 (e.g., one or more processors) and memory 1732, and one or more storage media 1730 (e.g., one or more mass storage devices) for storing application programs 1742 or data 1744. The memory 1732 and storage media 1730 can be temporary or persistent storage. The program stored in the storage media 1730 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the training device. Furthermore, the CPU 1717 may be configured to communicate with the storage media 1730 and execute the series of instruction operations in the storage media 1730 on the training device 1700.

[0292] The training device 1700 may also include one or more power supplies 1726, one or more wired or wireless network interfaces 1750, one or more input / output interfaces 1758; or, one or more operating systems 1741, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0293] Specifically, the training device can perform steps 801 to 804 in the above embodiments.

[0294] This application also provides a computer program product that, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0295] This application also provides a computer-readable storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0296] The execution device, training device, or terminal device provided in this application embodiment can specifically be a chip. The chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip within the execution device to execute the data processing method described in the above embodiments, or to cause the chip within the training device to execute the data processing method described in the above embodiments. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0297] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.

[0298] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0299] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0300] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0301] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. An image processing method, characterized in that, The method includes: A first feature representation of the blurred image and a second feature representation of the event data acquired by the event camera are obtained; the first feature representation and the second feature representation have the same size; Based on the first feature representation of the blurred image and the second feature representation of the event data, a scene flow prediction network is used to obtain a first scene flow corresponding to the blurred image and a second scene flow corresponding to the event data. The first scene flow and the first feature representation have the same size. Each pixel feature in the first scene flow indicates motion information from the pixel feature at the corresponding pixel position in the first feature representation to the pixel feature at the corresponding pixel position in the second feature representation. The second scene flow and the second feature representation have the same size. Each pixel feature in the second scene flow indicates motion information from the pixel feature at the corresponding pixel position in the second feature representation to the pixel feature at the corresponding pixel position in the first feature representation. Based on the first scene flow, an affine transformation (warp) is performed on the first feature representation to obtain the third feature representation; Based on the second scene flow, an affine transformation is performed on the second feature representation to obtain a fourth feature representation; the third feature representation and the fourth feature representation are used to deblur the blurred image.

2. The method according to claim 1, characterized in that, The blurred image and the event data were collected within the same time period for the same scene.

3. The method according to claim 1 or 2, characterized in that, The scene flow prediction network includes a first encoding module, a second encoding module, a fusion module, a first decoding module, and a second decoding module; The step of obtaining a first scene flow corresponding to the blurred image and a second scene flow corresponding to the event data through a scene flow prediction network based on a first feature representation of the blurred image and a second feature representation of the event data includes: Based on the first feature representation, a first encoding result is obtained through the first encoding module; Based on the second feature representation, the second encoding result is obtained through the second encoding module; Based on the first encoding result and the second encoding result, a fusion result is obtained through the fusion module; Based on the fusion result, the first scene stream corresponding to the blurred image and the second scene stream corresponding to the event data are obtained through the first decoding module and the second decoding module, respectively.

4. The method according to claim 1 or 2, characterized in that, The method further includes: A second occlusion region is identified based on the fourth feature representation and the first feature representation, wherein image data of the second occlusion region in the blurred image is valid in the second occlusion region of the event data; The feature representations in the first feature representation excluding the second occluded region and the feature representations in the fourth feature representation of the second occluded region are fused together to obtain a second fused feature representation.

5. The method according to claim 4, characterized in that, The second occlusion area is represented by a second mask, which has the same size as the fourth feature representation. Each pixel in the second mask is used to indicate whether the pixel feature at the corresponding position in the first feature representation is valid in the blurred image.

6. The method according to claim 1 or 2, characterized in that, The method further includes: A first occlusion region is determined based on the third feature representation and the second feature representation, wherein image data of the first occlusion region in the event data is valid in the first occlusion region of the blurred image; The feature representations in the second feature representation excluding the first occluded region and the feature representations in the third feature representation of the first occluded region are fused together to obtain the first fused feature representation.

7. The method according to claim 6, characterized in that, The first occlusion area is represented by a first mask, which has the same size as the third feature representation. Each pixel in the first mask is used to indicate whether the pixel feature at the corresponding position in the third feature representation is valid in the event data.

8. The method according to claim 4, characterized in that, The second fusion is an addition operation of corresponding pixel positions.

9. The method according to claim 1 or 2, characterized in that, The method further includes: The feature representations of the blurred image and the event data are processed through N concatenated feature nesting blocks to obtain the processing result for deblurring; wherein each feature nesting block is used to execute the image processing method as claimed in claim 1, the first feature nesting block is used to obtain the feature representations extracted from the blurred image and the event data by the feature extraction network, the nth feature nesting block is used to obtain the feature representation output by the (n-1)th feature nesting block, where n is less than N and greater than 1.

10. The method according to claim 9, characterized in that, The feature representation output by the Nth nested feature block is used to fuse with the feature representation extracted from the blurred image by the feature extraction network to obtain residual information. The residual information is used to fuse with the blurred image to achieve deblurring of the blurred image.

11. An image processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire a first feature representation of the blurred image and a second feature representation of the event data acquired by the event camera; the first feature representation and the second feature representation have the same size; A scene flow prediction module is used to obtain a first scene flow corresponding to the blurred image and a second scene flow corresponding to the event data through a scene flow prediction network based on a first feature representation of the blurred image and a second feature representation of the event data. The first scene flow and the first feature representation have the same size. Each pixel feature in the first scene flow indicates motion information from the pixel feature at the corresponding pixel position in the first feature representation to the pixel feature at the corresponding pixel position in the second feature representation. The second scene flow and the second feature representation have the same size. Each pixel feature in the second scene flow indicates motion information from the pixel feature at the corresponding pixel position in the second feature representation to the pixel feature at the corresponding pixel position in the first feature representation. The affine transformation module is used to perform an affine transformation on the first feature representation based on the first scene flow to obtain the third feature representation; Based on the second scene flow, an affine transformation is performed on the second feature representation to obtain a fourth feature representation; the third feature representation and the fourth feature representation are used to deblur the blurred image.

12. The apparatus according to claim 11, characterized in that, The blurred image and the event data were collected within the same time period for the same scene.

13. The apparatus according to claim 11 or 12, characterized in that, The scene flow prediction network includes a first encoding module, a second encoding module, a fusion module, a first decoding module, and a second decoding module; The scene flow prediction module is specifically used for: Based on the first feature representation, a first encoding result is obtained through the first encoding module; Based on the second feature representation, the second encoding result is obtained through the second encoding module; Based on the first encoding result and the second encoding result, a fusion result is obtained through the fusion module; Based on the fusion result, the first scene stream corresponding to the blurred image and the second scene stream corresponding to the event data are obtained through the first decoding module and the second decoding module, respectively.

14. The apparatus according to claim 11 or 12, characterized in that, The device further includes: An occlusion region identification module is used to identify a second occlusion region based on the fourth feature representation and the first feature representation, wherein image data of the second occlusion region in the blurred image is valid in the second occlusion region of the event data; The feature representations in the first feature representation excluding the second occluded region and the feature representations in the fourth feature representation of the second occluded region are fused together to obtain a second fused feature representation.

15. The apparatus according to claim 14, characterized in that, The second occlusion area is represented by a second mask, which has the same size as the fourth feature representation. Each pixel in the second mask is used to indicate whether the pixel feature at the corresponding position in the first feature representation is valid in the blurred image.

16. The apparatus according to claim 11 or 12, characterized in that, The device further includes: An occlusion region identification module is used to determine a first occlusion region based on the third feature representation and the second feature representation, wherein the image data of the first occlusion region in the event data is valid in the first occlusion region of the blurred image; The feature representations in the second feature representation excluding the first occluded region and the feature representations in the third feature representation of the first occluded region are fused together to obtain the first fused feature representation.

17. The apparatus according to claim 16, characterized in that, The first occlusion area is represented by a first mask, which has the same size as the third feature representation. Each pixel in the first mask is used to indicate whether the pixel feature at the corresponding position in the third feature representation is valid in the event data.

18. The apparatus according to claim 14, characterized in that, The second fusion is an addition operation of corresponding pixel positions.

19. The apparatus according to claim 11 or 12, characterized in that, The apparatus further includes: a feature nesting module, used to process the feature representation of the blurred image and the feature representation of the event data through N serially connected feature nesting blocks to obtain a processing result for deblurring; wherein each of the feature nesting blocks is used to execute the image processing method as claimed in claim 1, the first feature nesting block is used to obtain the feature representation extracted from the blurred image and the event data by the feature extraction network, the nth feature nesting block is used to obtain the feature representation output by the (n-1)th feature nesting block, and n is less than N.

20. The apparatus according to claim 19, characterized in that, The feature representation output by the Nth nested feature block is used to fuse with the feature representation extracted from the blurred image by the feature extraction network to obtain residual information. The residual information is used to fuse with the blurred image to achieve deblurring of the blurred image.

21. A computing device, characterized in that, The computing device includes a memory and a processor; the memory stores code, and the processor is configured to retrieve the code and execute the method as described in any one of claims 1 to 10.

22. A computer storage medium, characterized in that, The computer storage medium stores one or more instructions that, when executed by one or more computers, cause the one or more computers to perform the method of any one of claims 1 to 10.

23. A computer program product, comprising code, characterized in that, When the code is executed, it is used to implement the method as described in any one of claims 1 to 10.

Images