Motion deblurring method and device based on hybrid visual sensor
By establishing a hybrid vision sensor defuzzing data set and learning spatiotemporal features, the performance degradation and scale mismatch of image motion defuzzing in low-light and high-speed scenarios in the prior art is solved, and efficient and accurate image defuzzing is achieved. .
Patent Information
- Application Number
- CN202410967338.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-12-06
AI Technical Summary
The prior art image motion defuzzing performance in low-light and high-speed motion scenarios is degraded, and the space-time aligned image and event stream mixed data sets are lacking in real scenes. The single image defuzzing algorithm lacks motion information, and the image and event stream mixed sensors are removed. Fuzzy algorithms cannot handle situations where images and events have different scales.
By establishing a hybrid vision sensor that meets the preset resolution conditions, we learn the spatial implicit expression features of the target image and the temporal implicit representation features of the target event, and perform feature mixing processing to obtain the spatial and spatial domain features for the image motion defuzzing.
It effectively improves the efficiency and accuracy of image motion defuzzing, solves the problem of lack of motion information and mismatch between scales, and improves the applicability and accuracy of defuzzing in real scenes.
Smart Images

Figure CN119090770A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of underlying visual technology, and in particular to a motion deblurring method and device based on a hybrid visual sensor. Background Art
[0002] Image motion deblurring is a long-standing challenging problem with important applications in autonomous driving, mobile photography, video surveillance, etc. Image deblurring algorithms based on traditional CIS (CMOS Image Sensor) image frames have developed rapidly, but the performance of traditional image frame paradigms often degrades in low-light and high-speed motion scenes. The fundamental reason is that the camera sensor cannot effectively collect scene information in these extreme cases, which hinders the normal operation of the algorithm.
[0003] In recent years, computer vision tasks based on event cameras are developing rapidly. As a biologically inspired neuromorphic sensor, the event stream output by the event camera has extremely high temporal resolution and a large dynamic range. It still performs well in low-light and high-speed scenes, which helps to solve various computer vision tasks in extreme scenes. EVS (event vision sensor) has the advantages of high temporal resolution, high dynamic range, low redundancy and low power consumption. Due to the microsecond low latency of EVS, the necessary motion information can be captured from the event to help reduce motion blur in the CIS frame, providing a new solution for target motion deblurring in extreme scenes such as high-speed motion. The current motion deblurring methods mainly include the following three types of technologies.
[0004] Among them, (1) Deblurring datasets: At present, event-based deblurring datasets are mainly simulated and synthesized, for example, using the ESIM event simulator to synthesize event streams from image data. Due to the late development of event cameras, there are currently only a few datasets that contain real event streams, such as BlurDVS and REBlur, which use image and event sensors to form a binocular structure to obtain real scene images and event streams. However, current event-based deblurring datasets are usually collected from low-spatial-resolution cameras (e.g., DAVIS346, 346×260) or binocular camera systems using beam splitters, and it is necessary to manually align the CIS and EVS in space and synchronize them in time.
[0005] (2) Single image deblurring algorithm: Traditional motion deblurring methods based on single image frames aim to reconstruct clear and sharp images from blurred images. Some early methods adopted the classic image deconvolution algorithm and introduced additional priors (such as total variation) to reduce the pathology of the reconstructed image. Recently, deep learning-based methods have achieved better performance by directly learning the mapping from blurred images to clear images. For example, by designing a multi-level architecture to gradually learn spatial details and high-level contextual information, using a simple non-linear non-activation network, and improving reconstruction performance by removing non-linear activation functions. The latest Transformer-based model improves the effect of the deblurring algorithm by designing multi-head attention and feedforward networks to capture long-range pixel interactions.
[0006] (3) Deblurring algorithms for hybrid sensors of image and event streams: In recent years, event-based motion deblurring has attracted widespread attention due to the high temporal resolution of EVS. Deep learning-based motion deblurring methods significantly improve the deblurring performance guided by event streams. For example, an event-based double integral model is used to learn the mapping relationship between blurred images, events, and clear frames, thereby obtaining more accurate details than frame-based methods. By designing a self-supervised deblurring network, the optical flow of events is predicted to exploit blur consistency and photometric consistency. A cross-modal attention module is designed and applied to multiple levels of the network, which allows attention to relevant features of the event branch to improve the deblurring effect. Based on a bidirectional recurrent network, the information in the image and event is adaptively mixed according to their temporal proximity.
[0007] However, the deblurring dataset in related technologies still lacks a mixed dataset of images and event streams that are temporally aligned in real scenarios to serve as a training set and evaluation benchmark for the algorithm, which has limitations; the single image deblurring algorithm is only driven by a single frame of data and lacks motion information, resulting in limited recovery and reconstruction performance; the deblurring algorithm of the image and event stream hybrid sensor assumes that the image has the same spatial and temporal scale as the event. When the image and event have different scales, it cannot handle real scenes, reducing the applicability and accuracy of image motion deblurring, which urgently needs to be improved. Summary of the invention
[0008] The present application provides a motion deblurring method and device based on a hybrid vision sensor to solve the problems in the related art such as lack of motion information, resulting in limited restoration and reconstruction performance, and inability to process real scenes when images and events have different scales, reducing the applicability and accuracy of image motion deblurring, etc.
[0009] The first aspect of the present application provides a motion deblurring method based on a hybrid vision sensor, comprising the following steps: establishing a hybrid vision sensor deblurring data set that meets a first preset resolution condition; based on the hybrid vision sensor deblurring data set, learning the spatial implicit expression features of the target image and the temporal implicit representation features of the target event; performing feature mixing processing on the spatial implicit expression features and the temporal implicit representation features to obtain spatiotemporal domain features, and using the spatiotemporal domain features to perform motion deblurring processing on the target image to obtain a clear image that meets a second preset resolution condition.
[0010] Optionally, in one embodiment of the present application, establishing a hybrid visual sensor deblurring dataset that satisfies a first preset resolution condition includes: determining various indoor and outdoor target scenes with different spatial resolutions and exposure times; and establishing the hybrid visual sensor deblurring dataset that satisfies the first preset resolution condition in the target scenes.
[0011] Optionally, in one embodiment of the present application, the learning of the spatial implicit expression characteristics of the target image and the temporal implicit representation characteristics of the target event includes: learning the spatial correspondence of any spatial scale in the target image to obtain the spatial implicit expression characteristics of the target image; establishing the temporal correspondence of any temporal scale in the target event to obtain the temporal implicit representation characteristics of the target event.
[0012] Optionally, in one embodiment of the present application, the spatial correspondence of any spatial scale in the learning target image is obtained to obtain the spatial implicit expression features of the target image, including: collecting the target resolution features required for the target image, and adding the target resolution features to the image features to obtain the added image features; converting the added image features into a continuous feature representation to obtain the spatial implicit expression features.
[0013] Optionally, in one embodiment of the present application, the establishing of the time correspondence of any time scale in the target event to obtain the time implicit representation feature of the target event includes: obtaining a bidirectional event selection feature based on target time perception event selection; performing target time grouping displacement processing on the bidirectional time selection feature to obtain a time displacement feature; and obtaining the time implicit representation feature using the time displacement feature and the target mixed feature.
[0014] The second aspect of the present application provides a motion deblurring device based on a hybrid visual sensor, including: an establishment module, used to establish a hybrid visual sensor deblurring data set that meets a first preset resolution condition; a learning module, used to learn the spatial implicit expression features of the target image and the temporal implicit representation features of the target event based on the hybrid visual sensor deblurring data set; a processing module, used to perform feature mixing processing on the spatial implicit expression features and the temporal implicit representation features to obtain spatiotemporal domain features, and use the spatiotemporal domain features to perform motion deblurring processing on the target image to obtain a clear image that meets a second preset resolution condition.
[0015] Optionally, in one embodiment of the present application, the establishment module includes: a determination unit, used to determine various indoor and outdoor target scenes with different spatial resolutions and exposure times; an establishment unit, used to establish the hybrid visual sensor deblurring data set in the target scene that satisfies the first preset resolution condition.
[0016] Optionally, in one embodiment of the present application, the learning module includes: a learning unit, used to learn the spatial correspondence of any spatial scale in the target image to obtain the spatial implicit expression characteristics of the target image; and an establishing unit, used to establish the temporal correspondence of any temporal scale in the target event to obtain the temporal implicit representation characteristics of the target event.
[0017] Optionally, in one embodiment of the present application, the learning unit includes: a first determination subunit, used to collect target resolution features required for the target image, and add the target resolution features to the image features to obtain the added image features; a second determination subunit, used to convert the added image features into a continuous feature representation to obtain the spatial implicit expression features.
[0018] Optionally, in one embodiment of the present application, the establishment unit includes: a first acquisition subunit, used to obtain a bidirectional event selection feature based on target time-aware event selection; a second acquisition subunit, used to perform target time grouping displacement processing on the bidirectional time selection feature to obtain a time displacement feature; and a third acquisition subunit, used to obtain the time implicit representation feature using the time displacement feature and the target mixed feature.
[0019] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the motion deblurring method based on a hybrid vision sensor as described in the above embodiment.
[0020] The fourth aspect of the present application provides a computer-readable storage medium, which stores a computer program. When the program is executed by a processor, it implements the above-mentioned motion deblurring method based on a hybrid vision sensor.
[0021] The fifth aspect of the present application provides a computer program product, including a computer program, which, when executed, is used to implement the above-mentioned motion deblurring method based on a hybrid vision sensor.
[0022] The embodiment of the present application can learn the spatial implicit expression features of the target image and the temporal implicit representation features of the target event based on the hybrid visual sensor deblurring data set that meets the first resolution condition, perform feature mixing processing on the spatial implicit expression features and the temporal implicit representation features, obtain spatiotemporal features, and use the spatiotemporal features to perform motion deblurring processing on the target image to obtain a clear image that meets the second resolution condition, thereby restoring clear image content and effectively improving the efficiency and accuracy of image motion deblurring. Thus, the problem of limited restoration and reconstruction performance due to lack of motion information and the inability to process real scenes when images and events have different scales, which reduces the applicability and accuracy of image motion deblurring, is solved.
[0023] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0025] Figure 1 A flowchart of a motion deblurring method based on a hybrid vision sensor provided according to an embodiment of the present application;
[0026] Figure 2 A schematic diagram of a hybrid vision sensor outputting an EVS event stream and an RGB image frame according to a specific embodiment of the present application;
[0027] Figure 3 A schematic diagram of scale-aware image motion deblurring according to a specific embodiment of the present application;
[0028] Figure 4 A spatiotemporal super-resolution framework diagram of a joint event stream and image frame according to a specific embodiment of the present application;
[0029] Figure 5 A schematic diagram of a spatial implicit expression module according to a specific embodiment of the present application;
[0030] Figure 6A schematic diagram of a time implicit expression module according to a specific embodiment of the present application;
[0031] Figure 7 A schematic diagram of visualization of a GoPro (sports camera) simulation synthetic data set according to a specific embodiment of the present application;
[0032] Figure 8 A schematic diagram of visualization of a real H2D (High-resolution Hybrid Deblur) data set according to a specific embodiment of the present application;
[0033] Fig. 9 A schematic diagram of quantitative results at different time and space scales of a specific embodiment of the present application;
[0034] Fig.10 A schematic diagram of qualitative results on a GoPro synthetic dataset according to a specific embodiment of the present application;
[0035] Fig.11 A schematic diagram of qualitative results on an H2D synthetic data set according to a specific embodiment of the present application;
[0036] Fig.12 A schematic diagram of qualitative results at different movement speeds of a specific embodiment of the present application;
[0037] Fig.13 A schematic diagram of the structure of a motion deblurring device based on a hybrid vision sensor provided according to an embodiment of the present application;
[0038] Fig.14 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION
[0039] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.
[0040] The following describes the motion deblurring method and device based on a hybrid visual sensor of the embodiment of the present application with reference to the accompanying drawings. In view of the lack of motion information mentioned in the above background technology, resulting in limited recovery and reconstruction performance, and when the image and the event have different scales, it is impossible to process the real scene, reducing the applicability and accuracy of the image motion deblurring, etc., the present application provides a motion deblurring method based on a hybrid visual sensor, in which the spatial implicit expression features of the target image and the temporal implicit representation features of the target event can be learned based on the hybrid visual sensor deblurring data set that meets the first resolution condition, and the spatial implicit expression features and the temporal implicit representation features are subjected to feature mixing processing to obtain spatiotemporal domain features, and the spatiotemporal domain features are used to perform motion deblurring processing on the target image to obtain a clear image that meets the second resolution condition, thereby restoring the clear image content, and effectively improving the efficiency and accuracy of the image motion deblurring. Thus, the lack of motion information, resulting in limited recovery and reconstruction performance, and when the image and the event have different scales, it is impossible to process the real scene, reducing the applicability and accuracy of the image motion deblurring, etc. are solved.
[0041] Specifically, Figure 1 A flow chart of a motion deblurring method based on a hybrid vision sensor provided in an embodiment of the present application.
[0042] like Figure 1 As shown, the motion deblurring method based on the hybrid vision sensor includes the following steps:
[0043] In step S101, a hybrid vision sensor deblurring data set that meets a first preset resolution condition is established.
[0044] In the embodiment of the present application, the hybrid vision sensor that meets the first preset resolution condition is a sensor with a new hybrid EVS / CIS that meets the high-resolution condition. The specific resolution can be set by those skilled in the art and is not specifically limited here.
[0045] It can be understood that the embodiment of the present application can establish a hybrid visual sensor deblurring data set that meets the first preset resolution condition, that is, construct a high-quality deblurring data set, which effectively improves the feasibility of motion deblurring.
[0046] In the actual implementation process, since the event stream and image sequence of the current data set are captured by a spectroscopic dual sensor system, they need to be synchronized in time and aligned in space in advance; it is difficult for related technologies to use event data of different spatial and temporal scales in real scenes to process different degrees of motion blur. These defects make it difficult for current event-based deblurring methods to fully meet the application in actual scenes; Figure 2As shown in the figure, the Hybrid camera is a hybrid dual-mode vision sensor that combines dynamic vision circuits and traditional images. It can simultaneously output two visual information streams with complementary perceptual advantages, namely EVS event stream information with high temporal resolution and traditional RGB images with high spatial resolution.
[0047] Among them, Figure 3 As shown, according to the scale problem faced by DVS event stream and image frame fusion super-resolution in practice, the embodiment of the present application can define a new arbitrary scale spatiotemporal super-resolution task, so as to process events and image data of different time-space scales, and realize the recovery of high-resolution and high-definition image frames from any LR (low-resolution) event stream and HR (high-resolution) image frames with different degrees of motion blur. Therefore, the embodiment of the present application can use a sensor with a new hybrid EVS / CIS to obtain the H2D data set in the following steps, so as to ensure the diversity and applicability of the data set.
[0048] In one embodiment of the present application, a hybrid visual sensor deblurring dataset that satisfies a first preset resolution condition is established, including: determining various indoor and outdoor target scenes with different spatial resolutions and exposure times; and establishing a hybrid visual sensor deblurring dataset that satisfies the first preset resolution condition in the target scene.
[0049] For example, the embodiment of the present application first uses a sensor with a novel hybrid EVS / CIS to collect a high-resolution hybrid deblurring (H2D) dataset in various indoor and outdoor scenes with different spatial resolutions and exposure times, wherein H2D contains 2836 clear images with a resolution of 1920 × 1080, and naturally aligned and synchronized LR events from a hybrid CIS with embedded EVS, of which 1233 frames are used for training and 603 frames are used for testing. That is, the bionic hybrid camera makes H2D a competitive dataset with multiple features, including high spatial resolution, natural calibration of spatial and temporal domains of images and events, and real-world scenes, and rich diversity in scene categories, light changes, and motion speeds, thereby ensuring the diversity and applicability of the dataset.
[0050] In step S102, based on the hybrid visual sensor deblurring dataset, the spatial implicit expression features of the target image and the temporal implicit representation features of the target event are learned.
[0051] It can be understood that the embodiments of the present application can learn the spatial implicit expression features of the target image and the temporal implicit representation features of the target event in the following steps based on the hybrid visual sensor deblurring dataset, wherein the spatial implicit expression features focus on the structure and details of the image at different spatial scales, while the temporal implicit representation features focus on the dynamic changes in the event flow, such as motion direction and speed, thereby effectively improving the accuracy and efficiency of motion deblurring.
[0052] In some embodiments, the embodiments of the present application can utilize two physical characteristics of blurred scenes, namely, the spatial correlation between event occurrence and local blurred area, and the temporal correlation between exposure time and global motion blur amplitude to eliminate motion blur in a targeted manner; specifically, Figure 4 As shown in the figure, in order to eliminate local and global motion blur, SASNet (Scale-aware Spatiotemporal Network) can be used, and an implicit representation method can be used to aggregate the spatial and temporal domain corresponding features of images and events of any scale, wherein SASNet can be composed of three stages: feature extraction composed of RCB (Residual ConvolutionalBlock) and CAB (Channel Attention Block); spatial and temporal implicit representation composed of SIRM (Spatial Implicit Representation Module) and TIRM (Time Implicit Representation Module); reconstruction using convolutional layers. Among them, the two most important core modules are: spatial implicit representation module and temporal implicit representation module.
[0053] Among them, in one embodiment of the present application, learning the spatial implicit expression characteristics of the target image and the temporal implicit representation characteristics of the target event includes: learning the spatial correspondence of any spatial scale in the target image to obtain the spatial implicit expression characteristics of the target image; establishing the temporal correspondence of any temporal scale in the target event to obtain the temporal implicit representation characteristics of the target event.
[0054] As a possible implementation method, the embodiment of the present application can learn the spatial correspondence of any spatial scale in the target image in the following steps to obtain the spatial implicit expression characteristics of the target image, thereby capturing the intrinsic spatial structure of the image; the embodiment of the present application can also establish the temporal correspondence of any time scale in the target event in the following steps to obtain the temporal implicit representation characteristics of the target event, and then reflect the dynamic information in the time series, so as to more accurately remove blur and restore clear image content.
[0055] Optionally, in one embodiment of the present application, the spatial correspondence of any spatial scale in the target image is learned to obtain the spatial implicit expression features of the target image, including: collecting the target resolution features required for the target image, and adding the target resolution features to the image features to obtain the added image features; converting the added image features into continuous feature representations to obtain the spatial implicit expression features.
[0056] In some embodiments, due to the influence of the object's motion speed and depth, the intensity of blur is unevenly distributed in space. Considering the spatial correlation characteristics of the blurry scene, that is, the position of the emission event is highly correlated with the local blurry area, the embodiments of the present application can use the spatial implicit expression module to learn the spatial correspondence of any scale to focus on the highly blurred area, such as Figure 5 As shown, the embodiment of the present application can adopt an efficient spatial-aware coding sampling scheme to upsample the features to the required high-resolution features and add them to the image features, and then use an implicit decoder parameterized as an MLP (Multilayer Perceptron) to convert the added features into a continuous feature representation.
[0057] Next, the embodiment of the present application can perform spatial perception coding sampling, first using the encoder E(F E ; φ) Encode event features into event space features F ESR , then according to the input spatial scale R s , generate the query coordinate grid E through coordinate projection coord ; For each coordinate, from the event space feature F ESR Select the features of the eight points closest to the coordinates and perform two-dimensional feature sampling to obtain the sampled spatial features F SESR :
[0058] F SESR =f sample (E(F E ; φ), E coord )
[0059] Among them, f sample (·) represents the sampling function used to dynamically represent features at any spatial scale, E represents the encoder, and E coord Represents the query coordinate grid.
[0060] Secondly, the embodiment of the present application can perform implicit coding, that is, the sampled spatial feature F SESR and image features F I After being added, it is input to the implicit decoder D(.; η), that is:
[0061]
[0062] Among them, for simplicity and efficiency, the decoder with parameter η consists of two layers of convolution, and SIRM can model spatial correlation through implicit representations at different spatial scales to effectively capture the spatial structure information of the image.
[0063] Optionally, in one embodiment of the present application, a time correspondence relationship of any time scale in the target event is established to obtain a time implicit representation feature of the target event, including: obtaining a bidirectional event selection feature based on target time perception event selection; performing target time grouping displacement processing on the bidirectional time selection feature to obtain a time displacement feature; and obtaining a time implicit representation feature using the time displacement feature and the target mixed feature.
[0064] In some embodiments, the embodiments of the present application may set a temporal implicit expression module. Since the key to temporal information representation depends on the use of inter-frame features, most of them currently rely on complex network architectures, such as optical flow networks, resulting in high computational costs. In addition, when the unknown exposure duration leads to uncertain event time frame selection, the performance is limited. Therefore, the embodiments of the present application first select beneficial temporal features from events corresponding to uncertain time scales (exposure durations), and then aggregate the selected temporal features and use their correlation with motion blur to achieve more effective global deblurring.
[0065] Therefore, inspired by the temporal correlation between exposure time and motion blur degree, the embodiment of the present application can implicitly establish a temporal correspondence on any time scale through TIRM, such as Figure 6 As shown in Figure 1, TIRM adopts a time-aware grouped shift operation to effectively aggregate the corresponding features of bidirectional temporal information. Specifically, the input features are first segmented into pre-exposure neighboring features and post-exposure neighboring features Based on the time scale R t Long-term aggregated time-aware event selection, obtaining bidirectional event selection features from adjacent features and Then, the bidirectional selection is input into the temporal grouping shift operation with a larger receptive field to obtain the temporal shift feature and Finally, these displacement features and the connected hybrid features F fuse is input into a CAB (Channel Attential Block) to output the temporal implicit representation feature F TIR , the above two operations will be introduced in detail below.
[0066] Among them, the embodiment of the present application can perform time-aware event selection. In order to effectively maintain the time characteristics within the actual exposure time τ, the embodiment of the present application can use time-aware event selection according to the input time scale Events are selectively modulated; to achieve long-term bidirectional aggregation, each time unit of shutter neighboring features is compressed by applying point-wise convolution, which is formulated as:
[0067]
[0068] Among them, ε represents the replication operation along the time dimension to select time-propagated features from adjacent features; represents the time scale, Represents pre-exposure neighboring features.
[0069] The above formulation allows adaptive extraction of temporal information in an arbitrary exposure interval τ, thus alleviating the difficulty of long-term motion estimation.
[0070] Secondly, the embodiment of the present application can also perform time group shifting, that is, adopt a simple time group shifting operation to implicitly learn the correlation between event time and motion through a large range of spatial shifting. Specifically, the selected feature F ES Divided into m groups along the time dimension, each unit feature has a different shift length and direction, namely:
[0071]
[0072] Where Δx m , Δy m Represent m groups of pixels moving in the x and y directions respectively.
[0073] In the embodiment of the present application, m=4 and Δx can be set m , Δy m ∈{-7, -3, 3, 7} to expand the receptive field of temporal information aggregation to handle a wide range of motion blur; group shift provides multiple potential displacements to match corresponding features of different scales. In order to seamlessly integrate various shift groups, CAB with a kernel size equal to the shift length is adopted, thus achieving a large receptive field and long-term aggregation for global motion deblurring.
[0074] In step S103, the spatial implicit expression features and the temporal implicit representation features are subjected to feature mixing processing to obtain spatiotemporal domain features, and the spatiotemporal domain features are used to perform motion deblurring processing on the target image to obtain a clear image that meets the second preset resolution condition.
[0075] In the embodiment of the present application, the clear image that meets the second preset resolution condition is a clear image that meets the high resolution condition. The specific resolution can be set by those skilled in the art and is not specifically limited here.
[0076] It can be understood that the embodiments of the present application can perform feature mixing processing on the spatial implicit expression features and the temporal implicit representation features to obtain spatiotemporal domain features, and use the spatiotemporal domain features to perform motion deblurring processing on the target image to obtain a clear image that meets the second resolution condition, that is, restore clear image content, effectively improving the efficiency and accuracy of image motion deblurring.
[0077] For example, the working principle of the embodiments of the present application is explained in detail below with some specific embodiments.
[0078] In order to bridge the gap between reality and synthesis, the present application embodiment first uses a new generation of hybrid EVS / CIS sensors to establish a real-world high-resolution hybrid deblurring (H2D) dataset with a resolution of 1920×1080, which contains natural spatially aligned images and time-synchronized events of different scales. The dataset is as follows:
[0079] like Figure 7 As shown in the figure, the simulation dataset GOPRO has a resolution of 1280×720, and uses the ICNS (Information-Centric Networking) simulator to synthesize events. It consists of 3214 clear images with a resolution of 1280×720, of which 2103 are used for training and 1111 are used for testing. The HR clear images are downsampled into LR images of different scales, and then the event camera simulator ICNS is used to simulate LR events. At the same time, the HR blurred images are synthesized by averaging adjacent HR clear images.
[0080] The real dataset H2D has a resolution of 1920×1080 and is collected in a variety of real scenes using the OV60B-Hybrid sensor. Factors such as speed changes, changes in lighting conditions, changes in target scale, and target type diversity are considered during the collection process. H2D contains 2836 clear images with a resolution of 1920×1080, as well as Figure 8 Shown are naturally aligned and synchronized LR events from a hybrid CIS with embedded EVS, where 1233 frames are used for training and 603 frames are used for testing. Overall, this novel biomimetic hybrid camera makes H2D a competitive dataset with multiple features.
[0081] In addition, detailed qualitative and quantitative comparisons were made between the embodiments of the present application and the best existing similar methods, proving the beneficial effects of the present application.
[0082] Specifically, the embodiment of the present application selected an image-based method (i.e., only taking the image as input and ignoring the event stream input) and an event-based optimal method (the image frame and the event stream are input simultaneously) for comparison. The quantitative comparison results are shown in Table 1, where Table 1 is a quantitative result table of the SASNet algorithm on the GoPro and H2D datasets, as follows:
[0083] Table 1
[0084]
[0085] Among them, the evaluation indicators are consistent with previous work. It can be seen that the embodiment of the present application has the most superior performance in multiple indicators. As shown in Table 1, the present application has a lower model size and computational cost, and is much better than the frame-based and event-based methods on two datasets. Compared with the frame-based method, the number of parameters in the present application is less than one-tenth, and the average PSNR scores on the GoPro and H2D datasets are improved by 1.79dB and 2.79dB, respectively. Compared with the event-guided method, the embodiment of the present application has always performed well, with an average improvement of 1.96dB on the GoPro dataset and an average improvement of 2.5dB on the H2D dataset. The SASNet in the embodiment of the present application achieved the best performance at all scales and datasets, which shows that the embodiment of the present application has achieved better results on challenging motion blurred frames, especially in the real H2D dataset with rich textures.
[0086] Next, in order to test the generalization performance at different scales, the embodiment of the present application also evaluates it at the out-of-distribution spatial and temporal scales. s ∈[1,2],R t ∈(0.5,1], it can be extended to other scale ranges without retraining and fine-tuning, making it beneficial for practical applications. It should be noted that other comparison methods must train multiple models for different scales, which makes other comparison methods unsuitable for testing on scales outside the distribution; e.g. Fig. 9 From the analysis of (a) and (b), it can be seen that the embodiment of the present application maintains excellent performance at all scales of a single model, and the performance decreases less as the scale increases, which shows that the scale-aware spatiotemporal implicit expression of the embodiment of the present application has excellent generalization ability for practical applications.
[0087] like Fig.10 and Fig.11As described, a qualitative comparison between the algorithm in the embodiment of the present application and other algorithms is shown, as shown in the enlarged area in the red box. SASNet has a more complete structure, less blur, and better perception of detail fidelity in various types of scenes. It can be seen that HINet (Hierarchical Interaction Network) and Restormer have lost a lot of detail texture and structural distortion, while EFNet (efficient network) has blurred distortion and noise. On the contrary, SASNet reduces the appearance of these distortions and blurs, which is attributed to the implicit representation method of the embodiment of the present application that aggregates spatial and temporal corresponding features at any scale.
[0088] In addition, the present application also compares the effects of the method under a fast motion speed (i.e., >50 rpm). Fig.12 As shown, the embodiment of the present application still maintains structural integrity at all speeds, especially in highly blurred areas under high-speed motion (e.g., 150rpm). Therefore, the embodiment of the present application can effectively improve the efficiency and accuracy of image motion deblurring.
[0089] According to the motion deblurring method based on hybrid visual sensor proposed in the embodiment of the present application, the spatial implicit expression features of the target image and the temporal implicit representation features of the target event can be learned based on the hybrid visual sensor deblurring data set that meets the first resolution condition, and the spatial implicit expression features and the temporal implicit representation features are mixed to obtain spatiotemporal features, and the spatiotemporal features are used to perform motion deblurring on the target image to obtain a clear image that meets the second resolution condition, thereby restoring clear image content and effectively improving the efficiency and accuracy of image motion deblurring. Thus, the problems of limited restoration and reconstruction performance due to lack of motion information and the inability to process real scenes when images and events have different scales, which reduces the applicability and accuracy of image motion deblurring, are solved.
[0090] Next, a motion deblurring device based on a hybrid vision sensor proposed in accordance with an embodiment of the present application will be described with reference to the accompanying drawings.
[0091] Fig.13 It is a block diagram of a motion deblurring device based on a hybrid vision sensor according to an embodiment of the present application.
[0092] like Fig.13 As shown, the motion deblurring device 10 based on the hybrid vision sensor includes: an establishment module 100 , a learning module 200 and a processing module 300 .
[0093] Specifically, the establishment module 100 is used to establish a hybrid vision sensor deblurring data set that meets a first preset resolution condition.
[0094] The learning module 200 is used to learn the spatial implicit expression features of the target image and the temporal implicit representation features of the target event based on the hybrid visual sensor deblurring data set.
[0095] The processing module 300 is used to perform feature mixing processing on the spatial implicit expression feature and the temporal implicit representation feature to obtain spatiotemporal domain features, and use the spatiotemporal domain features to perform motion deblurring processing on the target image to obtain a clear image that meets the second preset resolution condition.
[0096] Optionally, in one embodiment of the present application, the establishing module 100 includes: a determining unit and an establishing unit.
[0097] The determination unit is used to determine various indoor and outdoor target scenes with different spatial resolutions and exposure times.
[0098] An establishing unit is used to establish the hybrid vision sensor deblurring data set that meets the first preset resolution condition in the target scene.
[0099] Optionally, in one embodiment of the present application, the learning module 200 includes: a learning unit and a establishing unit.
[0100] The learning unit is used to learn the spatial correspondence of any spatial scale in the target image to obtain the spatial implicit expression characteristics of the target image.
[0101] The establishing unit is used to establish a time correspondence relationship of any time scale in the target event to obtain the time implicit representation feature of the target event.
[0102] Optionally, in one embodiment of the present application, the learning unit includes: a first determining subunit and a second determining subunit.
[0103] The first determination subunit is used to collect target resolution features required for the target image, and add the target resolution features to image features to obtain added image features.
[0104] The second determining subunit is used to convert the added image features into continuous feature representations to obtain the spatial implicit expression features.
[0105] Optionally, in one embodiment of the present application, the establishing unit includes: a first acquiring subunit, a second acquiring subunit and a third acquiring subunit.
[0106] Among them, the first acquisition subunit is used to obtain a bidirectional event selection feature based on target time perception event selection.
[0107] The second acquisition subunit is used to perform target time grouping displacement processing on the bidirectional time selection feature to obtain a time displacement feature.
[0108] The third acquisition subunit is used to obtain the temporal implicit representation feature by using the temporal displacement feature and the target mixed feature.
[0109] It should be noted that the aforementioned explanation of the embodiment of the motion deblurring method based on the hybrid vision sensor is also applicable to the motion deblurring device based on the hybrid vision sensor of this embodiment, and will not be repeated here.
[0110] According to the motion deblurring device based on hybrid visual sensor proposed in the embodiment of the present application, the spatial implicit expression features of the target image and the temporal implicit representation features of the target event can be learned based on the hybrid visual sensor deblurring data set that meets the first resolution condition, and the spatial implicit expression features and the temporal implicit representation features are mixed to obtain spatiotemporal features, and the spatiotemporal features are used to perform motion deblurring on the target image to obtain a clear image that meets the second resolution condition, thereby restoring clear image content and effectively improving the efficiency and accuracy of image motion deblurring. Thus, the problems of limited restoration and reconstruction performance due to lack of motion information and the inability to process real scenes when images and events have different scales, which reduces the applicability and accuracy of image motion deblurring, are solved.
[0111] Fig.14 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:
[0112] A memory 1401 , a processor 1402 , and a computer program stored in the memory 1401 and executable on the processor 1402 .
[0113] When the processor 1402 executes the program, the motion deblurring method based on the hybrid vision sensor provided in the above embodiment is implemented.
[0114] Furthermore, the electronic device further comprises:
[0115] The communication interface 1403 is used for communication between the memory 1401 and the processor 1402 .
[0116] The memory 1401 is used to store computer programs that can be executed on the processor 1402 .
[0117] The memory 1401 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.
[0118] If the memory 1401, the processor 1402 and the communication interface 1403 are implemented independently, the communication interface 1403, the memory 1401 and the processor 1402 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.14 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0119] Optionally, in a specific implementation, if the memory 1401, the processor 1402 and the communication interface 1403 are integrated on a chip, the memory 1401, the processor 1402 and the communication interface 1403 can communicate with each other through an internal interface.
[0120] The processor 1402 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.
[0121] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the motion deblurring method based on a hybrid vision sensor is implemented as described above.
[0122] This embodiment also provides a computer program product, including a computer program. When the computer program is executed, it is used to implement the above motion deblurring method based on a hybrid vision sensor.
[0123] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.
[0124] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0125] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.
[0126] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.
[0127] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one or a combination of the following technologies known in the art: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0128] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.
[0129] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.
[0130] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.
Claims
1. A motion deblurring method based on a hybrid vision sensor, characterized in that: The following steps are involved: Establishing a hybrid vision sensor deblurring data set that meets a first preset resolution condition; Based on the hybrid visual sensor deblurring dataset, learning the spatial implicit expression features of the target image and the temporal implicit representation features of the target event; The spatial implicit expression feature and the temporal implicit representation feature are subjected to feature mixing processing to obtain spatiotemporal domain features, and the spatiotemporal domain features are used to perform motion deblurring processing on the target image to obtain a clear image that meets a second preset resolution condition.
2. The method according to claim 1, characterized in that The step of establishing a hybrid visual sensor deblurring data set that satisfies a first preset resolution condition includes: Determine a variety of indoor and outdoor target scenes with different spatial resolutions and exposure times; A hybrid vision sensor deblurring data set is established in the target scene that satisfies the first preset resolution condition.
3. The method according to claim 1, characterized in that The learning of the spatial implicit expression features of the target image and the temporal implicit representation features of the target event includes: Learning the spatial correspondence of any spatial scale in the target image to obtain the spatial implicit expression features of the target image; A time correspondence relationship of any time scale in the target event is established to obtain the time implicit representation feature of the target event.
4. The method according to claim 3, characterized in that: The learning of the spatial correspondence of any spatial scale in the target image to obtain the spatial implicit expression features of the target image includes: Acquiring target resolution features required for the target image, and adding the target resolution features to image features to obtain added image features; The added image features are converted into continuous feature representations to obtain the spatial implicit expression features.
5. The method according to claim 3, characterized in that: The step of establishing a time correspondence relationship of any time scale in the target event to obtain a time implicit representation feature of the target event includes: Based on the target time-aware event selection, a bidirectional event selection feature is obtained; The bidirectional time selection feature is subjected to target time grouping displacement processing to obtain a time displacement feature; The temporal implicit representation feature is obtained by using the temporal displacement feature and the target mixed feature.
6. A motion deblurring device based on a hybrid vision sensor, characterized in that: include: An establishing module, used to establish a hybrid visual sensor deblurring data set that meets a first preset resolution condition; A learning module, used for learning spatial implicit expression features of target images and temporal implicit representation features of target events based on the hybrid visual sensor deblurring dataset; A processing module is used to perform feature mixing processing on the spatial implicit expression feature and the temporal implicit representation feature to obtain spatiotemporal domain features, and use the spatiotemporal domain features to perform motion deblurring processing on the target image to obtain a clear image that meets a second preset resolution condition.
7. The device according to claim 6, characterized in that The establishment module includes: A determination unit, used to determine various indoor and outdoor target scenes with different spatial resolutions and exposure times; An establishing unit is used to establish the hybrid vision sensor deblurring data set that meets the first preset resolution condition in the target scene.
8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the motion deblurring method based on a hybrid vision sensor as described in any one of claims 1 to 5.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the motion deblurring method based on a hybrid vision sensor as described in any one of claims 1 to 5.
10. A computer program product, comprising a computer program, characterized in that The computer program is executed by a processor to implement the motion deblurring method based on a hybrid vision sensor as described in any one of claims 1 to 5.
Citation Information
Cited By
Self-adaptive exposure circuit for fusion type visual sensor
CN119854650A
Cross-scene action recognition method
CN120014706A
Event information guided image deblurring and high frame rate reconstruction method
CN120543417A
An event information guided image deblurring and high frame rate reconstruction method
CN120543417B
Event data set construction method and system
CN122002146A
