Video deblurring method and system based on roadside parking scene

Abstract

The invention discloses a video deblurring method and system based on a roadside parking scene, and relates to the field of intelligent parking management.The method comprises the steps that the advantages of an event camera are utilized, an intra-frame cross-modal feature enhancement model based on cyclic attention is constructed, feature enhancement in a single fuzzy frame is achieved, and a video deblurring result is obtained; and iteratively enhancing two modal features of the visible light camera and the event camera in the frame by using a circulation mode, constructing an event guide inter-frame time feature alignment model, realizing inter-frame time feature alignment, and collecting valuable remote time information to a target blurred frame, thereby better realizing a video deblurring effect, and improving the video deblurring efficiency. And the traffic management of the roadside parking scene is facilitated.

Description

Technical Field

[0001] The present invention relates to the field of intelligent parking management, and in particular to a video deblurring method and system based on a roadside parking scene. Background Art

[0002] In recent years, with the development of social economy, people's demand for convenience in daily travel has continued to increase, and the number of cars has continued to increase. However, the increase in the number of parking spaces is far less than the increase in the number of vehicles, resulting in difficulty in parking and disorderly parking, which has become one of the main problems that need to be solved in the development of urban transportation in my country. At present, traffic control departments in various places are also actively carrying out and promoting solutions to the problems of difficulty and disorder in parking for the majority of car owners. Especially in recent years, with the development of intelligent technology, in order to effectively alleviate this problem, traffic control departments in various places have tried and built various intelligent parking management technologies. Among them, by setting up parking spaces on the roadside, using intelligent sensing equipment to collect data such as video images, and processing and analyzing video image data through computer vision algorithms, intelligent parking management is currently a mainstream solution. At present, it is mainly achieved by installing high-position video cameras on the roadside for data collection, and then using visual algorithms such as semantic segmentation and target detection algorithms to realize the analysis of roadside parking scenes.

[0003] However, compared with other spectra, the visible light imaging range is significantly narrower, and it is only effective in environments with good lighting and high visibility. It may fail in special environments such as night. In low-light scenes such as night, the video quality captured by the camera is low, the visibility of targets such as vehicles and pedestrians is low, and there is too much noise information in the video, resulting in low accuracy in the application of visual algorithms. Video deblurring is usually used to improve video quality. The purpose of video deblurring is to effectively collect information from adjacent video frames to make up for the lack of data in a single blurred frame, thereby improving the quality of motion blurred video restoration results.

[0004] However, when there are multiple consecutive frames with severe motion blur in the video, the frame-based video deblurring method often cannot find the accurate time correspondence between adjacent video frames, resulting in a decrease in the performance of video deblurring, which in turn affects the accuracy of parking management. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a video deblurring method and system based on roadside parking scenes, which can solve the problem of low accuracy of existing video deblurring based on roadside parking scenes.

[0006] To achieve the above object, the present invention provides a video deblurring method based on a roadside parking scene, the method comprising:

[0007] Constructing pairs of blurred visible light video data and unblurred visible light video data in a roadside parking scene, and event data corresponding to the visible light video data;

[0008] Using the paired blurred visible light video data and the unblurred visible light video data, and the event data corresponding to the visible light video data as inputs of an intra-frame cross-modal feature enhancement model based on recurrent attention, to obtain enhanced feature information of the event data and feature information of the visible light video data;

[0009] The enhanced feature information of the event data and the feature information of the visible light video data are used as inputs of an event-guided inter-frame temporal feature alignment model to obtain alignment features corresponding to two adjacent blurred frame video data;

[0010] The aligned features corresponding to the video data of two adjacent blurred frames are used as the input of the feature decoding model to obtain the deblurred video frame;

[0011] A loss function is constructed according to the deblurred video frame, and a network model including an intra-frame cross-modal feature enhancement model based on cyclic attention, an event-guided inter-frame temporal feature alignment model, and a feature decoding model is corrected by the loss function.

[0012] Furthermore, the step of constructing a pair of blurred visible light video data and unblurred visible light video data in a roadside parking scene, and event data corresponding to the visible light video data includes:

[0013] The blurred visible light video image frame data and the corresponding event stream data of each video image frame are respectively configured as {B k} and {E k}, where k∈{tP,...,t,...,t+P}, P is the set time span range;

[0014] According to formula e i =(x i ,y i ,t i ,p i ) constructs each event and generates a voxel grid from the event data k ,E k ∈R CXHXW , where x i ,y i Indicates the pixel coordinate position, t i Indicates the time when the event is triggered, p i Indicates the polarity of the event, p i =+1 or -1, the corresponding pixel outputs an event +1 for increased brightness and an event -1 for decreased brightness.

[0015] Furthermore, the step of using the paired blurred visible light video data and the unblurred visible light video data, and the event data corresponding to the visible light video data, as inputs of the intra-frame cross-modal feature enhancement model based on recurrent attention to obtain enhanced feature information of the event data and feature information of the visible light video data includes:

[0016] According to the formula

[0017] Obtaining enhanced feature information of the event data and feature information of the visible light video data;

[0018] The index of each blurred visible light video frame is k∈{t-2,...,t,...,t+2}, and the exposure time is T k Divided into N time intervals, defined as Δt = T k / N, event data E corresponding to each visible light video frame k ∈R CXHXW , in each time interval, it is divided into N event voxels, expressed as The blurred visible light video image frame data is {B k}, the output feature is represented as F(B) k , event data The output features are expressed as Conv() represents the convolution operation, || represents concatenation in the channel dimension, and F R () represents a series of residual convolution operations, represents the initial query features of the recurrent attention operation, represents the cross attention value at the nth step, Respectively represent the output mapping features of the reconstructed key and value; W K , W V Represents the feature mapping process, which is implemented using a 1X1 convolutional layer; α is a hyperparameter, Indicates the mapping characteristics of Key and value, F KV Represents the residual convolution module, MLP() represents the multi-layer perceptron network, and the query feature after cyclic update obtained through N cyclic iterations is expressed as DeConv() represents the deconvolution operation.

[0019] Furthermore, the step of using the enhanced feature information of the event data and the feature information of the visible light video data as inputs of the event-guided inter-frame temporal feature alignment model to obtain alignment features corresponding to two adjacent blurred frame video data includes:

[0020] According to the formula

[0021] Obtaining alignment features corresponding to two adjacent blurred frame video data;

[0022] Among them, the time index is k, k∈{t-2,...,t,...,t+2}, and the time features of two adjacent frames are combined to obtain {ε [k,k+1]}, k∈{t-2,...,t,...,t+2}, the feature representation after feature enhancement of visible light video features is {F k}, k∈{t-2,...,t,...,t+2}, ResBlock() represents the residual convolution module, which includes multiple convolution layers, deconvolution layers, activation function layers, and normalization layers. The features of two adjacent blurred frames are and The alignment feature is

[0023] Furthermore, the step of using the alignment features corresponding to the video data of two adjacent blurred frames as input of the feature decoding model to obtain the deblurred video frame includes:

[0024] According to the formula The deblurred video frame is obtained, where Conv represents the convolutional neural network and the deblurred video frame.

[0025] Furthermore, the present invention provides a video deblurring system based on a roadside parking scene, the system comprising:

[0026] A construction module, used to construct paired blurred visible light video data and unblurred visible light video data in a roadside parking scene, and event data corresponding to the visible light video data;

[0027] The construction module is further used to use the paired blurred visible light video data and the unblurred visible light video data, and the event data corresponding to the visible light video data as inputs of the intra-frame cross-modal feature enhancement model based on recurrent attention, to obtain enhanced feature information of the event data and feature information of the visible light video data;

[0028] The construction module is further used to use the enhanced feature information of the event data and the feature information of the visible light video data as inputs of an event-guided inter-frame temporal feature alignment model to obtain alignment features corresponding to two adjacent blurred frame video data;

[0029] The construction module is further used to use the alignment features corresponding to the video data of two adjacent blurred frames as inputs of the feature decoding model to obtain a deblurred video frame;

[0030] A processing module is used to construct a loss function according to the deblurred video frame, and to correct a network model including an intra-frame cross-modal feature enhancement model based on cyclic attention, an event-guided inter-frame temporal feature alignment model, and a feature decoding model through the loss function.

[0031] Furthermore, the construction module is specifically used to configure the blurred visible light video image frame data and the event stream data of each corresponding video image frame into {B k} and {E k}, where k∈{tP,...,t,...,t+P}, P is the set time span range;

[0032] According to formula e i =(x i ,y i ,t i ,p i ) constructs each event and generates a voxel grid from the event data k ,E k ∈R CXHXW , where x i ,y i Indicates the pixel coordinate position, t i Indicates the time when the event is triggered, p i Indicates the polarity of the event, p i =+1 or -1, the corresponding pixel outputs an event +1 for increased brightness and an event -1 for decreased brightness.

[0033] Furthermore, the building block is specifically used to

[0034]

[0035] Obtaining enhanced feature information of the event data and feature information of the visible light video data;

[0036] The index of each blurred visible light video frame is k∈{t-2,...,t,...,t+2}, and the exposure time is T k Divided into N time intervals, defined as Δt = T k / N, event data E corresponding to each visible light video frame k ∈R CXHXW , is divided into N event voxels in each time interval, expressed as The blurred visible light video image frame data is {B k}, the output feature is represented as F(B) k , event data The output features are expressed as Conv() represents the convolution operation, || represents concatenation in the channel dimension, and F R () represents a series of residual convolution operations, represents the initial query features of the recurrent attention operation, represents the cross attention value at the nth step, Respectively represent the output mapping features of the reconstructed key and value; W K , W V Represents the feature mapping process, which is implemented using a 1X1 convolutional layer; α is a hyperparameter, Indicates the mapping characteristics of Key and value, F KV Represents the residual convolution module, MLP() represents the multi-layer perceptron network, and the query feature after cyclic update obtained through N cyclic iterations is expressed as DeConv() represents the deconvolution operation.

[0037] Furthermore, the building block is specifically used to

[0038]

[0039] Obtaining alignment features corresponding to two adjacent blurred frame video data;

[0040] Among them, the time index is k, k∈{t-2,...,t,...,t+2}, and the time features of two adjacent frames are combined to obtain {ε [k,k+1]}, k∈{t-2,...,t,...,t+2}, the feature representation after feature enhancement of visible light video features is {F k}, k∈{t-2,...,t,...,t+2}, ResBlock() represents the residual convolution module, which includes multiple convolution layers, deconvolution layers, activation function layers, and normalization layers. The features of two adjacent blurred frames are and The alignment feature is

[0041] Furthermore, the building block is specifically used to Get the deblurred video frame, where Conv represents the convolutional neural network, S t is the deblurred video frame.

[0042] The present invention provides a video deblurring method and system based on a roadside parking scene. By taking advantage of the event camera, an intra-frame cross-modal feature enhancement model based on cyclic attention is constructed to achieve feature enhancement within a single blurred frame. Then, the two modal features of the visible light camera and the event camera in the frame are iteratively enhanced in a cyclic manner, and an event-guided inter-frame temporal feature alignment model is constructed to achieve inter-frame temporal feature alignment. Valuable remote time information is collected into the target blurred frame, thereby better achieving the video deblurring effect and facilitating traffic management in roadside parking scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flow chart of a video deblurring method based on a roadside parking scene provided by the present invention;

[0044] Figure 2 It is a schematic diagram of a video deblurring system based on a roadside parking scene provided by the present invention. DETAILED DESCRIPTION

[0045] The device structure and implementation of the present invention are further described in detail below through the accompanying drawings and embodiments.

[0046] The present invention provides a video deblurring method based on a roadside parking scene. Figure 1 As shown, the specific steps include:

[0047] 101. Construct pairs of blurred visible light video data and unblurred visible light video data in a roadside parking scenario, as well as event data corresponding to the visible light video data.

[0048] Among them, visible light video data can be represented as RGB video data. Specifically, the data set can be acquired in two ways: one is the collection of real data, and the other is through the use of open source data or simulation data; by using event cameras and visible light RGB cameras, the two cameras are fixed on a stable tripod or other equipment to ensure that the captured images are aligned at the pixel level, and then the roadside parking scene is shot to obtain the corresponding RGB video and event data pairs, and the corresponding blurred and unblurred data are generated by adjusting parameters such as exposure time.

[0049] For open source data, it includes but is not limited to RGB video and event camera datasets such as GoPro datasets and HighREV datasets; the roadside parking scenes include major urban roads, intersections, roadside parking areas, key areas such as parking spaces at the entrances of schools and hospitals, and other traffic locations.

[0050] 102. Use the paired blurred visible light video data and unblurred visible light video data, and event data corresponding to the visible light video data, as inputs to an intra-frame cross-modal feature enhancement model based on recurrent attention to obtain enhanced feature information of the event data and enhanced feature information of the visible light video data.

[0051] Specifically, the blurred visible light video image frame data and the corresponding event stream data of each video image frame are respectively configured as {B k} and {E k}, where k∈{tP,...,t,...,t+P}, P is the set time span range; according to formula e i =(x i ,y i ,t i ,p i ) constructs each event and generates a voxel grid from the event data k ,E k ∈R CXHXW , where x i ,y i Indicates the pixel coordinate position, t i Indicates the time when the event is triggered, p i Indicates the polarity of the event, p i =+1 or -1, the corresponding pixel outputs an event +1 for increased brightness and an event -1 for decreased brightness.

[0052] For example, for blurred RGB video image frame data, it is defined as {B k}, corresponding to the event stream data of each video image frame, defined as {E k}, where k∈{tP,...,t,...,t+P}, P is the set time span range, P can be set to 2; the goal is to generate a blur-free RGB video frame S k Specifically, for event data {E k}, each event can be expressed as:

[0053] e i =(x i ,y i ,t i ,p i )

[0054] Among them, x i ,y i Indicates the pixel coordinate position, t i Indicates the time when the event is triggered, p i Indicates the polarity of the event, p i= +1 or -1, the corresponding pixel outputs an event +1 with increased brightness, otherwise it outputs an event -1 with decreased brightness; specifically, according to the conventional event camera data processing method, the event data is represented as a voxel grid E k ,E k ∈R CXHXW .

[0055] Specifically, according to the formula

[0056] The alignment features corresponding to two adjacent blurred frame video data are obtained.

[0057] The index of each blurred visible light video frame is k∈{t-2,...,t,...,t+2}, and the exposure time is T k Divided into N time intervals, defined as Δt = T k / N, event data E corresponding to each visible light video frame k ∈R CXHXW , in each time interval, it is divided into N event voxels, expressed as The blurred visible light video image frame data is {B k}, the output feature is represented as F(B) k , event data The output features are expressed as Conv() represents the convolution operation, || represents concatenation in the channel dimension, and F R () represents a series of residual convolution operations, represents the initial query features of the recurrent attention operation, represents the cross attention value at the nth step, Respectively represent the output mapping features of the reconstructed key and value; W K , W V represents the feature mapping process, α is a hyperparameter, Indicates the mapping characteristics of Key and value, F KV Represents the residual convolution module, MLP() represents the multi-layer perceptron network, and the query feature after cyclic update obtained through N cyclic iterations is expressed as DeConv() represents the deconvolution operation.

[0058] It should be noted that since the event camera provides rich temporal information of brightness changes, it is crucial to effectively utilize this dense event time information within the exposure time. By utilizing the event data information in continuous time, the interaction between the blurred frame RGB features and the event features is performed based on the recurrent attention mechanism, and finally the cross-modal feature enhancement within the frame is achieved; for each blurred RGB video frame, the index is k∈{t-2,...,t,...,t+2}, and the exposure time T is k Divided into N time intervals, defined as Δt = T k / N, corresponding event data E k ∈R CXHXW , in each time interval, it is divided into N event voxels, expressed as According to the above definition, firstly, the blurred RGB video image frame data {B k} and event data For feature extraction, a convolutional neural network with a residual module is used. Each residual module includes multiple convolutional layers, pooling layers, and normalization layers. For blurred RGB video image frame data {B k}, the output feature is represented as F(B) k ; For event data The output features are expressed as

[0059] Furthermore, in order to implement the cross-attention mechanism, we first construct a query based on the global information of the blurred image frame features and the event features segmented in time, which is specifically expressed as: Among them, Conv() represents the convolution operation, || represents concatenation in the channel dimension, and F R () represents a series of residual convolution operations, represents the initial query features of the recurrent attention operation; the attention mechanism is calculated as: in, Represents the cross attention value at the nth step; Respectively represent the output mapping features of the reconstructed key and value; W K , W V Represents the feature mapping process, which is implemented using a 1X1 convolutional layer; α is a hyperparameter; Indicates the mapping characteristics of Key and value, F KV represents the residual convolution module; the cyclic calculation process of cross attention is expressed as: Among them, MLP() represents a multi-layer perceptron network;

[0060] Furthermore, through N cycles of iteration, the query features after cycle update are obtained, which can be expressed as By using the deconvolution operation to restore the feature size, Q k Aggregate and obtain the final enhanced features, expressed as: Where DeConv() represents the deconvolution operation. Based on the above steps, intra-frame cross-modal feature enhancement based on cyclic attention is achieved.

[0061] 103. Use the enhanced feature information of the event data and the feature information of the visible light video data as inputs of an event-guided inter-frame temporal feature alignment model to obtain alignment features corresponding to two adjacent blurred frame video data.

[0062] Specifically, according to the formula

[0063] Obtaining alignment features corresponding to two adjacent blurred frame video data;

[0064] Among them, the time index is k, k∈{t-2,...,t,...,t+2}, and the time features of two adjacent frames are combined to obtain {ε [k,k+1]}, k∈{t-2,...,t,...,t+2}, the feature representation after feature enhancement of visible light video features is {F k}, k∈{t-2,...,t,...,t+2}, ResBlock() represents the residual convolution module, which includes multiple convolution layers, deconvolution layers, activation function layers, and normalization layers. The features of two adjacent blurred frames are and The alignment feature is

[0065] 104. Using the aligned features corresponding to the video data of two adjacent blurred frames as input of a feature decoding model to obtain a deblurred video frame.

[0066] Specifically, according to the formula Construct a feature decoding model, where Conv represents a convolutional neural network, S t is the feature decoding model.

[0067] 105. A loss function is constructed according to the deblurred video frame, and a network model including an intra-frame cross-modal feature enhancement model based on cyclic attention, an event-guided inter-frame temporal feature alignment model, and a feature decoding model is corrected through the loss function.

[0068] Specifically, the training loss function of the model consists of the following two parts: L = ε 1 L l1 +ε 2 L ssim , where α1 , α 2 represents the weight parameter; specifically, for L l1 and L ssim , as pixel fidelity loss, can be expressed as: in, y represents the two vectors that need to be calculated, which are the estimated value and target value of the model respectively. m represents the total amount of data. The ultimate goal of the loss function is to minimize the absolute difference between the total estimated value and the target value. Among them, SSIM represents the structural similarity of images.

[0069] An embodiment of the present invention provides a video deblurring method based on a roadside parking scene. By taking advantage of the event camera, an intra-frame cross-modal feature enhancement model based on cyclic attention is constructed to achieve feature enhancement within a single blurred frame. Then, the two modal features of the visible light camera and the event camera in the frame are iteratively enhanced in a cyclic manner, and an event-guided inter-frame temporal feature alignment model is constructed to achieve inter-frame temporal feature alignment. Valuable remote time information is collected into the target blurred frame, thereby better achieving the video deblurring effect and facilitating traffic management in roadside parking scenes.

[0070] As Figure 1 The specific implementation of the method shown in the figure, the embodiment of the present invention provides a video deblurring system based on a roadside parking scene, such as Figure 2 As shown, the system includes: a construction module 21, which is used to construct a pair of blurred visible light video data and unblurred visible light video data in a roadside parking scene, and event data corresponding to the visible light video data;

[0071] The construction module 21 is further used to use the paired blurred visible light video data and the unblurred visible light video data, and the event data corresponding to the visible light video data as inputs of the intra-frame cross-modal feature enhancement model based on recurrent attention, to obtain enhanced feature information of the event data and feature information of the visible light video data;

[0072] The construction module 21 is further used to use the enhanced feature information of the event data and the feature information of the visible light video data as inputs of the event-guided inter-frame temporal feature alignment model to obtain alignment features corresponding to two adjacent blurred frame video data;

[0073] The construction module 21 is further used to use the alignment features corresponding to the video data of two adjacent blurred frames as inputs of the feature decoding model to obtain a deblurred video frame;

[0074] The processing module 22 is used to deblur the blurred video frames through the trained intra-frame cross-modal feature enhancement model based on recurrent attention, the event-guided inter-frame temporal feature alignment model, and the feature decoding model.

[0075] Furthermore, the construction module 21 is specifically used to configure the blurred visible light video image frame data and the event stream data of each corresponding video image frame into {B k} and {E k}, where k∈{tP,...,t,...,t+P}, P is the set time span range;

[0076] According to formula e i =(x i ,y i ,t i ,p i ) constructs each event and generates a voxel grid from the event data k ,E k ∈R CXHXW , where x i ,y i Indicates the pixel coordinate position, t i Indicates the time when the event is triggered, p i Indicates the polarity of the event, p i =+1 or -1, the corresponding pixel outputs an event +1 for increased brightness and an event -1 for decreased brightness.

[0077] Furthermore, the construction module 21 is further configured to:

[0078]

[0079] Construct an intra-frame cross-modal feature enhancement model based on recurrent attention;

[0080] The index of each blurred visible light video frame is k∈{t-2,...,t,...,t+2}, and the exposure time is T k Divided into N time intervals, defined as Δt = T k / N, event data E corresponding to each visible light video frame k ∈R CXHXW , in each time interval, it is divided into N event voxels, expressed as The blurred visible light video image frame data is {B k}, the output feature is represented as F(B) k , event data The output features are expressed as Conv() represents the convolution operation, || represents concatenation in the channel dimension, and F R () represents a series of residual convolution operations, represents the initial query features of the recurrent attention operation, represents the cross attention value at the nth step, Respectively represent the output mapping features of the reconstructed key and value; W K , W V Represents the feature mapping process, which is implemented using a 1X1 convolutional layer; α is a hyperparameter, Indicates the mapping characteristics of Key and value, F KV Represents the residual convolution module, MLP() represents the multi-layer perceptron network, and the query feature after cyclic update obtained through N cyclic iterations is expressed as DeConv() represents the deconvolution operation.

[0081] Furthermore, the construction module 21 is further configured to:

[0082]

[0083] Construct an event-guided inter-frame temporal feature alignment model;

[0084] Among them, the time index is k, k∈{t-2,...,t,...,t+2}, and the time features of two adjacent frames are combined to obtain {ε [k,k+1]}, k∈{t-2,...,t,...,t+2}, the feature representation after feature enhancement of visible light video features is {F k}, k∈{t-2,...,t,...,t+2}, ResBlock() represents the residual convolution module, which includes multiple convolution layers, deconvolution layers, activation function layers, and normalization layers. The features of two adjacent blurred frames are and The alignment feature is

[0085] Furthermore, the construction module 21 is further configured to: Construct a feature decoding model, where Conv represents a convolutional neural network, S t is the feature decoding model.

[0086] An embodiment of the present invention provides a video deblurring system based on a roadside parking scene. By taking advantage of the event camera, an intra-frame cross-modal feature enhancement model based on cyclic attention is constructed to achieve feature enhancement within a single blurred frame. Then, the two modal features of the visible light camera and the event camera in the frame are iteratively enhanced in a cyclic manner, and an event-guided inter-frame temporal feature alignment model is constructed to achieve inter-frame temporal feature alignment. Valuable remote time information is collected into the target blurred frame, thereby better achieving the video deblurring effect and facilitating traffic management in roadside parking scenes.

[0087] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of protection of the present disclosure. The attached method claims present the elements of the various steps in an exemplary order and are not intended to be limited to the specific order or hierarchy described.

[0088] In the above detailed description, various features are grouped together in a single embodiment to simplify the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the embodiments of the claimed subject matter require more features than are clearly stated in each claim. On the contrary, as reflected in the appended claims, the invention is in a state of having less than all the features of the disclosed individual embodiments. Therefore, the appended claims are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate preferred embodiment of the invention.

[0089] The disclosed embodiments are described above to enable any person skilled in the art to implement or use the present invention. Various modifications of these embodiments are obvious to those skilled in the art, and the general principles defined herein may also be applied to other embodiments without departing from the spirit and scope of the present disclosure. Therefore, the present disclosure is not limited to the embodiments given herein, but is consistent with the broadest scope of the principles and novel features disclosed in this application.

[0090] The above description includes examples of one or more embodiments. Of course, it is impossible to describe all possible combinations of components or methods for the purpose of describing the above embodiments, but it should be recognized by those skilled in the art that the various embodiments may be further combined and arranged. Therefore, the embodiments described herein are intended to cover all such changes, modifications and variations that fall within the scope of protection of the appended claims. In addition, with respect to the term "comprising" used in the specification or claims, the word is covered in a manner similar to the term "including", just as "including," is explained as a transitional word in the claims. In addition, any term "or" used in the specification of the claims is intended to mean "non-exclusive or".

[0091] Those skilled in the art may also understand that the various illustrative logical blocks, units, and steps listed in the embodiments of the present invention may be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly demonstrate the interchangeability of hardware and software, the various illustrative components, units, and steps described above have generally described their functions. Whether such functions are implemented by hardware or software depends on the specific application and the design requirements of the entire system. Those skilled in the art may use various methods to implement the described functions for each specific application, but such implementation should not be understood as exceeding the scope of protection of the embodiments of the present invention.

[0092] The various illustrative logic blocks or units described in the embodiments of the present invention can be implemented or operated by a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of the above. The general-purpose processor can be a microprocessor, and optionally, the general-purpose processor can also be any conventional processor, controller, microcontroller or state machine. The processor can also be implemented by a combination of computing devices, such as a digital signal processor and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.

[0093] The steps of the method or algorithm described in the embodiments of the present invention can be directly embedded in hardware, a software module executed by a processor, or a combination of the two. The software module can be stored in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or other storage media of any form in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and the storage medium can be arranged in an ASIC, and the ASIC can be arranged in a user terminal. Optionally, the processor and the storage medium can also be arranged in different components in the user terminal.

[0094] In one or more exemplary designs, the above functions described in the embodiments of the present invention can be implemented in hardware, software, firmware or any combination of the three. If implemented in software, these functions can be stored on a computer-readable medium, or transmitted in the form of one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media that facilitate the transfer of computer programs from one place to another. The storage medium can be any available medium that can be accessed by any general or special computer. For example, such computer-readable media can include but are not limited to RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store program codes in the form of instructions or data structures and other forms that can be read by general or special computers, or general or special processors. In addition, any connection can be appropriately defined as a computer-readable medium, for example, if the software is transmitted from a website site, server or other remote resource through a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wirelessly, such as infrared, wireless and microwave, it is also included in the defined computer-readable medium. The disk and disc include compact disk, laser disk, optical disk, DVD, floppy disk and blue-ray disk. Disks usually copy data magnetically, while discs usually copy data optically with lasers. The above combination can also be included in computer readable media.

[0095] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A video deblurring method based on roadside parking scene, characterized in that: The method comprises: Constructing pairs of blurred visible light video data and unblurred visible light video data in a roadside parking scene, and event data corresponding to the visible light video data; Using the paired blurred visible light video data and the unblurred visible light video data, and the event data corresponding to the visible light video data as inputs of an intra-frame cross-modal feature enhancement model based on recurrent attention, to obtain enhanced feature information of the event data and feature information of the visible light video data; The enhanced feature information of the event data and the feature information of the visible light video data are used as inputs of an event-guided inter-frame temporal feature alignment model to obtain alignment features corresponding to two adjacent blurred frame video data; The aligned features corresponding to the video data of two adjacent blurred frames are used as the input of the feature decoding model to obtain the deblurred video frame; A loss function is constructed according to the deblurred video frame, and a network model including an intra-frame cross-modal feature enhancement model based on cyclic attention, an event-guided inter-frame temporal feature alignment model, and a feature decoding model is corrected by the loss function.

2. The video deblurring method based on roadside parking scene according to claim 1 is characterized in that: The step of constructing paired blurred visible light video data and unblurred visible light video data in a roadside parking scene, and event data corresponding to the visible light video data comprises: The blurred visible light video image frame data and the corresponding event stream data of each video image frame are respectively configured as {B k } and {E k }, where k∈{tP,...,t,...,t+P}, P is the set time span range; According to formula e i =(x i ,y i ,t i ,p i ) constructs each event and generates a voxel grid from the event data k ,E k ∈R CXHXW , where x i ,y i Indicates the pixel coordinate position, t i Indicates the time when the event is triggered, p i Indicates the polarity of the event, p i =+1 or -1, the corresponding pixel outputs an event +1 for increased brightness and an event -1 for decreased brightness.

3. The video deblurring method based on roadside parking scene according to claim 2 is characterized in that: The step of using the paired blurred visible light video data and the unblurred visible light video data, and the event data corresponding to the visible light video data, as inputs of the intra-frame cross-modal feature enhancement model based on recurrent attention to obtain enhanced feature information of the event data and feature information of the visible light video data comprises: According to the formula Obtaining enhanced feature information of the event data and feature information of the visible light video data; The index of each blurred visible light video frame is k∈{t-2,...,t,...,t+2}, and the exposure time is T k Divided into N time intervals, defined as Δt = T k / N, event data E corresponding to each visible light video frame k ∈R CXHXW , in each time interval, it is divided into N event voxels, expressed as The blurred visible light video image frame data is {B k }, the output feature is represented as F(B) k , event data The output features are expressed as Conv() represents the convolution operation, || represents concatenation in the channel dimension, and F R () represents a series of residual convolution operations, represents the initial query features of the recurrent attention operation, represents the cross attention value at the nth step, Respectively represent the output mapping features of the reconstructed key and value; W K , W V Represents the feature mapping process, which is implemented using a 1X1 convolutional layer; α is a hyperparameter, Indicates the mapping characteristics of Key and value, F KV Represents the residual convolution module, MLP() represents the multi-layer perceptron network, and the query feature after cyclic update obtained through N cyclic iterations is expressed as DeConv() represents the deconvolution operation.

4. The video deblurring method based on roadside parking scene according to claim 3 is characterized in that: The step of using the enhanced feature information of the event data and the feature information of the visible light video data as inputs of the event-guided inter-frame temporal feature alignment model to obtain alignment features corresponding to two adjacent blurred frame video data includes: According to the formula Obtaining alignment features corresponding to two adjacent blurred frame video data; Among them, the time index is k, k∈{t-2,...,t,...,t+2}, and the time features of two adjacent frames are combined to obtain {ε [k,k+1] }, k∈{t-2,...,t,...,t+2}, the feature representation after feature enhancement of visible light video features is {F k }, k∈{t-2,...,t,...,t+2}, ResBlock() represents the residual convolution module, which includes multiple convolution layers, deconvolution layers, activation function layers, and normalization layers. The features of two adjacent blurred frames are and The alignment feature is 5. The video deblurring method based on roadside parking scene according to claim 4 is characterized in that: The step of using the alignment features corresponding to the video data of two adjacent blurred frames as input of the feature decoding model to obtain the deblurred video frame comprises: According to the formula Get the deblurred video frame, where Conv represents the convolutional neural network, S t is the deblurred video frame.

6. A video deblurring system based on roadside parking scenes, characterized in that: The system comprises: A construction module, used to construct paired blurred visible light video data and unblurred visible light video data in a roadside parking scene, and event data corresponding to the visible light video data; The construction module is further used to use the paired blurred visible light video data and the unblurred visible light video data, and the event data corresponding to the visible light video data as inputs of the intra-frame cross-modal feature enhancement model based on recurrent attention, to obtain enhanced feature information of the event data and feature information of the visible light video data; The construction module is further used to use the enhanced feature information of the event data and the feature information of the visible light video data as inputs of an event-guided inter-frame temporal feature alignment model to obtain alignment features corresponding to two adjacent blurred frame video data; The construction module is further used to use the alignment features corresponding to the video data of two adjacent blurred frames as inputs of the feature decoding model to obtain a deblurred video frame; A processing module is used to construct a loss function according to the deblurred video frame, and to correct a network model including an intra-frame cross-modal feature enhancement model based on cyclic attention, an event-guided inter-frame temporal feature alignment model, and a feature decoding model through the loss function.

7. The video deblurring system based on roadside parking scene according to claim 6, characterized in that: The construction module is specifically used to configure the blurred visible light video image frame data and the event stream data of each corresponding video image frame into {B k } and {E k }, where k∈{tP,...,t,...,t+P}, P is the set time span range; According to formula e i =(x i ,y i ,t i ,p i ) constructs each event and generates a voxel grid from the event data k ,E k ∈R CXHXW , where x i ,y i Indicates the pixel coordinate position, t i Indicates the time when the event is triggered, p i Indicates the polarity of the event, p i =+1 or -1, the corresponding pixel outputs an event +1 for increased brightness and an event -1 for decreased brightness.

8. A video deblurring system based on roadside parking scenes according to claim 6 or 7, characterized in that: The building block is further used according to the formula Obtaining enhanced feature information of the event data and feature information of the visible light video data; The index of each blurred visible light video frame is k∈{t-2,...,t,...,t+2}, and the exposure time is T k Divided into N time intervals, defined as Δt = T k / N, event data E corresponding to each visible light video frame k ∈R CXHXW , in each time interval, it is divided into N event voxels, expressed as The blurred visible light video image frame data is {B k }, the output feature is represented as F(B) k , event data The output features are expressed as Conv() represents the convolution operation, || represents concatenation in the channel dimension, and F R () represents a series of residual convolution operations, represents the initial query features of the recurrent attention operation, represents the cross attention value at the nth step, Respectively represent the output mapping features of the reconstructed key and value; W K , W V Represents the feature mapping process, which is implemented using a 1X1 convolutional layer; α is a hyperparameter, Indicates the mapping characteristics of Key and value, F KV Represents the residual convolution module, MLP() represents the multi-layer perceptron network, and the query feature after cyclic update obtained through N cyclic iterations is expressed as DeConv() represents the deconvolution operation.

9. The video deblurring system based on roadside parking scene according to claim 8, characterized in that: The building block is further used according to the formula Obtaining alignment features corresponding to two adjacent blurred frame video data; Among them, the time index is k, k∈{t-2,...,t,...,t+2}, and the time features of two adjacent frames are combined to obtain {ε [k,k+1] }, k∈{t-2,...,t,...,t+2}, the feature representation after feature enhancement of visible light video features is {F k }, k∈{t-2,...,t,...,t+2}, ResBlock() represents the residual convolution module, which includes multiple convolution layers, deconvolution layers, activation function layers, and normalization layers. The features of two adjacent blurred frames are and The alignment feature is 10. The video deblurring system based on roadside parking scene according to claim 9, characterized in that: The building block is further used according to the formula Get the deblurred video frame, where Conv represents the convolutional neural network, S t is the deblurred video frame.

Images