A video deblurring method and system based on a roadside parking scene

Abstract

The application discloses a video deblurring method and system based on a roadside parking scene, and relates to the field of intelligent parking management.The method comprises the following steps: constructing an intra-frame cross-modal feature enhancement model based on a cyclic attention by taking advantage of an event camera, realizing feature enhancement in a single blurred frame, iteratively enhancing the two modal features of the visible light camera and the event camera in the frame by using a cyclic mode, constructing an event-guided inter-frame time feature alignment model, realizing time feature alignment between frames, collecting valuable remote time information to a target blurred frame, and thus better realizing a video deblurring effect, which is helpful for traffic management in the roadside parking scene.

Description

Technical Field

[0001] This invention relates to the field of intelligent parking management, and in particular to a video deblurring method and system based on roadside parking scenarios. Background Technology

[0002] In recent years, with the development of the social economy, people's demand for convenient daily travel has been increasing, leading to a continuous increase in the number of cars. However, the increase in the number of parking spaces has lagged far behind the increase in the number of vehicles, making parking difficulties and disorderly parking one of the major problems that urgently need to be solved in the development of urban traffic in my country. Currently, traffic management departments in various regions are actively working to address the parking difficulties and disorderly parking problems faced by car owners. In particular, with the development of intelligent technology in recent years, traffic management departments in various regions have tried and constructed various intelligent parking management technologies to effectively alleviate this problem. Among them, setting up parking spaces on the roadside, using intelligent sensing equipment to collect video images and other data, and processing and analyzing the video image data through computer vision algorithms to achieve intelligent parking management is a mainstream solution at present. Currently, this is mainly achieved by installing high-position video cameras on the roadside for data collection, and then using visual algorithms such as semantic segmentation and object detection to analyze the roadside parking scene.

[0003] However, compared with other spectra, the visible light imaging range is significantly narrower, and it is only effective in well-lit environments with high visibility. It may fail in special environments such as at night. In low-light scenes such as at night, the video quality captured by the camera is low, the visibility of targets such as vehicles and pedestrians is low, and there is too much noise in the video, resulting in low accuracy of the application of visual algorithms. Video deblurring is usually used to improve video quality. The purpose of video deblurring is to effectively collect information from adjacent video frames to make up for the lack of data in a single blurred frame, thereby improving the quality of motion-blurred video restoration results.

[0004] However, when faced with multiple consecutive frames of video with severe motion blur, frame-based video deblurring methods often fail to find accurate temporal correspondences between adjacent video frames, resulting in decreased video deblurring performance and consequently affecting the accuracy of parking management. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a video deblurring method and system based on roadside parking scenarios, which can solve the problem of low accuracy in existing video deblurring methods based on roadside parking scenarios.

[0006] To achieve the above objectives, the present invention provides a video deblurring method based on roadside parking scenarios, the method comprising:

[0007] Construct pairs of fuzzy and unfuzzy visible light video data and event data corresponding to the visible light video data in the roadside parking scenario;

[0008] The paired blurred visible light video data, unblurred visible light video data, and event data corresponding to the visible light video data are used as inputs to an intra-frame cross-modal feature enhancement model based on recurrent attention to obtain the enhanced feature information of the event data and the feature information of the visible light video data.

[0009] The enhanced feature information of the event data and the feature information of the visible light video data are used as inputs to the event-guided inter-frame temporal feature alignment model to obtain the alignment features corresponding to two adjacent blurred frame video data.

[0010] The alignment features corresponding to two adjacent blurred video frames are used as input to the feature decoding model to obtain the deblurred video frames.

[0011] A loss function is constructed based on the deblurred video frames, and the network model containing the intra-frame cross-modal feature enhancement model based on recurrent attention, the event-guided inter-frame temporal feature alignment model, and the feature decoding model is modified using the loss function.

[0012] Furthermore, the step of constructing pairs of blurred visible light video data and unblurred visible light video data in the roadside parking scenario, as well as event data corresponding to the visible light video data, includes:

[0013] Configure the blurred visible light video image frame data and the corresponding event stream data for each video image frame as {B} k} and {E k}, where k∈{tP,...,t,...,t+P}, and P is the set time span range;

[0014] According to formula e i =(x i ,y i ,t i ,p i E) constructs each event and generates a voxel mesh from the event data. k E k ∈R CXHXW , where x i ,y i t represents the pixel coordinate position. i p represents the time when the event was triggered. i p represents the polarity of an event. i = +1 or -1, corresponding to the pixel outputting an event of increasing brightness +1, and an event of decreasing brightness -1.

[0015] Further, the step of using the paired blurred visible light video data and unblurred visible light video data, as well as the event data corresponding to the visible light video data, as input to the intra-frame cross-modal feature enhancement model based on recurrent attention to obtain the enhanced feature information of the event data and the feature information of the visible light video data includes:

[0016] According to the formula

[0017] The enhanced feature information of the event data and the feature information of the visible light video data are obtained;

[0018] Wherein, the index of each blurred visible light video frame is k∈{t-2,...,t,...,t+2}, and the exposure time T k Divided into N time intervals, defined as Δt = T k / N, Event data E corresponding to each visible light video frame k ∈R CXHXW Within each time interval, the data is divided into N event voxels, represented as follows: Blurred visible light video image frame data is {B k The output feature is represented as F(B). k Event data The output features are represented as Conv() represents a convolution operation, || represents concatenation along the channel dimension, and F R () represents a series of residual convolution operations. This represents the initial query features for the recurrent attention operation. This represents the cross-attention value at step n. W represents the output mapping features of the reconstructed key and value, respectively. K W V The feature mapping process is represented by a 1x1 convolutional layer; α is a hyperparameter. F represents the mapping characteristics between key and value. KV MLP() represents a residual convolutional module, and MLP() represents a multilayer perceptron network. The query features are obtained through N iterations and are represented as follows: DeConv() represents the deconvolution operation.

[0019] Further, the step of using the enhanced feature information of the event data and the feature information of the visible light video data as input to the event-guided inter-frame temporal feature alignment model to obtain the alignment features corresponding to two adjacent blurred frame video data includes:

[0020] According to the formula

[0021] Obtain the alignment features corresponding to two adjacent blurred frame video data;

[0022] Where the time index is k, k∈{t-2,...,t,...,t+2}, and the time features of two adjacent frames are combined to obtain {ε [k,k+1]}, k∈{t-2,...,t,...,t+2}, the feature representation of visible light video features after feature enhancement is {F k}, k∈{t-2,...,t,...,t+2}, ResBlock() represents a residual convolutional module, including multiple convolutional layers, deconvolutional layers, activation function layers, and normalization layers. The features of two adjacent blurred frames are respectively and Alignment features are

[0023] Furthermore, the step of using the alignment features corresponding to two adjacent blurred video frames as input to the feature decoding model to obtain the deblurred video frames includes:

[0024] According to the formula The deblurred video frames are obtained, where Conv represents the convolutional neural network and the deblurred video frames.

[0025] Furthermore, the present invention provides a video deblurring system based on roadside parking scenarios, the system comprising:

[0026] The construction module is used to construct pairs of blurred and unblurred visible light video data and event data corresponding to the visible light video data in the roadside parking scenario.

[0027] The construction module is further configured to take the paired fuzzy visible light video data and unfuzzy visible light video data, as well as the event data corresponding to the visible light video data, as input to the intra-frame cross-modal feature enhancement model based on recurrent attention, to obtain the enhanced feature information of the event data and the feature information of the visible light video data.

[0028] The construction module is also used to take the enhanced feature information of the event data and the feature information of the visible light video data as input to the event-guided inter-frame temporal feature alignment model to obtain the alignment features corresponding to two adjacent blurred frame video data.

[0029] The construction module is also used to take the alignment features corresponding to two adjacent blurred video frames as input to the feature decoding model to obtain the deblurred video frames.

[0030] The processing module is used to construct a loss function based on the deblurred video frames, and to modify the network model containing the intra-frame cross-modal feature enhancement model based on recurrent attention, the event-guided inter-frame temporal feature alignment model, and the feature decoding model through the loss function.

[0031] Furthermore, the construction module is specifically configured to configure the blurred visible light video image frame data and the corresponding event stream data of each video image frame as {B}. k} and {E k}, where k∈{tP,...,t,...,t+P}, and P is the set time span range;

[0032] According to formula e i =(x i ,y i ,t i ,p i E) constructs each event and generates a voxel mesh from the event data. k E k ∈R CXHXW , where x i ,y i t represents the pixel coordinate position. i p represents the time when the event was triggered. i p represents the polarity of an event. i = +1 or -1, corresponding to the pixel outputting an event of increasing brightness +1, and an event of decreasing brightness -1.

[0033] Furthermore, the building module is specifically used to calculate according to the formula.

[0034]

[0035] The enhanced feature information of the event data and the feature information of the visible light video data are obtained;

[0036] Wherein, the index of each blurred visible light video frame is k∈{t-2,...,t,...,t+2}, and the exposure time T k Divided into N time intervals, defined as Δt = T k / N, Event data E corresponding to each visible light video frame k ∈R CXHXW Each time interval is divided into N event voxels, denoted as Blurred visible light video image frame data is {B k The output feature is represented as F(B). k Event data The output features are represented as Conv() represents a convolution operation, || represents concatenation along the channel dimension, and F R () represents a series of residual convolution operations. This represents the initial query features for the recurrent attention operation. This represents the cross-attention value at step n. W represents the output mapping features of the reconstructed key and value, respectively. K W V The feature mapping process is represented by a 1x1 convolutional layer; α is a hyperparameter. F represents the mapping characteristics between key and value. KV MLP() represents a residual convolutional module, and MLP() represents a multilayer perceptron network. The query features are obtained through N iterations and are represented as follows: DeConv() represents the deconvolution operation.

[0037] Furthermore, the building module is specifically used to calculate according to the formula.

[0038]

[0039] Obtain the alignment features corresponding to two adjacent blurred frame video data;

[0040] Where the time index is k, k∈{t-2,...,t,...,t+2}, and the time features of two adjacent frames are combined to obtain {ε [k,k+1]}, k∈{t-2,...,t,...,t+2}, the feature representation of visible light video features after feature enhancement is {F k}, k∈{t-2,...,t,...,t+2}, ResBlock() represents a residual convolutional module, including multiple convolutional layers, deconvolutional layers, activation function layers, and normalization layers. The features of two adjacent blurred frames are respectively and Alignment features are

[0041] Furthermore, the building module is specifically used to calculate according to the formula. The deblurred video frames are obtained, where Conv represents the convolutional neural network, and S... t These are the video frames after deblurring.

[0042] This invention provides a video deblurring method and system based on roadside parking scenarios. By leveraging the advantages of event cameras, it constructs an intra-frame cross-modal feature enhancement model based on cyclic attention to enhance features within a single blurred frame. Then, by iteratively enhancing the two modal features of the visible light camera and the event camera within the frame using a cyclic approach, it constructs an event-guided inter-frame temporal feature alignment model to achieve inter-frame temporal feature alignment. This collects valuable remote temporal information into the target blurred frame, thereby achieving better video deblurring results and contributing to traffic management in roadside parking scenarios. Attached Figure Description

[0043] Figure 1 This is a flowchart of a video deblurring method based on roadside parking scenarios provided by the present invention;

[0044] Figure 2 This is a schematic diagram of a video deblurring system based on roadside parking scenarios provided by the present invention. Detailed Implementation

[0045] The device structure and implementation of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0046] This invention provides a video deblurring method based on roadside parking scenarios, such as... Figure 1 As shown, the specific steps include the following:

[0047] 101. Construct pairs of fuzzy visible light video data and unfuzzy visible light video data in roadside parking scenarios, as well as event data corresponding to the visible light video data.

[0048] Visible light video data can be represented as RGB video data. Specifically, the dataset can be acquired in two ways: one is by collecting real data, and the other is by using open source data or simulated data. By using an event camera and a visible light RGB camera, the two cameras are fixed on a stable tripod or other equipment to ensure that the captured images are aligned at the pixel level. Then, the roadside parking scene is captured to obtain the corresponding RGB video and event data pairs. By adjusting parameters such as exposure time, the corresponding blurred and unblurred data can be generated.

[0049] For open-source data, including but not limited to RGB video and event camera datasets such as the GoPro dataset and the HighREV dataset; the roadside parking scenarios include major urban roads, intersections, roadside parking areas, and key areas such as parking areas in front of schools and hospitals.

[0050] 102. The paired blurred visible light video data, unblurred visible light video data, and event data corresponding to the visible light video data are used as inputs to an intra-frame cross-modal feature enhancement model based on recurrent attention to obtain the enhanced feature information of the event data and the feature information of the visible light video data.

[0051] Specifically, the blurred visible light video image frame data and the corresponding event stream data for each video image frame are configured as {B k} and {E k}, where k∈{tP,...,t,...,t+P}, and P is the set time span; according to formula e i =(x i ,y i ,t i ,p i E) constructs each event and generates a voxel mesh from the event data. k E k ∈R CXHXW , where x i ,y i t represents the pixel coordinate position. i p represents the time when the event was triggered. i p represents the polarity of an event. i = +1 or -1, corresponding to the pixel outputting an event of increasing brightness +1, and an event of decreasing brightness -1.

[0052] For example, for blurred RGB video image frame data, it is defined as {B k}, corresponding to the event stream data of each video image frame, is defined as {E k}, where k∈{tP,...,t,...,t+P}, P is the set time span range, P can be set to 2; the goal is to generate blur-free RGB video frames S. k Specifically, for event data {E} k Each event can be represented as:

[0053] e i =(x i ,y i ,t i ,p i )

[0054] Where, x i ,y i t represents the pixel coordinate position. i p represents the time when the event was triggered. i p represents the polarity of an event. i= +1 or -1 corresponds to an event of increased brightness (+1) for the corresponding pixel, and conversely, an event of decreased brightness (-1) for the opposite. Specifically, according to the conventional event camera data processing method, the event data is represented as a voxel grid (E). k E k ∈R CXHXW .

[0055] Specifically, according to the formula

[0056] The alignment features corresponding to two adjacent blurred frame video data are obtained.

[0057] Wherein, the index of each blurred visible light video frame is k∈{t-2,...,t,...,t+2}, and the exposure time T k Divided into N time intervals, defined as Δt = T k / N, Event data E corresponding to each visible light video frame k ∈R CXHXW Within each time interval, the data is divided into N event voxels, represented as follows: Blurred visible light video image frame data is {B k The output feature is represented as F(B). k Event data The output features are represented as Conv() represents a convolution operation, || represents concatenation along the channel dimension, and F R () represents a series of residual convolution operations. This represents the initial query features for the recurrent attention operation. This represents the cross-attention value at step n. W represents the output mapping features of the reconstructed key and value, respectively. K W V This represents the feature mapping process, where α is a hyperparameter. F represents the mapping characteristics between key and value. KV MLP() represents a residual convolutional module, and MLP() represents a multilayer perceptron network. The query features are obtained through N iterations and are represented as follows: DeConv() represents the deconvolution operation.

[0058] It should be noted that since event cameras provide rich temporal information about brightness changes, effectively utilizing this dense event temporal information within the exposure time is crucial. By leveraging event data within a continuous timeframe, and based on a recurrent attention mechanism, the interaction between the RGB features of the blurred frame and event features is performed, ultimately achieving intra-frame cross-modal feature enhancement. For each blurred RGB video frame, the index is k∈{t-2,...,t,...,t+2}, and the exposure time T is... k Divided into N time intervals, defined as Δt = T k / N, corresponding to event data E k ∈R CXHXW Within each time interval, the data is divided into N event voxels, represented as follows: Based on the above definition, firstly, the blurred RGB video image frame data {B k} and event data Feature extraction is performed using a convolutional neural network with residual modules, each residual module comprising multiple convolutional layers, pooling layers, and normalization layers; for blurred RGB video image frame data {B k The output feature is represented as F(B). k For event data The output features are represented as

[0059] Furthermore, to enable the cross-attention mechanism, a query for global information based on blurred image frame features and temporally segmented event features is first constructed, specifically represented as follows: Where Conv() represents the convolution operation, || represents concatenation along the channel dimension, and F R () represents a series of residual convolution operations. The initial query features represent the recurrent attention operation; the attention mechanism is calculated as follows: in, This represents the cross-attention value at step n; W represents the output mapping features of the reconstructed key and value, respectively. K W V The feature mapping process is represented by a 1x1 convolutional layer; α is a hyperparameter. F represents the mapping characteristics between key and value. KV This represents the residual convolution module; the iterative computation process of cross-attention is represented as follows: Where MLP() represents a multilayer perceptron network;

[0060] Furthermore, through N iterations, the updated query features are obtained, represented as follows: By utilizing deconvolution operations to recover feature sizes, and with Q... k The features are aggregated to obtain the final enhanced features, represented as: DeConv() represents the deconvolution operation; based on the above steps, intra-modal feature enhancement based on recurrent attention is achieved.

[0061] 103. Using the enhanced feature information of the event data and the feature information of the visible light video data as input to the event-guided inter-frame temporal feature alignment model, the alignment features corresponding to two adjacent blurred frame video data are obtained.

[0062] Specifically, according to the formula

[0063] Obtain the alignment features corresponding to two adjacent blurred frame video data;

[0064] Where the time index is k, k∈{t-2,...,t,...,t+2}, and the time features of two adjacent frames are combined to obtain {ε [k,k+1]}, k∈{t-2,...,t,...,t+2}, the feature representation of visible light video features after feature enhancement is {F k}, k∈{t-2,...,t,...,t+2}, ResBlock() represents a residual convolutional module, including multiple convolutional layers, deconvolutional layers, activation function layers, and normalization layers. The features of two adjacent blurred frames are respectively and Alignment features are

[0065] 104. Use the alignment features corresponding to two adjacent blurred video frames as input to the feature decoding model to obtain the deblurred video frames.

[0066] Specifically, according to the formula Construct a feature decoding model, where Conv represents a convolutional neural network, and S... t This is a feature decoding model.

[0067] 105. Construct a loss function based on the deblurred video frames, and use the loss function to correct the network model that includes the intra-frame cross-modal feature enhancement model based on recurrent attention, the event-guided inter-frame temporal feature alignment model, and the feature decoding model.

[0068] Specifically, the training loss function of the model consists of the following two parts: L = ε1L l1 +ε2L ssim Where α1 and α2 represent weight parameters; specifically, for L l1and L ssim As a pixel fidelity loss, it can be expressed as: in, y represents the two vectors to be calculated, and the ratio is the model's estimated value and the target value. m represents the total amount of data. The ultimate goal of this loss function is to minimize the absolute difference between the total estimated value and the target value. SSIM stands for Structural Similarity of Images.

[0069] This invention provides a video deblurring method for roadside parking scenarios. By leveraging the advantages of event cameras, it constructs an intra-frame cross-modal feature enhancement model based on cyclic attention to enhance features within a single blurred frame. Then, by iteratively enhancing the two modal features of the visible light camera and the event camera within the frame using a cyclic approach, it constructs an event-guided inter-frame temporal feature alignment model to achieve inter-frame temporal feature alignment. This collects valuable remote temporal information into the target blurred frame, thereby achieving better video deblurring results and contributing to traffic management in roadside parking scenarios.

[0070] As Figure 1 The specific implementation of the method shown in this invention provides a video deblurring system based on roadside parking scenarios, such as... Figure 2 As shown, the system includes: a construction module 21, used to construct pairs of fuzzy visible light video data and unfuzzy visible light video data in a roadside parking scenario, as well as event data corresponding to the visible light video data;

[0071] The construction module 21 is further configured to take the paired fuzzy visible light video data and unfuzzy visible light video data, as well as the event data corresponding to the visible light video data, as input to the intra-frame cross-modal feature enhancement model based on recurrent attention, to obtain the feature information of the enhanced event data and the feature information of the visible light video data.

[0072] The construction module 21 is also used to take the feature information of the enhanced event data and the feature information of the visible light video data as input to the event-guided inter-frame temporal feature alignment model to obtain the alignment features corresponding to two adjacent blurred frame video data.

[0073] The construction module 21 is also used to take the alignment features corresponding to two adjacent blurred video frames as input to the feature decoding model to obtain the deblurred video frames.

[0074] Processing module 22 is used to deblur the blurred video frames using a trained intra-modal feature enhancement model based on recurrent attention, an event-guided inter-frame temporal feature alignment model, and a feature decoding model.

[0075] Furthermore, the construction module 21 is specifically configured to configure the blurred visible light video image frame data and the corresponding event stream data of each video image frame as {B k} and {E k}, where k∈{tP,...,t,...,t+P}, and P is the set time span range;

[0076] According to formula e i =(x i ,y i ,t i ,p i E) constructs each event and generates a voxel mesh from the event data. k E k ∈R CXHXW , where x i ,y i t represents the pixel coordinate position. i p represents the time when the event was triggered. i p represents the polarity of an event. i = +1 or -1, corresponding to the pixel outputting an event of increasing brightness +1, and an event of decreasing brightness -1.

[0077] Furthermore, the construction module 21 is specifically used to calculate according to the formula.

[0078]

[0079] Construct an intra-frame cross-modal feature enhancement model based on recurrent attention;

[0080] Wherein, the index of each blurred visible light video frame is k∈{t-2,...,t,...,t+2}, and the exposure time T k Divided into N time intervals, defined as Δt = T k / N, Event data E corresponding to each visible light video frame k ∈R CXHXW Within each time interval, the data is divided into N event voxels, represented as follows: Blurred visible light video image frame data is {B k The output feature is represented as F(B). k Event data The output features are represented as Conv() represents a convolution operation, || represents concatenation along the channel dimension, and F R () represents a series of residual convolution operations. This represents the initial query features for the recurrent attention operation. This represents the cross-attention value at step n. W represents the output mapping features of the reconstructed key and value, respectively. K W V The feature mapping process is represented by a 1x1 convolutional layer; α is a hyperparameter. F represents the mapping characteristics between key and value. KV MLP() represents a residual convolutional module, and MLP() represents a multilayer perceptron network. The query features are obtained through N iterations and are represented as follows: DeConv() represents the deconvolution operation.

[0081] Furthermore, the construction module 21 is specifically used to calculate according to the formula.

[0082]

[0083] Construct an event-guided inter-frame temporal feature alignment model;

[0084] Where the time index is k, k∈{t-2,...,t,...,t+2}, and the time features of two adjacent frames are combined to obtain {ε [k,k+1]}, k∈{t-2,...,t,...,t+2}, the feature representation of visible light video features after feature enhancement is {F k}, k∈{t-2,...,t,...,t+2}, ResBlock() represents a residual convolutional module, including multiple convolutional layers, deconvolutional layers, activation function layers, and normalization layers. The features of two adjacent blurred frames are respectively and Alignment features are

[0085] Furthermore, the construction module 21 is specifically used to calculate according to the formula. Construct a feature decoding model, where Conv represents a convolutional neural network, and S... t This is a feature decoding model.

[0086] This invention provides a video deblurring system for roadside parking scenarios. By leveraging the advantages of event cameras, it constructs an intra-frame cross-modal feature enhancement model based on cyclic attention to enhance features within a single blurred frame. Then, by iteratively enhancing the two modal features of the visible light camera and the event camera within the frame using a cyclic approach, it constructs an event-guided inter-frame temporal feature alignment model to achieve inter-frame temporal feature alignment. This collects valuable remote temporal information into the target blurred frame, thereby achieving better video deblurring results and contributing to traffic management in roadside parking scenarios.

[0087] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process may be rearranged without departing from the scope of this disclosure. The appended method claims provide elements of various steps in an exemplary order and are not intended to limit the scope to the specific order or hierarchy described.

[0088] In the above detailed description, various features are combined together in a single embodiment to simplify this disclosure. This approach to disclosure should not be construed as reflecting an intention that embodiments of the claimed subject matter require more features than are explicitly stated in each claim. Rather, as reflected in the appended claims, the invention is presented with fewer features than all of the features of the single disclosed embodiment. Therefore, the appended claims are hereby explicitly incorporated into the detailed description, wherein each claim stands alone as a preferred embodiment of the invention.

[0089] The disclosed embodiments have been described above to enable any person skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments without departing from the spirit and scope of this disclosure. Therefore, this disclosure is not limited to the embodiments given herein, but is consistent with the broadest scope of the principles and novel features disclosed in this application.

[0090] The foregoing description includes examples of one or more embodiments. It is certainly impossible to describe all possible combinations of components or methods in order to describe the above embodiments, but those skilled in the art will recognize that further combinations and arrangements of the various embodiments are possible. Therefore, the embodiments described herein are intended to cover all such changes, modifications, and variations that fall within the scope of the appended claims. Furthermore, the term "comprising" as used in the specification or claims is interpreted in a manner similar to the term "including," as interpreted when used as a conjunction in the claims. Additionally, the use of any term "or" in the specification of the claims is intended to mean "non-exclusive or."

[0091] Those skilled in the art will also understand that the various illustrative logical blocks, units, and steps listed in the embodiments of the present invention can be implemented by electronic hardware, computer software, or a combination of both. To clearly demonstrate the interchangeability of hardware and software, the functions of the various illustrative components, units, and steps described above have been generally described. Whether such functionality is implemented through hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the described functions using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of the present invention.

[0092] The various illustrative logic blocks or units described in the embodiments of this invention can be implemented or operate the described functions using a general-purpose processor, digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The general-purpose processor can be a microprocessor; alternatively, it can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented using a combination of computing devices, such as a digital signal processor and a microprocessor, multiple microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.

[0093] The steps of the methods or algorithms described in the embodiments of this invention can be directly embedded in hardware, a software module executed by a processor, or a combination of both. The software module can be stored in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from and write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and storage medium can be housed in an ASIC, which can be housed in a user terminal. Optionally, the processor and storage medium can also be housed in different components of the user terminal.

[0094] In one or more exemplary designs, the functions described in the embodiments of the present invention can be implemented in hardware, software, firmware, or any combination of these three. If implemented in software, these functions can be stored on a computer-readable medium or transmitted on a computer-readable medium in the form of one or more instructions or code. Computer-readable media include computer storage media and communication media that facilitate the transfer of computer programs from one place to another. Storage media can be any available media that can be accessed by a general-purpose or special-purpose computer. For example, such computer-readable media can include, but is not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store program code in the form of instructions or data structures and other forms that can be read by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Furthermore, any connection can be suitably defined as a computer-readable medium, for example, if the software is transmitted from a website, server or other remote resource via a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wirelessly, such as infrared, wireless and microwave, it is also included in the defined computer-readable medium. The disks and discs mentioned include compressed disks, laser discs, optical discs, DVDs, floppy disks, and Blu-ray discs. Disks typically copy data magnetically, while disks typically copy data optically using lasers. Combinations of the above can also be contained in computer-readable media.

[0095] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A video deblurring method based on roadside parking scenarios, characterized in that, The method includes: Construct pairs of fuzzy and unfuzzy visible light video data and event data corresponding to the visible light video data in the roadside parking scenario; The paired blurred visible light video data, unblurred visible light video data, and event data corresponding to the visible light video data are used as inputs to an intra-frame cross-modal feature enhancement model based on recurrent attention to obtain the enhanced feature information of the event data and the feature information of the visible light video data. The enhanced feature information of the event data and the feature information of the visible light video data are used as inputs to the event-guided inter-frame temporal feature alignment model to obtain the alignment features corresponding to two adjacent blurred frame video data. The alignment features corresponding to two adjacent blurred video frames are used as input to the feature decoding model to obtain the deblurred video frames. A loss function is constructed based on the deblurred video frames, and the network model containing the intra-frame cross-modal feature enhancement model based on recurrent attention, the event-guided inter-frame temporal feature alignment model, and the feature decoding model is modified using the loss function.

2. The video deblurring method based on roadside parking scenarios according to claim 1, characterized in that, The steps for constructing pairs of blurred and unblurred visible light video data and corresponding event data in the roadside parking scenario include: Configure the blurred visible light video image frame data and the corresponding event stream data for each video image frame as {B} k } and {E k }, where k∈{tP,...,t,...,t+P}, and P is the set time span range; According to formula e i =(x i ,y i ,t i ,p i E) constructs each event and generates a voxel mesh from the event data. k E k ∈R CXHXW , where x i ,y i t represents the pixel coordinate position. i p represents the time when the event was triggered. i p represents the polarity of an event. i = +1 or -1, corresponding to the pixel outputting an event of increasing brightness +1, and an event of decreasing brightness -1.

3. The video deblurring method based on roadside parking scenarios according to claim 2, characterized in that, The step of using the paired blurred visible light video data, unblurred visible light video data, and event data corresponding to the visible light video data as input to an intra-frame cross-modal feature enhancement model based on recurrent attention to obtain the enhanced feature information of the event data and the feature information of the visible light video data includes: According to the formula The enhanced feature information of the event data and the feature information of the visible light video data are obtained; Wherein, the index of each blurred visible light video frame is k∈{t-2,...,t,...,t+2}, and the exposure time T k Divided into N time intervals, defined as Δt = T k / N, Event data E corresponding to each visible light video frame k ∈R CXHXW Within each time interval, the data is divided into N event voxels, represented as follows: Blurred visible light video image frame data is {B k The output feature is represented as F(B). k Event data The output features are represented as Conv() represents a convolution operation, || represents concatenation along the channel dimension, and F R () represents a series of residual convolution operations. This represents the initial query features for the recurrent attention operation. This represents the cross-attention value at step n. W represents the output mapping features of the reconstructed key and value, respectively. K W V The feature mapping process is represented by a 1x1 convolutional layer; α is a hyperparameter. F represents the mapping characteristics between key and value. KV MLP() represents a residual convolutional module, and MLP() represents a multilayer perceptron network. The query features are obtained through N iterations and are represented as follows: DeConv() represents the deconvolution operation.

4. The video deblurring method based on roadside parking scenarios according to claim 3, characterized in that, The step of using the enhanced feature information of the event data and the feature information of the visible light video data as input to the event-guided inter-frame temporal feature alignment model to obtain the alignment features corresponding to two adjacent blurred frame video data includes: According to the formula Obtain the alignment features corresponding to two adjacent blurred frame video data; Where the time index is k, k∈{t-2,...,t,...,t+2}, and the time features of two adjacent frames are combined to obtain {ε [k,k+1] }, k∈{t-2,...,t,...,t+2}, the feature representation of visible light video features after feature enhancement is {F k }, k∈{t-2,...,t,...,t+2}, ResBlock() represents a residual convolutional module, including multiple convolutional layers, deconvolutional layers, activation function layers, and normalization layers. The features of two adjacent blurred frames are respectively and Alignment features are 5. A video deblurring method based on roadside parking scenarios according to claim 4, characterized in that, The step of using the alignment features corresponding to two adjacent blurred video frames as input to the feature decoding model to obtain the deblurred video frames includes: According to the formula The deblurred video frames are obtained, where Conv represents the convolutional neural network, and S... t These are the video frames after deblurring.

6. A video deblurring system based on roadside parking scenarios, characterized in that, The system includes: The construction module is used to construct pairs of blurred and unblurred visible light video data and event data corresponding to the visible light video data in the roadside parking scenario. The construction module is further configured to take the paired fuzzy visible light video data and unfuzzy visible light video data, as well as the event data corresponding to the visible light video data, as input to the intra-frame cross-modal feature enhancement model based on recurrent attention, to obtain the enhanced feature information of the event data and the feature information of the visible light video data. The construction module is also used to take the enhanced feature information of the event data and the feature information of the visible light video data as input to the event-guided inter-frame temporal feature alignment model to obtain the alignment features corresponding to two adjacent blurred frame video data. The construction module is also used to take the alignment features corresponding to two adjacent blurred video frames as input to the feature decoding model to obtain the deblurred video frames. The processing module is used to construct a loss function based on the deblurred video frames, and to modify the network model containing the intra-frame cross-modal feature enhancement model based on recurrent attention, the event-guided inter-frame temporal feature alignment model, and the feature decoding model through the loss function.

7. A video deblurring system based on roadside parking scenarios according to claim 6, characterized in that, The construction module is specifically used to configure the blurred visible light video image frame data and the corresponding event stream data of each video image frame as {B}. k } and {E k }, where k∈{tP,...,t,...,t+P}, and P is the set time span range; According to formula e i =(x i ,y i ,t i ,p i E) constructs each event and generates a voxel mesh from the event data. k E k ∈R CXHXW , where x i ,y i t represents the pixel coordinate position. i p represents the time when the event was triggered. i p represents the polarity of an event. i = +1 or -1, corresponding to the pixel outputting an event of increasing brightness +1, and an event of decreasing brightness -1.

8. A video deblurring system based on roadside parking scenarios according to claim 6 or 7, characterized in that, The construction module is further specifically used to calculate according to the formula. The enhanced feature information of the event data and the feature information of the visible light video data are obtained; Wherein, the index of each blurred visible light video frame is k∈{t-2,...,t,...,t+2}, and the exposure time T k Divided into N time intervals, defined as Δt = T k / N, Event data E corresponding to each visible light video frame k ∈R CXHXW Within each time interval, the data is divided into N event voxels, represented as follows: Blurred visible light video image frame data is {B k The output feature is represented as F(B). k Event data The output features are represented as Conv() represents a convolution operation, || represents concatenation along the channel dimension, and F R () represents a series of residual convolution operations. This represents the initial query features for the recurrent attention operation. This represents the cross-attention value at step n. W represents the output mapping features of the reconstructed key and value, respectively. K W V The feature mapping process is represented by a 1x1 convolutional layer; α is a hyperparameter. F represents the mapping characteristics between key and value. KV MLP() represents a residual convolutional module, and MLP() represents a multilayer perceptron network. The query features are obtained through N iterations and are represented as follows: DeConv() represents the deconvolution operation.

9. A video deblurring system based on roadside parking scenarios according to claim 8, characterized in that, The construction module is further specifically used to calculate according to the formula. Obtain the alignment features corresponding to two adjacent blurred frame video data; Where the time index is k, k∈{t-2,...,t,...,t+2}, and the time features of two adjacent frames are combined to obtain {ε [k,k+1] }, k∈{t-2,...,t,...,t+2}, the feature representation of visible light video features after feature enhancement is {F k }, k∈{t-2,...,t,...,t+2}, ResBlock() represents a residual convolutional module, including multiple convolutional layers, deconvolutional layers, activation function layers, and normalization layers. The features of two adjacent blurred frames are respectively and Alignment features are 10. A video deblurring system based on roadside parking scenarios according to claim 9, characterized in that, The construction module is further specifically used to calculate according to the formula. The deblurred video frames are obtained, where Conv represents the convolutional neural network, and S... t These are the video frames after deblurring.

Images