Video processing method and apparatus, device, storage medium, and computer program product

Abstract

The present application discloses a video processing method and apparatus, a device, a storage medium, and a computer program product. The method comprises: acquiring a video to be processed captured by an event camera and event data corresponding to the video to be processed; performing event feature extraction on the event data to obtain a motion region feature corresponding to the video to be processed; performing frame feature extraction on the video to be processed and the event data to obtain a motion holistic feature corresponding to the video to be processed, wherein the motion holistic feature is used for representing a dependency relationship between time and space of the video to be processed; performing feature fusion on the motion region feature and the motion holistic feature to obtain a fused feature; and performing video reconstruction on the basis of the fused feature to obtain a target video. The method provided in the present application can improve the resolution and frame rate of a restored video, thereby enhancing video quality.

Description

Video processing methods, apparatus, equipment, storage media, and computer program products Technical Field

[0001] This application belongs to the field of computer vision, and in particular relates to a video processing method, apparatus, device, storage medium and computer program product. Background Technology

[0002] Implicit Neural Representation (INR) is a method that uses neural networks to represent input information such as images, audio, and point clouds as functions. It can be used in scenarios such as video restoration by representing images or videos as implicit fields for decoding at arbitrary resolutions.

[0003] However, in related technologies, INR-based C-STVSR (Continuous Space-Time Video Super-Resolution) methods typically assume linear motion between frames and use interpolation or feature warping to generate features at arbitrary spatial and temporal locations between two frames. This approach limits C-STVSR's ability to capture fast and nonlinear motion as well as long-term dependencies in complex dynamic scenes, thereby reducing video resolution and frame rate. Summary of the Invention

[0004] This application provides a video processing method, apparatus, device, storage medium, and computer program product that can improve the resolution and frame rate of the processed video and improve video quality.

[0005] In a first aspect, embodiments of this application provide a video processing method, the method comprising: acquiring a video to be processed captured by an event camera and event data corresponding to the video to be processed; extracting event features from the event data to obtain motion region features corresponding to the video to be processed; extracting frame features from the video to be processed and the event data to obtain overall motion features corresponding to the video to be processed, wherein the overall motion features are used to characterize the temporal and spatial dependencies of the video to be processed; fusing the motion region features and the overall motion features to obtain fused features; and reconstructing the video based on the fused features to obtain a target video.

[0006] Secondly, embodiments of this application provide a video processing apparatus, comprising: a data acquisition module for acquiring a video to be processed captured by an event camera and event data corresponding to the video to be processed; a first feature extraction module for extracting event features from the event data to obtain motion region features corresponding to the video to be processed; a second feature extraction module for extracting frame features from the video to be processed and the event data to obtain overall motion features corresponding to the video to be processed, wherein the overall motion features are used to characterize the temporal and spatial dependencies of the video to be processed; a feature fusion module for fusing the motion region features and the overall motion features to obtain fused features; and a video reconstruction module for reconstructing the video based on the fused features to obtain a target video.

[0007] Thirdly, embodiments of this application provide an electronic device, which includes: a processor and a memory storing computer program instructions; the processor executes the computer program instructions to implement the video processing method as described in the first aspect.

[0008] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the video processing method as described in the first aspect.

[0009] Fifthly, embodiments of this application provide a computer program product in which instructions, when executed by a processor of an electronic device, cause the electronic device to perform the video processing method as described in the first aspect.

[0010] As described above, in this embodiment, the event data corresponding to the video to be processed contains rich event information. By extracting event features from the event data in the video to be processed, the nonlinear motion and long-term dependencies of moving objects in the video can be captured, providing a foundation for improving the resolution and frame rate of the video. Furthermore, in this embodiment, feature extraction from event data can capture dynamic motion and edge details in the video to be processed. Extracting overall motion features from both the video and event data can model long-term motion dependencies. Combining the features extracted by these two methods allows for a comprehensive understanding of the dynamic changes in the video content, thereby improving the resolution and frame rate of the reconstructed target video.

[0011] Therefore, it can be seen that the method proposed in this application can obtain a target video with high resolution and frame rate by processing the video to be processed. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 is a flowchart illustrating a video processing method provided in an embodiment of this application;

[0014] Figure 2 is a schematic diagram of the architecture of a video processing model provided in one embodiment of this application;

[0015] Figure 3 is a schematic diagram of the structure of a pyramid model provided in an embodiment of this application;

[0016] Figure 4 is a schematic diagram of experimental visualization results on an Adobe dataset provided in an embodiment of this application;

[0017] Figure 5 is a schematic diagram of experimental visualization results on the TimeLens++ dataset provided in an embodiment of this application;

[0018] Figure 6 is a schematic diagram of experimental visualization results on the CED dataset provided in an embodiment of this application;

[0019] Figure 7 is a schematic diagram of experimental visualization results on the TimeLens++ dataset provided in an embodiment of this application;

[0020] Figure 8 is a schematic diagram of the structure of a video processing apparatus provided in another embodiment of this application;

[0021] Figure 9 is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0022] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0023] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0024] To facilitate understanding, before explaining the solution provided in this application, the background of the solution provided in this application will be explained first.

[0025] In related technologies, due to limitations in I / O (Input / Output) bandwidth and sensor size, camera systems can only record video at a fixed resolution and frame rate. This impacts computer vision applications, such as immersive experiences in virtual reality and traffic analysis in autonomous driving. To overcome this limitation, Continuous Space-Time Video Super-Resolution (C-STVSR) can be used to simultaneously improve video resolution and frame rate.

[0026] In recent years, implicit neural representations have been applied to video inpainting, which can decode at arbitrary resolutions by representing images or videos as implicit fields. However, existing INR-based C-STVSR methods have limitations. They typically assume linear motion between frames and use interpolation or feature warping to generate features at arbitrary spatial and temporal locations between two frames. This approach limits C-STVSR's ability to capture fast and nonlinear motion as well as long-term dependencies (involving more than two frames) in complex dynamic scenes.

[0027] To address the problems of the prior art, embodiments of this application provide a video processing method, apparatus, device, storage medium, and computer program product. The video processing method provided in this application embodiment will be described first below.

[0028] The method proposed in this application aims to capture global dependencies and regional motion based on INR, assisted by an event camera with high temporal resolution and low latency. By designing a feature extraction framework, it fully utilizes the rich temporal information provided by event data to capture regional nonlinear motion and long-term dependencies, and achieves the capture of long-term dependencies through spatial-temporal embedding.

[0029] Figure 1 shows a flowchart of a video processing method provided in an embodiment of this application. As shown in Figure 1, the method includes the following steps:

[0030] Step S101: Obtain the video to be processed captured by the event camera and the event data corresponding to the video to be processed.

[0031] In step S101, the event camera is a camera that acquires images through a dynamic visual sensor. The event camera can acquire "events" (i.e., event data) during the image generation process, such as changes in pixel brightness. In this embodiment, each event data point is used to characterize pixel intensity changes at a specific time and location in the video.

[0032] In step S101, the video to be processed can be a video to be repaired, such as a blurry video captured by an event camera. In this embodiment, the video to be processed includes multiple RGB frames.

[0033] It should be noted that, in this embodiment of the application, the event camera has high temporal resolution and low latency characteristics. Based on this, by using the event camera to collect the video to be processed and the event data of the video to be processed, subtle motion changes in the video to be processed can be obtained, providing a basis for the recovery of the video to be processed.

[0034] Step S102: Extract event features from the event data to obtain the motion region features corresponding to the video to be processed.

[0035] In step S102, a region event feature extractor can be used to extract features from the event data, obtaining motion region features that include dynamic motion and edge details. That is, by extracting features from the event data in step S102, detailed motion information in the video to be processed can be obtained. This detailed information can improve the accuracy and precision of video restoration, ensuring the video's resolution and frame rate.

[0036] Step S103: Extract frame features from the video to be processed and the event data to obtain the overall motion features corresponding to the video to be processed.

[0037] In step S103, the overall motion features are used to characterize the temporal and spatial dependencies of the video to be processed.

[0038] As an example, a global event feature extractor can be used to extract the overall motion features of the video to be processed. The global event feature extractor can be a feature extractor based on the Swing Transformer, which can model long-term temporal and spatial dependencies over a large range.

[0039] Step S104: Perform feature fusion on the motion region features and the overall motion features to obtain fused features.

[0040] It should be noted that motion region features and overall motion features can reflect the dynamic changes in the content of the video to be processed. Therefore, after extracting motion region features and overall motion features, feature fusion is performed on the above two features. Based on the fused features, video reconstruction can be performed, which can clearly show the motion details in the video and improve the resolution and frame rate of the video.

[0041] Step S105: Reconstruct the video based on the fusion features to obtain the target video.

[0042] In step S105, video reconstruction can be achieved by using INR technology to perform spatial-temporal decoding of the fused features. During video reconstruction, a learned temporal embedding is introduced as an attention vector, along with spatial embedding and decoding techniques. This allows for efficient reconstruction of video frames without the need to estimate optical flow, improving the efficiency and accuracy of video reconstruction and enhancing the resolution and frame rate of the reconstructed video.

[0043] Based on the scheme defined in steps S101 to S105 above, it can be understood that in this embodiment, the event data corresponding to the video to be processed contains rich event information. By extracting event features from the event data in the video to be processed, the nonlinear motion and long-term dependencies of moving objects in the video to be processed can be captured, providing a foundation for improving the resolution and frame rate of the video to be processed. Furthermore, in this embodiment, feature extraction from event data can capture dynamic motion and edge details in the video to be processed, while extracting overall motion features from the video to be processed and event data can model long-term motion dependencies. Combining the features extracted by the above two feature extraction methods allows for a comprehensive understanding of the dynamic changes in video content, thereby improving the resolution and frame rate of the reconstructed target video.

[0044] Therefore, it can be seen that the method proposed in this application can obtain a target video with high resolution and frame rate by processing the video to be processed.

[0045] In one embodiment, Figure 2 illustrates the architecture of a video processing model implementing the method proposed in this application. As shown in Figure 2, the video processing model mainly includes a data representation unit, a region event feature extractor, a global event feature extractor, and a decoding unit. The data representation unit is used to represent event data to more accurately capture motion changes in the video; the region event feature extractor is used to extract local dynamic change information and edge information from the event data; the global event feature extractor is used to model long-term temporal and spatial dependencies; and the decoding unit is used to decode the data to achieve video reconstruction.

[0046] The implementation steps of the method proposed in the embodiments of this application will be explained below with reference to Figure 2.

[0047] First, after acquiring the video to be processed and the event data captured by the event camera, event features can be extracted from the event data to obtain the motion feature information corresponding to the video to be processed; then, regional event features can be extracted from the motion feature information to obtain the motion region features corresponding to the video to be processed.

[0048] In one embodiment, the event data can be processed by a data representation unit to extract event features, thereby obtaining motion feature information corresponding to the video to be processed. Specifically, the data representation unit uses a time granularity corresponding to each pyramid level in the time pyramid model to perform multi-level time pyramid representation on the event data, thereby obtaining motion feature information corresponding to the video to be processed.

[0049] For example, in Figure 2, after acquiring the video to be processed and event data, the event features are represented by a time pyramid using a data representation unit. The pyramid model includes multiple pyramid levels, and the duration of each pyramid level is a portion of the duration of the previous pyramid level. For instance, in the structural diagram of the pyramid model shown in Figure 3, the duration of the first pyramid level is... The duration of the second pyramid level is It is the duration of the first pyramid level. The duration of the third pyramid level is It is the duration of the second pyramid level. Where r is a positive integer.

[0050] In this embodiment, the pyramid model can have a structure of seven pyramid levels. In this scenario, the pyramid model can most effectively capture both rapid and subtle dynamic changes. Each pyramid level represents a different temporal granularity, thereby ensuring a comprehensive understanding of motion information in the video.

[0051] In addition, for each pyramid level, a dynamically adjusted resolution (based on a reduction of 1 / 2, 1 / 4, 1 / 8 of the original video resolution) can be used to reduce the computational burden while preserving sufficient motion details.

[0052] It should be noted that in this embodiment of the application, a pyramid model is used to represent the event data in a time pyramid manner, which can achieve a finer time granularity, effectively reduce computational complexity, and at the same time retain important motion information.

[0053] To further improve the accuracy of the pyramid model in representing the time pyramid of event data, in practical applications, the pyramid model can be dynamically adjusted according to the video content of the video to be processed.

[0054] In one embodiment, dynamic characteristic data from the video to be processed is obtained, and then the level parameters corresponding to each pyramid level in the time pyramid model are adjusted based on the dynamic characteristic data.

[0055] In the above embodiments, dynamic characteristic data is used to characterize the motion data of moving objects in the video to be processed, such as motion duration, motion area, and motion speed. The parameters of the pyramid hierarchy include at least one of the following: time span and time granularity.

[0056] As an example, the time span and / or temporal granularity of each pyramid level can be adaptively adjusted based on the dynamic characteristics data in the video to be processed, thereby capturing motion changes in the video more accurately.

[0057] In another embodiment, a multi-scale time window can also be used to extract event features from the event data to obtain motion feature information corresponding to the video to be processed.

[0058] Specifically, the dynamic characteristic data of at least one moving object in the video to be processed is obtained. Then, based on the relationship between the dynamic characteristic data and the time window, the time window corresponding to the dynamic characteristic data of at least one moving object is determined. Then, the event data is encoded through the time window corresponding to the dynamic characteristic data of at least one moving object to obtain the motion feature information corresponding to the video to be processed.

[0059] In the above embodiments, dynamic characteristic data is used to characterize the motion data of moving objects in the video to be processed. In this embodiment, event data can be encoded simultaneously using time windows of different sizes to capture various motion speeds from fast to slow, thereby enhancing the adaptability of the video processing model to different motion patterns. Furthermore, in practical applications, after acquiring event data, the video processing model can obtain a target time window that matches the event data from multiple time windows, and then use the target time window to encode the event data, thereby obtaining the motion feature information corresponding to the video to be processed.

[0060] Furthermore, as shown in Figure 2, after obtaining the motion feature information corresponding to the video to be processed, a region event feature extractor is used to extract region event features from the motion feature information. Specifically, the region event feature extractor uses the convolutional layer and multi-head self-attention mechanism of the neural network to extract local dynamic change information and edge information from the motion feature information, thereby obtaining the motion region features corresponding to the video to be processed.

[0061] It should be noted that, because the regional event feature extractor uses convolutional layers and multi-head attention mechanisms, it can capture local dynamic change information and edge information, providing a foundation for improving the resolution and frame rate of the recovered video.

[0062] Furthermore, it should be noted that in practical applications, region event feature extractors can also be hybrid feature extraction networks or modular feature extractors. Hybrid feature extraction networks combine different types of network structures, such as convolutional neural networks and graph neural networks, to fully utilize their advantages in processing spatial information and modeling long-range dependencies. Modular feature extractors, through modular design, allow different feature extraction modules to be dynamically combined and selected based on the characteristics of the input data, adapting to diverse video content and scenarios.

[0063] In one embodiment, as shown in Figure 2, the overall event feature extractor extracts the long-term temporal and spatial dependencies between the video to be processed and the event data to determine the overall motion trend of moving objects in the video. After obtaining the motion region features and the overall motion features, the two are fused to obtain fused features, which are then input into the decoding unit to achieve video reconstruction.

[0064] Specifically, the decoding unit acquires the target frame rate and target resolution corresponding to the target video as input, and determines the display time corresponding to each frame in the target video based on the target frame rate; then, it performs a temporal embedding operation on the fusion features to obtain the first fusion feature corresponding to the display time; next, it performs a spatial embedding operation on the first fusion feature to obtain the second fusion feature corresponding to the target resolution; finally, it performs a decoding operation on the second fusion feature to obtain the target video with the target frame rate and target resolution.

[0065] In the above embodiments, the target frame rate and target resolution can be input by the user according to actual needs. The target frame rate and target resolution are the frame rate and resolution of the target video after video reconstruction of the video to be processed.

[0066] Taking Figure 2 as an example, after obtaining the fusion feature R t Next, a temporal embedding operation is performed on the fused features, that is, the timestamp corresponding to the target frame rate is embedded into the fused features to capture features at the aforementioned timestamp; then, the target resolution is embedded into the first fused feature R. ts In the process, spatial embedding operation is performed to obtain the second fused feature; then, the second fused feature is decoded by a multilayer perceptron (MLP) (for example, a four-layer perceptron is used in Figure 2) to obtain the target video.

[0067] It should be noted that in practical applications, an iterative refinement approach can be used for decoding. That is, based on the initial space-time decoding, an iterative refinement mechanism is introduced, which involves repeating the decoding process multiple times in the above manner. Through multiple rounds of refinement processing, the quality and accuracy of the reconstructed video frames are gradually improved.

[0068] In another embodiment, an image segmentation-based decoding strategy can also be used to decode the second fusion feature. Specifically, firstly, the video to be processed is segmented to obtain multiple regional videos; then, based on the correlation between the regional videos and the second fusion feature, the second fusion feature corresponding to each regional video is determined; next, based on the correlation between the decoding strategy and the second fusion feature, the corresponding decoding strategy is used to decode the second fusion feature of the corresponding regional video, resulting in a target regional video for each regional video; finally, the target regional videos corresponding to multiple regional videos are combined to obtain the target video.

[0069] It should be noted that the image segmentation-based decoding strategy first segments the video frames and then uses different decoding strategies for different regions, especially for dynamic and static backgrounds, in order to achieve higher quality video reconstruction.

[0070] The method proposed in this application can be applied to, but is not limited to, scenarios such as urban traffic monitoring video enhancement, sports event live video processing, and virtual reality content generation. The following uses three scenarios as examples to illustrate the method proposed in this application.

[0071] In the scenario of enhancing urban traffic monitoring videos, low-resolution, low-frame-rate urban traffic monitoring videos and related event data captured by event cameras are acquired. The event data is processed using a temporal pyramid representation, and features are extracted from the video and event data through a hybrid feature extraction network. Finally, a high-resolution, high-frame-rate video output is generated through an iterative refinement decoding strategy, resulting in traffic monitoring videos with significantly improved clarity. These videos can clearly identify license plates and pedestrian facial features, which helps to improve the accuracy and efficiency of traffic accident analysis.

[0072] In sports event live video processing scenarios, the system acquires live sports event video and event streams captured by high-speed event cameras. It then processes the input video using a dynamic temporal pyramid and a modular feature extractor, employs an image segmentation-based decoding strategy to process athletes and backgrounds separately, and outputs high-quality video through INR-based spatial-temporal decoding. This approach maintains the real-time nature of the live sports video while simultaneously improving its quality, resulting in smoother motion (including enhanced resolution and motion smoothing), clearer images, and a significantly enhanced viewing experience.

[0073] In virtual reality content generation scenarios, video data captured by event cameras is acquired, and 3D rendering content is processed using temporal pyramid representation and hybrid feature extraction networks. INR-based spatial-temporal decoding technology is used to enhance resolution and smoothness. The generated VR (Virtual Reality) content has high definition and smooth motion, significantly improving the user's immersion and experience quality.

[0074] In this application embodiment, the performance of the method proposed in this application embodiment is also verified through verification experiments.

[0075] In this validation experiment, a diverse dataset was used as the training dataset for the video processing model. This diverse dataset contains rich dynamic scenes and different lighting conditions, such as the Adobe 240fps and GoPro datasets, to enhance the model's generalization ability and adaptability. Additionally, data augmentation techniques such as random cropping, rotation, and flipping were employed in this validation experiment to increase the diversity of the training data, thereby improving the model's ability to handle complex scenes.

[0076] In training video processing models, a phased training approach can be adopted. First, pre-training is performed at a lower resolution, then fine-tuning is done by gradually increasing the resolution. This phased training method can effectively accelerate the training process and improve the model's accuracy. For the loss function, a combination of Charbonnier loss and perceptual loss can be used, considering not only pixel-level reconstruction accuracy but also the perceptual quality of the image to achieve better visual results.

[0077] For the hardware and software environment, choose high-performance computing resources and use a computing platform with high-speed memory and multi-core processors, such as a workstation equipped with an NVIDIA RTX 3090 or higher-performance GPU, to ensure training and inference efficiency. For the software environment, development can be carried out under deep learning frameworks such as PyTorch or TensorFlow, which provide rich APIs and optimization tools to facilitate model design and experimental verification.

[0078] In the verification experiments, experiments were conducted using the Adobe dataset, GoPro dataset, TimeLens++ dataset, and CED dataset.

[0079] The results of the super-resolution and frame interpolation experiments on the Adobe dataset and the GoPro dataset are shown in Table 1:

[0080] Table 1

[0081]

[0082]

[0083] In Table 1, M represents the number of parameters; HR-INR represents the video processing method proposed in the embodiments of this application.

[0084] The results of super-resolution and frame interpolation experiments at different magnifications on the Adobe dataset are shown in Table 2:

[0085] Table 2

[0086]

[0087] The results of the frame interpolation experiments at different magnifications on the TimeLens++ dataset are shown in Table 3:

[0088] Table 3

[0089]

[0090] The results of super-resolution experiments at different magnifications on the CED dataset are shown in Table 4:

[0091] Table 4

[0092]

[0093] Figures 4 to 7 show the experimental visualization results corresponding to different datasets. Figure 4 shows the experimental visualization results on the Adobe dataset, Figure 5 shows the experimental visualization results on the TimeLens++ dataset, Figure 6 shows the experimental visualization results on the CED dataset, and Figure 7 shows the experimental visualization results on the TimeLens++ dataset.

[0094] As can be seen from the experimental results shown in Tables 1 to 4 and Figures 5 to 7 above, the method proposed in this application captures the dynamics of a rotating tire more effectively than VideoINR, demonstrating the advantages of the HR-INR framework in handling fast motion and detail recovery.

[0095] The above verification experiments and results demonstrate that the method proposed in this application has achieved significant progress and effects in multiple aspects, including technical, economic, and social aspects, as detailed below:

[0096] The method proposed in this application significantly improves video resolution and frame rate while addressing the challenges of capturing fast and non-linear motion. Compared to existing technologies, the method proposed in this application can achieve arbitrary-scale decoding of video resolution while maintaining high temporal resolution and low latency. Moreover, experimental results from verification experiments show that the method proposed in this application outperforms traditional INR-based methods on four datasets, especially when handling fast motion and complex scenes, with a 35dB improvement in PSNR (Peak Signal-to-Noise Ratio) and a 0.05-0.1 improvement in SSIM (Structural Similarity Index), demonstrating the video processing capabilities and generalization of the method proposed in this application.

[0097] From an economic perspective, by improving the efficiency and quality of video processing technology, the method proposed in this application can bring significant economic benefits to multiple fields such as video content providers, security monitoring systems, and autonomous vehicles. For example, in the field of security monitoring, using the method proposed in this application can provide clearer video images without increasing additional hardware costs, thereby improving the accuracy and efficiency of event detection and reducing economic losses caused by poor monitoring quality.

[0098] In a social context, the method proposed in this application not only improves video quality but also plays a significant role in promoting the development of virtual reality and augmented reality technologies. High-quality video input is key to enhancing the realism of virtual reality / augmented reality experiences. The method proposed in this application can provide users with a more immersive and realistic virtual experience, driving progress in social applications across multiple fields such as education, entertainment, and healthcare.

[0099] In tests on the Adobe 240 dataset, the method proposed in this application significantly outperforms existing technologies in terms of 4x spatial upsampling and 8x video frame interpolation (as shown in Tables 1-4). The method proposed in this application is significantly superior to traditional video super-resolution and frame rate enhancement techniques. Furthermore, in application tests on actual surveillance video datasets, the method proposed in this application was able to recover clearly identifiable facial and license plate information under extremely low light and high-speed motion conditions, demonstrating its effectiveness and reliability in practical applications.

[0100] This application embodiment also provides a video processing device, as shown in FIG8. The device 800 includes: a data acquisition module 801, a first feature extraction module 802, a second feature extraction module 803, a feature fusion module 804, and a video reconstruction module 803.

[0101] The data acquisition module 801 is used to acquire the video to be processed captured by the event camera and the event data corresponding to the video to be processed;

[0102] The first feature extraction module 802 is used to extract event features from the event data to obtain the motion region features corresponding to the video to be processed.

[0103] The second feature extraction module 803 is used to extract frame features from the video to be processed and event data to obtain the overall motion features corresponding to the video to be processed. The overall motion features are used to characterize the temporal and spatial dependencies of the video to be processed.

[0104] The feature fusion module 804 is used to fuse the features of the motion region and the overall motion features to obtain fused features;

[0105] The video reconstruction module 805 is used to reconstruct the video based on fusion features to obtain the target video.

[0106] In one example, the first feature extraction module includes a third feature extraction module and a fourth feature extraction module. The third feature extraction module is used to extract event features from the event data to obtain motion feature information corresponding to the video to be processed; the fourth feature extraction module is used to extract regional event features from the motion feature information to obtain motion region features corresponding to the video to be processed.

[0107] In one example, the third feature extraction module is specifically used to perform multi-level temporal pyramid representation of the event data using the temporal granularity corresponding to each pyramid level in the temporal pyramid model, so as to obtain the motion feature information corresponding to the video to be processed.

[0108] In one example, the video processing device further includes: a parameter adjustment module for acquiring dynamic characteristic data in the video to be processed, wherein the dynamic characteristic data is used to characterize the motion data of moving objects in the video to be processed; and adjusting the level parameters corresponding to each pyramid level in the time pyramid model according to the dynamic characteristic data.

[0109] In one example, the third feature extraction module is specifically used to acquire dynamic characteristic data of at least one moving object in the video to be processed, wherein the dynamic characteristic data is used to characterize the motion data of the moving object in the video to be processed; based on the relationship between the dynamic characteristic data and the time window, a time window corresponding to the dynamic characteristic data of at least one moving object is determined; and the event data is encoded through the time window corresponding to the dynamic characteristic data of at least one moving object to obtain the motion feature information corresponding to the video to be processed.

[0110] In one example, the fourth feature extraction module is specifically used to extract local dynamic change information and edge information from motion feature information through the convolutional layer and multi-head self-attention mechanism of the neural network, so as to obtain the motion region features corresponding to the video to be processed.

[0111] In one example, the video reconstruction module includes: an information acquisition module, a time determination module, a first embedding module, a second embedding module, and a decoding module. The information acquisition module acquires the target frame rate and target resolution of the target video; the time determination module determines the display time of each frame in the target video based on the target frame rate; the first embedding module performs a time embedding operation on the fusion features to obtain a first fusion feature corresponding to the display time; the second embedding module performs a spatial embedding operation on the first fusion feature to obtain a second fusion feature corresponding to the target resolution; and the decoding module decodes the second fusion feature to obtain the target video with the target frame rate and target resolution.

[0112] In one example, the decoding module is specifically used to segment the video to be processed, obtaining multiple regional videos; determine the second fusion feature corresponding to each regional video based on the correlation between the regional videos and the second fusion feature; based on the correlation between the decoding strategy and the second fusion feature, use the corresponding decoding strategy to decode the second fusion feature of the corresponding regional video, resulting in the target regional video for each regional video; and combine the target regional videos corresponding to multiple regional videos to obtain the target video.

[0113] The video processing apparatus provided in this application embodiment can implement all the processes implemented in the foregoing method embodiments, and will not be repeated here to avoid repetition.

[0114] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0115] Figure 9 shows a schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application.

[0116] The electronic device may include a processor 901 and a memory 902 storing computer program instructions.

[0117] Specifically, the processor 901 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0118] Memory 902 may include mass storage for data or instructions. For example, and not limitingly, memory 902 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 902 may include removable or non-removable (or fixed) media. Where appropriate, memory 902 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 902 is non-volatile solid-state memory.

[0119] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the methods according to one aspect of this disclosure.

[0120] The processor 901 implements any of the video processing methods described in the above embodiments by reading and executing computer program instructions stored in the memory 902.

[0121] In one example, the electronic device may also include a communication interface 903 and a bus 910. As shown in Figure 9, the processor 901, memory 902, and communication interface 903 are connected via the bus 910 and communicate with each other.

[0122] The communication interface 903 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0123] Bus 910 includes hardware, software, or both, that couples components of an electronic device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 910 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0124] Furthermore, in conjunction with the video processing methods in the above embodiments, this application embodiment can provide a computer-readable storage medium for implementation. This computer-readable storage medium stores computer program instructions; when executed by a processor, these computer program instructions implement any of the video processing methods in the above embodiments.

[0125] Furthermore, in conjunction with the video processing methods described in the above embodiments, this application can provide a computer program product for implementation. When the instructions in this computer program product are executed by the processor of an electronic device, the electronic device performs any of the video processing methods described in the above embodiments.

[0126] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0127] The functional modules shown in the above-described block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0128] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0129] The foregoing flowcharts and / or block diagrams of video processing methods, apparatuses, devices, storage media, and computer program products according to embodiments of the present disclosure have described various aspects of the present disclosure. It should be understood that each block in the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to create a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowcharts and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0130] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A video processing method, characterized in that, include: Acquire the video to be processed captured by the event camera and the event data corresponding to the video to be processed; Event features are extracted from the event data to obtain the motion region features corresponding to the video to be processed; Frame features are extracted from the video to be processed and the event data to obtain the overall motion features corresponding to the video to be processed, wherein the overall motion features are used to characterize the temporal and spatial dependencies of the video to be processed. The motion region features and the overall motion features are fused to obtain fused features; The target video is obtained by reconstructing the video based on the fusion features.

2. The method according to claim 1, characterized in that, The event data is subjected to event feature extraction to obtain the motion region features corresponding to the video to be processed, including: Event features are extracted from the event data to obtain motion feature information corresponding to the video to be processed; The motion feature information is used to extract regional event features to obtain the motion region features corresponding to the video to be processed.

3. The method according to claim 2, characterized in that, The event data is subjected to event feature extraction to obtain motion feature information corresponding to the video to be processed, including: By employing the temporal granularity corresponding to each pyramid level in the temporal pyramid model, the event data is represented by a multi-level temporal pyramid to obtain the motion feature information corresponding to the video to be processed.

4. The method according to claim 3, characterized in that, Before performing multi-level temporal pyramid representation on the event data using a temporal granularity corresponding to each pyramid level in the temporal pyramid model to obtain the motion feature information corresponding to the video to be processed, the method further includes: The dynamic characteristic data in the video to be processed is obtained, wherein the dynamic characteristic data is used to characterize the motion data of moving objects in the video to be processed; The level parameters corresponding to each pyramid level in the time pyramid model are adjusted based on the dynamic characteristic data.

5. The method according to claim 2, characterized in that, The event data is subjected to event feature extraction to obtain motion feature information corresponding to the video to be processed, including: The dynamic characteristic data of at least one moving object in the video to be processed are obtained, wherein the dynamic characteristic data is used to characterize the motion data of the moving object in the video to be processed; Based on the relationship between the dynamic characteristic data and the time window, determine the time window corresponding to the dynamic characteristic data of the at least one moving object; By encoding the event data using a time window corresponding to the dynamic characteristic data of the at least one moving object, motion feature information corresponding to the video to be processed is obtained.

6. The method according to claim 2, characterized in that, The motion feature information is used to extract regional event features to obtain the motion region features corresponding to the video to be processed, including: By using the convolutional layers and multi-head self-attention mechanism of the neural network, local dynamic change information and edge information are extracted from the motion feature information to obtain the motion region features corresponding to the video to be processed.

7. The method according to claim 1, characterized in that, Based on the fusion features, video reconstruction is performed to obtain the target video, including: Obtain the target frame rate and target resolution corresponding to the target video; The display time corresponding to each frame in the target video is determined based on the target frame rate; Perform a time embedding operation on the fusion feature to obtain the first fusion feature corresponding to the display time; Perform a spatial embedding operation on the first fused feature to obtain the second fused feature corresponding to the target resolution; The second fusion feature is decoded to obtain a target video with the target frame rate and the target resolution.

8. The method according to claim 7, characterized in that, Decoding the second fused feature to obtain a target video with the target frame rate and the target resolution includes: The video to be processed is segmented to obtain multiple video regions; Based on the correlation between the regional video and the second fusion feature, determine the second fusion feature corresponding to each regional video; Based on the correlation between the decoding strategy and the second fusion feature, the corresponding decoding strategy is used to decode the second fusion feature of the corresponding regional video, wherein each regional video corresponds to the target regional video; The target video is obtained by combining the target region videos corresponding to the multiple region videos.

9. A video processing apparatus, characterized in that, include: The data acquisition module is used to acquire the video to be processed captured by the event camera and the event data corresponding to the video to be processed; The first feature extraction module is used to extract event features from the event data to obtain the motion region features corresponding to the video to be processed. The second feature extraction module is used to extract frame features from the video to be processed and the event data to obtain the overall motion features corresponding to the video to be processed, wherein the overall motion features are used to characterize the temporal and spatial dependencies of the video to be processed. The feature fusion module is used to fuse the motion region features and the overall motion features to obtain fused features; The video reconstruction module is used to reconstruct the video based on the fusion features to obtain the target video.

10. An electronic device, characterized in that, Electronic devices include: processors and memory storing computer program instructions; When the processor executes the computer program instructions, it implements the video processing method as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, A computer-readable storage medium stores computer program instructions that, when executed by a processor, implement the video processing method as described in any one of claims 1-8.

12. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device causes the electronic device to perform the video processing method as described in any one of claims 1-8.

Images