Video processing method, apparatus, device, storage medium and computer program product

Abstract

A video processing method, device, equipment, storage medium and computer program product are disclosed. The method comprises: obtaining a to-be-processed video collected by an event camera and event data corresponding to the to-be-processed video; performing event feature extraction on the event data to obtain motion region features corresponding to the to-be-processed video; performing frame feature extraction on the to-be-processed video and the event data to obtain motion overall features corresponding to the to-be-processed video, wherein the motion overall features are used to represent a dependency between time and space of the to-be-processed video; performing feature fusion on the motion region features and the motion overall features to obtain fused features; and performing video reconstruction based on the fused features to obtain a target video. The method disclosed in the present application can improve the resolution and frame rate of the video after recovery processing, and improve the video quality.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision, and particularly relates to a video processing method and device, equipment, storage medium and computer program product. BACKGROUND

[0002] Implicit Neural Representation (INR) is a method of representing input images, audio and point clouds and the like information as a function through a neural network, which can be used in video repair and the like scenarios, and represents images or videos as implicit fields to decode at any resolution.

[0003] However, in the related art, the INR-based C-STVSR (Continuous Space-Time Video Super-Resolution) method usually assumes that there is linear motion between frames, and uses interpolation or feature warping to generate features at any spatial-temporal location between two frames. This way limits the ability of C-STVSR to capture long-term dependencies in fast and non-linear motion and complex dynamic scenes, thereby reducing the resolution and frame rate of the video. SUMMARY

[0004] The embodiments of the present application provide a video processing method, device, equipment, storage medium and computer program product, which can improve the resolution and frame rate of the processed video and improve the video quality.

[0005] In a first aspect, the embodiments of the present application provide a video processing method, which comprises: acquiring a to-be-processed video collected by an event camera and event data corresponding to the to-be-processed video; performing event feature extraction on the event data to obtain motion region features corresponding to the to-be-processed video; performing frame feature extraction on the to-be-processed video and the event data to obtain motion overall features corresponding to the to-be-processed video, wherein the motion overall features are used to represent the dependency between time and space of the to-be-processed video; performing feature fusion on the motion region features and the motion overall features to obtain fused features; and performing video reconstruction based on the fused features to obtain a target video.

[0006] In a second aspect, an embodiment of the present application provides a video processing apparatus, comprising: a data acquisition module configured to acquire a to-be-processed video captured by an event camera and event data corresponding to the to-be-processed video; a first feature extraction module configured to perform event feature extraction on the event data to obtain motion region features corresponding to the to-be-processed video; a second feature extraction module configured to perform frame feature extraction on the to-be-processed video and the event data to obtain motion overall features corresponding to the to-be-processed video, wherein the motion overall features are used to represent a dependency between time and space of the to-be-processed video; a feature fusion module configured to perform feature fusion on the motion region features and the motion overall features to obtain fused features; and a video reconstruction module configured to perform video reconstruction based on the fused features to obtain a target video.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory storing computer program instructions; and the processor executes the computer program instructions to implement the video processing method according to the first aspect.

[0008] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the video processing method according to the first aspect.

[0009] In a fifth aspect, an embodiment of the present application provides a computer program product, and instructions in the computer program product are executed by a processor of an electronic device to cause the electronic device to perform the video processing method according to the first aspect.

[0010] From the above, in the embodiment of the present application, the event data corresponding to the to-be-processed video contains rich event information, and by performing event feature extraction on the event data in the to-be-processed video, the nonlinear motion and long-term dependency of the moving object in the to-be-processed video can be captured, which provides a basis for improving the resolution and frame rate of the to-be-processed video. In addition, in the embodiment of the present application, by performing feature extraction on the event data, the dynamic motion and edge details in the to-be-processed video can be captured, and by performing motion overall feature extraction on the to-be-processed video and the event data, long-term motion dependency can be modeled, and the features extracted by the above two feature extraction methods can comprehensively understand the dynamic changes of the video content, thereby improving the resolution and frame rate of the reconstructed target video.

[0011] Therefore, the method provided in the embodiment of the present application can obtain a target video with high resolution and frame rate by processing the to-be-processed video. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments of the present application will be briefly introduced. Those drawings can help the ordinary skilled in the art to obtain other drawings without creative effort.

[0013] Figure 1 is a flow diagram of a video processing method provided by an embodiment of the present application;

[0014] Figure 2 is an architecture diagram of a video processing model provided by an embodiment of the present application;

[0015] Figure 3 is a structure diagram of a pyramid model provided by an embodiment of the present application;

[0016] Figure 4 is a visualization result diagram of an experiment on an Adobe dataset provided by an embodiment of the present application;

[0017] Figure 5 is a visualization result diagram of an experiment on a TimeLens++ dataset provided by an embodiment of the present application;

[0018] Figure 6 is a visualization result diagram of an experiment on a CED dataset provided by an embodiment of the present application;

[0019] Figure 7 is a visualization result diagram of an experiment on a TimeLens++ dataset provided by an embodiment of the present application;

[0020] Figure 8 is a structure diagram of a video processing device provided by another embodiment of the present application;

[0021] Figure 9 is a structure diagram of an electronic device provided by yet another embodiment of the present application. DETAILED DESCRIPTION

[0022] The features and exemplary embodiments of various aspects of the present application will be described in detail below, in order to make the purposes, technical solutions and advantages of the present application more clear and apparent, the present application will be further described in detail below in combination with the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, but not to limit the present application. The present application can be implemented without some of these specific details by those skilled in the art. The following description of the embodiments is only to provide a better understanding of the present application by showing examples of the present application.

[0023] It is to be noted that, in the present document, relational terms such as first and second, and the like, are used solely to distinguish one entity or action from another entity or action, without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0024] For the purpose of facilitating understanding, before the scheme provided by the present application is explained and described, the background of the scheme provided by the present application is first explained and described.

[0025] In the related art, due to the limitation of I / O (Input / Output) bandwidth and sensor size, the camera system can only record video at a fixed resolution and frame rate, which causes an impact on the application of computer vision, for example, immersive experience in virtual reality, traffic analysis in autonomous driving, etc. To overcome this limitation, in the related art, continuous space-time video super-resolution (C-STVSR) can be used to simultaneously improve the resolution and frame rate of the video.

[0026] In recent years, implicit neural representations have been applied to video inpainting, by representing images or videos as implicit fields, which can be decoded at arbitrary resolutions. However, the existing INR-based C-STVSR methods have limitations, which usually assume linear motion between frames and use interpolation or feature warping to generate features at arbitrary space-time locations between two frames, which limits the ability of C-STVSR to capture fast and non-linear motion and long-term dependencies (involving more than two frames) in complex dynamic scenes.

[0027] To solve the technical problems in the prior art, the embodiments of the present application provide a video processing method, device, equipment, storage medium and computer program product. First, the video processing method provided by the embodiments of the present application is introduced.

[0028] The method provided in the embodiments of the present application aims to capture the overall dependence and regional motion based on INR, with the assistance of an event camera with high temporal resolution and low delay. By designing a feature extraction framework, the rich temporal information provided by event data is fully utilized to capture regional nonlinear motion and long-term dependence, and the capture of long-term dependence is realized through spatial-temporal embedding.

[0029] Figure 1 A flowchart of a video processing method provided by an embodiment of the present application is shown. As shown in the figure, the method comprises the following steps: Figure 1

[0030] In step S101, event data corresponding to the to-be-processed video is obtained.

[0031] In step S101, the event camera is a camera that collects images through a dynamic vision sensor, wherein the event camera can collect "events" (i.e., event data) in the image generation process, for example, the change of pixel brightness. In the embodiments of the present application, each event data is used to represent the pixel intensity change information at a specific time and location in the video.

[0032] In step S101, the to-be-processed video can be a video to be repaired, for example, a blurred video collected by the event camera. In the embodiments of the present application, the to-be-processed video comprises a plurality of RGB frames.

[0033] It should be noted that in the embodiments of the present application, the event camera has the characteristics of high temporal resolution and low delay, and based on this, the use of the event camera to collect the to-be-processed video and the event data of the to-be-processed video can obtain the subtle motion changes in the to-be-processed video, thereby providing a basis for the recovery of the to-be-processed video.

[0034] In step S102, event feature extraction is performed on the event data to obtain motion region features corresponding to the to-be-processed video.

[0035] In step S102, a regional event feature extractor can be used to extract features from the event data, and motion region features containing dynamic motion and edge details can be obtained. That is, by performing feature extraction on the event data in step S102, the detailed information of the motion in the to-be-processed video can be obtained, which can improve the accuracy and precision of video recovery and ensure the resolution and frame rate of the video.

[0036] In step S103, frame feature extraction is performed on the to-be-processed video and the event data to obtain motion overall features corresponding to the to-be-processed video.

[0037] In step S103, the motion overall features are used to represent the dependence between the time and space of the to-be-processed video.​

[0038] As an example, a global event feature extractor can be employed to extract motion global features corresponding to the to-be-processed video, where the global event feature extractor can be a Swin Transformer-based feature extractor capable of modeling long-term dependencies in time and space in a larger range.

[0039] In step S104, the motion region features and the motion global features are fused to obtain fused features.

[0040] It should be noted that the motion region features and the motion global features can reflect the content dynamic change information of the to-be-processed video, and thus, after the motion region features and the motion global features are extracted, the two features are fused, and the video is reconstructed based on the fused features, which can make the reconstructed video clearly show the motion details in the video, and improve the resolution and frame rate of the video.

[0041] In step S105, the video is reconstructed based on the fused features to obtain a target video.

[0042] In step S105, the video can be reconstructed by using the INR technology to spatially and temporally decode the fused features. In the process of video reconstruction, the learned temporal embedding is introduced as an attention vector, and the spatial embedding and decoding technology are introduced, so that the video frames can be effectively reconstructed without estimating the optical flow, thereby improving the efficiency and accuracy of video reconstruction and improving the resolution and frame rate of the reconstructed video.

[0043] Based on the scheme defined in steps S101 to S105, it can be known that in the embodiments of the present application, the event data corresponding to the to-be-processed video contains rich event information, and by extracting event features from the event data in the to-be-processed video, the nonlinear motion and long-term dependencies of the moving objects in the to-be-processed video can be captured, thereby providing a basis for improving the resolution and frame rate of the to-be-processed video. In addition, in the embodiments of the present application, the dynamic motion and edge details in the to-be-processed video can be captured by extracting features from the event data, and the long-term motion dependency relationship can be modeled by extracting motion global features from the to-be-processed video and the event data. By combining the features extracted by the above two feature extraction methods, the dynamic changes of the video content can be comprehensively understood, thereby improving the resolution and frame rate of the reconstructed target video.

[0044] As can be seen, the method proposed in the embodiments of the present application can obtain a target video with high resolution and frame rate by processing the to-be-processed video.

[0045] In one embodiment, Figure 2An architecture diagram of a video processing model implementing the method proposed in the embodiments of the present application is shown, which comprises Figure 2 It can be seen that the video processing model mainly comprises a data representation unit, a regional event feature extractor, an overall event feature extractor, and a decoding unit. The data representation unit is configured to perform data representation on event data to more accurately capture motion changes in the video. The regional event feature extractor is configured to extract local dynamic change information and edge information in the event data. The overall event feature extractor is configured to model long-term dependencies in time and space. The decoding unit is configured to decode data to implement video reconstruction.

[0046] The implementation steps of the method proposed in the embodiments of the present application are explained below. Figure 2 The implementation steps of the method proposed in the embodiments of the present application are explained below.

[0047] First, after obtaining the to-be-processed video and event data collected by the event camera, event feature extraction is performed on the event data to obtain motion feature information corresponding to the to-be-processed video. Then, regional event feature extraction is performed on the motion feature information to obtain motion region features corresponding to the to-be-processed video.

[0048] In one embodiment, the data representation unit can be used to perform event feature extraction on the event data to obtain motion feature information corresponding to the to-be-processed video. Specifically, the data representation unit uses a time granularity corresponding to each pyramid level in a time pyramid model to perform multi-level time pyramid representation on the event data to obtain motion feature information corresponding to the to-be-processed video.

[0049] For example, in the embodiment shown in FIG. 1, the data representation unit 110 is configured to perform event feature extraction on the event data collected by the event camera 100 to obtain motion feature information corresponding to the to-be-processed video. Figure 2 For example, in the embodiment shown in FIG. 1, the data representation unit 110 is configured to perform event feature extraction on the event data collected by the event camera 100 to obtain motion feature information corresponding to the to-be-processed video. Figure 3 For example, in the embodiment shown in FIG. 1, the data representation unit 110 is configured to perform event feature extraction on the event data collected by the event camera 100 to obtain motion feature information corresponding to the to-be-processed video. For example, in the embodiment shown in FIG. 1, the data representation unit 110 is configured to perform event feature extraction on the event data collected by the event camera 100 to obtain motion feature information corresponding to the to-be-processed video. For example, in the embodiment shown in FIG. 1, the data representation unit 110 is configured to perform event feature extraction on the event data collected by the event camera 100 to obtain motion feature information corresponding to the to-be-processed video. For example, in the embodiment shown in FIG. 1, the data representation unit 110 is configured to perform event feature extraction on the event data collected by the event camera 100 to obtain motion feature information corresponding to the to-be-processed video. For example, in the embodiment shown in FIG. 1, the data representation unit 110 is configured to perform event feature extraction on the event data collected by the event camera 100 to obtain motion feature information corresponding to the to-be-processed video. For example, in the embodiment shown in FIG. 1, the data representation unit 110 is configured to perform event feature extraction on the event data collected by the event camera 100 to obtain motion feature information corresponding to the to-be-processed video.

[0050] In the embodiments of the present application, the pyramid model can have a structure of 7 pyramid levels, in which case the pyramid model can most effectively capture fast and small dynamic changes. Each pyramid level represents a different time granularity, thereby ensuring that the motion information in the video can be comprehensively understood.

[0051] In addition, a dynamically adjusted resolution (based on 1 / 2, 1 / 4, 1 / 8, etc. scaling of the original video resolution) can also be used for each pyramid level to reduce the computational burden while preserving sufficient motion details.

[0052] It should be noted that, in the embodiments of the present application, the pyramid model is used to perform time pyramid representation on the event data, which can achieve a finer time granularity, effectively reduce the computational complexity, and meanwhile preserve important motion information.

[0053] To further improve the accuracy of the time pyramid representation of the event data by the pyramid model, in actual applications, the pyramid model can also be dynamically adjusted according to the video content of the to-be-processed video.

[0054] In one embodiment, dynamic characteristic data in the to-be-processed video is acquired, and then the level parameters corresponding to each pyramid level in the time pyramid model are adjusted according to the dynamic characteristic data.

[0055] In the above embodiment, the dynamic characteristic data is used to represent the motion data of the motion object in the to-be-processed video, such as motion duration, motion region, etc., and the speed of the motion. The parameters of the pyramid level include at least one of the following: time span, time granularity.

[0056] As an example, the time span and / or the time granularity of each pyramid level can be adaptively adjusted according to the dynamic characteristic data in the to-be-processed video, so as to more accurately capture the motion changes in the video.

[0057] In another embodiment, a multi-scale time window can also be used to perform event feature extraction on the event data to obtain the motion feature information corresponding to the to-be-processed video.

[0058] Specifically, dynamic characteristic data of at least one motion object in the to-be-processed video is acquired, and then a time window corresponding to the dynamic characteristic data of the at least one motion object is determined according to the relationship between the dynamic characteristic data and the time window. The event data is encoded through the time window corresponding to the dynamic characteristic data of the at least one motion object to obtain the motion feature information corresponding to the to-be-processed video.

[0059] In the above embodiment, the dynamic characteristic data is used to represent the motion data of the moving object in the to-be-processed video. In this embodiment, the event data can be encoded using different sizes of time windows at the same time to capture various motion speeds from fast to slow, thereby enhancing the adaptability of the video processing model to different motion patterns. Further, in actual applications, after obtaining the event data, the video processing model can obtain a target time window matched with the event data from a plurality of time windows, and then use the target time window to encode the event data, thereby obtaining the motion characteristic information corresponding to the to-be-processed video.

[0060] Further, as shown in Figure 2 After obtaining the motion characteristic information corresponding to the to-be-processed video, the regional event feature extractor extracts the regional event feature of the motion characteristic information. Specifically, the regional event feature extractor extracts the local dynamic change information and edge information in the motion characteristic information through the convolution layer and the multi-head self-attention mechanism of the neural network, thereby obtaining the motion regional feature corresponding to the to-be-processed video.

[0061] It should be noted that, since the regional event feature extractor adopts the convolution layer and the multi-head attention mechanism, the local dynamic change information and the edge information can be captured, thereby providing a basis for improving the resolution and frame rate of the recovered video.

[0062] In addition, it should be further noted that, in actual applications, the regional event feature extractor can also be a hybrid feature extraction network or a modular feature extractor. The hybrid feature extraction network combines different types of network structures such as convolutional neural networks and graph neural networks to fully utilize their advantages in processing spatial information and modeling long-distance dependencies. The modular feature extractor allows different feature extraction modules to be dynamically combined and selected according to the characteristics of the input data, so as to adapt to diversified video content and scenes through modular design.

[0063] In one embodiment, as shown in Figure 2 The overall event feature extractor extracts the long-term dependencies between time and space in the to-be-processed video and the event data to determine the overall motion trend of the moving object in the to-be-processed video. After obtaining the motion regional feature and the motion overall feature, the two features are fused to obtain the fused feature, which is then input into the decoding unit to realize video reconstruction.

[0064] Specifically, the decoding unit acquires the target frame rate and target resolution corresponding to the target video as input, and determines the display time corresponding to each frame in the target video based on the target frame rate; then, it performs a temporal embedding operation on the fusion features to obtain the first fusion feature corresponding to the display time; next, it performs a spatial embedding operation on the first fusion feature to obtain the second fusion feature corresponding to the target resolution; finally, it performs a decoding operation on the second fusion feature to obtain the target video with the target frame rate and target resolution.

[0065] In the above embodiments, the target frame rate and target resolution can be input by the user according to actual needs. The target frame rate and target resolution are the frame rate and resolution of the target video after video reconstruction of the video to be processed.

[0066] by Figure 2 For example, after obtaining the fusion feature R t Next, a temporal embedding operation is performed on the fused features, that is, the timestamp corresponding to the target frame rate is embedded into the fused features to capture features at the aforementioned timestamp; then, the target resolution is embedded into the first fused feature R. ts In this process, spatial embedding is performed to obtain the second fused feature; then, through a multilayer MLP (Multilayer Perceptron) (for example, ...), Figure 2 The target video can be obtained by decoding the second fused feature using a four-layer perceptron.

[0067] It should be noted that in practical applications, an iterative refinement approach can be used for decoding. That is, based on the initial space-time decoding, an iterative refinement mechanism is introduced, which involves repeating the decoding process multiple times in the above manner. Through multiple rounds of refinement processing, the quality and accuracy of the reconstructed video frames are gradually improved.

[0068] In another embodiment, an image segmentation-based decoding strategy can also be used to decode the second fusion feature. Specifically, firstly, the video to be processed is segmented to obtain multiple regional videos; then, based on the correlation between the regional videos and the second fusion feature, the second fusion feature corresponding to each regional video is determined; next, based on the correlation between the decoding strategy and the second fusion feature, the corresponding decoding strategy is used to decode the second fusion feature of the corresponding regional video, resulting in a target regional video for each regional video; finally, the target regional videos corresponding to multiple regional videos are combined to obtain the target video.

[0069] It should be noted that the image segmentation-based decoding strategy first segments the video frames and then uses different decoding strategies for different regions, especially for dynamic and static backgrounds, in order to achieve higher quality video reconstruction.

[0070] The method proposed in the embodiments of the present application can be applied to but is not limited to the urban traffic monitoring video enhancement scene, the sports event live video processing scene and the virtual reality content generation scene. The method proposed in the embodiments of the present application is exemplarily described below taking the three scenes as examples.

[0071] In the urban traffic monitoring video enhancement scene, a low-resolution and low-frame-rate urban traffic monitoring video and related event data captured by an event camera are obtained, the event data are processed by using a temporal pyramid representation, the video and the event data are subjected to feature extraction by using a hybrid feature extraction network, and finally a high-resolution and high-frame-rate video output is generated by using an iterative refinement decoding strategy, so that a traffic monitoring video with significantly improved definition is obtained, and the license plate and the facial features of pedestrians can be clearly identified, which helps to improve the accuracy and efficiency of traffic accident analysis.

[0072] In the sports event live video processing scene, a live sports event video and an event stream thereof captured by a high-speed event camera are obtained, the input video is processed by using a dynamic temporal pyramid and a modular feature extractor, the players and the background are processed respectively by using an image segmentation-based decoding strategy, and a high-quality video is output by using an INR-based spatial-temporal decoding. In the above manner, the quality of the sports event live video is improved in real time while the real-time performance of the live video is maintained, the action (including enhanced resolution and smooth action) is more fluent, the picture is clearer, and the viewing experience of the audience is greatly enhanced.

[0073] In the virtual reality content generation scene, video data captured by an event camera are obtained, 3D rendering content is processed by using a temporal pyramid representation and a hybrid feature extraction network, and the resolution and the smoothness are enhanced by using an INR-based spatial-temporal decoding technology, so that the generated VR (Virtual Reality) content has high definition and fluent action, and the immersion and the quality of experience of the user are significantly improved.

[0074] In the embodiments of the present application, the performance of the method proposed in the embodiments of the present application is also verified by a verification experiment.

[0075] In the verification experiment, a diversity dataset is used as a training dataset of a video processing model, the diversity dataset contains rich dynamic scenes and different light conditions, for example, Adobe240fps, GoPro and other datasets, so as to enhance the generalization ability and adaptability of the model. In addition, in the verification experiment, random cropping, rotation and flipping and other data enhancement techniques can be used to increase the diversity of the training data, so as to improve the processing capability of the model for complex scenes.

[0076] In the process of training the video processing model, a phased training method can be used, which first pre-trains at a lower resolution and then fine-tunes gradually by increasing the resolution. This phased training method can effectively accelerate the training process and improve the accuracy of the model. For the loss function, a combination of Charbonnier loss and perceptual loss can be used, which not only considers the pixel-level reconstruction accuracy, but also pays attention to the perceptual quality of the image, to achieve better visual effects.

[0077] For hardware and software environments, high-performance computing resources are selected, and computing platforms with high-speed memory and multi-core processors are used, such as workstations equipped with NVIDIA RTX 3090 or higher performance GPUs, to ensure the efficiency of training and inference. For software environments, deep learning frameworks such as PyTorch or TensorFlow can be used for development, which provide rich APIs and optimization tools for model design and experimental verification.

[0078] In the verification experiment, Adobe dataset, GoPro dataset, TimeLens++ dataset, and CED dataset were used for experiments.

[0079] The results of super-resolution and frame interpolation experiments on the Adobe dataset and GoPro dataset are shown in Table 1:

[0080] Table 1

[0081]

[0082]

[0083] In Table 1, M is the number of parameters; HR-INR is the video processing method proposed in the present application.

[0084] The results of super-resolution and frame interpolation experiments of different multiples on the Adobe dataset are shown in Table 2:

[0085] Table 2

[0086]

[0087] The results of frame interpolation experiments of different multiples on the TimeLens++ dataset are shown in Table 3:

[0088] Table 3

[0089]

[0090] The results of super-resolution experiments of different multiples on the CED dataset are shown in Table 4:

[0091] Table 4

[0092]

[0093] Figures 4 to 7 The experimental visualization results corresponding to different data sets are shown, wherein, Figure 4 The experimental visualization results on the Adobe data set are shown, Figure 5 The experimental visualization results on the TimeLens++ data set are shown, Figure 6 The experimental visualization results on the CED data set are shown, Figure 7 The experimental visualization results on the TimeLens++ data set are shown.

[0094] From the experimental results shown in Tables 1-4 above and Figures 5 to 7 It can be seen from the experimental results shown above that the method proposed in the application can more effectively capture the dynamics of the rotating tire than VideoINR, and demonstrates the advantages of the HR-INR framework in handling fast action and detail recovery.

[0095] From the above verification experiments and experimental results, the method proposed in the embodiments of the application has made significant progress and effect in technology, economy and society, as follows:

[0096] The method proposed in the embodiments of the application significantly improves the resolution and frame rate of the video, while solving the challenges of fast and nonlinear motion capture. Compared with the prior art, the method proposed in the embodiments of the application can realize arbitrary scale decoding of video resolution while maintaining high temporal resolution and low delay. Moreover, as can be seen from the experimental results of the verification experiments, the performance of the method proposed in the embodiments of the application on the four data sets exceeds that of the traditional INR-based method, especially in handling fast motion and complex scenes, the PSNR (Peak Signal-to-Noise Ratio) is improved by 35dB, and the SSIM (Structural Similarity) is improved by 0.050.1, proving the video processing capability and generalization of the method proposed in the embodiments of the application.

[0097] In terms of economy, by improving the efficiency and quality of video processing technology, the method proposed in the embodiments of the application can bring significant economic benefits to video content providers, security monitoring systems, autonomous vehicles and other fields. For example, in the field of security monitoring, using the method proposed in the embodiments of the application can provide clearer video pictures without increasing additional hardware costs, thereby improving the accuracy and efficiency of event detection and reducing economic losses caused by poor monitoring quality.

[0098] In terms of society, the method proposed in the embodiments of the present application has important significance in improving video quality and promoting the development of virtual reality and augmented reality technology. High-quality video input is the key to enhancing the realism of virtual reality / augmented reality experience. The method proposed in the embodiments of the present application can provide users with a more immersive and realistic virtual experience, and promote the progress of social applications in education, entertainment, medical treatment and other fields.

[0099] In the test on the Adobe240 dataset, the method proposed in the embodiments of the present application has a significant improvement compared with the prior art when performing 4 times spatial upsampling and 8 times video frame interpolation (as shown in Tables 1-4). The method proposed in the embodiments of the present application is significantly better than the traditional video super-resolution and frame rate improvement technology. In addition, in the application test on the actual monitoring video dataset, the method proposed in the embodiments of the present application can restore clear and distinguishable face and license plate information under extremely low light and high-speed motion conditions, which confirms its effectiveness and reliability in actual application.

[0100] The embodiments of the present application also provide a video processing device, as shown in Figure 8 The device 800 includes a data acquisition module 801, a first feature extraction module 802, a second feature extraction module 803, a feature fusion module 804, and a video reconstruction module 805.

[0101] The data acquisition module 801 is configured to acquire a to-be-processed video collected by an event camera and event data corresponding to the to-be-processed video.

[0102] The first feature extraction module 802 is configured to perform event feature extraction on the event data to obtain motion region features corresponding to the to-be-processed video.

[0103] The second feature extraction module 803 is configured to perform frame feature extraction on the to-be-processed video and the event data to obtain motion overall features corresponding to the to-be-processed video, wherein the motion overall features are used to represent the dependency between time and space of the to-be-processed video.

[0104] The feature fusion module 804 is configured to perform feature fusion on the motion region features and the motion overall features to obtain fused features.

[0105] The video reconstruction module 805 is configured to perform video reconstruction based on the fused features to obtain a target video.

[0106] In an example, the first feature extraction module comprises: a third feature extraction module and a fourth feature extraction module. The third feature extraction module is configured to perform event feature extraction on the event data to obtain motion feature information corresponding to the to-be-processed video. The fourth feature extraction module is configured to perform regional event feature extraction on the motion feature information to obtain motion region features corresponding to the to-be-processed video.

[0107] In an example, the third feature extraction module is specifically configured to perform multi-level time pyramid representation on the event data by using a time granularity corresponding to each pyramid level in the time pyramid model, to obtain the motion feature information corresponding to the to-be-processed video.

[0108] In an example, the video processing apparatus further comprises a parameter adjustment module configured to obtain dynamic characteristic data in the to-be-processed video, wherein the dynamic characteristic data is used to represent motion data of a motion object in the to-be-processed video; and adjust a level parameter corresponding to each pyramid level in the time pyramid model according to the dynamic characteristic data.

[0109] In an example, the third feature extraction module is specifically configured to obtain dynamic characteristic data of at least one motion object in the to-be-processed video, wherein the dynamic characteristic data is used to represent motion data of a motion object in the to-be-processed video; determine a time window corresponding to the dynamic characteristic data of the at least one motion object according to a relationship between the dynamic characteristic data and the time window; and encode the event data by using the time window corresponding to the dynamic characteristic data of the at least one motion object to obtain the motion feature information corresponding to the to-be-processed video.

[0110] In an example, the fourth feature extraction module is specifically configured to extract local dynamic change information and edge information in the motion feature information by using a convolution layer and a multi-head self-attention mechanism of a neural network, to obtain the motion region features corresponding to the to-be-processed video.

[0111] In an example, the video reconstruction module comprises: an information acquisition module, a time determination module, a first embedding module, a second embedding module, and a decoding module. The information acquisition module is configured to obtain a target frame rate and a target resolution corresponding to a target video. The time determination module is configured to determine a display time corresponding to each image in the target video according to the target frame rate. The first embedding module is configured to perform time embedding operation on the fusion feature to obtain first fusion feature corresponding to the display time. The second embedding module is configured to perform spatial embedding operation on the first fusion feature to obtain second fusion feature corresponding to the target resolution. The decoding module is configured to perform decoding operation on the second fusion feature to obtain the target video with the target frame rate and the target resolution.

[0112] In one example, the decoding module is specifically configured to perform segmentation processing on the video to be processed to obtain a plurality of region videos; determine the second fusion feature corresponding to each region video according to the association relationship between the region video and the second fusion feature; perform decoding operation on the second fusion feature of the corresponding region video according to the association relationship between the decoding strategy and the second fusion feature, to obtain a target region video corresponding to each region video; and perform combination processing on the target region videos corresponding to the plurality of region videos to obtain the target video.

[0113] The video processing apparatus provided by the embodiments of the present application can realize each process realized by the method embodiments, and thus, for the sake of brevity and conciseness, the details are not described herein.

[0114] Those skilled in the art can clearly understand that, for the sake of brevity and conciseness, only the division of the above functional units and modules is exemplified, and in actual applications, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiments can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit, and the integrated unit can be realized in the form of hardware or in the form of software functional unit. In addition, the specific names of the functional units and modules are only for the purpose of mutual distinction, and do not limit the protection scope of the present application. The specific working process of the units and modules in the system can refer to the corresponding process in the foregoing method embodiments, and thus, the details are not described herein.

[0115] Figure 9 A hardware structure schematic diagram of an electronic device provided by the embodiments of the present application is shown.

[0116] The electronic device can include a processor 901 and a memory 902 having computer program instructions stored therein.

[0117] Specifically, the processor 901 can include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or can be configured as one or more integrated circuits that implement one or more embodiments of the present application.

[0118] The memory 902 can include mass storage for data or instructions. As an example and not by way of limitation, the memory 902 can include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disc (e.g., a compact disc (CD) or a digital versatile disc (DVD)), a solid-state drive (SSD), a USB drive, or a combination of two or more of these. Where appropriate, the memory 902 can include removable or non-removable (or fixed) media, where appropriate. The memory 902 can be internal or external to the integrated gateway disaster recovery appliance. In particular embodiments, the memory 902 is non-volatile, solid-state memory.

[0119] The memory can include read-only memory (ROM), random-access memory (RAM), magnetic disk storage mediums, optical storage mediums, flash memory devices, electrical, optical, or other physically tangible / moφhological memory storage devices. Thus, in general, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software that, when executed (by one or more processors), is operable to

[0120] The processor 901 implements any of the video processing methods in the above embodiments by reading and executing computer program instructions stored in the memory 902.

[0121] In one example, the electronic device further includes a communication interface 903 and a bus 910. As shown, the processor 901, the memory 902, and the communication interface 903 are connected through the bus 910 and complete communication among each other. Figure 9

[0122] The communication interface 903 is mainly used to realize the communication between the modules, devices, units and / or equipment in the embodiments of the present application.

[0123] ​Bus 910 includes a hardware, software, or both that couples components of electronic device to each other. As an example and not by way of limitation, bus can include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand (IB) interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or another suitable bus or a combination of two or more of these. Where appropriate, bus 910 can include one or more buses. Although this application describes and shows a particular bus, this application contemplates any suitable bus or interconnect.

[0124] In addition, in combination with the video processing method in the above embodiments, the embodiments of the present application can provide a computer readable storage medium to implement. The computer readable storage medium has computer program instructions stored thereon; the computer program instructions are executed by a processor to implement any one of the video processing methods in the above embodiments.

[0125] In addition, in combination with the video processing method in the above embodiments, the embodiments of the present application can provide a computer program product to implement. The instructions in the computer program product are executed by the processor of the electronic device, so that the electronic device executes any one of the video processing methods as described in the above embodiments.

[0126] It needs to be clear that the present application is not limited to the specific configurations and processes described above and shown in the drawings. For the sake of brevity, detailed descriptions of well-known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between steps, after understanding the spirit of the present application.

[0127] The functional modules shown in the structural block diagram described above can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a functional card, and the like. When implemented in software, the elements of the present application are program or code segments that are used to perform the required tasks. The program or code segments can be stored in a machine-readable medium or transmitted through a data signal carried in a carrier wave over a transmission medium or communication link. The "machine-readable medium" can include any medium that can store or transfer information. Examples of the machine-readable medium include an electronic circuit, a semiconductor memory device, a ROM, a flash memory, an erasable ROM (EROM), a floppy diskette, a CD-ROM, an optical disk, a hard disk, a fiber optic medium, a radio frequency (RF) link, and the like. The code segments can be downloaded via a computer network, such as the Internet, an intranet, and the like.

[0128] It is also necessary to note that the exemplary embodiments mentioned in the present application describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiments, or in an order different from the embodiments, or several steps can be performed simultaneously.

[0129] The above describes aspects of the present disclosure with reference to flowcharts and / or block diagrams of a video processing method, apparatus, device, storage medium, and computer program product according to embodiments of the present disclosure. It should be understood that each block in the flowchart and / or block diagram, and combinations of blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus enable the implementation of the functions / acts specified in one or more blocks of the flowchart and / or block diagram. The processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can also be implemented by special hardware for performing specified functions or acts, or can be implemented by a combination of special hardware and computer instructions.

[0130] The above merely describes a specific implementation of the present application. Those skilled in the art can clearly understand the specific working processes of the system, modules and units described above for the convenience and brevity of description, and can refer to the corresponding processes in the foregoing method embodiments, which will not be described herein again. It should be understood that the protection scope of the present application is not limited to this, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application.

Claims

1. A video processing method, characterized in that, include: Acquire the video to be processed captured by the event camera and the event data corresponding to the video to be processed; Event features are extracted from the event data to obtain the motion region features corresponding to the video to be processed; Frame features are extracted from the video to be processed and the event data to obtain the overall motion features corresponding to the video to be processed, wherein the overall motion features are used to characterize the temporal and spatial dependencies of the video to be processed. The motion region features and the overall motion features are fused to obtain fused features; The target video is obtained by reconstructing the video based on the fusion features.

2. The method according to claim 1, characterized in that, The event data is subjected to event feature extraction to obtain the motion region features corresponding to the video to be processed, including: Event features are extracted from the event data to obtain motion feature information corresponding to the video to be processed; The motion feature information is used to extract regional event features to obtain the motion region features corresponding to the video to be processed.

3. The method according to claim 2, characterized in that, The event data is subjected to event feature extraction to obtain motion feature information corresponding to the video to be processed, including: By employing the temporal granularity corresponding to each pyramid level in the temporal pyramid model, the event data is represented by a multi-level temporal pyramid to obtain the motion feature information corresponding to the video to be processed.

4. The method according to claim 3, characterized in that, Before performing multi-level temporal pyramid representation on the event data using a temporal granularity corresponding to each pyramid level in the temporal pyramid model to obtain the motion feature information corresponding to the video to be processed, the method further includes: The dynamic characteristic data in the video to be processed is obtained, wherein the dynamic characteristic data is used to characterize the motion data of moving objects in the video to be processed; The level parameters corresponding to each pyramid level in the time pyramid model are adjusted based on the dynamic characteristic data.

5. The method according to claim 2, characterized in that, The event data is subjected to event feature extraction to obtain motion feature information corresponding to the video to be processed, including: The dynamic characteristic data of at least one moving object in the video to be processed are obtained, wherein the dynamic characteristic data is used to characterize the motion data of the moving object in the video to be processed; Based on the relationship between the dynamic characteristic data and the time window, determine the time window corresponding to the dynamic characteristic data of the at least one moving object; By encoding the event data using a time window corresponding to the dynamic characteristic data of the at least one moving object, motion feature information corresponding to the video to be processed is obtained.

6. The method according to claim 2, characterized in that, The motion feature information is used to extract regional event features to obtain the motion region features corresponding to the video to be processed, including: By using the convolutional layers and multi-head self-attention mechanism of the neural network, local dynamic change information and edge information are extracted from the motion feature information to obtain the motion region features corresponding to the video to be processed.

7. The method according to claim 1, characterized in that, Based on the fusion features, video reconstruction is performed to obtain the target video, including: Obtain the target frame rate and target resolution corresponding to the target video; The display time corresponding to each frame in the target video is determined based on the target frame rate; Perform a time embedding operation on the fusion feature to obtain the first fusion feature corresponding to the display time; Perform a spatial embedding operation on the first fused feature to obtain the second fused feature corresponding to the target resolution; The second fusion feature is decoded to obtain a target video with the target frame rate and the target resolution.

8. The method according to claim 7, characterized in that, Decoding the second fused feature to obtain a target video with the target frame rate and the target resolution includes: The video to be processed is segmented to obtain multiple video regions; Based on the correlation between the regional video and the second fusion feature, determine the second fusion feature corresponding to each regional video; Based on the correlation between the decoding strategy and the second fusion feature, the corresponding decoding strategy is used to decode the second fusion feature of the corresponding regional video, wherein each regional video corresponds to the target regional video; The target video is obtained by combining the target region videos corresponding to the multiple region videos.

9. A video processing apparatus, characterized in that, include: The data acquisition module is used to acquire the video to be processed captured by the event camera and the event data corresponding to the video to be processed; The first feature extraction module is used to extract event features from the event data to obtain the motion region features corresponding to the video to be processed. The second feature extraction module is used to extract frame features from the video to be processed and the event data to obtain the overall motion features corresponding to the video to be processed, wherein the overall motion features are used to characterize the temporal and spatial dependencies of the video to be processed. The feature fusion module is used to fuse the motion region features and the overall motion features to obtain fused features; The video reconstruction module is used to reconstruct the video based on the fusion features to obtain the target video.

10. An electronic device, characterized in that, Electronic devices include: processors and memory storing computer program instructions; When the processor executes the computer program instructions, it implements the video processing method as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, A computer-readable storage medium stores computer program instructions that, when executed by a processor, implement the video processing method as described in any one of claims 1-8.

12. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device causes the electronic device to perform the video processing method as described in any one of claims 1-8.

Images