A method and device for processing real-time video streams

Abstract

An embodiment of the present application provides a method and device for processing a real-time video stream, which are applied in the technical field of multimedia resource processing, including: for the i-th video frame in the real-time video stream, extracting features of the i-th video frame through a feature extraction module to obtain spatial features of the i-th video frame; based on the previous video frame of the i-th video frame and the spatial features of the i-th video frame, obtaining spatio-temporal features of the i-th video frame through a spatio-temporal consistency module; the spatio-temporal consistency module is used to extract the temporal dependence relationship in space between the i-th video frame and the previous video frame; based on the spatio-temporal features of the i-th video frame, repairing the spatial features of the i-th video frame to obtain the repaired i-th video frame; according to the repaired video frames in the real-time video stream, obtaining the repaired real-time video stream. The repaired real-time video stream finally output can significantly improve the quality of the video stream while maintaining the overall visual coherence of the video, and reduce noise and blur phenomena.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of multimedia resource processing, and in particular, to a method and device for processing real-time video streams. Background Art

[0002] With the rapid development of multimedia technology, video content has become an indispensable part of people's daily life and business activities. Video repair technology, as an important technology in the field of video processing, mainly focuses on repairing and improving damaged or low-quality video files. This technology is widely used in multiple fields such as movie restoration, surveillance video enhancement, and personal video editing.

[0003] The prior art mainly relies on traditional interpolation and filtering techniques for video repair: 1. Intra-frame repair: For damages within a single frame, such as noise, blur, and occlusion. Commonly used techniques include digital filters, image interpolation techniques, and content-based image repair methods. 2. Inter-frame repair: Handling frame loss problems caused by transmission errors or storage damages. Such techniques usually rely on the information between adjacent frames and use motion estimation and frame interpolation techniques to reconstruct the lost frames. These methods can achieve certain effects when dealing with non-dynamic scenes, but the effects are usually not good when facing complex dynamic scenes and detailed information.

[0004] Although the prior art has achieved certain achievements in video repair, there are still problems in processing real-time videos. It cannot effectively handle the spatio-temporal relationships in video data, resulting in poor video quality and visual coherence after repair. Summary of the Invention

[0005] The embodiments of the present application provide a method and device for processing real-time video streams, which are used to repair real-time video streams, so that the finally output sequence of repaired video frames can significantly improve the video quality, reduce noise and blur phenomena while maintaining the overall visual coherence of the video.

[0006] In a first aspect, the embodiments of the present application provide a method for processing a real-time video stream, including:

[0007] For the i-th video frame in the real-time video stream, the feature extraction module extracts features from the i-th video frame to obtain the spatial features of the i-th video frame;

[0008] Based on the previous video frame of the i-th video frame and the spatial features of the i-th video frame, the spatio-temporal consistency module obtains the spatio-temporal features of the i-th video frame; the spatio-temporal consistency module is used to extract the temporal dependence relationship in space between the i-th video frame and the previous video frame;

[0009] Based on the spatio-temporal features of the i-th video frame, the spatial features of the i-th video frame are repaired by a repair and reconstruction module to obtain the repaired i-th video frame;

[0010] According to the repaired video frames in the real-time video stream, a repaired real-time video stream is obtained.

[0011] In this application, a trained deep learning model is applied to an actual video repair task. The sequence of video frames to be repaired is input into the trained deep learning model frame by frame. The model extracts the spatial features of each frame, and at the same time uses the spatio-temporal features of the previous video frames to update the spatial features to ensure frame-to-frame coherence. Then, the extracted spatial features and the consistent spatio-temporal features before and after are repaired and reconstructed to generate repaired frames. This process is repeated for each frame, and finally a sequence of repaired video frames is output. The model can significantly improve the video quality, reduce noise and blur while maintaining the overall visual coherence of the video.

[0012] Optionally, the spatio-temporal consistency module is trained separately in the first training stage;

[0013] The feature extraction module and the repair and reconstruction module are jointly trained in the second training stage; the spatio-temporal consistency module used in the second training stage is obtained through the first training stage, and the parameters of the spatio-temporal consistency module remain unchanged in the second training stage.

[0014] By introducing the spatio-temporal consistency module and deep learning technology, the effect and efficiency of real-time video repair are significantly improved. First, by using the spatio-temporal consistency module, the spatio-temporal relationship between the front and back frames can be fully considered when processing video frames, ensuring the coherence of the visual effect when the video is played continuously, and greatly reducing the visual jumps and discomfort caused by frame-to-frame inconsistency.

[0015] Optionally, the spatio-temporal consistency module is trained by using some or all of the following loss factors:

[0016] Content style feature loss factor, used to characterize the correlation of content and style between videos;

[0017] Part-whole feature loss factor, used to characterize the correlation between partial video features and overall video features in the video;

[0018] Feature offset loss factor, used to characterize the offset degree of features in the video.

[0019] Optionally, the spatio-temporal consistency module adjusts the parameters of the spatio-temporal consistency module according to the comprehensive loss value of each round of training until a spatio-temporal consistency module that meets the training termination conditions is obtained; where the comprehensive loss value of each round of training is obtained by the following formula:

[0020] L = αL content + βL part-whole + γL shift + L reg

[0021] Wherein, L content is the content style feature loss factor, L part-whole is the part-whole feature loss factor, L shift is the feature offset loss factor, L reg is the regularization term, and the regularization term prevents the spatio-temporal consistency module from overfitting. α, β, and γ are the weight parameters of each loss factor.

[0022] Optionally, the content style feature loss factor is determined in the following manner:

[0023]

[0024] Wherein, V1 and V2 are two different videos that are similar in content style, is the first feature of the i-th video frame in V1 after being processed by the spatio-temporal consistency module, is the second feature of the i-th video frame in V2 after being processed by the spatio-temporal consistency module, and N is the number of video frames.

[0025] Optionally, the part-whole feature loss factor is determined in the following manner:

[0026]

[0027] Wherein, M is the number of frames of a partial video, P is the partial video, O is the overall video containing the partial video P, and F(P i ) is the third feature of the i-th video frame in the partial video after being processed by the spatio-temporal consistency module, and F(O) is the fourth feature of each video frame in the overall video after being processed by the spatio-temporal consistency module.

[0028] Optionally, the feature offset loss factor is determined in the following manner:

[0029]

[0030] Wherein, T is the total number of frames in the video, and F t is the fifth feature of the video frame that has not been processed by the spatio-temporal consistency module, and F t+1 is the sixth feature of the video frame that has been processed by the spatio-temporal consistency module.

[0031] In a second aspect, an embodiment of the present application provides a processing device for a real-time video stream, including

[0032] An extraction module, configured to perform feature extraction on the i-th video frame in a real-time video stream through a feature extraction module to obtain spatial features of the i-th video frame;

[0033] A processing module, configured to obtain spatio-temporal features of the i-th video frame through a spatio-temporal consistency module based on the previous video frame of the i-th video frame and the spatial features of the i-th video frame; the spatio-temporal consistency module is configured to extract the temporal dependence relationship in space between the i-th video frame and the previous video frame;

[0034] A repair module, configured to repair the spatial features of the i-th video frame through a repair and reconstruction module based on the spatio-temporal features of the i-th video frame to obtain the repaired i-th video frame;

[0035] A synthesis module, configured to obtain a repaired real-time video stream according to the repaired video frames in the real-time video stream.

[0036] In this application, a trained deep learning model is applied to an actual video repair task. The video frame sequence to be repaired is input into the trained deep learning model frame by frame. The model extracts the spatial features of each frame, and at the same time uses the spatio-temporal features of the previous video frame to update the spatial features to ensure the coherence between frames. Then, the extracted spatial features and the consistent spatio-temporal features are repaired and reconstructed to generate repaired frames. This process is repeated for each frame, and finally a sequence of repaired video frames is output. The model can significantly improve the video quality, reduce noise and blur phenomena while maintaining the overall visual coherence of the video.

[0037] Optionally, the processing module is specifically configured to:

[0038] The spatio-temporal consistency module is trained separately in the first training stage;

[0039] The feature extraction module and the repair and reconstruction module are jointly trained in the second training stage; the spatio-temporal consistency module used in the second training stage is obtained through the first training stage, and the parameters of the spatio-temporal consistency module remain unchanged in the second training stage.

[0040] Optionally, the processing module is specifically configured to:

[0041] The spatio-temporal consistency module is trained through the following partial or all loss factors:

[0042] A content style feature loss factor, configured to characterize the feature correlation between different videos with similar content styles after being processed by the spatio-temporal consistency module;

[0043] The partial-global feature loss factor is used to characterize the feature correlation between some video frames and the overall video frames in the same video after being processed by the spatio-temporal consistency module;

[0044] The feature offset loss factor is used to characterize the feature offset degree of the video after being processed by the spatio-temporal consistency module.

[0045] Optionally, the spatio-temporal consistency module adjusts the parameters of the spatio-temporal consistency module through the comprehensive loss value of each round of training until a spatio-temporal consistency module that meets the training termination condition is obtained; wherein, the comprehensive loss value of each round of training is obtained through the following formula:

[0046] L = αL content + βL part-whole + γL shift + L reg

[0047] wherein, L content is the content style feature loss factor, L part-whole is the partial-global feature loss factor, L shift is the feature offset loss factor, L reg is the regularization term, and the regularization term prevents the spatio-temporal consistency module from overfitting. α, β, and γ are the weight parameters of each loss factor.

[0048] Optionally, the processing module is specifically configured to:

[0049] The content style feature loss factor is determined by the following method:

[0050]

[0051] wherein, V1 and V2 are two different videos that are similar in content style, is the first feature of the i-th video frame in V1 after being processed by the spatio-temporal consistency module, is the second feature of the i-th video frame in V2 after being processed by the spatio-temporal consistency module, and N is the number of video frames.

[0052] Optionally, the processing module is specifically configured to:

[0053] The partial-global feature loss factor is determined by the following method:

[0054]

[0055] wherein, M is the number of frames of the partial video, P is the partial video, O is the overall video containing the partial video P, F(P i) is the third feature of the i-th video frame in a partial video, and F(O) is the fourth feature of each video frame in the overall video after being processed by the spatio-temporal consistency module.

[0056] Optionally, the processing module is specifically configured to:

[0057] The feature offset loss factor is determined by the following method:

[0058]

[0059] where T is the total number of frames in the video, F t is the fifth feature of the video frame without being processed by the spatio-temporal consistency module, and F t+1 is the sixth feature of the video frame after being processed by the spatio-temporal consistency module.

[0060] In a third aspect, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor executes the method for processing a real-time video stream according to any one of the first aspects above.

[0061] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program executable by a computer device. When the program runs on the computer device, the computer device executes the method for processing a real-time video stream according to any one of the first aspects above.

[0062] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, the method for processing a real-time video stream according to any one of the first aspects above is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the accompanying drawings required for description in the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0064] Figure 1 is a schematic diagram of a model architecture for processing a real-time video stream provided by an embodiment of the present application;

[0065] Figure 2 is a schematic flowchart of a method for processing a real-time video stream provided by an embodiment of the present application;

[0066] Figure 3 is a schematic diagram of a model architecture for processing a real-time video stream provided by an embodiment of the present application;

[0067] Figure 4 It is a schematic structural diagram of a processing device for real-time video streams provided by an embodiment of the present application;

[0068] Figure 5 It is a schematic structural diagram of a computer device provided by an embodiment of the present application. Specific embodiments

[0069] In order to make the objectives, technical solutions and beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0070] As Figure 1 shown, it is a model architecture for processing real-time video streams provided by the present application, including five parts: an input layer, a feature extraction module, a spatio-temporal consistency module, a repair and reconstruction module, and an output layer.

[0071] As Figure 2 shown, it is a schematic flowchart of a method for processing real-time video streams provided by the present application, specifically including the following steps:

[0072] Step S201: For the i-th video frame in the real-time video stream, the feature extraction module extracts features from the i-th video frame to obtain the spatial features of the i-th video frame.

[0073] Specifically, the spatial features of a video frame mainly refer to the characteristics exhibited by the image of the i-th video frame in the two-dimensional space. The spatial features may include color features, texture features, and shape features, etc. Color features: The distribution combination of different colors reflects the image content. For example, in a video frame of a forest, color features such as green leaves and brown tree trunks can help identify objects in the scene. Texture features: Pixel arrangement and repetition patterns. Such as the delicate and random texture of grassland and the rough texture of rocks, etc., can be used to distinguish objects of different materials. Shape features: The outline and shape of an object. Such as a round sun and a square building, etc., help to identify specific objects.

[0074] Step S202: Based on the previous video frame of the i-th video frame and the spatial features of the i-th video frame, the spatio-temporal consistency module obtains the spatio-temporal features of the i-th video frame; the spatio-temporal consistency module is used to extract the temporal dependence relationship in space between the i-th video frame and the previous video frame.

[0075] Specifically, the spatio-temporal features of a video frame refer to the feature representation that combines both the temporal dimension and the spatial dimension. Among them, the temporal features are the changing information reflected in a continuous sequence of video frames, which can include motion features, dynamic textures, and temporal coherence. Motion features: The position changes of an object between different frames. For example, a running person moves from the left to the right of a video frame, and information such as its motion trajectory and speed is a temporal feature. Motion features can be extracted through techniques such as optical flow method, which can describe the motion direction and speed of pixels over time. Dynamic textures: Repeated or regular dynamic patterns presented by certain objects or scenes over time. For example, the undulation of ocean waves, the flickering of flames, etc., have specific dynamic texture features. Temporal coherence: The similarity and continuity between adjacent frames. If the motion of an object in a video is smooth, then the change between adjacent frames is relatively small, having a high temporal coherence.

[0076] By introducing the spatial features of the previous video frames and combining them with the spatial features of the i-th video frame, the spatio-temporal features of the current i-th video frame are obtained. The spatio-temporal consistency features of the i-th video frame are updated using the spatial features of the previous video frames to ensure coherence between frames. For example, taking the i-th video frame as a reference, the previous video frames of the i-th video frame can be the (i - 1)-th video frame, the (i - 2)-th video frame, the (i - 3)-th video frame, or a video stream composed of consecutive frames before the i-th video frame. For example, a video stream composed of the (i - 1)-th video frame, the (i - 2)-th video frame, and the (i - 3)-th video frame.

[0077] Step S203, based on the spatio-temporal features of the i-th video frame, the spatial features of the i-th video frame are repaired through a repair and reconstruction module to obtain the repaired i-th video frame.

[0078] Specifically, the extracted spatial features of the i-th video frame and the spatio-temporal features of the i-th video frame are jointly input into the repair and reconstruction layer. Since the spatio-temporal features of the i-th video frame introduce the spatial features of the previous video frames, the spatio-temporal features of the i-th video frame at this time are coherent with its previous video frames, so that the generated repaired i-th video frame is coherent with its previous video frames, significantly improving the video quality and reducing noise and blurring phenomena.

[0079] Step S204, based on the repaired video frames in the real-time video stream, a repaired real-time video stream is obtained.

[0080] Specifically, each repaired video frame is merged to obtain a repaired real-time video stream.

[0081] In this application, a trained deep learning model is applied to an actual video restoration task. The video frame sequence to be restored is input into the trained deep learning model frame by frame. The model extracts the spatial features of each frame and simultaneously updates the spatial features using the spatio-temporal features of the previous video frames to ensure frame-to-frame coherence. Then, the extracted spatial features and consistent spatio-temporal features are repaired and reconstructed to generate restored frames. This process is repeated for each frame, and finally, a sequence of restored video frames is output. The model can significantly improve the video quality, reduce noise and blur while maintaining the overall visual coherence of the video.

[0082] This application involves the training of a feature extraction module, a spatio-temporal consistency module, and a repair and reconstruction module. The training process includes the following:

[0083] 1. Prepare video data:

[0084] Before training, video sample data needs to be prepared. First, collect video data from a specified source. This data can be from video capture devices such as cameras, videos taken by mobile phones, or pre-existing video databases. The collected video data needs to go through a series of preprocessing steps: perform video encoding checks to ensure that the format and encoding method of the video data are unified for subsequent processing. Then, convert the video format to a unified processing format, such as converting various video encoding formats to a unified H.264 or H.265 format. These processing steps ensure data compatibility and processing efficiency, providing a high-quality data source for subsequent deep learning model training.

[0085] 2. Generate training data:

[0086] To enable the model to handle various actually damaged videos, we need to construct training data containing noise. First, add random noise to the original video to simulate actual damage. We use the Gaussian noise model, i.e., N_noise = N(0,σ^2), and add it to the video frames to generate a noisy dataset D_noise. Next, divide the video data into a training set and a test set to ensure the generalization ability of the model. The training set is used for the training process of the model, while the test set is used to verify the performance and effectiveness of the model. In addition, to enhance the robustness of the model, we perform data augmentation on the training data, including operations such as random cropping, rotation, and translation, to increase data diversity. Through these preparation steps, we obtain high-quality and diverse training data, laying a foundation for the effective training of the model. Through the above process, classified video data can be obtained, divided into a noisy dataset D_noise and a noise-free dataset D_original.

[0087] 3. Standardize the data:

[0088] Take the obtained noisy dataset D_noise as the video data to be normalized, and perform normalization processing on the noisy dataset D_noise. First, perform frame segmentation on the video data. Assume that the video stream has T frames, the size of each frame image is H×W pixels, and the number of color channels is C. The segmented frames can be represented as a four-dimensional tensor: X = {X1, X2, …, XT}; where, Xt represents the t-th frame, with a size of H×W×C. Then perform pixel value normalization. Normalize the pixel values in each frame image. Calculate the mean μ and standard deviation σ of the entire dataset. Normalize each pixel value Xt[i,j,k]:

[0089]

[0090] Finally, obtain the normalized dataset (D_noise').

[0091] 4. Randomly crop and augment the noisy and noiseless video frame datasets

[0092] Perform the following operations on each video in the noisy video frame dataset D_noise′ and the noiseless original video frame dataset D_original: Randomly select a rectangular area from within it and apply it to all frames for cropping. To ensure data alignment, the cropping operation must be performed simultaneously on the noisy data and the noiseless data. First, determine the size (hcrop, wcrop) of the cropping area, where the size of the cropping area is between 80% and 100% of the original image size. Randomly select the starting position (xstart, ystart) of the cropping area to ensure that the cropping area is valid within the frame. Perform the cropping operation on each frame through the following calculation formula:

[0093] Assume that the size of the original frame is H×W and the size of the cropping area is hcrop×wcrop, then:

[0094] hcrop = rand(0.8H, H)

[0095] wcrop = rand(0.8W, W)

[0096] Starting position:

[0097] xstart = rand(0, H - hcrop)

[0098] ystart = rand(0, W - wcrop)

[0099] Cropped frame:

[0100] Xtcrop = Xt[xstart:xstart + hcrop, ystart:ystart + wcrop]

[0101] Among them, Xt can be Xt_noise or Xt_original.

[0102] Through this, the noisy data D_noise after random cropping augmentation and the noise-free data D_original' after random cropping augmentation can be obtained.

[0103] Through these steps, a diverse and representative video stream training dataset is generated, further improving the performance of the model in actual video restoration tasks.

[0104] 5. Model Initialization

[0105] Due to the levels and depths of the convolutional neural network involved in video restoration, a multi-layer convolutional network can be selected to extract features from video frames. Among them, the input data of the output layer is a sequence of video frames. The input size is T×H×W×C (T is the number of time steps, H is the frame height, W is the frame width, and C is the number of color channels).

[0106] Preferably, as Figure 3 shown in the model architecture, it includes five parts: an input layer, a feature extraction module, a spatio-temporal consistency module, a restoration and reconstruction module, and an output layer. Among them, the feature extraction module can be designed as the following 6-layer structure:

[0107] (1) Convolutional layer 1: The convolutional kernel size is 3x3, the number of convolutional kernels is 64, the stride is 1, the padding method is same, and the activation function is ReLU. The output result of this layer is the preliminary spatial features. Among them, the padding method of same in the convolutional layer is to ensure that after the convolutional operation, the spatial size of the output feature map is the same as or as close as possible to the spatial size of the input feature map. This helps to avoid the sharp reduction of the feature map size as the network depth increases when constructing a deep neural network, thus retaining more spatial information and details.

[0108] (2) Convolutional layer 2: The convolutional kernel size is 3x3, the number of convolutional kernels is 128, the stride is 1, the padding method is same, and the activation function is ReLU. The output result of this layer is deeper spatial features.

[0109] (3) Pooling layer 1: The pooling window size is 2x2, the stride is 2, and max pooling is performed. The purpose of this layer is to reduce the size of the feature map and retain important spatial features.

[0110] (4) Convolutional layer 3: The convolutional kernel size is 3x3, the number of convolutional kernels is 256, the stride is 1, the padding method is same, and the activation function is ReLU. The output result of this layer is higher-level spatial features.

[0111] (5) Convolutional layer 4: The convolutional kernel size is 3x3, the number of convolutional kernels is 256, the stride is 1, the padding method is same, and the activation function is ReLU. The purpose of this layer is to further refine the high-level spatial features.

[0112] (6) Pooling layer 2: The pooling window size is 2x2, the stride is 2, and max pooling is performed. The purpose of this layer is to further reduce the size of the feature map and retain the key spatial features.

[0113] It should be noted that the structure of the feature extraction module may also include other possible implementation methods, not limited to the above module structure.

[0114] Preferably, in the model architecture as Figure 3 shown, the spatio-temporal consistency module can be designed as the following three-layer structure:

[0115] (1) LSTM layer: The number of hidden layer units is 256, and the number of time steps is T (the number of time steps). The purpose of this layer is to extract the temporal features of the video frames and capture the temporal dependencies between frames.

[0116] (2) Fully connected layer: The number of output units is 512, and the activation function is ReLU. The purpose of this layer is to further process the temporal features and generate high-dimensional spatio-temporal features.

[0117] (3) Weighted update layer: The features of historical frames are weighted and updated to generate the spatio-temporal consistency features of the current frame. The purpose of this layer is to ensure the coherence and consistency between video frames.

[0118] It should be noted that the structure of the spatio-temporal consistency module may also include other possible implementation methods, not limited to the above module structure.

[0119] Preferably, in the model architecture for processing real-time video streams as Figure 3 shown, the repair and reconstruction module can be designed as the following four-layer structure:

[0120] (1) Transposed convolutional layer 1: Apply a 3x3 convolutional kernel, with a number of 256, a stride of 2, and a padding method of same. The same padding makes the output feature map of the transposed convolution operation have the same size as the input feature map in the spatial dimension. This can retain the spatial information of the input feature map to a certain extent and is helpful for feature recovery and upsampling operations in the network. The activation function is ReLU. The purpose of this layer is to enlarge the size of the feature map and restore more details.

[0121] (2) Convolutional layer 5: Apply a 3x3 convolutional kernel, with a quantity of 128, a stride of 1, and a padding method of same. The same padding ensures that after the convolution operation, the spatial dimensions of the output feature map are the same as or as close as possible to those of the input feature map. This helps to avoid a sharp reduction in the size of the feature map as the network depth increases when constructing a deep neural network, thereby retaining more spatial information and details. The activation function is ReLU. The purpose of this layer is to further refine the features and enhance detail restoration.

[0122] (3) Transposed convolutional layer 2: Apply a 3x3 convolutional kernel, with a quantity of 64, a stride of 2, a padding method of same, and an activation function of ReLU. The purpose of this layer is to enlarge the feature map again to approach the size of the original video frame.

[0123] (4) Convolutional layer 6: Apply a 3x3 convolutional kernel, with a quantity of 3 (number of output channels), a stride of 1, a padding method of same, and an activation function of tanh. The purpose of this layer is to generate the restored frame and output the restored video frame.

[0124] It should be noted that the structure of the repair and reconstruction module may also include other possible implementation methods, not limited to the above module structure.

[0125] In the above step S202, the spatio-temporal features of the i-th video frame are obtained through the previous video frame of the i-th video frame and the spatial features of the i-th video frame, including: passing the previous video frame of the i-th video frame and the spatial features of the i-th video frame through the spatio-temporal consistency module. The spatio-temporal consistency module updates the spatial features of the i-th video frame based on the previous video frame of the i-th video frame, so that the i-th video frame obtains a temporal dependence relationship in space with the previous video frame; through the temporal dependence relationship and the spatial features of the i-th video frame, the spatio-temporal features of the i-th video frame are obtained.

[0126] Specifically, the spatio-temporal consistency module uses the spatial features of the previous video frame of the i-th video frame to perform weighted calculation on the spatial features of the current i-th video frame. This weighting process takes into account the information of the previous video frame to ensure the spatio-temporal consistency of the current frame features. At this time, the spatio-temporal features of the i-th video frame are coherent with its previous video frame, so that the restored i-th video frame is coherent with its previous video frame, significantly improving the video quality and reducing noise and blurring phenomena.

[0127] By introducing the spatio-temporal consistency module and deep learning technology, the effect and efficiency of real-time video repair have been significantly improved. First, using the spatio-temporal consistency module, when processing video frames, it can fully consider the spatio-temporal relationship between the front and back frames, ensuring the coherence of the visual effect during continuous video playback, and greatly reducing visual jumps and discomfort caused by frame-to-frame inconsistencies.

[0128] The spatio-temporal consistency module is trained separately in the first training stage; the feature extraction module and the restoration and reconstruction module are jointly trained in the second training stage; the spatio-temporal consistency module used in the second training stage is obtained through the first training stage, and the parameters of the spatio-temporal consistency module remain unchanged in the second training stage.

[0129] In some embodiments, the spatio-temporal consistency module is trained by the following partial or all loss factors: the content style feature loss factor, which is used to characterize the feature correlation after different videos with similar content styles are processed by the spatio-temporal consistency module; the part-whole feature loss factor, which is used to characterize the feature correlation between partial video frames and the whole video frame in the same video after being processed by the spatio-temporal consistency module; the feature offset loss factor, which is used to characterize the feature offset degree after the video is processed by the spatio-temporal consistency module.

[0130] Specifically, designing the content style feature loss factor to train the spatio-temporal consistency module is to ensure that even if two videos have different contents, their features in terms of visual style can be effectively captured and distinguished by the model. To ensure that the video features extracted by the trained spatio-temporal consistency model can capture the core content and style of the video, rather than just local or surface information. Among them, the content of the video refers to the actual image information in the video, such as the shape, color, and texture of objects. The style of the video refers to the overall visual effect and consistency of the video, such as lighting, hue, and texture consistency.

[0131] Designing the part-whole feature loss factor to train the spatio-temporal consistency module is to ensure that the features in terms of time and space within a video are continuous, ensure that the content and style of each part within the video frame are consistent, and avoid the situation where the style of a local area does not match the overall style after restoration. In video restoration, the temporal continuity means that the features between adjacent frames should maintain consistency in the time dimension, and this consistency ensures a smooth transition during video playback, avoiding visual jumps or incoherence. The spatial continuity means that the features within a single video frame should maintain consistency in the spatial dimension. When processing a video, the features of some parts of the video are similar to those of the whole video, thus ensuring consistency, that is, ensuring the continuity of features in terms of time and space within the video, making the restored video more natural, coherent, and with better visual effects when played.

[0132] Designing the feature offset loss factor to train the spatio-temporal consistency module is to ensure that the features of the video stream do not undergo significant offsets during the process of feature extraction by the spatio-temporal consistency module.

[0133] The comprehensive loss factor is obtained through the content style feature loss factor, the part-whole feature loss factor, and the feature offset loss factor, and the comprehensive loss factor is comprehensively adjusted until the comprehensive loss factor converges.

[0134] In some embodiments, the content style feature loss factor is determined by the following formula 1:

[0135]

[0136] where V1 and V2 are two different videos that are similar in content style, is the first feature of the i-th video frame in V1 after being processed by the spatio-temporal consistency module, is the second feature of the i-th video frame in V2 after being processed by the spatio-temporal consistency module, and N is the number of video frames.

[0137] Specifically, when training the spatio-temporal consistency module, two videos V1 and V2 are randomly obtained from the video stream dataset as training samples, and at the same time, for the video features and of the i-th frame in V1 and V2, the calculation as in the above formula 1 is performed, and each frame in V1 and V2 is processed in turn.

[0138] In some embodiments, the part-whole feature loss factor is determined by the following formula 2:

[0139]

[0140] where M is the number of frames of the partial video, P is the partial video, O is the overall video containing the partial video P, and F(P i ) is the third feature of the i-th video frame in the partial video after being processed by the spatio-temporal consistency module, and F(O) is the fourth feature of each video frame in the overall video after being processed by the spatio-temporal consistency module.

[0141] Specifically, when training the spatio-temporal consistency module, a video V is randomly obtained from the video stream dataset as a training sample, the video V is used as the overall video O, a segment randomly selected from the overall video O is used as the partial video, and a frame extracted from the partial video is used as the partial frame P i . The features of the overall video O and the partial frame P i are extracted to obtain F(O) and F(P i ). The part-whole feature loss factor L part-whole is adjusted by the above formula 2 to make the feature F(P i ) of the partial frame and the feature F(O) of the overall video similar, so as to ensure the consistency of features within a video.

[0142] In some embodiments, to ensure that the features in the video stream feature extraction process do not undergo significant offsets, the feature offset loss factor is designed to be determined by the following formula 3:

[0143]

[0144] where T is the total number of frames in the video, and F t is the fifth feature of the video frame that has not been processed by the spatio-temporal consistency module, and F t+1 is the sixth feature of the video frame that has been processed by the spatio-temporal consistency module.

[0145] It should be noted that the samples used in the training processes of the content style feature loss factor, the part-whole feature loss factor, and the feature offset loss factor are not related.

[0146] In some embodiments, a regularization term factor is also added to prevent overfitting of the spatio-temporal consistency module, specifically as follows in formula 4:

[0147] L reg = λ∑ ω∈W ω 2 ......Formula 4

[0148] where λ is the regularization parameter and ω is the set of model weights. Among them, the acquisition and update process of the model weight ω 2 is as follows:

[0149] 1. Initialization: The model weights are initialized by the He initialization method before the start of training. The He initialization method can effectively prevent gradient vanishing or explosion, and its formula is: W = randn(size) × 2 / sqrt(n), where n is the number of nodes in the previous layer.

[0150] 2. Training update: During the training process, the model weights propagate the error backward from the output layer to the input layer through the backpropagation algorithm. In each layer, the error gradient of that layer is calculated based on the error and the current activation value. This gradient indicates how the weights should be adjusted to reduce the error. Specifically, for each weight, the adjustment direction and magnitude of the weight are determined by calculating the partial derivative of the error with respect to that weight. The optimization algorithm used is the Adam optimizer, which combines the advantages of momentum and adaptive learning rate. In each iteration, the model weights are updated according to the gradient of the loss function to minimize the loss function.

[0151] Then, according to the content style feature loss factor L content , the part-whole feature loss factor L part-whole , the feature offset loss factor L shift , a weight coefficient is assigned, and in combination with the regularization term factor L regThe comprehensive loss factor L is obtained, and the specific expression of L is as follows in Formula 5:

[0152] L = αL content + βL part-whole + γL shift + L reg ......Formula 5

[0153] The comprehensive loss factor L is comprehensively adjusted until it converges. The parameters of the spatio-temporal consistency module can be continuously adjusted through the backpropagation algorithm and the Adam optimizer until the comprehensive loss factor L converges. During the training process, the performance of the spatio-temporal consistency module on the video stream validation set is verified regularly to ensure the spatio-temporal consistency feature extraction ability of the spatio-temporal consistency module.

[0154] The spatio-temporal features of the i-th video frame and the spatial features of the i-th video frame are input into the restoration and reconstruction module to obtain the restored i-th video frame. Specifically, the noisy dataset D_noise is used as the input of the restoration and reconstruction module, and the noiseless dataset D_original is used as the target output for training. During the data preparation stage, training samples (x i , y i ) are constructed, where x i ∈ D_noise and y i ∈ D_original. The video frames x i of the noisy dataset are input into a convolutional neural network (CNN) to generate the restored video frame output Calculate the loss between the restoration output and y i . The mean squared error (MSE) can be used as the loss function, which is specifically shown in the following Formula 6:

[0155]

[0156] And through the backpropagation algorithm, calculate the gradient of the loss function with respect to the parameters of the restoration and reconstruction module, and update the weights of the restoration and reconstruction module. Using the Adam optimizer, the update formula is as follows:

[0157] θ t+1 = θ t - η......Formula 7

[0158] A deep learning model using a Convolutional Neural Network (CNN) can effectively extract and repair complex detail information in videos and perform excellently even when dealing with dynamic scenes. In addition, through real-time processing technology, this method can instantly repair and enhance video quality, reduce processing latency, and provide an immediate visual improvement experience. This efficient and low-latency processing method not only improves the quality of video repair but also enhances the practical application value of the system and is applicable to multiple fields such as movie repair, surveillance video enhancement, and personal video editing. In summary, while improving the video repair effect, the present invention ensures the real-time nature and stability of processing, having significant practical application prospects and promotion value.

[0159] As Figure 4 shown, an embodiment of the present application provides a processing device 400 for a real-time video stream. The device 400 includes

[0160] an extraction module 401, configured to perform feature extraction on the i-th video frame in the real-time video stream through a feature extraction module to obtain the spatial features of the i-th video frame;

[0161] a processing module 402, configured to obtain the spatio-temporal features of the i-th video frame through a spatio-temporal consistency module based on the previous video frame of the i-th video frame and the spatial features of the i-th video frame; the spatio-temporal consistency module is configured to extract the temporal dependence relationship in space between the i-th video frame and the previous video frame;

[0162] a repair module 403, configured to repair the spatial features of the i-th video frame through a repair and reconstruction module based on the spatio-temporal features of the i-th video frame to obtain the repaired i-th video frame;

[0163] a synthesis module 404, configured to obtain a repaired real-time video stream according to the repaired video frames in the real-time video stream.

[0164] In the present application, by applying a trained deep learning model to an actual video repair task, the sequence of video frames to be repaired is input into the trained deep learning model frame by frame. The model extracts the spatial features of each frame, and at the same time uses the spatio-temporal features of the previous video frame to update the spatial features to ensure the coherence between frames. Then, the extracted spatial features and the consistent spatio-temporal features are repaired and reconstructed to generate repaired frames. This process is repeated for each frame, and finally a sequence of repaired video frames is output. The model can significantly improve the video quality while maintaining the overall visual coherence of the video, reducing noise and blur phenomena.

[0165] Optionally, the processing module 402 is specifically configured to:

[0166] The spatio-temporal consistency module is trained separately in the first training stage;

[0167] The feature extraction module and the repair and reconstruction module are jointly trained in the second training stage; the spatio-temporal consistency module used in the second training stage is obtained through the first training stage, and the parameters of the spatio-temporal consistency module remain unchanged in the second training stage.

[0168] Optionally, the processing module 402 is specifically configured to:

[0169] The spatio-temporal consistency module is trained through some or all of the following loss factors:

[0170] The content style feature loss factor is used to characterize the feature correlation of different videos with similar content styles after being processed by the spatio-temporal consistency module;

[0171] The part-whole feature loss factor is used to characterize the feature correlation between partial video frames and the overall video frames in the same video after being processed by the spatio-temporal consistency module;

[0172] The feature offset loss factor is used to characterize the feature offset degree of the video after being processed by the spatio-temporal consistency module.

[0173] Optionally, the spatio-temporal consistency module adjusts the parameters of the spatio-temporal consistency module through the comprehensive loss value of each round of training until a spatio-temporal consistency module that meets the training termination condition is obtained; wherein, the comprehensive loss value of each round of training is obtained through the following formula:

[0174] L = αL content + βL part-whole + γL shift + L reg

[0175] wherein, L content is the content style feature loss factor, L part-whole is the part-whole feature loss factor, L shift is the feature offset loss factor, L reg is the regularization term, and the regularization term prevents the spatio-temporal consistency module from overfitting, and α, β, and γ are the weight parameters of each loss factor.

[0176] Optionally, the processing module 402 is specifically configured to:

[0177] The content style feature loss factor is determined in the following manner:

[0178]

[0179] wherein, V1 and V2 are two different videos that are similar in content style, is the first feature of the i-th video frame in V1 after being processed by the spatio-temporal consistency module, is the second feature of the i-th video frame in V2 after being processed by the spatio-temporal consistency module, and N is the number of video frames.

[0180] Optionally, the processing module 402 is specifically configured to:

[0181] The partial-global feature loss factor is determined by the following method:

[0182]

[0183] where M is the number of video frames of the partial video, P is the partial video, O is the overall video containing the partial video P, F(P i ) is the third feature of the i-th video frame in the partial video after being processed by the spatio-temporal consistency module, and F(O) is the fourth feature of each video frame in the overall video after being processed by the spatio-temporal consistency module.

[0184] Optionally, the processing module 402 is specifically configured to:

[0185] The feature offset loss factor is determined by the following method:

[0186]

[0187] where T is the total number of video frames in the video, F t is the fifth feature of the video frame that has not been processed by the spatio-temporal consistency module, and F t+1 is the sixth feature of the video frame after being processed by the spatio-temporal consistency module.

[0188] Based on the same technical concept, an embodiment of the present application provides a computer device, as Figure 5 shown, including at least one processor 501 and a memory 502 connected to the at least one processor. In the embodiment of the present application, the specific connection medium between the processor 501 and the memory 502 is not limited, Figure 5 and taking the case where the processor 501 and the memory 502 are connected by a bus as an example. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0189] In the embodiment of the present application, the memory 502 stores instructions executable by the at least one processor 501. The at least one processor 501 can execute the steps of the above method for processing a real-time video stream by executing the instructions stored in the memory 502.

[0190] Among them, the processor 501 is the control center of the computer device. It can connect various parts of the computer device through various interfaces and circuits. By running or executing the instructions stored in the memory 502 and invoking the data stored in the memory 502, it can process the video stream and ensure the spatio-temporal coherence and consistency between the frames of the real-time video stream after the repair process. Optionally, the processor 501 may include one or more processing units. The processor 501 may integrate an application processor and a modem processor. Among them, the application processor mainly processes the operating system, user interface, application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor may not be integrated into the processor 501. In some embodiments, the processor 501 and the memory 502 may be implemented on the same chip. In some embodiments, they may also be separately implemented on independent chips.

[0191] The processor 501 may be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application specific integrated circuit (ASIC), a field programmable gate array, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application may be directly embodied as being executed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0192] The memory 502 serves as a non-volatile computer-readable storage medium and can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. The memory 502 may include at least one type of storage medium. For example, it may include flash memory, hard disks, multimedia cards, card-type memories, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memories, magnetic disks, optical disks, and so on. The memory 502 is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 502 in the embodiments of the present application may also be a circuit or any other device capable of implementing a storage function, for storing program instructions and / or data.

[0193] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium that stores a computer program executable by a computer device. When the program runs on the computer device, it causes the computer device to execute the steps of the above-described method for processing real-time video streams.

[0194] Based on the same inventive concept, an embodiment of the present application further provides a computer program product. The computer program product includes: computer program code. When the computer program code runs on a computer, it causes the computer to execute the interface test method as described in any of the foregoing discussions. Since the principle of solving problems by the above computer program product is similar to that of the interface test method, the implementation of the above computer program product can refer to the implementation of the method, and the repeated parts will not be described again.

[0195] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program code.

[0196] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to the application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processors of general-purpose computers, special-purpose computers, embedded processors, or other programmable data processing devices to produce a machine, such that the instructions executed by the processors of the computer or other programmable data processing devices produce means for implementing the functions specified in the flow Figure 1 one or more flows and / or blocks Figure 1 or means for implementing the functions specified in one or more blocks.

[0197] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufactured article including instruction means that implement the functions specified in the flow Figure 1 one or more flows and / or blocks Figure 1 or means for implementing the functions specified in one or more blocks.

[0198] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in the flow Figure 1 one or more flows and / or blocks Figure 1 or means for implementing the functions specified in one or more blocks.

[0199] Obviously, those skilled in the art can make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalent technologies, this application is also intended to include these changes and modifications.

Claims

1. A method for processing real-time video streams, characterized in that, Including: For the i-th video frame in the real-time video stream, the feature extraction module extracts features from the i-th video frame to obtain the spatial features of the i-th video frame; Based on the previous video frame of the i-th video frame and the spatial features of the i-th video frame, the spatio-temporal consistency module obtains the spatio-temporal features of the i-th video frame; the spatio-temporal consistency module is used to extract the temporal dependence relationship in space between the i-th video frame and the previous video frame; Based on the spatio-temporal features of the i-th video frame, the restoration and reconstruction module repairs the spatial features of the i-th video frame to obtain the repaired i-th video frame; According to the repaired video frames in the real-time video stream, a repaired real-time video stream is obtained; The spatio-temporal consistency module is trained separately in the first training stage; The feature extraction module and the restoration and reconstruction module are jointly trained in the second training stage; the spatio-temporal consistency module used in the second training stage is obtained through the first training stage, and the parameters of the spatio-temporal consistency module remain unchanged in the second training stage; The spatio-temporal consistency module is trained through some or all of the following loss factors: The content style feature loss factor is used to characterize the feature correlation after different videos with similar content styles are processed by the spatio-temporal consistency module; The part-whole feature loss factor is used to characterize the feature correlation between some video frames and the whole video frame in the same video after being processed by the spatio-temporal consistency module; The feature offset loss factor is used to characterize the feature offset degree of the video after being processed by the spatio-temporal consistency module.

2. The method according to claim 1, characterized in that, The spatio-temporal consistency module adjusts the parameters of the spatio-temporal consistency module through the comprehensive loss value of each round of training until a spatio-temporal consistency module that meets the training termination condition is obtained; among them, the comprehensive loss value of each round of training is obtained through the following formula: L = αL content + βL part-whole + γL shift + L reg Among them, L content is the content style feature loss factor, L part-whole is the partial overall feature loss factor, L shift is the feature offset loss factor, L reg is the regularization term, and the regularization term prevents the spatio-temporal consistency module from overfitting. α, β, and γ are the weight parameters of each loss factor.

3. The method according to claim 2, characterized in that, The content style feature loss factor is determined in the following way: Among them, V1 and V2 are two different videos with similar content styles. is the first feature of the i-th video frame in V1 after being processed by the spatio-temporal consistency module. is the second feature of the i-th video frame in V2 after being processed by the spatio-temporal consistency module, and N is the number of video frames.

4. The method according to claim 2, wherein The part-whole feature loss factor is determined in the following way: Wherein, M is the number of frames of a partial video, P is the partial video, O is the overall video containing the partial video P, and F(P i ) is the third feature of the i-th video frame in the partial video through the spatio-temporal consistency module, and F(O) is the fourth feature of each video frame in the overall video after being processed by the spatio-temporal consistency module.

5. The method according to claim 2, wherein The feature offset loss factor is determined in the following way: where T is the total number of frames in the video, and F t is the fifth feature of the video frame that has not been processed by the spatio-temporal consistency module, and F t+1 is the sixth feature of the video frame that has been processed by the spatio-temporal consistency module.

6. A processing device for real-time video streams, characterized in that, Including: An extraction module, which is used to extract features from the i-th video frame in the real-time video stream through the feature extraction module to obtain the spatial features of the i-th video frame; A processing module, which is used to obtain the spatio-temporal features of the i-th video frame through the spatio-temporal consistency module based on the previous video frame of the i-th video frame and the spatial features of the i-th video frame; the spatio-temporal consistency module is used to extract the temporal dependence relationship in space between the i-th video frame and the previous video frame; A repair module, which is used to repair the spatial features of the i-th video frame through the restoration and reconstruction module based on the spatio-temporal features of the i-th video frame to obtain the repaired i-th video frame; A synthesis module, which is used to obtain a repaired real-time video stream according to the repaired video frames in the real-time video stream; The spatio-temporal consistency module is trained separately in the first training stage; The feature extraction module and the repair and reconstruction module are jointly trained in the second training stage; the spatio-temporal consistency module used in the second training stage is obtained through the first training stage, and the parameters of the spatio-temporal consistency module remain unchanged in the second training stage; The spatio-temporal consistency module is trained through some or all of the following loss factors: The content style feature loss factor is used to characterize the feature correlation after different videos with similar content styles are processed by the spatio-temporal consistency module; The part-whole feature loss factor is used to characterize the feature correlation between some video frames and the whole video frames in the same video after being processed by the spatio-temporal consistency module; The feature offset loss factor is used to characterize the feature offset degree of the video after being processed by the spatio-temporal consistency module.

7. A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, It stores a computer program executable by a computer device. When the program runs on the computer device, the computer device is caused to execute the steps of the method according to any one of claims 1 to 5.

9. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images