Real-time video stream processing method and device

Abstract

The embodiment of the invention provides a real-time video stream processing method and device, which are applied to the technical field of multimedia resource processing, and the method comprises the steps: for an ith video frame in a real-time video stream, carrying out the feature extraction of the ith video frame through a feature extraction module, and obtaining the spatial feature of the ith video frame; on the basis of a preorder video frame of the ith video frame and the spatial characteristics of the ith video frame, obtaining the spatial-temporal characteristics of the ith video frame through a spatial-temporal consistency module; the space-time consistency module is used for extracting a time sequence dependency relationship of the ith video frame and a preorder video frame in space; based on the spatial-temporal characteristics of the ith video frame, repairing the spatial characteristics of the ith video frame to obtain a repaired ith video frame; and obtaining the repaired real-time video stream according to each repaired video frame in the real-time video stream. Therefore, the quality of the video stream can be remarkably improved and the noise and blurring phenomena can be reduced while the overall visual coherence of the video can be maintained by the finally output restored real-time video stream.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of multimedia resource processing, and in particular to a method and device for processing a real-time video stream. Background Art

[0002] With the rapid development of multimedia technology, video content has become an indispensable part of people's daily life and business activities. Video repair technology, as an important technology in the field of video processing, is mainly dedicated to repairing and improving damaged or low-quality video files. This technology is widely used in many fields such as film restoration, surveillance video enhancement, and personal video editing.

[0003] Existing technologies mainly rely on traditional interpolation and filtering techniques for video restoration: 1. Intra-frame restoration: for damage within a single frame, such as noise, blur, and occlusion. Commonly used technologies include digital filters, image interpolation technology, and content-based image restoration methods. 2. Inter-frame restoration: dealing with frame loss problems caused by transmission errors or storage damage. This type of technology usually relies on information between adjacent frames, using motion estimation and frame interpolation technology to reconstruct lost frames. These methods can achieve certain results when dealing with non-dynamic scenes, but the results are usually not good when facing complex dynamic scenes and detailed information.

[0004] Although existing technologies have made certain achievements in video restoration, there is still a problem that they cannot effectively handle the spatiotemporal relationship in video data when processing real-time video, resulting in poor quality and visual coherence of the restored video. Summary of the invention

[0005] The embodiments of the present application provide a real-time video stream processing method and device for repairing the real-time video stream so that the final output repaired video frame sequence can significantly improve the video quality and reduce noise and blur while maintaining the overall visual coherence of the video.

[0006] In a first aspect, an embodiment of the present application provides a method for processing a real-time video stream, comprising:

[0007] For an i-th video frame in a real-time video stream, extracting features of the i-th video frame by a feature extraction module to obtain spatial features of the i-th video frame;

[0008] Based on the spatial features of the preceding video frame of the i-th video frame and the i-th video frame, the spatial and temporal features of the i-th video frame are obtained by a spatial and temporal consistency module; the spatial and temporal consistency module is used to extract the temporal dependency relationship between the i-th video frame and the preceding video frame in space;

[0009] Based on the spatiotemporal features of the i-th video frame, the spatial features of the i-th video frame are repaired by a repairing and reconstructing module to obtain a repaired i-th video frame;

[0010] A restored real-time video stream is obtained according to each restored video frame in the real-time video stream.

[0011] This application applies the trained deep learning model to the actual video restoration task, inputs the video frame sequence to be restored frame by frame into the trained deep learning model, and the model extracts the spatial features of each frame, and updates the spatial features using the spatiotemporal features of the previous video frame to ensure the coherence between frames, and then restores and reconstructs the extracted spatial features and the consistent spatiotemporal features to generate the restored frame. This process is repeated on each frame, and finally the restored video frame sequence is output. The model can significantly improve the video quality and reduce noise and blur while maintaining the overall visual coherence of the video.

[0012] Optionally, the spatiotemporal consistency module is trained separately in the first training phase;

[0013] The feature extraction module and the restoration and reconstruction module are jointly trained in the second training stage; the spatiotemporal consistency module used in the second training stage is obtained through the first training stage, and the parameters of the spatiotemporal consistency module remain unchanged in the second training stage.

[0014] By introducing the spatiotemporal consistency module and deep learning technology, the effect and efficiency of real-time video restoration have been significantly improved. First, the spatiotemporal consistency module can fully consider the spatiotemporal relationship between the previous and next frames when processing video frames, ensuring the continuity of visual effects when the video is played continuously, and greatly reducing the visual jumps and discomfort caused by inconsistency between frames.

[0015] Optionally, the spatiotemporal consistency module is trained by using some or all of the following loss factors:

[0016] Content-style feature loss factor, used to characterize the relevance of content and style between videos;

[0017] Partial overall feature loss factor, used to characterize the correlation between partial video features and overall video features in the video;

[0018] Feature shift loss factor, used to characterize the degree of shift of features in the video.

[0019] Optionally, the spatiotemporal consistency module adjusts the parameters of the spatiotemporal consistency module by the comprehensive loss value of each round of training until a spatiotemporal consistency module that meets the training termination condition is obtained; wherein the comprehensive loss value of each round of training is obtained by the following formula:

[0020] L=αL content +βL part-whole +γL shift +L reg

[0021] Among them, L content is the content style feature loss factor, L part-whole is the overall characteristic loss factor of the part, L shift is the characteristic shift loss factor, L reg is a regularization term, which prevents the spatiotemporal consistency module from overfitting, and α, β, and γ are weight parameters of each loss factor.

[0022] Optionally, the content style feature loss factor is determined by:

[0023]

[0024] Among them, V1 and V2 are two different videos with similar content styles. is the first feature of the i-th video frame in V1 after being processed by the spatiotemporal consistency module. is the second feature of the i-th video frame in V2 after being processed by the spatiotemporal consistency module, and N is the number of video frames.

[0025] Optionally, the partial overall feature loss factor is determined by:

[0026]

[0027] Where M is the number of frames of the partial video, P is the partial video, O is the entire video containing the partial video P, and F(P i ) is the third feature of the i-th video frame in the partial video after being processed by the spatiotemporal consistency module, and F(O) is the fourth feature of each video frame in the overall video after being processed by the spatiotemporal consistency module.

[0028] Optionally, the characteristic shift loss factor is determined by:

[0029]

[0030] Where T is the total number of frames in the video, F t is the fifth feature of the video frame not processed by the spatiotemporal consistency module, F t+1 It is the sixth feature of the video frame processed by the spatiotemporal consistency module.

[0031] In a second aspect, an embodiment of the present application provides a real-time video stream processing device, comprising:

[0032] An extraction module is used to extract features of an i-th video frame in a real-time video stream through a feature extraction module to obtain spatial features of the i-th video frame;

[0033] A processing module, configured to obtain the spatiotemporal features of the i-th video frame through a spatiotemporal consistency module based on the spatial features of the preceding video frame of the i-th video frame and the i-th video frame; the spatiotemporal consistency module is configured to extract the temporal dependency relationship between the i-th video frame and the preceding video frame in space;

[0034] A repair module, used for repairing the spatial features of the i-th video frame through a repair and reconstruction module based on the spatiotemporal features of the i-th video frame to obtain a repaired i-th video frame;

[0035] The synthesis module is used to obtain a restored real-time video stream according to each restored video frame in the real-time video stream.

[0036] This application applies the trained deep learning model to the actual video restoration task, inputs the video frame sequence to be restored frame by frame into the trained deep learning model, and the model extracts the spatial features of each frame, and updates the spatial features using the spatiotemporal features of the previous video frame to ensure the coherence between frames, and then restores and reconstructs the extracted spatial features and the consistent spatiotemporal features to generate the restored frame. This process is repeated on each frame, and finally the restored video frame sequence is output. The model can significantly improve the video quality and reduce noise and blur while maintaining the overall visual coherence of the video.

[0037] Optionally, the processing module is specifically used for:

[0038] The spatiotemporal consistency module is trained separately in the first training phase;

[0039] The feature extraction module and the restoration and reconstruction module are jointly trained in the second training stage; the spatiotemporal consistency module used in the second training stage is obtained through the first training stage, and the parameters of the spatiotemporal consistency module remain unchanged in the second training stage.

[0040] Optionally, the processing module is specifically used for:

[0041] The spatiotemporal consistency module is trained by some or all of the following loss factors:

[0042] Content style feature loss factor, which is used to characterize the feature correlation of different videos with similar content styles after being processed by the spatiotemporal consistency module;

[0043] Partial-overall feature loss factor, used to characterize the feature correlation between partial video frames and overall video frames in the same video after being processed by the spatiotemporal consistency module;

[0044] The feature shift loss factor is used to characterize the feature shift degree of the video after being processed by the spatiotemporal consistency module.

[0045] Optionally, the spatiotemporal consistency module adjusts the parameters of the spatiotemporal consistency module by the comprehensive loss value of each round of training until a spatiotemporal consistency module that meets the training termination condition is obtained; wherein the comprehensive loss value of each round of training is obtained by the following formula:

[0046] L=αL content +βL part-whole +γL shift +L reg

[0047] Among them, L content is the content style feature loss factor, L part-whole is the overall characteristic loss factor of the part, L shift is the characteristic shift loss factor, L reg is a regularization term, which prevents the spatiotemporal consistency module from overfitting, and α, β, and γ are weight parameters of each loss factor.

[0048] Optionally, the processing module is specifically used for:

[0049] The content style feature loss factor is determined by:

[0050]

[0051] Among them, V1 and V2 are two different videos with similar content styles. is the first feature of the i-th video frame in V1 after being processed by the spatiotemporal consistency module. is the second feature of the i-th video frame in V2 after being processed by the spatiotemporal consistency module, and N is the number of video frames.

[0052] Optionally, the processing module is specifically used for:

[0053] The partial overall characteristic loss factor is determined by:

[0054]

[0055] Where M is the number of frames of the partial video, P is the partial video, O is the entire video containing the partial video P, and F(P i) is the third feature of the i-th video frame in the partial video after being processed by the spatiotemporal consistency module, and F(O) is the fourth feature of each video frame in the overall video after being processed by the spatiotemporal consistency module.

[0056] Optionally, the processing module is specifically used for:

[0057] The characteristic shift loss factor is determined by:

[0058]

[0059] Where T is the total number of frames in the video, F t is the fifth feature of the video frame not processed by the spatiotemporal consistency module, F t+1 It is the sixth feature of the video frame processed by the spatiotemporal consistency module.

[0060] In a third aspect, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes any of the methods for processing real-time video streams described in the first aspect.

[0061] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program executable by a computer device. When the program runs on the computer device, the computer device executes any method for processing a real-time video stream as described in the first aspect above.

[0062] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program / instruction, and when the computer program / instruction is executed by a processor, it implements any method for processing a real-time video stream as described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0064] Figure 1 A schematic diagram of a model architecture for processing real-time video streams provided in an embodiment of the present application;

[0065] Figure 2 A flowchart of a method for processing a real-time video stream provided in an embodiment of the present application;

[0066] Figure 3 A schematic diagram of a model architecture for processing real-time video streams provided in an embodiment of the present application;

[0067] Figure 4 A schematic diagram of the structure of a real-time video stream processing device provided in an embodiment of the present application;

[0068] Figure 5 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and beneficial effects of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0070] like Figure 1 As shown, a model architecture for processing real-time video streams provided by the present application includes five parts: an input layer, a feature extraction module, a spatiotemporal consistency module, a repair and reconstruction module, and an output layer.

[0071] like Figure 2 FIG. 1 is a flow chart of a method for processing a real-time video stream provided by the present application, which specifically includes the following steps:

[0072] Step S201: for the i-th video frame in the real-time video stream, a feature extraction module is used to extract features of the i-th video frame to obtain spatial features of the i-th video frame.

[0073] Specifically, the spatial features of a video frame mainly refer to the characteristics of the image of the i-th video frame in two-dimensional space. Spatial features may include color features, texture features, and shape features. Color features: The distribution and combination of different colors that reflect the image content. For example, in a video frame of a forest, color features such as green leaves and brown tree trunks can help identify objects in the scene. Texture features: Pixel arrangement and repetitive patterns. Such as the fine and random texture of grass and the rough texture of rocks, which can be used to distinguish objects of different materials. Shape features: The outline and shape of an object. Like a round sun, a square building, etc., it helps to identify specific objects.

[0074] Step S202, based on the spatial features of the preceding video frame and the i-th video frame, the spatiotemporal features of the i-th video frame are obtained through a spatiotemporal consistency module; the spatiotemporal consistency module is used to extract the spatial temporal dependency between the i-th video frame and the preceding video frame.

[0075] Specifically, the spatiotemporal features of a video frame refer to the feature representation that combines both the temporal dimension and the spatial dimension. Among them, the temporal feature is the change information reflected in a continuous sequence of video frames, which can include motion features, dynamic textures, and temporal coherence. Motion features: The position change of an object between different frames. For example, a running person moves from the left to the right of a video frame, and its motion trajectory, speed, and other information are temporal features. Motion features can be extracted using techniques such as optical flow, which can describe the direction and speed of movement of pixels in time. Dynamic texture: The repetitive or regular dynamic patterns presented by certain objects or scenes over time. For example, the rise and fall of waves, the jumping of flames, etc., have specific dynamic texture characteristics. Temporal coherence: The similarity and continuity between adjacent frames. If the movement of objects in the video is smooth, then the changes between adjacent frames are relatively small, with high temporal coherence.

[0076] By introducing the spatial features of the previous video frame and combining them with the spatial features of the i-th video frame, the spatiotemporal features of the current i-th video frame are obtained. The spatiotemporal consistency features of the i-th video frame are updated using the spatial features of the previous video frame to ensure the continuity between frames. For example, taking the i-th video frame as a reference, the previous video frame of the i-th video frame can be the i-1th video frame, the i-2th video frame, the i-3th video frame, or a video stream composed of continuous frames before the i-th video frame, for example, a video stream composed of the i-1th video frame, the i-2th video frame and the i-3th video frame.

[0077] Step S203: based on the spatiotemporal features of the i-th video frame, the spatial features of the i-th video frame are repaired by a repairing and reconstructing module to obtain a repaired i-th video frame.

[0078] Specifically, the extracted spatial features of the i-th video frame and the spatiotemporal features of the i-th video frame are jointly input into the restoration and reconstruction layer. Since the spatiotemporal features of the i-th video frame introduce the spatial features of the previous video frame, the spatiotemporal features of the i-th video frame at this time are consistent with those of the previous video frame, so that the restored i-th video frame is consistent with its previous video frame, which significantly improves the video quality and reduces noise and blur.

[0079] Step S204, obtaining a restored real-time video stream according to the restored video frames in the real-time video stream.

[0080] Specifically, each restored video frame is merged to obtain a restored real-time video stream.

[0081] This application applies the trained deep learning model to the actual video restoration task, inputs the video frame sequence to be restored frame by frame into the trained deep learning model, and the model extracts the spatial features of each frame, and updates the spatial features using the spatiotemporal features of the previous video frame to ensure the coherence between frames, and then restores and reconstructs the extracted spatial features and the consistent spatiotemporal features to generate the restored frame. This process is repeated on each frame, and finally the restored video frame sequence is output. The model can significantly improve the video quality and reduce noise and blur while maintaining the overall visual coherence of the video.

[0082] This application involves training of a feature extraction module, a spatiotemporal consistency module, and a repair and reconstruction module. The training process includes the following:

[0083] 1. Prepare video data:

[0084] Video sample data needs to be prepared before training. First, video data is collected from a specified source. This data can come from video capture devices, such as cameras, videos shot by mobile phones, or pre-existing video databases. The collected video data needs to go through a series of preprocessing steps: video encoding checks are performed to ensure that the format and encoding method of the video data are unified for subsequent processing. Next, the video format is converted to convert videos of different formats into a unified processing format, such as converting multiple video encoding formats into a unified H.264 or H.265 format. These processing steps ensure data compatibility and processing efficiency, providing a high-quality data source for subsequent deep learning model training.

[0085] 2. Generate training data:

[0086] In order to enable the model to handle various actual damaged videos, we need to construct training data containing noise. First, random noise is added to the original video to simulate the actual damage. We adopt the Gaussian noise model, that is, N_noise=N(0,σ^2), and add it to the video frame to generate a noisy dataset D_noise. Next, the video data is divided into a training set and a test set to ensure the generalization ability of the model. The training set is used for the training process of the model, while the test set is used to verify the performance and effect of the model. In addition, in order to enhance the robustness of the model, we perform data augmentation on the training data, including random cropping, rotation, translation and other operations to increase the diversity of the data. Through these preparatory steps, we obtain high-quality and diverse training data, which lays the foundation for the effective training of the model. Through the above process, the classified video data can be obtained, which is divided into a noisy dataset D_noise and a noise-free dataset D_original.

[0087] 3. Standardize the data:

[0088] The obtained noisy data set D_noise is used as the video data to be standardized. The noisy data set D_noise is standardized. First, the video data is frame segmented. Assume that the video stream has T frames, the size of each frame is H×W pixels, and the number of color channels is C. The segmented frame can be represented as a four-dimensional tensor: X={X1,X2,…,XT}; where Xt represents the tth frame, with a size of H×W×C. Then the pixel values ​​are standardized. The pixel values ​​in each frame are standardized. Calculate the mean μ and standard deviation σ of the entire data set. Standardize each pixel value Xt[i,j,k]:

[0089]

[0090] Finally, a standardized data set (D_noise') is obtained.

[0091] 4. Random cropping and enhancement of noisy and noise-free video frame datasets

[0092] The following operations are performed on each video of the noisy video frame dataset D_noise′ and the noise-free original video frame dataset D_original: a rectangular area is randomly selected from its interior and applied to all frames for cropping. In order to ensure data alignment, the cropping operation must be performed on both the noisy data and the noise-free data. First, the size of the cropping area (hcrop, wcrop) is determined, and the size of the cropping area is between 80% and 100% of the original image size. The starting position of the cropping area (xstart, ystart) is randomly selected to ensure that the cropping area is valid within the frame. The cropping operation is performed on each frame using the following calculation formula:

[0093] Assuming the original frame size is H×W and the crop area size is hcrop×wcrop, then:

[0094] hcrop = rand(0.8H, H)

[0095] wcrop = rand(0.8W, W)

[0096] Starting Position:

[0097] xstart = rand(0, H - hcrop)

[0098] ystart = rand(0, W-wcrop)

[0099] Cropped frame:

[0100] Xtcrop=Xt[xstart:xstart+hcrop,ystart:ystart+wcrop]

[0101] where Xt can be either Xt_noise or Xt_original.

[0102] The noisy data D_noise' after random cropping enhancement and the noise-free data D_original' after random cropping enhancement can be obtained.

[0103] Through these steps, a diverse and representative video stream training dataset is generated, which further improves the performance of the model in practical video restoration tasks.

[0104] 5. Model initialization

[0105] Since the convolutional neural network level and depth of video restoration are involved, a multi-layer convolutional network can be selected to extract features from video frames. Among them, the input data of the output layer is a video frame sequence. The input size is T×H×W×C (T is the number of time steps, H is the frame height, W is the frame width, and C is the number of color channels).

[0106] Preferably, Figure 3 The model architecture shown in the figure includes five parts: input layer, feature extraction module, spatiotemporal consistency module, repair and reconstruction module and output layer. The feature extraction module can be designed as the following 6-layer structure:

[0107] (1) Convolutional layer 1: The convolution kernel size is 3x3, the number of convolution kernels is 64, the stride is 1, the padding method is same, and the activation function is ReLU. The output of this layer is the preliminary spatial features. The padding method in the convolution layer is same to ensure that after the convolution operation, the spatial size of the output feature map is consistent with the spatial size of the input feature map, or as close as possible. This helps to avoid a sharp decrease in the size of the feature map as the network layer deepens when building a deep neural network, thereby retaining more spatial information and details.

[0108] (2) Convolutional layer 2: The convolution kernel size is 3x3, the number of convolution kernels is 128, the step size is 1, the padding method is the same, and the activation function is ReLU. The output of this layer is a deeper spatial feature.

[0109] (3) Pooling layer 1: The pooling window size is 2x2, the stride is 2, and the maximum pooling is performed. The purpose of this layer is to reduce the size of the feature map and retain important spatial features.

[0110] (4) Convolutional layer 3: The convolution kernel size is 3x3, the number of convolution kernels is 256, the step size is 1, the padding method is same, and the activation function is ReLU. The output of this layer is a higher-level spatial feature.

[0111] (5) Convolutional layer 4: The convolution kernel size is 3x3, the number of convolution kernels is 256, the step size is 1, the padding method is same, and the activation function is ReLU. The purpose of this layer is to further refine the high-level spatial features.

[0112] (6) Pooling layer 2: The pooling window size is 2x2, the stride is 2, and the maximum pooling is performed. The purpose of this layer is to further reduce the size of the feature map and retain the key spatial features.

[0113] It should be noted that the structure of the feature extraction module also includes other possible implementation methods and is not limited to the above module structure.

[0114] Preferably, Figure 3 In the model architecture shown, the spatiotemporal consistency module can be designed as the following three-layer structure:

[0115] (1) LSTM layer: The number of hidden layer units is 256, and the number of time steps is T (time steps). The purpose of this layer is to extract the temporal features of video frames and capture the temporal dependencies between frames.

[0116] (2) Fully connected layer: The number of output units is 512, and the activation function is ReLU. The purpose of this layer is to further process the time series features and generate high-dimensional spatiotemporal features.

[0117] (3) Weighted update layer: The features of historical frames are weighted and updated to generate the spatiotemporal consistency features of the current frame. The purpose of this layer is to ensure the coherence and consistency between video frames.

[0118] It should be noted that the structure of the spatiotemporal consistency module also includes other possible implementation methods and is not limited to the above module structure.

[0119] Preferably, Figure 3 In the real-time video stream processing model architecture shown in FIG, the repair and reconstruction module can be designed as the following 4-layer structure:

[0120] (1) Deconvolution layer 1: Apply 3x3 convolution kernels, the number is 256, the stride is 2, and the padding method is same. Same padding is to make the feature map output after the deconvolution operation consistent with the size of the input feature map in terms of spatial dimension. This can preserve the spatial information of the input feature map to a certain extent, which is helpful for feature recovery and upsampling operations in the network. The activation function is ReLU. The purpose of this layer is to enlarge the size of the feature map and restore more details.

[0121] (2) Convolutional layer 5: 3x3 convolution kernels are applied, the number is 128, the stride is 1, and the padding method is same. Same padding ensures that after the convolution operation, the spatial size of the output feature map is consistent with the spatial size of the input feature map, or as close as possible. This helps to avoid the sharp decrease in the size of the feature map as the network layer deepens when building a deep neural network, thereby retaining more spatial information and details. The activation function is ReLU. The purpose of this layer is to further refine the features and enhance detail recovery.

[0122] (3) Deconvolution layer 2: Apply 3x3 convolution kernels, the number is 64, the stride is 2, the padding method is the same, and the activation function is ReLU. The purpose of this layer is to enlarge the feature map again to approach the size of the original video frame.

[0123] (4) Convolutional layer 6: Apply 3x3 convolution kernels, the number is 3 (the number of output channels), the step size is 1, the padding method is same, and the activation function is tanh. The purpose of this layer is to generate the repaired frame and output the repaired video frame.

[0124] It should be noted that the structure of the repair and reconstruction module also includes other possible implementation methods and is not limited to the above-mentioned module structure.

[0125] In the above step S202, the spatiotemporal features of the ith video frame are obtained through the spatial features of the preceding video frame of the ith video frame and the ith video frame, including: passing the preceding video frame of the ith video frame and the spatial features of the ith video frame through a spatiotemporal consistency module, the spatiotemporal consistency module updates the spatial features of the ith video frame based on the preceding video frame of the ith video frame, so that the ith video frame obtains a temporal dependency relationship with the preceding video frame in space; the spatiotemporal features of the ith video frame are obtained through the temporal dependency relationship and the spatial features of the ith video frame.

[0126] Specifically, the spatiotemporal consistency module uses the spatial features of the previous video frame of the i-th video frame to perform weighted calculation on the spatial features of the current i-th video frame. The weighting process takes into account the information of the previous video frame to ensure the spatiotemporal consistency of the current frame features. At this time, the spatiotemporal features of the i-th video frame are consistent with its previous video frame, so that the generated restored i-th video frame is consistent with its previous video frame, significantly improving the video quality and reducing noise and blur.

[0127] By introducing the spatiotemporal consistency module and deep learning technology, the effect and efficiency of real-time video restoration have been significantly improved. First, the spatiotemporal consistency module can fully consider the spatiotemporal relationship between the previous and next frames when processing video frames, ensuring the continuity of visual effects when the video is played continuously, and greatly reducing the visual jumps and discomfort caused by inconsistency between frames.

[0128] The spatiotemporal consistency module is trained separately in the first training stage; the feature extraction module and the repair and reconstruction module are trained together in the second training stage; the spatiotemporal consistency module used in the second training stage is obtained through the first training stage, and the parameters of the spatiotemporal consistency module remain unchanged in the second training stage.

[0129] In some embodiments, the spatiotemporal consistency module is trained by training some or all of the following loss factors: a content-style feature loss factor, which is used to characterize the feature correlation of different videos with similar content styles after being processed by the spatiotemporal consistency module; a partial-overall feature loss factor, which is used to characterize the feature correlation between some video frames and the overall video frames in the same video after being processed by the spatiotemporal consistency module; a feature offset loss factor, which is used to characterize the feature offset degree of the video after being processed by the spatiotemporal consistency module.

[0130] Specifically, the content-style feature loss factor is designed to train the spatiotemporal consistency module to ensure that even if the content of two videos is different, their visual style features can be effectively captured and distinguished by the model. This ensures that the video features extracted by the trained spatiotemporal consistency model can capture the core content and style of the video, rather than just local or surface information. The content of the video refers to the actual image information in the video, such as the shape, color, and texture of the object. The style of the video refers to the overall visual effect and consistency of the video, such as lighting, hue, and texture consistency.

[0131] The purpose of designing a partial-overall feature loss factor to train the spatiotemporal consistency module is to ensure that the temporal and spatial features within a video are continuous, to ensure that the content and style of each part of the video frame are consistent, and to avoid the situation where the style of the local area does not match the overall style after restoration. In video restoration, temporal continuity means that the features between adjacent frames should be consistent in the temporal dimension. This consistency ensures a smooth transition during video playback and avoids visual jumps or incoherence. Spatial continuity means that the features within a single video frame should be consistent in the spatial dimension. When processing videos, the features of some videos are similar to those of the entire video, thereby ensuring consistency, that is, ensuring the continuity of temporal and spatial features within the video, making the restored video more natural and coherent during playback, and having better visual effects.

[0132] The purpose of designing a feature shift loss factor to train the spatiotemporal consistency module is to ensure that the features of the video stream do not shift significantly during the process of extracting features by the spatiotemporal consistency module.

[0133] The comprehensive loss factor is obtained through the content style feature loss factor, the partial overall feature loss factor and the feature offset loss factor, and the comprehensive loss factor is comprehensively adjusted until the comprehensive loss factor converges.

[0134] In some embodiments, the content style feature loss factor is determined by the following formula 1:

[0135]

[0136] Among them, V1 and V2 are two different videos with similar content styles. is the first feature of the i-th video frame in V1 after being processed by the spatiotemporal consistency module. is the second feature of the i-th video frame in V2 after being processed by the spatiotemporal consistency module, and N is the number of video frames.

[0137] Specifically, when training the spatiotemporal consistency module, two videos V1 and V2 are randomly obtained from the video stream dataset as training samples, and the video features of the i-th frame in V1 and V2 are and Perform calculations as in formula 1 above, and process each frame in V1 and V2 in turn.

[0138] In some embodiments, the partial overall characteristic loss factor is determined by the following formula 2:

[0139]

[0140] Where M is the number of frames of the partial video, P is the partial video, O is the entire video containing the partial video P, and F(P i ) is the third feature of the i-th video frame in the partial video after being processed by the spatiotemporal consistency module, and F(O) is the fourth feature of each video frame in the overall video after being processed by the spatiotemporal consistency module.

[0141] Specifically, when training the spatiotemporal consistency module, a video V is randomly obtained from the video stream dataset as a training sample, the video V is used as the whole video O, a randomly selected segment from the whole video O is used as the partial video, and a frame extracted from the partial video is used as the partial frame P. i . Extract the overall video O and partial frames P i The characteristics of F(O) and F(P i ). Adjust the partial overall feature loss factor L through the above formula 2 part-whole Let the feature F(P i ) is close to the overall video feature F(O), thus ensuring the consistency of features within a video.

[0142] In some embodiments, in order to ensure that the features in the video stream feature extraction process do not have a significant offset, the feature offset loss factor is designed to be determined by the following formula 3:

[0143]

[0144] Where T is the total number of frames in the video, F t is the fifth feature of the video frame not processed by the spatiotemporal consistency module, F t+1 It is the sixth feature of the video frame processed by the spatiotemporal consistency module.

[0145] It should be noted that there is no correlation between the samples used in the training process of the content style feature loss factor, the part-whole feature loss factor, and the feature shift loss factor.

[0146] In some embodiments, a regularization factor is added to prevent the spatiotemporal consistency module from overfitting, as shown in Formula 4:

[0147] L reg =λ∑ ω∈W ω 2 ...Formula 4

[0148] Where λ is the regularization parameter and ω is the model weight set. 2 The acquisition and update process is as follows:

[0149] 1. Initialization: The model weights are initialized by the He initialization method before training begins. The He initialization method can effectively prevent the gradient from disappearing or exploding. The formula is: W = randn(size) × 2 / nW = randn(size) × 2 / n, where nn is the number of nodes in the previous layer.

[0150] 2. Training update: During the training process, the model weights propagate the error backwards from the output layer to the input layer through the back-propagation algorithm. At each layer, the error gradient of the layer is calculated based on the error and the current activation value. This gradient indicates how the weight should be adjusted to reduce the error. Specifically, for each weight, the direction and size of the weight adjustment are determined by calculating the partial derivative of the error with respect to the weight. The optimization algorithm used is the Adam optimizer, which combines the advantages of momentum and adaptive learning rate. In each iteration, the model weights are updated according to the gradient of the loss function to minimize the loss function.

[0151] Then, according to the content style feature loss factor L content , partial overall feature loss factor L part-whole , feature shift loss factor L shift Assign a weight coefficient, combined with the regularization factor L regThe comprehensive loss factor L is obtained, and the specific expression of L is as follows:

[0152] L=αL content +βL part-whole +γL shift +L reg ...Formula 5

[0153] The comprehensive loss factor L is adjusted comprehensively until the comprehensive loss factor converges. The parameters of the spatiotemporal consistency module can be continuously adjusted through the back propagation algorithm and the Adam optimizer until the comprehensive loss factor L converges. During the training process, the performance of the spatiotemporal consistency module on the video stream verification set is regularly verified to ensure the spatiotemporal consistency feature extraction capability of the spatiotemporal consistency module.

[0154] The spatiotemporal features of the i-th video frame and the spatial features of the i-th video frame are input into the repair and reconstruction module to obtain the repaired i-th video frame. Specifically, the noise dataset D_noise is used as the input of the repair and reconstruction module, and the noise-free dataset D_original is used as the target output for training. In the data preparation stage, the training sample (x i ,y i ), where x i ∈D_noise,y i ∈D_original. The video frame x of the noise dataset i Input into the convolutional neural network (CNN) to generate the restored video frame output Compute repair output and i The loss between , the mean square error (MSE) can be used as the loss function, as shown in the following formula 6:

[0155]

[0156] And through the back propagation algorithm, the gradient of the loss function to the parameters of the repair and reconstruction module is calculated, and the weight of the repair and reconstruction module is updated. Using the Adam optimizer, the update formula is as follows:

[0157] θ t+1 =θ t -η...Formula 7

[0158] The deep learning model of convolutional neural network (CNN) is used to effectively extract and repair complex detail information in the video, and it can perform well even when processing dynamic scenes. In addition, the method can instantly repair and enhance the video quality, reduce processing delays, and provide an immediate visual improvement experience through real-time processing technology. This efficient and low-latency processing method not only improves the quality of video restoration, but also enhances the practical application value of the system. It is suitable for multiple fields such as movie restoration, surveillance video enhancement, and personal video editing. In summary, the present invention improves the video restoration effect while ensuring the real-time and stability of the processing, and has significant practical application prospects and promotion value.

[0159] like Figure 4 As shown, a real-time video stream processing device 400 is provided in an embodiment of the present application. The device 400 includes

[0160] The extraction module 401 is used for extracting features of the i-th video frame in the real-time video stream through the feature extraction module to obtain the spatial features of the i-th video frame;

[0161] A processing module 402 is used to obtain the spatiotemporal features of the i-th video frame through a spatiotemporal consistency module based on the spatial features of the preceding video frame of the i-th video frame and the i-th video frame; the spatiotemporal consistency module is used to extract the temporal dependency relationship between the i-th video frame and the preceding video frame in space;

[0162] A repair module 403 is used to repair the spatial features of the i-th video frame through a repair and reconstruction module based on the spatiotemporal features of the i-th video frame to obtain a repaired i-th video frame;

[0163] The synthesis module 404 is used to obtain a restored real-time video stream according to each restored video frame in the real-time video stream.

[0164] This application applies the trained deep learning model to the actual video restoration task, inputs the video frame sequence to be restored frame by frame into the trained deep learning model, and the model extracts the spatial features of each frame, and updates the spatial features using the spatiotemporal features of the previous video frame to ensure the coherence between frames, and then restores and reconstructs the extracted spatial features and the consistent spatiotemporal features to generate the restored frame. This process is repeated on each frame, and finally the restored video frame sequence is output. The model can significantly improve the video quality and reduce noise and blur while maintaining the overall visual coherence of the video.

[0165] Optionally, the processing module 402 is specifically configured to:

[0166] The spatiotemporal consistency module is trained separately in the first training phase;

[0167] The feature extraction module and the restoration and reconstruction module are jointly trained in the second training stage; the spatiotemporal consistency module used in the second training stage is obtained through the first training stage, and the parameters of the spatiotemporal consistency module remain unchanged in the second training stage.

[0168] Optionally, the processing module 402 is specifically configured to:

[0169] The spatiotemporal consistency module is trained by some or all of the following loss factors:

[0170] Content style feature loss factor, which is used to characterize the feature correlation of different videos with similar content styles after being processed by the spatiotemporal consistency module;

[0171] Partial-overall feature loss factor, used to characterize the feature correlation between partial video frames and overall video frames in the same video after being processed by the spatiotemporal consistency module;

[0172] The feature shift loss factor is used to characterize the feature shift degree of the video after being processed by the spatiotemporal consistency module.

[0173] Optionally, the spatiotemporal consistency module adjusts the parameters of the spatiotemporal consistency module by the comprehensive loss value of each round of training until a spatiotemporal consistency module that meets the training termination condition is obtained; wherein the comprehensive loss value of each round of training is obtained by the following formula:

[0174] L=αL content +βL part-whole +γL shift +L reg

[0175] Among them, L content is the content style feature loss factor, L part-whole is the overall characteristic loss factor of the part, L shift is the characteristic shift loss factor, L reg is a regularization term, which prevents the spatiotemporal consistency module from overfitting, and α, β, and γ are weight parameters of each loss factor.

[0176] Optionally, the processing module 402 is specifically configured to:

[0177] The content style feature loss factor is determined by:

[0178]

[0179] Among them, V1 and V2 are two different videos with similar content styles. is the first feature of the i-th video frame in V1 after being processed by the spatiotemporal consistency module. is the second feature of the i-th video frame in V2 after being processed by the spatiotemporal consistency module, and N is the number of video frames.

[0180] Optionally, the processing module 402 is specifically configured to:

[0181] The partial overall characteristic loss factor is determined by:

[0182]

[0183] Where M is the number of frames of the partial video, P is the partial video, O is the entire video containing the partial video P, and F(P i ) is the third feature of the i-th video frame in the partial video after being processed by the spatiotemporal consistency module, and F(O) is the fourth feature of each video frame in the overall video after being processed by the spatiotemporal consistency module.

[0184] Optionally, the processing module 402 is specifically configured to:

[0185] The characteristic shift loss factor is determined by:

[0186]

[0187] Where T is the total number of frames in the video, F t is the fifth feature of the video frame not processed by the spatiotemporal consistency module, F t+1 It is the sixth feature of the video frame processed by the spatiotemporal consistency module.

[0188] Based on the same technical concept, the embodiment of the present application provides a computer device, such as Figure 5 As shown, it includes at least one processor 501 and a memory 502 connected to the at least one processor. The specific connection medium between the processor 501 and the memory 502 is not limited in the embodiment of the present application. Figure 5 For example, the processor 501 and the memory 502 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0189] In the embodiment of the present application, the memory 502 stores instructions that can be executed by at least one processor 501, and the at least one processor 501 can perform the steps of the above-mentioned real-time video stream processing method by executing the instructions stored in the memory 502.

[0190] Among them, the processor 501 is the control center of the computer device, which can use various interfaces and lines to connect various parts of the computer device, and process the video stream by running or executing instructions stored in the memory 502 and calling data stored in the memory 502, and ensure the temporal and spatial coherence and consistency between the frames of the real-time video stream after the repair process. Optionally, the processor 501 may include one or more processing units, and the processor 501 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 501. In some embodiments, the processor 501 and the memory 502 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on independent chips.

[0191] The processor 501 may be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0192] The memory 502 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 502 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 502 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 502 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0193] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program executable by a computer device. When the program runs on the computer device, the computer device executes the steps of the above-mentioned real-time video stream processing method.

[0194] Based on the same inventive concept, the embodiment of the present application also provides a computer program product, which includes: computer program code, when the computer program code is run on a computer, the computer executes any of the interface testing methods discussed above. Since the principle of solving the problem by the above computer program product is similar to that of the interface testing method, the implementation of the above computer program product can refer to the implementation of the method, and the repeated parts will not be repeated.

[0195] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0196] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0197] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0198] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0199] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for processing a real-time video stream, characterized in that: include: For an i-th video frame in a real-time video stream, extracting features of the i-th video frame by a feature extraction module to obtain spatial features of the i-th video frame; Based on the spatial features of the preceding video frame of the i-th video frame and the i-th video frame, the spatial and temporal features of the i-th video frame are obtained by a spatial and temporal consistency module; the spatial and temporal consistency module is used to extract the temporal dependency relationship between the i-th video frame and the preceding video frame in space; Based on the spatiotemporal features of the i-th video frame, the spatial features of the i-th video frame are repaired by a repairing and reconstructing module to obtain a repaired i-th video frame; A restored real-time video stream is obtained according to each restored video frame in the real-time video stream.

2. The method according to claim 1, characterized in that The spatiotemporal consistency module is trained separately in the first training phase; The feature extraction module and the restoration and reconstruction module are jointly trained in the second training stage; the spatiotemporal consistency module used in the second training stage is obtained through the first training stage, and the parameters of the spatiotemporal consistency module remain unchanged in the second training stage.

3. The method according to claim 1, characterized in that The spatiotemporal consistency module is trained by some or all of the following loss factors: Content style feature loss factor, which is used to characterize the feature correlation of different videos with similar content styles after being processed by the spatiotemporal consistency module; Partial-overall feature loss factor, used to characterize the feature correlation between partial video frames and overall video frames in the same video after being processed by the spatiotemporal consistency module; The feature shift loss factor is used to characterize the feature shift degree of the video after being processed by the spatiotemporal consistency module.

4. The method according to claim 3, characterized in that The spatiotemporal consistency module adjusts the parameters of the spatiotemporal consistency module by the comprehensive loss value of each round of training until a spatiotemporal consistency module that meets the training termination condition is obtained; wherein the comprehensive loss value of each round of training is obtained by the following formula: L=αL content +βL part-whole +γL shift +L reg Among them, L content is the content style feature loss factor, L part-whole is the overall characteristic loss factor of the part, L shift is the characteristic offset loss factor, L reg is a regularization term, which prevents the spatiotemporal consistency module from overfitting, and α, β, and γ are weight parameters of each loss factor.

5. The method according to claim 4, characterized in that The content style feature loss factor is determined by: Among them, V1 and V2 are two different videos with similar content styles. is the first feature of the i-th video frame in V1 after being processed by the spatiotemporal consistency module. is the second feature of the i-th video frame in V2 after being processed by the spatiotemporal consistency module, and N is the number of video frames.

6. The method according to claim 4, characterized in that The partial overall characteristic loss factor is determined by: Where M is the number of frames of the partial video, P is the partial video, O is the entire video containing the partial video P, and F(P i ) is the third feature of the i-th video frame in the partial video after being processed by the spatiotemporal consistency module, and F(O) is the fourth feature of each video frame in the overall video after being processed by the spatiotemporal consistency module.

7. The method according to claim 4, characterized in that The characteristic shift loss factor is determined by: Where T is the total number of frames in the video, F t is the fifth feature of the video frame not processed by the spatiotemporal consistency module, F t+1 It is the sixth feature of the video frame processed by the spatiotemporal consistency module.

8. A real-time video stream processing device, characterized in that: include: An extraction module is used to extract features of an i-th video frame in a real-time video stream through a feature extraction module to obtain spatial features of the i-th video frame; A processing module, configured to obtain the spatiotemporal features of the i-th video frame through a spatiotemporal consistency module based on the spatial features of the preceding video frame of the i-th video frame and the i-th video frame; the spatiotemporal consistency module is configured to extract the temporal dependency relationship between the i-th video frame and the preceding video frame in space; A repair module, used for repairing the spatial features of the i-th video frame through a repair and reconstruction module based on the spatiotemporal features of the i-th video frame to obtain a repaired i-th video frame; The synthesis module is used to obtain a restored real-time video stream according to each restored video frame in the real-time video stream.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of any one of the methods of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: It stores a computer program executable by a computer device. When the program is run on the computer device, the computer device executes the steps of any method described in claims 1 to 7.

11. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of any method described in claims 1 to 7 are implemented.

Images