Video processing method and device based on neural network, equipment and medium

Abstract

This invention discloses a video processing method based on a neural network. It includes: at the transmitting end, decomposing the original video data to obtain semantic condition information Z, motion parameters M, and residual information R, where Z, M, and R satisfy decoupling constraints in the feature space; compressing Z to obtain a semantically compressed representation Z′; predicting motion parameters Mpred based on Z and calculating the motion residual; sparsely encoding R to obtain a sparse residual representation; and jointly encoding Z′, the motion residual, and the sparse residual representation to generate compressed data C; at the receiving end, recovering Z, M, and R from C; generating a reconstructed video V^ using a generative model G; and correcting V^ based on R to obtain the output video V^′. By converting the video representation into a parameterized representation based on the generative driving signal, and combining the decoupling constraint coding mechanism with the generative model reconstruction mechanism, data redundancy is effectively reduced and coding efficiency is improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of video technology, and more specifically, to a video processing method, apparatus, device, and medium based on neural networks. Background Technology

[0002] With the rise of emerging applications such as metaverse, real-time digital human interaction, and cloud gaming, unprecedentedly stringent requirements have been placed on the real-time performance of video coding. Existing video coding methods mainly suffer from the following problems:

[0003] 1) Pixel-based compression suffers from high data redundancy and limited encoding efficiency;

[0004] 2) Poor system scalability;

[0005] 3) Insufficient real-time processing capability.

[0006] Existing video processing methods still have significant room for improvement in terms of low bandwidth, low latency, and system scalability. Summary of the Invention

[0007] This invention provides a video processing method, apparatus, device, and storage medium based on neural networks. By transforming video data from traditional pixel-level representation to parameterized representation based on generation driving signals, and combining decoupled constraint coding mechanism with generative model reconstruction mechanism, redundant information transmission is reduced, coding efficiency is improved, and video reconstruction quality is enhanced.

[0008] In a first aspect, embodiments of the present invention provide a video processing method based on a neural network, applied at a transmitting end, comprising:

[0009] Obtain the raw video data to be transmitted;

[0010] The original video data is decomposed to obtain a generation driving signal; the generation driving signal includes: semantic condition information Z, motion parameters M, and residual information R, and Z, M, and R satisfy decoupling constraint relationship in feature space; Z is used to describe the semantic structure of the video content, M is used to describe the spatial and temporal dynamic changes of objects in the video, and R is used to represent high-frequency detail compensation information;

[0011] The semantic condition information Z is compressed and encoded to obtain the semantic compressed representation Z′;

[0012] Based on the semantic conditional information Z, prediction is performed to obtain the predicted motion parameters Mpred, and the motion residuals Mres are calculated; where... ;

[0013] The residual information R is sparsely encoded to obtain the sparse residual representation Rsparse;

[0014] Z′, Mres, and Rsparse are jointly encoded to generate compressed data C.

[0015] As one embodiment, the joint encoding of Z′, Mres, and Rsparse to generate compressed data C includes:

[0016] L = D + λB;

[0017] in:

[0018] L represents the value of the encoding optimization objective function;

[0019] D represents the distortion measure between the reconstructed video and the original video;

[0020] B represents the bitrate or data volume of the encoded data;

[0021] λ represents the weighting parameter used to balance the relationship between distortion and bit rate.

[0022] As one embodiment, the motion residual Mres is obtained in the following way:

[0023] Based on semantic conditional information Z, the motion parameter M is predicted to obtain the predicted motion parameter Mpred.

[0024] And Mres is obtained based on the difference between M and Mpred;

[0025] During the encoding process, only the motion residual Mres is encoded, and the motion parameter M is not directly involved in the encoding.

[0026] As one embodiment, the semantic condition information Z includes at least one of text information and audio information;

[0027] The audio information includes speech semantic features or environmental sound features;

[0028] The motion parameter M includes key point motion, posture changes, or facial expression parameters.

[0029] Secondly, embodiments of the present invention provide a video processing method based on a neural network, applied at a receiving end, comprising:

[0030] Receive compressed data; the compressed data is obtained based on the aforementioned method; the compressed data includes: a compressed representation Z′ of semantic condition information Z, motion residual Mres and a sparse residual representation Rsparse; the semantic condition information Z is extracted from the original video data, Mres is the prediction residual of motion parameters, Rsparse is the sparse representation of residual information, and the residual information R is used to represent high-frequency detail compensation information;

[0031] The driving signal is recovered and generated based on the compressed data; wherein the recovered driving signal includes: semantic condition information Z, motion parameters M, and residual information R;

[0032] Based on the semantic condition information Z and motion parameters M, the reconstructed video V^ is obtained using the generative model G. The reconstructed video V^ is then corrected based on the residual information R to obtain the final output video V^′.

[0033] The generative model is a conditional generative model, comprising a semantic control branch, a motion control branch, and a residual injection branch. The semantic control branch is used to generate the semantic content of the video based on the semantic condition information Z. The motion control branch is used to control the spatial position changes and temporal dynamics of objects in the video based on the motion parameters M. The residual injection branch is used to perform detail compensation on the intermediate features of the generative model based on the residual information R.

[0034] Thirdly, embodiments of the present invention provide a video processing method based on a neural network, including:

[0035] The received compressed data is decoded to obtain semantic condition information Z, motion parameters M, and residual information R;

[0036] Based on the semantic condition information Z, the motion parameters M, and the residual information R, a reconstructed video V^ is generated through a generative model.

[0037] The generative model is a conditional generative model, which includes a semantic control branch, a motion control branch, and a residual injection branch.

[0038] The semantic control branch is used to generate the semantic content of the video based on the semantic condition information Z;

[0039] The motion control branch is used to control the spatial position changes and temporal dynamics of objects in the video according to the motion parameter M;

[0040] The residual injection branch is used to perform detail compensation on the intermediate features of the generated model based on the residual information R;

[0041] The residual information R does not participate in the initial content construction process of the generated model.

[0042] Fourthly, embodiments of the present invention provide a video processing apparatus based on a neural network, applied at a transmitting end, comprising:

[0043] The acquisition module is used to acquire the raw video data to be transmitted;

[0044] The decomposition module is used to decompose the original video data to obtain a generation driving signal. The generation driving signal includes semantic condition information Z, motion parameters M, and residual information R, and Z, M, and R satisfy decoupling constraint relationships in the feature space. Z is used to describe the semantic structure of the video content, M is used to describe the spatial and temporal dynamic changes of objects in the video, and R is used to represent high-frequency detail compensation information.

[0045] The semantic compression module is used to compress and encode the semantic condition information Z to obtain a semantically compressed representation Z′;

[0046] The motion parameter calculation module is used to perform prediction based on the semantic condition information Z, obtain the predicted motion parameters Mpred, and calculate the motion residuals Mres; wherein... ;

[0047] The parameter information R encoding module is used to perform sparse encoding on the residual information R to obtain a sparse residual representation Rsparse;

[0048] The joint encoding module is used to jointly encode Z′, Mres, and Rsparse to generate compressed data C.

[0049] As an example, the joint encoding module is used to generate compressed data by minimizing the following optimization objective function:

[0050] L = D + λB;

[0051] in:

[0052] L: Represents the value of the encoding optimization objective function;

[0053] D: Represents the distortion measure between the reconstructed video and the original video;

[0054] B: Indicates the bitrate or data volume of the encoded data;

[0055] λ: Represents the weighting parameter used to balance the relationship between distortion and bit rate.

[0056] Fifthly, embodiments of the present invention provide a video processing apparatus based on a neural network, configured at a receiving end, comprising:

[0057] A receiving module is used to receive compressed data; the compressed data is obtained based on the method described in the first aspect; the compressed data includes: a compressed representation Z′ of semantic condition information Z, motion residual Mres, and sparse residual representation Rsparse; the semantic condition information Z is extracted from the original video data, Mres is the prediction residual of motion parameters, Rsparse is the sparse representation of residual information, and the residual information R is used to represent high-frequency detail compensation information.

[0058] The decoding module is used to recover the generated driving signal based on the compressed data; wherein the recovered generated driving signal includes: semantic condition information Z, motion parameters M, and residual information R;

[0059] The video generation module is used to obtain a reconstructed video V^ using a generation model G based on the semantic condition information Z and motion parameters M, and to correct the reconstructed video V^ based on the residual information R to obtain the final output video V^′.

[0060] The generative model is a conditional generative model, comprising a semantic control branch, a motion control branch, and a residual injection branch. The semantic control branch is used to generate the semantic content of the video based on the semantic condition information Z. The motion control branch is used to control the spatial position changes and temporal dynamics of objects in the video based on the motion parameters M. The residual injection branch is used to perform detail compensation on the intermediate features of the generative model based on the residual information R.

[0061] In a sixth aspect, embodiments of the present invention provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the program to implement the above-described method.

[0062] In a seventh aspect, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the above-described method.

[0063] Compared with the prior art, the embodiments of the present invention convert the video representation from pixel-level representation to a parameterized representation based on the generation driving signal, and realize the structural reconstruction of the video data representation method through decoupled modeling of semantic information, motion information and residual information;

[0064] Z′ is obtained by compressing and encoding semantic condition information, Mres is obtained by predicting residual information, and Rsparse is obtained by sparse representation of residual information. Z′, Mres and Rsparse are jointly encoded to effectively reduce data redundancy.

[0065] The video is reconstructed by a conditional generation model and detailed compensation is performed by combining residual information, which significantly reduces the amount of data transmitted while ensuring video quality.

[0066] This improves coding efficiency and the system's adaptability in low-bandwidth, low-latency scenarios. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 This is an overall flowchart of video processing based on the generation of driving signals provided in an embodiment of the present invention;

[0069] Figure 2 This is a diagram illustrating the architecture of a video processing system based on generated driving signals, provided in an embodiment of the present invention.

[0070] Figure 3 This is a video decomposition flowchart provided in an embodiment of the present invention;

[0071] Figure 4 A flowchart of the generation driver coding process provided in an embodiment of the present invention;

[0072] Figure 5 A flowchart of the generation and reconstruction process provided for embodiments of the present invention;

[0073] Figure 6 A system operation architecture diagram provided for embodiments of the present invention;

[0074] Figure 7 This is a schematic diagram of the electronic device structure provided in an embodiment of the present invention. Detailed Implementation

[0075] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0076] The principles and implementation methods of the video processing method, apparatus, system, electronic device and storage medium of this embodiment will be described in detail below, so that those skilled in the art can understand this embodiment without creative effort.

[0077] Example 1

[0078] Figure 1 The overall flowchart of the video processing method based on neural networks provided in the embodiments of the present invention is shown.

[0079] The core of this embodiment is to represent video as a parameterized expression composed of semantic conditional information, motion parameters, and residual information. This allows for video reconstruction using a generative model, rather than encoding pixel data frame by frame. This method is applied to video processing systems. For example... Figure 1 As shown, the video processing method includes the following steps:

[0080] Step 101: Input video V.

[0081] In step 101, the raw video data to be transmitted is obtained.

[0082] Step 102: Decompose the video to obtain Z, M, and R.

[0083] In step 102, the original video data is decomposed to obtain the generation driving signal. The generation driving signal includes semantic condition information Z, motion parameters M, and residual information R, and Z, M, and R satisfy decoupling constraints in the feature space. Z is used to describe the semantic structure of the video content, M is used to describe the spatial and temporal dynamic changes of objects in the video, and R is used to represent high-frequency detail compensation information.

[0084] Step 103: Encoding process.

[0085] In step 103, the semantic conditional information Z is compressed and encoded to obtain a semantically compressed representation Z′. Based on the semantic conditional information Z and the motion parameters M, prediction is performed to obtain the predicted motion parameters Mpred, and the motion residuals Mres are calculated; where, The residual information R is sparsely encoded to obtain the sparse residual representation Rsparse.

[0086] Step 104: Joint encoding.

[0087] In step 104, Z′, Mres, and Rsparse are jointly encoded to generate compressed data C.

[0088] Step 105: Data transmission.

[0089] The sending end transmits the compressed data C from step 104 to the receiving end via the transmission network. The compressed data includes: the compressed representation Z′ of the semantic condition information Z, the motion residual Mres, and the sparse residual representation Rsparse; the semantic condition information Z is extracted from the original video data, Mres is the prediction residual of the motion parameters, Rsparse is the sparse representation of the residual information, and the residual information R is used to represent the high-frequency detail compensation information.

[0090] Step 106: Decoding process.

[0091] After receiving the compressed data C, the receiving end decodes it.

[0092] Step 107: Restore and generate drive signals Z, M, and R.

[0093] In step 107, the driving signal is recovered and generated based on the compressed data. The recovered driving signal includes semantic condition information Z, motion parameters M, and residual information R.

[0094] Step 108: Reconstruct video V^ based on the generative model G.

[0095] In step 108, the reconstructed video is generated based on V^=G(Z,M,R).

[0096] Step 109: Residual Correction.

[0097] In step 108, the reconstructed video V^ is obtained using the generative model G based on the semantic conditional information Z and motion parameters M. In step 109, the reconstructed video V^ is corrected based on the residual information R to obtain the final output video V^′. The generative model is a conditional generative model, including a semantic control branch, a motion control branch, and a residual injection branch. The semantic control branch is used to generate the semantic content of the video based on the semantic conditional information Z. The motion control branch is used to control the spatial position changes and temporal dynamics of objects in the video based on the motion parameters M. The residual injection branch is used to perform detail compensation on the intermediate features of the generative model based on the residual information R.

[0098] Step 110: Output video.

[0099] The following provides a detailed description of steps 101 to 110 of the video processing method in this embodiment.

[0100] Step 101: Input video V.

[0101] Step 101 allows you to input any video data, such as digital human videos or other types of video data.

[0102] Step 102: Decompose the video to obtain Z, M, and R.

[0103] In step 102, the input raw video data is decomposed to obtain the generation drive signal.

[0104] The generation driving signal is used to reconstruct the video at the decoding end based on the generation model. In this embodiment, the generation driving signal may include: semantic condition information Z, motion parameters M, and residual information R, and Z, M, and R satisfy decoupling constraints in the feature space. Z is used to describe the semantic structure of the video content of the original video data, M is used to describe the spatial and temporal dynamic changes of objects in the original video, i.e., the motion state of the objects. R is used to represent high-frequency detail compensation information, and R is a parameterized feature. It can be understood that Z, M, and R are not pixel-level features of the video, but rather parameterized features of the video. The semantic condition information Z may include at least one of text information and audio information, and the audio information may include speech semantic features or ambient sound features. The motion parameters M include: keypoint motion, pose changes, or facial expression parameters.

[0105] In this step, deep feature analysis and decoupling processing are performed on the input raw video data to extract a set of compact and efficient generation driving signals. These generation driving signals are the core basis for subsequent high-fidelity video reconstruction using the generation model at the decoding end. This embodiment adopts a hierarchical decoupling strategy of "semantic-motion-detail" to highly abstract and parameterize the video content. Specifically, the generation driving signals are decomposed into three orthogonal or weakly correlated feature components: semantic conditional information Z, motion parameters M, and residual information R.

[0106] In this implementation, the semantic condition information Z, the motion parameters M, and the residual information R satisfy the decoupling constraint relationship:

[0107] Z is used to define the semantic content distribution of the video;

[0108] M is only used to describe spatial and temporal motion changes and does not participate in semantic construction;

[0109] R is only used to compensate for errors in high-frequency details of the generative model.

[0110] In the feature space, the semantic conditional information Z, the motion parameters M, and the residual information R satisfy a decoupling constraint relationship, where:

[0111] Corr(Z, M) ≈ 0, indicating that Z and M are approximately independent.

[0112] Corr(Z, R) ≈ 0, indicating that Z and R are approximately independent.

[0113] Corr (M, R) < ε, indicating a weak correlation between M and R;

[0114] The above constraints ensure that the generated driving signal has minimal redundancy in its representation, thereby improving compression efficiency.

[0115] Semantic conditional information Z is used to describe the intent of video content. It captures the static content, background environment, and high-level semantic logic of the video. Semantic conditional information Z can be designed as a multimodal embedding vector or feature map, specifically including at least one of the following forms:

[0116] Textual semantic features: Natural language processing models (such as Transformer-type models) are used to transform scene descriptions, action tags, or plot summaries from videos into high-dimensional text embeddings. For example, "a running dog" in the scene can be transformed into a corresponding text vector to guide the generative model in constructing basic object shapes.

[0117] Audio semantic features: Semantic information is extracted from audio using an audio encoder (such as Wav2Vec or CLAP). This includes not only transcribed text features of the speech content, but also ambient sounds (such as wind and rain) or speaker timbre features (Speaker Embedding). These features serve as conditional inputs, guiding the video generation model to generate visual content that matches the rhythm and emotion of the voice (e.g., generating corresponding lip-sync amplitude based on accents).

[0118] Motion parameters M describe the geometric state and dynamic trajectory of objects in the video as they change over time. This embodiment focuses more on the structured motion representation of objects to ensure the temporal coherence of the generated video. Motion parameters M may include:

[0119] Keypoint motion trajectory: Extract the coordinate changes of key feature points (such as 68 key points of a face, skeletal joints of the human body) of people or objects in the video over time. These sparse keypoints form the skeleton of the object's movement.

[0120] Pose and expression parameters: Parametric human or face models (such as SMPL or 3DMM models) are used to solve the motion in the video into low-dimensional pose and expression coefficients. For example, a set of values ​​can be used to accurately describe the raising of eyebrows, the raising of the corners of the mouth, or the angle of arm waving.

[0121] Depth and structural information: The motion parameter M can also include a monocular depth estimation map to describe the changes in the three-dimensional geometry of the scene, ensuring that the generative model can understand the foreground and background occlusion relationships of objects during reconstruction.

[0122] Residual Information R: Although semantic conditional information Z and motion parameters M can construct the contours and dynamics of a video, residual information R is introduced in this embodiment to help the generative model better reproduce all the texture details in the original video (such as the specific patterns of clothing and complex noise in the background). R is not a pixel difference, but a parameterized feature residual. It represents the difference in feature space between the initial output of the generative model and the original video, given Z and M. R can be a high-frequency feature compensation vector. R can be designed to awaken specific texture channels in the generative model, or to supplement lost local details back into the reconstruction process through feature injection. Compact Representation: Compared to the original pixel residual, R has undergone feature compression, retaining only the key high-frequency information that is sensitive to human vision or required for machine recognition, thereby greatly reducing the amount of data while ensuring image quality.

[0123] Step 103: Encoding process.

[0124] In step 103, the semantic conditional information Z is compressed and encoded to obtain a semantically compressed representation Z′. Based on the semantic conditional information Z, prediction is performed to obtain the predicted motion parameters Mpred, and the motion residuals Mres are calculated. The residual information R is sparsely encoded to obtain the sparse residual representation Rsparse.

[0125] In step 103, instead of directly encoding the original video pixels, the semantic compressed representation Z', motion residual Mres, and sparse residual representation Rsparse are jointly encoded to generate compressed data C.

[0126] The compressed data obtained is as follows:

[0127] C = Enc (Z', Mres, Rsparse)

[0128] In this embodiment, the encoding function Enc includes:

[0129] 1) Semantic encoding:

[0130] By performing low-dimensional embedding representation or discrete token encoding on Z, a semantically compressed representation Z' is obtained;

[0131] 2) Motion Predictive Coding:

[0132] Z-pred prediction of motion parameters Mpred based on semantic conditional information;

[0133] Calculate the motion residuals:

[0134] Mres = M - Mpred (Z);

[0135] Only the motion residuals Mres are encoded;

[0136] 3) Residual sparse coding:

[0137] High-frequency feature extraction is performed on the residual information R, retaining only the Top-K important features;

[0138] And it is compressed using entropy coding;

[0139] in:

[0140] Enc: The encoding function;

[0141] C: for compressing data;

[0142] Z': Represents the low-dimensional representation of semantic condition information Z after compression encoding, or a discrete token sequence;

[0143] Mres: Represents the residual of the motion parameter M relative to the predicted value Mpred, i.e.:

[0144] Mres = M - Mpred (Z);

[0145] Rsparse: Represents the high-frequency feature representation of residual information R after sparsification;

[0146] Z', Mres, and Rsparse correspond to semantic compression representation, motion prediction residual, and sparse detail compensation information, respectively.

[0147] Constraint coding mechanism for generating driving signals

[0148] In this embodiment, the core of the invention lies in achieving efficient compression of video data through a constraint coding mechanism that generates driving signals.

[0149] Specifically, it includes:

[0150] 1) Predict motion parameters Mpred using semantic conditional information Z to reduce redundant information;

[0151] 2) By using residual information R to compensate only for high-frequency details, the amount of data is reduced;

[0152] 3) Improve coding stability by decoupling the constraints between Z, M, and R;

[0153] 4) Achieve parameter-driven video reconstruction through a conditional generation model;

[0154] Through the above mechanism, the present invention realizes the transformation from pixel encoding to generation-driven encoding.

[0155] In this embodiment, Z, M, and R represent semantic condition information, motion parameters, and residual information, respectively, as defined above.

[0156] Specifically, compressed data can be obtained by minimizing the following optimization objective function:

[0157] L = D + λB;

[0158] in:

[0159] L: Represents the value of the encoding optimization objective function;

[0160] D: Represents the distortion measure between the reconstructed video and the original video;

[0161] B: Indicates the bitrate or data volume of the encoded data;

[0162] λ: Represents the weighting parameter used to balance the relationship between distortion and bit rate.

[0163] To optimize the loss function L, the network parameters need to be continuously adjusted during training to find the optimal balance between reconstruction quality (distortion D) and compression efficiency (bitrate B). This optimization process may include the following steps:

[0164] 1. Construct a differentiable overall architecture (end-to-end neural network)

[0165] An end-to-end neural network consists of an encoder that generates driving signals, a parameterized encoding module, and a decoder that reconstructs video from the signals. It is constructed as a globally differentiable deep neural network, and all parameters can be jointly optimized by calculating gradients.

[0166] The analysis network (Encoder) is responsible for extracting the generation driving signals Z, M, and R from the original video, and further obtaining the semantic compressed representation Z', motion residual Mres, and sparse residual representation Rsparse through a parameterized encoding process.

[0167] Specifically, the semantic condition information Z is compressed and encoded to obtain Z'; the motion parameter M is predicted based on the semantic information Z to obtain Mpred, and the motion residual Mres is calculated; the residual information R is sparsified to obtain Rsparse.

[0168] The synthesis network (Decoder / Generator) is responsible for generating the reconstructed video based on the decoded generation drive signals Z, M, and R, where:

[0169] Z is obtained by decoding Z';

[0170] M is obtained by recovering Mpred(Z) and Mres;

[0171] R is obtained by decoding with Rsparse;

[0172] The entropy model is used to probabilistically model the semantic compressed representation Z', motion residual Mres, and sparse residual representation Rsparse, and to estimate the code rate B required for encoding.

[0173] 2. Define the optimization function L as the loss function.

[0174] During the training phase, L = D + λB, and the optimization objective is the loss function of the entire neural network. The distortion term D (Perceptual Loss), since the goal is to generate realistic videos, can be a perceptual loss, calculated using a pre-trained network (such as VGG, CLIP, or a discriminator). It ensures high similarity between the original and generated videos by comparing the differences between the "original video" and the "reconstructed video" in the high-level feature space. The bitrate term B (Rate Estimation) is estimated using an entropy model. The entropy model learns the probability distributions of the semantic compressed representation Z', the motion residual Mres, and the sparse residual representation Rsparse, and estimates the number of bits required for encoding based on these distributions.

[0175] 3. End-to-end optimization using gradient descent

[0176] Using massive amounts of video data as the training set, all parameters of the entire network are updated through stochastic gradient descent (SGD) or its variants (such as Adam). The training process includes: Forward propagation: Inputting an original video, the network generates driving signals Z, M, and R, and further obtains a semantic compressed representation Z', motion residual Mres, and a sparse residual representation Rsparse. The decoder reconstructs the video based on these signals. Loss calculation: Calculating the perceptual difference D between the reconstructed video and the original video, and simultaneously calculating B using an entropy model, ultimately obtaining the total loss L = D + λB. Backpropagation: Calculating the gradient of the loss L with respect to each parameter in the network. Parameter update: Based on the calculated gradients, updating all parameters of the network using an optimizer (such as Adam). Through large-scale training, the network's L is minimized.

[0177] It is worth mentioning that encoding the generated driving signal to obtain compressed data may include: obtaining predicted motion information based on semantic conditional information Z, where motion parameters include real motion information; obtaining motion information residuals based on the predicted and real motion information; and encoding compressed data based on semantic conditional information Z and the motion information residuals. Thus, by jointly compressing Z and M, redundancy is further reduced and the compression ratio is increased. Real motion information refers to the motion information of objects in the video obtained from the analysis of the original video data, and this motion information may include various specific motion parameters of the objects.

[0178] Step 104: Joint encoding.

[0179] In step 104, Z′, Mres, and Rsparse are jointly encoded to generate compressed data C.

[0180] Step 105: Data transmission.

[0181] The sending end transmits the compressed data C from step 104 to the receiving end via the transmission network. The compressed data includes: the compressed representation Z′ of the semantic condition information Z, the motion residual Mres, and the sparse residual representation Rsparse; the semantic condition information Z is extracted from the original video data, Mres is the prediction residual of the motion parameters, Rsparse is the sparse representation of the residual information, and the residual information R is used to represent the high-frequency detail compensation information.

[0182] Step 106: Decoding process.

[0183] After receiving the compressed data, it is decoded.

[0184] The compressed data includes: semantic compressed representation Z', motion residual Mres, and sparse residual representation Rsparse.

[0185] The result obtained through decoding is:

[0186] Z = Dec_Z(Z');

[0187] M = Mpred(Z) + Mres;

[0188] R = Dec_R(Rsparse);

[0189] Step 107: Restore and generate drive signals Z, M, and R.

[0190] At the receiving end (also known as the decoding end), the compressed data is decoded to obtain the generation drive signal, wherein:

[0191] Semantic condition information Z is recovered from semantically compressed representation Z'.

[0192] The motion parameters M are recovered based on the semantic conditional information Z and the motion residuals Mres.

[0193] Residual information R is recovered from the sparse residual representation Rsparse;

[0194] At the receiving end, the video can be reconstructed using a generative model based on the recovered generative driving signal.

[0195] Step 108: Reconstruct video V^ based on the generative model G.

[0196] Generate the reconstructed video based on V^=G(Z,M,R).

[0197] In this embodiment, the generative model G is a conditional generative model, including:

[0198] 1) Semantic generation branch:

[0199] Receive semantic condition information Z to construct the basic content of the video;

[0200] 2) Motion control branch:

[0201] Receive motion parameters M to control the spatial and temporal changes of objects in the video;

[0202] 3) Residual injection branch:

[0203] The residual information R is received and applied to the intermediate layers of the generative model through feature injection or skip connections.

[0204] in:

[0205] Z is used to control the semantic content output of the generative model;

[0206] M is used to control the spatial structure and temporal dynamics of the generated results;

[0207] R is applied to the intermediate layers of the generative model through feature injection for detail compensation, and does not participate in the initial content generation process;

[0208] The residual information R does not participate in the initial content construction of the generative model, but is only used for detail enhancement in the later stages of generation.

[0209] in:

[0210] V^ represents the output video reconstructed using the generative model G;

[0211] G is a conditional generative model, which includes a semantic control branch, a motion control branch, and a residual injection branch;

[0212] Z represents semantic condition information, used to describe the semantic category and structure of video content;

[0213] M represents motion parameters, which are used to describe the spatial position changes and temporal dynamics of objects in the video.

[0214] R represents residual information, which is high-frequency compensation information in the feature space and is used to enhance the details of the generative model output.

[0215] In this invention, Z, M, and R correspond to semantic control, motion control, and detail compensation, respectively, and they have a clear division of labor and constraint relationship in the generation process.

[0216] Step 109: Residual Correction.

[0217] In step 109, the reconstructed video V^ is corrected based on the residual information R to obtain the final output video V^′. The generation model is a conditional generation model, including a semantic control branch, a motion control branch, and a residual injection branch. The semantic control branch generates the semantic content of the video based on the semantic condition information Z. The motion control branch controls the spatial position changes and temporal dynamics of objects in the video based on the motion parameters M. The residual injection branch performs detail compensation on the intermediate features of the generation model based on the residual information R.

[0218] To improve reconstruction accuracy, the reconstructed video is corrected based on the following formula:

[0219] V^′ = V^ + α*R;

[0220] In the formula:

[0221] V^′: Represents the final output video after correction based on residual information;

[0222] V^: Represents the reconstructed video obtained through generative model reconstruction;

[0223] R: Residual information (representing generation error);

[0224] α is an optional correction weighting coefficient (typically α=1, but in some embodiments it can be dynamically adjusted according to the signal-to-noise ratio).

[0225] By compensating for generation errors using residual information, it can be understood that both the video reconstruction based on the generation driving signal and the correction process for the reconstructed video are performed using parametric calculation methods. The corrected reconstructed video is then output.

[0226] The generative model G can be a deep neural network architecture, such as a Generative Adversarial Network (GAN), a Variational Autoencoder (VAE), or a Transformer-based video generation model. Internally, Z typically functions as a conditional embedding to control the semantic category of the generated content; M serves as a spatial transformation parameter (such as optical flow or keypoint displacement) to guide inter-frame motion; and R supplements the underlying details as part of feature injection or skip connections. The generative model G outputs a preliminary reconstructed video stream V^. At this point, V^ is structurally and semantically highly consistent with the original video, but may have slight differences in texture sharpness and high-frequency details. To eliminate the slight differences between the preliminary reconstructed video V^ and the original video, this embodiment introduces a post-processing correction mechanism based on residual information R.

[0227] It is important to emphasize that R here is not the difference in the original pixel space, but rather the parameterized residual in the feature space. R can be the feature offset extracted from the intermediate hidden layer (Latent Space) of the generative model. Compared to directly transmitting pixel residuals, this feature-level correction method requires very little data but can bring significant texture restoration effects. Although the formula appears as a linear addition, in actual network implementations, R can be applied to the feature map of V^ through gating mechanisms or modulation modules (such as AdaIN) to achieve more complex nonlinear detail enhancement.

[0228] Step 110: Output video.

[0229] The embodiments of the present invention can be applied to general video processing, and the specific processing flow is as follows:

[0230] At the encoding end, the input raw video is decomposed to obtain semantic condition information Z, motion parameters M, and residual information R. Specifically: semantic condition information Z describes the semantic structure of the video content; motion parameters M describes the spatial and temporal dynamic changes of objects in the video; and residual information R represents high-frequency detail compensation information.

[0231] The semantic conditional information Z is compressed and encoded to obtain a semantically compressed representation Z′. Based on the semantic conditional information Z, prediction is performed to obtain the predicted motion parameters Mpred, and the motion residuals Mres are calculated. The residual information R is sparsely encoded to obtain the sparse residual representation Rsparse.

[0232] Subsequently, Z′, Mres, and Rsparse are encoded to generate compressed data C, resulting in compressed data C:

[0233] C = Enc (Z', Mres, Rsparse)

[0234] Where: C: compressed data; Enc: encoding function; Z′: semantic compressed representation;

[0235] Mres: represents motion residuals; Rsparse: represents sparse residuals.

[0236] At the decoding end, the compressed data is restored using a decoding function, resulting in:

[0237] Z = Dec_Z (Z′);

[0238] M = Mpred (Z) + Mres;

[0239] R = Dec_R (Rsparse);

[0240] Subsequently, the video is reconstructed based on Z, M, and R using the generative model G:

[0241] V^ = G (Z, M, R).

[0242] Where: V^: represents the reconstructed video obtained through the generative model; G: is the generative model;

[0243] Z: Semantic conditional information (describes video content); M: Motion parameters (describes the motion state of an object);

[0244] R: Residual information (representing high-frequency detail compensation information); and the reconstructed video is corrected based on the residual information R to obtain the final video output.

[0245] The embodiments of the present invention can be applied to digital human video processing, and the specific processing flow is as follows:

[0246] The input video is a digital human video, in which the human figure is represented by a pre-trained generative model. At the encoding end, only the parameter information driving the generation of the digital human is extracted, including: semantic conditional information Z, motion parameters M, and residual information R; Z may include text or audio conditional information; M may include facial expression parameters and pose parameters, and R is used for high-frequency detail compensation information.

[0247] At the decoding end, the video is reconstructed based on Z, M, and R using the generation model G. In this scenario, the residual information R can optionally participate in detail compensation.

[0248] V^ = G (Z, M, R);

[0249] in:

[0250] V^: Represents the reconstructed video obtained through generative model reconstruction;

[0251] G: Generative model;

[0252] Z: Semantic conditional information (describing video content);

[0253] M: Motion parameters (describe the motion state of the object);

[0254] R: Residual information;

[0255] 1) Instead of directly encoding the original video pixels, the parameters that drive the video generation are encoded;

[0256] 2) Reconstruct video content at the decoding end using a generative model;

[0257] 3) Compensate for generation errors using residual information;

[0258] 4) To realize the transformation of video encoding method from "pixel compression" to "generative-driven compression".

[0259] Compared with existing technologies, this invention transforms video transmission data from pixel-level data to parameter-level data, thereby reducing redundant information transmission and improving coding efficiency. Simultaneously, since video content is reconstructed through a generative model, high-quality video content can be recovered under conditions of lower data volume, thus improving the system's adaptability in low-bandwidth environments.

[0260] This invention also provides a video processing system based on a neural network, such as... Figure 2 As shown, the system includes: a video decomposition module, an encoding module, a decoding module, a generation model module, and a reconstruction and output module. These modules are integrated into a unified system via computer programs. Specifically, the video decomposition and encoding modules are integrated at the sending end (server), while the decoding, generation model, and reconstruction and output modules are integrated at the receiving end (client).

[0261] Video decomposition module: Performs feature extraction and decoupling decomposition on the input video V to obtain semantic condition information Z, motion parameters M, and residual information R (high frequency compensation).

[0262] Encoding module: Z is compressed to obtain Z′; Mpred is obtained through the prediction model, and the motion residual Mres=M-Mpred is calculated for residual encoding; R is sparsely encoded to obtain Rsparse, and finally the compressed data C is generated by joint encoding.

[0263] Decoding module: Decodes C to obtain Z′, Mres, and Rsparse, and restores them to obtain Z, M, and R respectively, where M = Mpred(Z) + Mres.

[0264] Generative model module: Generates the initial video using Z as the semantic condition, drives motion generation using M, and performs residual injection and compensation only at the detail layer using R to generate the initial reconstructed video V^.

[0265] Reconstruction and Output Module: Uses R to perform residual correction (a·R) on V^ to obtain the final reconstructed video V^' and outputs it for playback or display.

[0266] The video decomposition process of this invention embodiment is as follows: Figure 3 As shown, it includes the following steps:

[0267] 1) Extract shared feature representations F from the input video V using the feature extraction module Esep(·);

[0268] 2) The content decomposition module includes semantic branches, motion branches, and residual branches, which generate Z, M, and R respectively;

[0269] 3) The decoupling constraint module performs decoupling constraints on Z, M, and R to minimize the correlation between the three.

[0270] 4) Constraint loss feedback optimizes the parameters of each branch and feature extraction module to achieve effective decoupling of Z, M, and R.

[0271] The generation and driving signal encoding process of this invention embodiment is as follows: Figure 4 As shown, it includes the following steps:

[0272] 1) Based on the semantic conditional information Z, the semantic feature F2 is obtained through the semantic encoder E2, and then quantized / entropy encoded to obtain the semantic compressed representation Z′;

[0273] 2) Based on the motion parameter M, the motion residual Mres is calculated using the predicted motion Mpred, and then quantized / entropy encoded to obtain the compressed representation of the motion residual Mres′.

[0274] 3) Based on the residual information R, perform sparsification (Top-K / reading) to obtain the sparse residual Rsparse, and then perform quantization / entropy encoding to obtain the sparse residual compressed representation Rsparse′;

[0275] 4) Combine and package Z′, Mres′, and Rsparse′ to generate driver encoded data C, and compress and output it for network transmission.

[0276] The video generation and reconstruction process of this invention is as follows: Figure 5 As shown, it includes the following steps:

[0277] 1) Decoding and decompression module: Decodes the received compressed data C to obtain compressed representations Z′, Mres′, and Rsparse′;

[0278] The motion residuals are recovered from the predicted motion Mpred (Z′) to obtain the complete motion parameters M; the residuals are then desparsed to obtain the residual information R.

[0279] 2) Condition generation module: Generates semantic feature map based on Z′, motion feature map based on M, and residual feature map based on R.

[0280] 3) Feature fusion and generation network: The three features are fused and the initial reconstructed video V^ is generated through the generation network G(·).

[0281] 4) Reconstruction Output Module: The initial video V^ is corrected using the residual information R to obtain the final reconstructed video V^′, which is then output to the terminal device for display or playback.

[0282] It should be noted that the following controls can be applied to the reconstructed video using the generator network G:

[0283] Resolution control, frame rate control, quality adjustment, style / content control, and other parameter controls are not subject to excessive restrictions here.

[0284] This invention also provides a video processing system, such as... Figure 6 As shown, the system includes: application layer, service layer, platform layer, infrastructure layer and terminal layer.

[0285] The application layer may include: video generation service, reconstruction and playback service, device management service, data management service, user and permission management service, etc.

[0286] The service layer may include:

[0287] 1. Decomposition Service (Video Decomposition): Semantic decomposition, motion estimation, and residual extraction, yielding Z, M, and R respectively.

[0288] 2. Encoding Services (Enc) (Generation-Driven Encoding): Semantic compression coding, motion residual coding, sparse residual coding, joint encapsulation and packing.

[0289] 3. Transmission Services (Data Transmission and Management): Compressed data transmission (Transmission C), transmission scheduling, bandwidth adaptation, and reliable transmission (QoS control).

[0290] 4. Decoding Service (Dec) (Decoding and Restoration): Decode C to obtain Z′, Mres, and Rsparse, and restore them to obtain Z, M, and R. The restoration steps are the same as in the previous embodiments, and will not be repeated here.

[0291] 5. Generate Service G (Conditional Generation and Feature Fusion): Generate Model G(Z,M,R), feature fusion and generation.

[0292] 6. Reconstruction Service (Reconstruction and Output): Residual correction, output reconstructed video V^′ for playback or storage.

[0293] 7. Management Services (System Operation and Management): Task scheduling, resource management, status monitoring, log management, security management, and other services.

[0294] The platform layer includes: microservice architecture, containerization platform (Docker / Kubernetes), middleware (message queue / caching / registry center), API gateway, configuration center, and service governance (discovery, circuit breaking, load balancing).

[0295] The foundation layer includes: computing resource pools (CPU / GPU / NPU), storage resource pools (block storage, object storage), network resources (5G / leaseline / Wi-Fi), security protection (firewall / encryption / access control), operation and maintenance monitoring (monitoring / alarms / logs) and other services.

[0296] Terminal layer: smartphones, tablets, laptops, desktop computers, smart TVs, VR / AR devices, in-vehicle devices, drones and other types of terminals.

[0297] This invention also provides an electronic device, such as... Figure 7 As shown, the device includes: a processor 701, a memory 702, an input device 703, an output device 704, a storage device 705, a sensor 706, a communication module 707, a system bus 708, and a power module 709.

[0298] The processor 701 includes an arithmetic logic unit (ALU) 7011, a control unit 7012, and an internal bus 7013. The memory 702 stores a computer program, which, when executed by the processor 701, implements functions such as… Figures 1 to 6 The video processing procedure shown includes: decomposing the input video V to obtain Z / M / R, encoding it to obtain Z′ / Mres / Rsparse, generating compressed data C:C=Enc(Z′,Mres,Rsparse), transmitting C through the communication module 707, decoding to recover Z / M / R, and generating the reconstructed video V^′ based on the generation model G(Z,M,R).

[0299] The communication module 707 is used to transmit compressed data C between electronic devices or between an electronic device and a server, and supports multiple communication methods, including but not limited to: 5G, Wi-Fi, Bluetooth, Ethernet, etc.

[0300] The processor 701 and memory 702 work together to execute programs and model parameters stored in memory, implementing core functions such as the encoding function Enc, the decoding function Dec, and the conditional generation function G. This is understandable. Figure 7 Only modules relevant to this invention are shown; the actual device may contain more or fewer components.

[0301] Example 2

[0302] This invention provides a video processing apparatus based on a neural network, configurable at a transmitting end. The video processing apparatus includes:

[0303] The acquisition module is used to acquire the raw video data to be transmitted;

[0304] The decomposition module is used to decompose the original video data to obtain a generation driving signal. The generation driving signal includes semantic condition information Z, motion parameters M, and residual information R, and Z, M, and R satisfy decoupling constraint relationships in the feature space. Z is used to describe the semantic structure of the video content, M is used to describe the spatial and temporal dynamic changes of objects in the video, and R is used to represent high-frequency detail compensation information.

[0305] The semantic compression module is used to compress and encode the semantic condition information Z to obtain a semantically compressed representation Z′;

[0306] The motion parameter calculation module is used to perform prediction based on the semantic condition information Z, obtain the predicted motion parameters Mpred, and calculate the motion residuals Mres; wherein... ;

[0307] The parameter information R encoding module is used to perform sparse encoding on the residual information R to obtain a sparse residual representation Rsparse;

[0308] The joint encoding module is used to jointly encode Z′, Mres, and Rsparse to generate compressed data C.

[0309] Example 3

[0310] Embodiment 3 of the present invention provides a video processing device based on a neural network, which can be configured at a receiving end. The video processing device includes:

[0311] A receiving module is used to receive compressed data; the compressed data is obtained based on the method of any one of claims 1 to 3; the compressed data includes: a compressed representation Z′ of semantic condition information Z, motion residual Mres and a sparse residual representation Rsparse; the semantic condition information Z is extracted from the original video data, Mres is the prediction residual of motion parameters, Rsparse is the sparse representation of the residual information, and the residual information R is used to represent high-frequency detail compensation information;

[0312] The decoding module is used to recover the generated driving signal based on the compressed data; wherein the recovered generated driving signal includes: semantic condition information Z, motion parameters M, and residual information R;

[0313] The video generation module is used to obtain a reconstructed video V^ using a generation model G based on the semantic condition information Z and motion parameters M, and to correct the reconstructed video V^ based on the residual information R to obtain the final output video V^′.

[0314] The generative model is a conditional generative model, comprising a semantic control branch, a motion control branch, and a residual injection branch. The semantic control branch is used to generate the semantic content of the video based on the semantic condition information Z. The motion control branch is used to control the spatial position changes and temporal dynamics of objects in the video based on the motion parameters M. The residual injection branch is used to perform detail compensation on the intermediate features of the generative model based on the residual information R.

[0315] Embodiment 4 of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a computer processor, is used to perform the technical solution of any method embodiment.

[0316] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or grid device, etc.) to execute the methods described in the various embodiments of the present invention.

[0317] It is worth noting that in the above-described embodiments of the sharing device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0318] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A video processing method based on neural networks, characterized in that, Applied to the sending end, the method includes: Obtain the raw video data to be transmitted; The original video data is decomposed to obtain a generation driving signal; the generation driving signal includes: semantic condition information Z, motion parameters M and residual information R, and Z, M and R satisfy decoupling constraint relationship in feature space; Z is used to describe the semantic structure of video content, M is used to describe the spatial and temporal dynamic changes of objects in video, and R is used to represent high-frequency detail compensation information. The semantic conditional information Z is compressed and encoded to obtain the semantic compressed representation Z′; Based on the semantic conditional information Z, prediction is performed to obtain the predicted motion parameters Mpred, and the motion residuals Mres are calculated; where... ; The residual information R is sparsely encoded to obtain the sparse residual representation Rsparse; Z′, Mres, and Rsparse are jointly encoded to generate compressed data C.

2. The method according to claim 1, characterized in that, The process of jointly encoding Z′, Mres, and Rsparse to generate compressed data C includes: Compressed data is obtained by minimizing the following optimization objective function: L = D + λB; in: L: Represents the value of the encoding optimization objective function; D: Represents the distortion measure between the reconstructed video and the original video; B: Indicates the bitrate or data volume of the encoded data; λ: Represents the weighting parameter used to balance the relationship between distortion and bit rate.

3. The method according to claim 1, characterized in that, The semantic condition information Z includes at least one of text information and audio information; the audio information includes speech semantic features or environmental sound features; the motion parameters M include key point motion, posture changes or facial expression parameters.

4. A video processing method based on neural networks, characterized in that, Applied to the receiving end, the method includes: Receive compressed data; the compressed data is obtained based on the method of any one of claims 1 to 3; the compressed data includes: a compressed representation Z′ of semantic condition information Z, motion residual Mres, and a sparse residual representation Rsparse; the semantic condition information Z is extracted from the original video data; Rsparse is the sparse representation of the residual information, and the residual information R is used to represent high-frequency detail compensation information; The driving signal is recovered and generated based on the compressed data; wherein the recovered driving signal includes: semantic condition information Z, motion parameters M, and residual information R; Based on the semantic condition information Z, motion parameters M, and residual information R, the reconstructed video V^ is obtained using the generative model G. The reconstructed video V^ is then corrected based on the residual information R to obtain the final output video V^′. The generative model is a conditional generative model, comprising a semantic control branch, a motion control branch, and a residual injection branch. The semantic control branch is used to generate the semantic content of the video based on the semantic condition information Z. The motion control branch is used to control the spatial position changes and temporal dynamics of objects in the video based on the motion parameters M. The residual injection branch is used to perform detail compensation on the intermediate features of the generative model based on the residual information R.

5. A video processing method based on neural networks, characterized in that, include: The received compressed data is decoded to obtain semantic condition information Z, motion parameters M, and residual information R; Based on the semantic condition information Z, the motion parameters M, and the residual information R, a reconstructed video V^ is generated through a generative model. The generative model is a conditional generative model, which includes a semantic control branch, a motion control branch, and a residual injection branch. The semantic control branch is used to generate the semantic content of the video based on the semantic condition information Z; The motion control branch is used to control the spatial position changes and temporal dynamics of objects in the video according to the motion parameter M; The residual injection branch is used to perform detail compensation on the intermediate features of the generated model based on the residual information R; The residual information R does not participate in the initial content construction process of the generated model.

6. A video processing device based on a neural network, characterized in that, Applied to the transmitting end, the device includes: The acquisition module is used to acquire the raw video data to be transmitted; The decomposition module is used to decompose the original video data to obtain a generation driving signal. The generation driving signal includes semantic condition information Z, motion parameters M, and residual information R, and Z, M, and R satisfy decoupling constraint relationships in the feature space. Z is used to describe the semantic structure of the video content, M is used to describe the spatial and temporal dynamic changes of objects in the video, and R is used to represent high-frequency detail compensation information. The semantic compression module is used to compress and encode the semantic condition information Z to obtain a semantically compressed representation Z′; The motion parameter calculation module is used to perform prediction based on the semantic condition information Z, obtain the predicted motion parameters Mpred, and calculate the motion residuals Mres; wherein... ; The parameter information R encoding module is used to perform sparse encoding on the residual information R to obtain a sparse residual representation Rsparse; The joint encoding module is used to jointly encode Z′, Mres, and Rsparse to generate compressed data C.

7. The apparatus according to claim 6, characterized in that, The joint encoding module is used to generate compressed data by minimizing the following optimization objective function: L = D + λB; in: L: Represents the value of the encoding optimization objective function; D: Represents the distortion measure between the reconstructed video and the original video; B: Indicates the bitrate or data volume of the encoded data; λ: Represents the weighting parameter used to balance the relationship between distortion and bit rate.

8. A video processing device based on a neural network, characterized in that, Configured at the receiving end, the device includes: A receiving module is used to receive compressed data; the compressed data is obtained based on the method of any one of claims 1 to 3; the compressed data includes: a compressed representation Z′ of semantic condition information Z, motion residual Mres and a sparse residual representation Rsparse; the semantic condition information Z is extracted from the original video data, Mres is the prediction residual of motion parameters, Rsparse is the sparse representation of the residual information, and the residual information R is used to represent high-frequency detail compensation information; The decoding module is used to recover the generated driving signal based on the compressed data; wherein the recovered generated driving signal includes: semantic condition information Z, motion parameters M, and residual information R; The video generation module is used to obtain a reconstructed video V^ using a generation model G based on the semantic condition information Z and motion parameters M, and to correct the reconstructed video V^ based on the residual information R to obtain the final output video V^′. The generative model is a conditional generative model, comprising a semantic control branch, a motion control branch, and a residual injection branch. The semantic control branch is used to generate the semantic content of the video based on the semantic condition information Z. The motion control branch is used to control the spatial position changes and temporal dynamics of objects in the video based on the motion parameters M. The residual injection branch is used to perform detail compensation on the intermediate features of the generative model based on the residual information R.

9. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the video processing method based on a neural network as described in any one of claims 1 to 5.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the video processing method based on neural networks as described in any one of claims 1 to 5.

Images