Video processing model training method, video processing method, electronic device, storage medium and computer program product

Abstract

The application provides a video processing model training method, a video processing method, an electronic device, a storage medium and a computer program product, and relates to the technical field of video processing technology. The method comprises the following steps: determining a second video based on a first model processing a first video and a first configuration parameter of an encoder; wherein the first configuration parameter is used to determine the processing intensity during the training of the first model; determining processing index information for the second video based on the encoder processing the second video; iteratively training the first model according to the processing index information and evaluation information of the first model for the processing quality of the first video, stopping the training until a predetermined training condition is reached, and determining a third model.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a video processing model training method, a video processing method, an electronic device, a storage medium, and a computer program product. Background Technology

[0002] In related technologies, traditional preprocessing algorithms based on manual features, such as Gaussian blur, sharpening filtering, and Motion Compensated Temporal Filter (MCTF), not only struggle to effectively eliminate redundant information in complex scenes but also lead to a decline in subjective quality, resulting in poor video quality after encoder processing. Furthermore, deep learning-based preprocessing algorithms learn the statistical patterns of video signals through data-driven methods, offering a new paradigm for overcoming the limitations of experience-based manual algorithm design. However, data-driven methods mostly simulate the degradation process during compression by manually designing multiple blur, scaling, and noise-adding operators, failing to actually perceive the encoding process. This also contributes to poor video quality after encoder processing. Summary of the Invention

[0003] This application provides a video processing model training method, a video processing method, an electronic device, a storage medium, and a computer program product.

[0004] The technical solution of this application is implemented as follows:

[0005] This application provides a video processing model training method, including:

[0006] A second video is determined based on the first model processing the first video and the first configuration parameters of the encoder; wherein, the first configuration parameters are used to determine the processing intensity during the training of the first model;

[0007] Based on the encoder, the second video is processed to determine the processing index information for the second video;

[0008] Based on the processing index information and the evaluation information of the first model on the processing quality of the first video, the first model is iteratively trained until the predetermined training conditions are met, at which point training stops, and the third model is determined.

[0009] In the above scheme, the step of iteratively training the first model based on the processing index information and the evaluation information of the first model for the processing quality of the first video, until training stops when a predetermined training condition is met, and then determining the third model, includes:

[0010] The second model is jointly and iteratively trained based on the evaluation information and the processing indicator information, and the first model is iteratively trained based on the evaluation information; wherein, the second model is used to determine the evaluation information;

[0011] Based on the trained first model, the next first video is processed to determine the next second video, and the encoder is used to process the next second video to determine the corresponding next processing index information. Based on the trained second model, the next first video and the next second video are processed to determine the next evaluation information.

[0012] The second model is jointly and iteratively trained based on the evaluation information and the processing index information described below, and the first model is iteratively trained based on the evaluation information described below until a predetermined number of training iterations are reached, at which point training stops, and the third model corresponding to the first model is determined.

[0013] In the above scheme, determining the second video based on the first model processing of the first video and the first configuration parameters of the encoder includes:

[0014] The first video frame and the first configuration parameters in the first video are input into the first model to determine the first feature corresponding to the first configuration parameters and the second feature of the first model calculation unit corresponding to the first video frame; wherein, the first feature is used to characterize the scaling factor of the channel processing intensity of the first model.

[0015] Based on the first feature, the second feature is subjected to intensity scaling processing to determine the second video frame corresponding to the first video frame, and the second video is determined based on the second video frame.

[0016] In the above scheme, determining the first feature corresponding to the first configuration parameter and the second feature corresponding to the first model calculation unit of the first video frame includes:

[0017] The first configuration parameter is processed into a first vector, and at least one of the luminance component and chrominance component in the first video frame is processed by the calculation unit to determine the corresponding second feature; wherein, the number of elements in the first vector is the same as the number of channels in the first model, and the elements are used to characterize the scaling factor of the corresponding channel.

[0018] In the above scheme, the step of performing intensity scaling processing on the second feature based on the first feature to determine the second video frame corresponding to the first video frame includes:

[0019] The second video frame is determined by multiplying the first vector with the second feature corresponding to the computing unit and then transforming the vector.

[0020] In the above scheme, the step of processing the next first video and the next second video based on the trained second model to determine the next evaluation information includes:

[0021] Align the first video frame of the next first video with the corresponding second video frame of the next second video, and then perform quality evaluation using the trained second model to determine the evaluation value of the corresponding first video frame; wherein, the evaluation information includes: the evaluation value of the first video frame.

[0022] In the above scheme, the first video includes: a first type of video and a second type of video with a training order; wherein, the first type of video is a preset sample video; the second type of video is a sample video obtained when the encoder is applied; the first configuration parameter includes: any one of the target bitrate and quantization parameter; the first configuration parameter is different for each iteration of training.

[0023] The method in the above scheme further includes:

[0024] The first video frame from the first video is input into the first model to determine the second video frame corresponding to the first video frame, and the second video is determined based on the second video frame; wherein, the first model is constructed by adjusting the processing intensity based on the first configuration parameters.

[0025] In the above scheme, the processing indicator information includes at least one of the following: true bitrate, encoding quality information, and pixel similarity; the pixel similarity is used to characterize the pixel similarity between the first video frame and the corresponding second video frame in the first video; the step of jointly iteratively training the second model based on the evaluation information and the processing indicator information, and iteratively training the first model based on the evaluation information, includes:

[0026] In response to user instructions, the weights corresponding to the actual bitrate, the encoding quality information, and the pixel similarity are determined, and a weighted sum is performed to determine the maximum reward function value; wherein, the weight of the actual bitrate is a negative number; each of the weights is determined by the user based on the magnitude of the actual bitrate, the encoding quality information, and the pixel similarity representation value to determine the maximum reward function value;

[0027] The parameters of the second model are updated based on the loss determined by the reward function value and the evaluation information, and the parameters of the first model are iteratively updated based on the loss determined by the negative value of the numerical value represented by the evaluation information.

[0028] This application also provides a video processing method, including:

[0029] The fourth video is determined based on the second configuration parameters of the encoder and the third video processed by the third model.

[0030] The step of determining the fourth video based on the second configuration parameters of the encoder and the third model processing includes:

[0031] The third video frame in the third video and the second configuration parameters are input into the third model to determine the third feature corresponding to the second configuration parameters and the fourth feature corresponding to the calculation unit of the third model; wherein, the third feature is used to characterize the scaling factor of the channel processing intensity of the third model;

[0032] The fourth feature is scaled based on the third feature to determine the fourth video frame corresponding to the third video frame, and the fourth video is determined based on the fourth video frame.

[0033] The fourth video is processed based on the encoder.

[0034] In the above scheme, the third model is iteratively trained based on processing index information and the evaluation information of the first model for the processing quality of the first video; the processing index information is determined by the encoder processing the second video; the second video is determined by the first model processing the first video and the first configuration parameters of the encoder; the first configuration parameters are used to determine the processing intensity when the first model is trained; the second configuration parameters are used to determine the processing intensity when the third model is applied.

[0035] In the above scheme, determining the third feature corresponding to the second configuration parameter and the fourth feature corresponding to the third model calculation unit of the third video frame includes:

[0036] The second configuration parameter is processed into a second vector, and at least one of the luminance component and chrominance component in the third video frame is processed by the calculation unit of the third model to determine the corresponding fourth feature; wherein the number of elements in the second vector is the same as the number of channels in the third model, and the elements are used to characterize the scaling factor of the corresponding channel.

[0037] In the above scheme, the step of performing intensity scaling processing on the fourth feature based on the third feature to determine the fourth video frame corresponding to the third video frame includes:

[0038] The fourth video frame is determined by multiplying the second vector with the corresponding fourth feature of the computing unit and then transforming the vector.

[0039] This application also provides a video processing apparatus, including:

[0040] Input unit, used to acquire third-party video;

[0041] The first preprocessing unit is used to determine the fourth video based on the second configuration parameters of the encoder and the third video processed by the third model; wherein, determining the fourth video based on the second configuration parameters of the encoder and the third video processed by the third model includes:

[0042] The third video frame in the third video and the second configuration parameters are input into the third model to determine the third feature corresponding to the second configuration parameters and the fourth feature corresponding to the calculation unit of the third model; wherein, the third feature is used to characterize the scaling factor of the channel processing intensity of the third model;

[0043] A scaling unit is configured to scale the fourth feature based on the third feature, determine the fourth video frame corresponding to the third video frame, and determine the fourth video frame based on the fourth video frame.

[0044] The first encoding unit is used to process the fourth video based on the encoder.

[0045] This application also provides an electronic device, including a first memory and a first processor. The first memory stores a computer program that can run on the first processor. When the first processor executes the computer program, it implements the steps in the video processing model training method.

[0046] This application also provides an electronic device, including a second memory and a second processor. The second memory stores a computer program that can run on the second processor, and the second processor executes the computer program to implement the steps in the video processing method.

[0047] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a first processor, implements the steps in the video processing model training method.

[0048] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a second processor, implements the steps in the video processing method.

[0049] This application also provides a computer program product, including a computer program that, when executed by a first processor, implements the steps in the video processing model training method.

[0050] This application also provides a computer program product, including a computer program that, when executed by a second processor, implements the steps of a video processing method.

[0051] In this embodiment, a second video is determined based on the first model processing the first video and the first configuration parameters of the encoder; wherein, the first configuration parameters are used to determine the processing intensity during the training of the first model; based on the encoder processing the second video, processing index information for the second video is determined; according to the processing index information and the evaluation information of the first model's processing quality for the first video, the first model is iteratively trained until a predetermined training condition is reached, at which point training stops, and a third model is determined. Thus, the first configuration parameters of the encoder and the corresponding processing index information are considered during the training of the third model, allowing the third model to fully perceive the encoder's encoding process during training. Furthermore, by combining the encoder's corresponding first configuration parameters and processing index information with the training scheme of the third model, an end-to-end joint training effect between the third model and the encoder can be achieved, enabling the third model and the encoder to be globally aligned in terms of perceptual enhancement targets and bitrate control. Consequently, the video output by the trained third model, after being processed by the encoder, exhibits high quality. Attached Figure Description

[0052] Figure 1 A flowchart illustrating the video processing method provided in this application embodiment. Figure 1 ;

[0053] Figure 2 Flowchart of the video processing method provided in the embodiments of this application Figure 2 ;

[0054] Figure 3 A flowchart illustrating the video processing model training method provided in this application embodiment. Figure 1 ;

[0055] Figure 4 This is a schematic diagram illustrating the effect of the video processing model training method provided in the embodiments of this application. Figure 1 ;

[0056] Figure 5 A flowchart illustrating the video processing model training method provided in this application embodiment. Figure 2 ;

[0057] Figure 6 This is a schematic diagram illustrating the effect of the video processing model training method provided in the embodiments of this application. Figure 2 ;

[0058] Figure 7 This is a schematic diagram illustrating the effect of the video processing model training method provided in the embodiments of this application. Figure 3 ;

[0059] Figure 8 A flowchart illustrating the video processing model training method provided in this application embodiment. Figure 4 ;

[0060] Figure 9 This is a schematic diagram illustrating the effect of the video processing model training method provided in the embodiments of this application. Figure 3 ;

[0061] Figure 10 A flowchart illustrating the video processing model training method provided in this application embodiment. Figure 5 ;

[0062] Figure 11 A flowchart illustrating the video processing model training method provided in this application embodiment. Figure 6 ;

[0063] Figure 12 This is a schematic diagram of the structure of the video processing apparatus provided in the embodiments of this application;

[0064] Figure 13 A schematic diagram of a hardware entity of an electronic device provided in an embodiment of this application;

[0065] Figure 14 This is a schematic diagram of the structure of the video processing model training device provided in the embodiments of this application;

[0066] Figure 15 This is a schematic diagram of a hardware entity of an electronic device provided in an embodiment of this application. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0068] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0069] If the application documents contain similar descriptions such as "first / second", the following explanation shall be added: In the following description, the terms "first / second / third" are used only to distinguish similar objects and do not represent a specific order of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0070] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0071] 1. Traditional algorithm.

[0072] In the field of video compression, traditional preprocessing algorithms based on handcrafted features, such as Gaussian blur, sharpening filtering, and temporal filtering, not only struggle to effectively eliminate redundant information in complex scenes but also lead to a decline in subjective quality, resulting in a bottleneck in improving compression efficiency. The core idea of ​​this technique is to directly process the original frame, using motion estimation and motion compensation methods to weight and average all the best-matching blocks from the four frames preceding and following the current frame with the current block, achieving noise reduction and thus improving coding efficiency. The drawback of this method is that due to the handcrafted weighting coefficients based on fixed rules, it suffers from generalization problems. As the demand for improved coding efficiency continues to increase, further optimization becomes increasingly difficult. Facing complex motion scenes and rich detail textures, it lacks adaptability to complex image content, making it difficult to achieve optimal results in different scenarios, leading to a decline in subjective quality and an inability to effectively preserve object edge information—a common problem in handcrafted algorithms. This results in poor encoded performance.

[0073] 2. Deep learning methods.

[0074] Deep learning-based preprocessing techniques learn the statistical patterns of video signals through a data-driven approach, providing a new paradigm for overcoming the limitations of manually designed algorithms based on experience. However, deep learning-based methods face the following problems in practical applications: 1) Bottlenecks in the application of neural network encoding tools: Due to the limited computing power of current terminal decoding devices, it is difficult to support the application of neural network tools. 2) Optimization fragmentation caused by non-differentiable encoders and decoders: The non-differentiability of traditional encoders makes it impossible to jointly train the preprocessing network with the compression process end-to-end, resulting in difficulty in globally aligning the perceptual enhancement goal with the bitrate control requirements. Furthermore, video encoding must strictly adhere to standards; modifying the bitstream or decoder will break compatibility, making it difficult to implement in the existing industrial ecosystem. 3) Limitations of existing deep learning preprocessing methods: Most existing data-driven methods simulate the degradation process in compression by manually designing multiple blurring, scaling, and noise-adding operators, and cannot actually perceive the encoding process.

[0075] Please see Figure 1 The above is a flowchart illustrating the video processing method provided in the embodiments of this application. Figure 1 , will combine Figure 1 The steps shown are explained below:

[0076] S101. Based on the third model, process the third video and the second configuration parameters of the encoder to determine the fourth video; wherein, the step of processing the third video and the second configuration parameters of the encoder to determine the fourth video includes: inputting the third video frame in the third video and the second configuration parameters into the third model to determine the third feature corresponding to the second configuration parameters and the fourth feature corresponding to the calculation unit of the third model; wherein, the third feature is used to characterize the scaling factor of the channel processing intensity of the third model.

[0077] In this embodiment, the third model is iteratively trained based on processing index information and the evaluation information of the first model for the processing quality of the first video; the processing index information is determined by the encoder processing the second video; the second video is determined by the first model processing the first video and the first configuration parameters of the encoder; the first configuration parameters are used to determine the processing intensity during the training of the first model; the second configuration parameters are used to determine the processing intensity when the third model is applied. In this embodiment, the video processing device can obtain the third model and the corresponding second configuration parameters of the encoder, and use the third model to process the third video and the second configuration parameters to determine the fourth video. Specifically, the video processing device can input each third video frame and the corresponding second configuration parameters from the third video into the third model to determine the third feature corresponding to the second configuration parameters and the fourth feature corresponding to the computational unit of the third model. The third feature is used to characterize the scaling factor of each channel of the third model.

[0078] In this embodiment of the application, the video processing device can be a model training device or other devices connected to the model training device.

[0079] The first model can include a preprocessing model preceding the encoder. This preprocessing model can include models based on algorithms such as Gaussian blur, sharpening filtering, and Motion Compensated Temporal Filter (MCTF). The preprocessing model is used to remove redundant information, remove noise, or compress the input video. The third model has the same model category as the first model.

[0080] The first configuration parameter is the initial configuration parameter set by the user for the encoder when training the first model. This first configuration parameter may include parameters the user expects to represent bitrate or coding quality. During training of the first model, training can be performed using multiple values ​​of the first configuration parameter to cover as much of the encoder's configurable parameter range as possible. The second configuration parameter is the configuration parameter set by the user for the encoder's application scenario. This second configuration parameter may include parameters the user expects to represent bitrate or coding quality. The second configuration parameter may be the same as or different from the first configuration parameter.

[0081] The first video may include pre-prepared videos for training, such as animated videos, live videos, or other types of videos. The third video may include live videos acquired during encoder application, landscape videos captured by the camera, or videos of other scenes.

[0082] In this embodiment, the third model is located before the entire encoding loop in the traditional video coding framework. It first performs preprocessing enhancement on each frame of the video to be encoded, and then directly inputs the enhanced video into the encoder for subsequent encoding. The method proposed in this application follows the original encoder's bitstream specification and does not alter the original bitstream. The model can be fine-tuned to adapt to codecs of different video coding standards.

[0083] S102. Based on the third feature, the fourth feature is scaled to determine the fourth video frame corresponding to the third video frame, and the fourth video is determined based on the fourth video frame.

[0084] In this embodiment, the processing device can use the third feature and the fourth feature, which characterize the channel scaling coefficient of the third model, to determine the final feature, and then use the final feature to perform pixel restoration processing to determine the fourth video frame. The fourth video frames are then combined in sequence to determine the fourth video.

[0085] In this embodiment, the processing device can multiply the third feature by the output features of each network processing module of the third model for the third video frame to determine the final feature. Then, pixel restoration processing is performed on the final feature to determine the fourth video frame.

[0086] S103. Process the fourth video based on the encoder.

[0087] In this embodiment of the application, the video processing device can use an encoder to encode and compress the fourth video to determine the fifth video.

[0088] In this embodiment, because the encoder's first configuration parameters and corresponding processing metrics are considered during the training of the third model, the third model can fully perceive the encoder's encoding process during training. Furthermore, by combining the encoder's first configuration parameters and processing metrics with the training scheme for the third model, an end-to-end joint training effect between the third model and the encoder can be achieved. This allows the third model and the encoder to be globally aligned in terms of perceptual enhancement targets and bitrate control, resulting in high-quality video output from the trained third model after processing by the encoder.

[0089] Please see Figure 2 The above is a flowchart illustrating the video processing method provided in the embodiments of this application. Figure 2 , Figure 1 S101 in the middle can also be implemented through S201 to S202, which will combine Figure 2 The steps shown are explained below:

[0090] S201. The second configuration parameters are processed into a second vector through the third model, and at least one of the luminance component and chrominance component in the third video frame is processed by the calculation unit of the third model to determine the corresponding fourth feature; wherein, the number of elements in the second vector is the same as the number of channels in the third model, and the elements are used to characterize the scaling factor of the corresponding channel.

[0091] In this embodiment, the video processing device can sequentially input each third video frame and the second configuration parameter into a third model according to the video frame order in the third video, and process the third video frame and the second configuration parameter respectively. A path in the third model can map the second configuration parameter to a second vector. A third feature characterizes the scaling factor of the processing intensity of each channel in the path where the third model processes the third video frame. The third model can process the third video frame through convolutional layers and various computational units (ResBlocks) to determine a fourth feature for each computational unit. The video processing device can process the second configuration parameter through an adaptive module in the third model, and process each third video frame through a video processing module in the third model. The second configuration parameter can be mapped to a second vector through the adaptive module. The number of elements in the second vector is the same as the number of channels in the video processing module, and each element is used to characterize the scaling factor of the corresponding channel. The corresponding fourth feature can be determined by processing at least one of the luminance component and chrominance component in each first video frame through multiple computational units (ResBlocks) in the video processing module. The scaling factor can be the opposite of the target bitrate corresponding to the encoder.

[0092] In this embodiment of the application, the second configuration parameter may include either the quantization parameter or the target bit rate.

[0093] The input scheme for each frame of the third video must be consistent with the input scheme used during the training of the first model. This can include the following different schemes: 1) Input only luminance Y; 2) Input only chrominance U and V; 3) Input only luminance Y and target bitrate; 4) Input only luminance Y and quantization parameter QP; 5) Input only chrominance U and V and target bitrate; 6) Input only chrominance U and V and quantization parameter QP; 7) Input luminance Y, chrominance U and V, and target bitrate; 8) Input luminance Y, chrominance U and V, and quantization parameter QP. The difference between the different schemes lies in their complexity and preprocessing effect. The quantization parameter and target bitrate are hyperparameters that adaptively adjust to control the processing quality of the third model under different application scenarios. For example, if the third model needs to preprocess the luminance of the video, schemes 3) and 4) can be selected: 3) Input only luminance Y and target bitrate; and 4) Input only luminance Y and quantization parameter QP.

[0094] S202. Based on the second vector multiplied by the corresponding fourth feature of each computing unit, the transformation is performed to determine each fourth video frame.

[0095] In this embodiment, the video processing device can perform intensity scaling processing on the fourth feature in each computing unit based on the second vector to determine a final feature, and then perform pixel recovery processing on the final feature to determine the fourth video frame. The fourth video frames are then combined sequentially to determine the fourth video. Specifically, the video processing device can perform pixel recovery transformation based on multiplying the second vector with the corresponding fourth feature of each computing unit to determine each fourth video frame.

[0096] In this embodiment, if the third model is obtained by pre-setting the parameters of the first model using the training device for the first configuration parameters, and only requires input video frames during training, then the video processing device can process each third video frame through the third model to determine the corresponding fourth video frame.

[0097] In this embodiment, the second configuration parameters are processed into a second vector by a third model, and at least one of the luminance and chrominance components in the third video frame is processed by the computing unit of the third model to determine the corresponding fourth feature. The number of elements in the second vector is the same as the number of channels in the third model, and the elements represent the scaling factor of the corresponding channel. Each fourth video frame is determined by multiplying the second vector by the corresponding fourth feature of each computing unit. In this way, the second configuration parameters of the encoder are considered during the video processing of the third model, allowing the third model to fully perceive the encoder's encoding process. This enables end-to-end joint processing between the third model and the encoder, thereby improving the performance of the third model and the quality of the output video after encoder processing.

[0098] This application provides a video processing model training method. Please refer to [link / reference]. Figure 3 This is a flowchart illustrating the video processing model training method provided in this application embodiment. Figure 1 , will combine Figure 3 The steps shown are explained below:

[0099] S301. Based on the first model processing of the first video and the first configuration parameters of the encoder, determine the second video; wherein, the first configuration parameters are used to determine the processing intensity during the training of the first model.

[0100] In this embodiment, the video processing model training device can pre-acquire first configuration parameters for the encoder. The training device can acquire a first video (sample video), input each frame of the first video and the first configuration parameters into a first model for processing, determine the video frame corresponding to each output frame, and then combine them into a second video. The first model processes the first configuration parameters to obtain corresponding features, which can be used to constrain or control the processing intensity of the first model for each frame, resulting in the second video after processing the first video.

[0101] The first model can include a preprocessing model preceding the encoder. This preprocessing model can include models based on algorithms such as Gaussian blur, sharpening filtering, and Motion Compensated Temporal Filter (MCTF). The preprocessing model is used to remove redundant information, remove noise, or compress the input video.

[0102] Here, the first configuration parameter is the initial configuration parameter set by the user for the encoder when training the first model. The first configuration parameter can include parameters that the user expects to represent the bitrate or coding quality. When training the first model, training can be performed using multiple values ​​of the first configuration parameter to cover as much of the configurable range of the encoder as possible.

[0103] The first video may include a pre-prepared video for training, such as an animated video, a live video, or other types of video.

[0104] In this embodiment, after the first model processes each first video frame of the first video, a corresponding second video frame is obtained. The training device can use the second model to evaluate the processing quality of the first and second video frames and determine evaluation information.

[0105] S302. Based on the encoder, process the second video to determine processing index information for the second video.

[0106] In this embodiment of the application, the training device can encode the processed second video into the encoder and output the corresponding video, as well as the processing index information for the second video.

[0107] Among them, the processing index information is used to characterize the encoder's information on the actual bitrate or encoding quality of the second video.

[0108] The processing indicator information may also include information used to characterize the similarity between the first video and the second video.

[0109] S303. Based on the processing index information and the evaluation information of the first model for the processing quality of the first video, iteratively train the first model until the predetermined training conditions are met, then stop training and determine the third model.

[0110] In this embodiment, the training device can combine the processing index information and the evaluation information of the first model on the processing quality of the first video to determine the loss of the first model, and then use the loss to strengthen the iterative training of the first model until a predetermined number of training times is reached or the loss converges to stop training, and determine the third model corresponding to the first model.

[0111] In this embodiment, the training device can jointly process index information and evaluation information to determine the loss of the second model, determine the loss of the first model based on the evaluation information, and then simultaneously perform iterative training on the first and second models. The second model after each training is used to determine the evaluation information for the next training, until the predetermined training conditions are met and training stops, and the third model corresponding to the first model is determined.

[0112] For example, in combination Figure 4 The trained first model is considered as an agent, and its preprocessed second video is considered as action a. t The open-source encoder acts as an environment, interacting with it through actions, and outputs compressed video as state s. t The compressed video can be used to determine processing metrics, including the encoder's output bitrate and coding quality: objective or subjective quality. Objective quality is defined as Peak Signal-to-Noise Ratio (PSNR), while subjective quality is defined as Video Multi-method Assessment Fusion (VMAF). The training device can construct a multi-objective reward function based on the bitrate and coding quality to determine the reward value R. This reward value and evaluation information are then used to achieve end-to-end optimization training of the encoder / decoder in a black-box scenario, compared to the first model. However, this training process only trains the first model, not the encoder.

[0113] In this embodiment, a second video is determined based on the first model processing the first video and the first configuration parameters of the encoder. The first configuration parameters determine the processing intensity during the training of the first model. Processing metrics for the second video are determined based on the encoder processing the second video. The first model is iteratively trained by combining the processing metrics with the evaluation information of the first model's processing quality for the first video, until training stops when predetermined training conditions are met, thus determining the third model. In this way, the first configuration parameters of the encoder and the corresponding processing metrics are considered during the training of the third model, allowing the third model to fully perceive the encoder's encoding process. Furthermore, by combining the encoder's first configuration parameters and processing metrics with the training scheme for the third model, an end-to-end joint training effect between the third model and the encoder can be achieved. This allows the third model and the encoder to be globally aligned in terms of perceptual enhancement targets and bitrate control, resulting in high-quality video output from the trained third model after further encoder processing.

[0114] Please see Figure 5 This is a flowchart illustrating the video processing model training method provided in this application embodiment. Figure 2 , Figure 3 S303 in the above can also be implemented through S401 to S403, combining Figure 5 The steps shown are explained below:

[0115] S401. The second model is jointly trained iteratively based on the evaluation information and the processing index information, and the first model is trained iteratively based on the evaluation information.

[0116] In this embodiment, the training device can determine the maximum reward function value based on processing index information, and update the parameters of the second model by combining the evaluation information and the maximum reward function value to obtain the second model after one training iteration. It can also update the parameters of the first model based on the evaluation information to obtain the first model after one training iteration.

[0117] In this embodiment, both the first model and the second model require multiple rounds of training using the first video (sample video) until training stops when predetermined training conditions are met. The second model may include a value network model, the network structure of which is as follows: Figure 6 As shown, it consists of convolutional layers (Conv 3x3), pooling layers (AvgPool2D), and fully connected layers (Linear). This value network model exists only during the training phase. The value network model plays a crucial role as the environment evaluator in Deep Deterministic Policy Gradient (DDPG), a deep reinforcement learning algorithm for continuous action control. Its core framework consists of the following key components: predicting the expected reward value obtained after performing an action in the current state. Specifically, it takes the luminance components (Input first video frame Y and Input second video frame Y) and chrominance components (Input first video frame UV and Input second video frame UV) corresponding to the first video frame and the second video frame after preprocessing by the network, respectively, to derive the value Q. The Q value determines the evaluation information and provides optimization direction for the first model.

[0118] S402. Based on the trained first model, process the next first video to determine the next second video, and use the encoder to process the next second video to determine the corresponding next processing index information. Based on the trained second model, process the next first video and the next second video to determine the next evaluation information.

[0119] In this embodiment of the application, the training device can process the next first video (sample video) based on the trained first model to determine the next second video, and use the encoder to process the next second video to determine the corresponding next processing index information, and process the next first video and the next second video based on the trained second model to determine the next evaluation information.

[0120] In this embodiment, the training device aligns each first video frame of the next first video with each corresponding second video frame of the next second video, and then performs quality evaluation using the trained second model to determine the evaluation value for each first video frame. The evaluation information includes the evaluation value for each first video frame. The quality evaluation can be performed by comparing the pixel similarity between the first and second video frames, or by comparing the similarity of the image content between the first and second video frames.

[0121] S403. The second model is jointly and iteratively trained based on the evaluation information and the processing index information described below, and the first model is iteratively trained based on the evaluation information described below until a predetermined number of training iterations are reached, at which point training is stopped, and the third model corresponding to the first model is determined.

[0122] In this embodiment, the training device can determine the maximum reward function value based on the processing index information, determine the loss of the second model based on the maximum reward function value and the evaluation information, determine the loss of the first model based on the evaluation information, update the parameters of the second model according to the loss gradient of the second model, update the parameters of the first model according to the loss gradient of the first model, and then perform iterative training of the first model and the second model simultaneously. The second model after each training is used to determine the evaluation information for the next time, until the predetermined number of training times is reached and training stops, and the third model corresponding to the first model is determined.

[0123] In this embodiment, a second model is iteratively trained based on evaluation information and processing index information, and a first model is iteratively trained based on the evaluation information. The second model is used to determine the evaluation information. The trained first model processes the next first video to determine the next second video, and the encoder processes the next second video to determine the corresponding next processing index information. The trained second model processes the next first video and the next second video to determine the next evaluation information. The second model is iteratively trained based on the next evaluation information and processing index information, and the first model is iteratively trained based on the next evaluation information until a predetermined number of iterations are reached, at which point training stops, and a third model corresponding to the first model is determined. Since the evaluation information is an assessment of the processing quality of the first video under the influence of the first configuration parameters, the training of the first and second models takes into account the encoder's first configuration parameters and corresponding processing index information. This scheme of training the third model by combining the encoder's first configuration parameters and processing index information achieves end-to-end joint training between the third model and the encoder. This allows the third model and the encoder to be globally aligned in terms of perceptual enhancement targets and bitrate control, resulting in high-quality video output from the trained third model after encoder processing.

[0124] Please see Figure 7 This is a flowchart illustrating the video processing model training method provided in this application embodiment. Figure 3 , Figure 3 S301 in the middle can also be implemented through S501 to S502, which will combine Figure 7 The steps shown are explained below:

[0125] S501. Input the first video frame and the first configuration parameters from the first video into the first model, and determine the first feature corresponding to the first configuration parameters and the second feature corresponding to the first model calculation unit of the first video frame; wherein, the first feature is used to characterize the scaling factor of the channel processing intensity of the first model.

[0126] In this embodiment, the first model may include two processing paths: one path processes the first configuration parameter, and the other path processes each first video frame. The training device can sequentially input each first video frame and the first configuration parameter into the first model according to the order of the video frames in the first video, using the two paths to process the first video frame and the first configuration parameter respectively. One path of the first model can map the first configuration parameter to a first feature, where the first feature represents the scaling factor of the processing intensity of each channel in the first video frame processing path of the first model. The first model can process the first video frame through convolutional layers and various computational units (ResBlocks) to determine the second feature processed by each computational unit.

[0127] In this embodiment, the first video includes: a first type of video and a second type of video with a training order; wherein, the first type of video is a preset sample video; the second type of video is a sample video obtained when the encoder is applied; the first configuration parameter includes either a target bitrate or a quantization parameter (QP). The first configuration parameter is different for each iteration of training.

[0128] In this embodiment, the training process of the training device for the first model may include two stages: the first stage uses multiple first-type videos for training, and the second stage uses multiple second-type videos for training. Each first-type video or each second-type video in each training stage can correspond to a batch. In each batch, the video frames of the corresponding video and the first configuration parameters (target bitrate or quantization parameters) can be input into the first model for training. The selection of the target bitrate or quantization parameters as the first configuration parameters during training is influenced by the application scenario of the encoder.

[0129] In this embodiment of the application, the training device can also pre-set the parameters of the first model for the first configuration parameters, so that training can be completed by inputting video frames during training.

[0130] In this embodiment, the first type of video may include: a new training dataset for deep video compression optimization (BVI-AOM) and a Tencent Video dataset (TVD). The second type of video may include: a live video dataset from a real-world application scenario of the encoder. The training device can first convert the BVI-AOM and TVD video datasets to YUV420p format as the training set. Using a deep deterministic policy gradient algorithm as a framework, the training set videos are first input into the first model video by video and frame by frame. Then, the enhanced video output by the first model is input into a non-differentiable traditional encoder (an encoder conforming to any video encoding standard is acceptable). The encoding configuration during encoding is the first configuration parameter of the specific real-world live application scenario. Finally, the actual bitrate consumption and decoding PSNR (objective quality) or VMAF (subjective quality) are received as reward signals to directly optimize the parameter update direction of the first model (Policy Gradient), achieving closed-loop optimization of encoding and decoding. The first model is trained based on the PyTorch (Python-based Torch) framework. There are two training phases, each containing 100 epochs. The first 100 epochs are used for training with the BVI-AOM and TVD video datasets, and the latter 100 epochs are used for fine-tuning the model with live streaming datasets from real-world application scenarios, for a total of 200 epochs. During training, each video is treated as a batch with a batch size of 1.

[0131] S502. Perform intensity scaling processing on the second feature based on the first feature to determine the second video frame corresponding to the first video frame, and determine the second video based on the second video frame.

[0132] In this embodiment, since the first feature is used to characterize the scaling factor of the processing intensity of each channel of the first model, this factor (i.e., the first feature) can be combined with the second feature of each computing unit of the first model to determine a final feature. Then, the final feature is converted into an image to determine the corresponding second video frame. The second video frames are combined in sequence to determine the second video.

[0133] In this embodiment, each first video frame and first configuration parameters from the first video are input into the first model to determine the first feature corresponding to the first configuration parameters and the second feature corresponding to each computational unit of the first model for each first video frame. The first feature characterizes the scaling factor of the processing intensity of each channel of the first model. Intensity scaling processing is performed on each second feature based on the first feature to determine the second video frame corresponding to each first video frame, and the second video is determined based on each second video frame. In this way, the first configuration parameters of the encoder are considered during the video processing of the first model, allowing the first model to fully perceive the encoder's encoding process during training. This enables end-to-end joint training between the first model and the encoder, thereby improving the performance of the third model corresponding to the first model and enhancing the quality of the output video of the third model after encoder processing.

[0134] Please see Figure 8 This is a flowchart illustrating the video processing model training method provided in this application embodiment. Figure 4 , Figure 7 S501 to S502 can also be implemented through S601 to S602, combining Figure 8 The steps shown are explained below:

[0135] S601. The first configuration parameter is processed into a first vector, and at least one of the luminance component and chrominance component in the first video frame is processed by the calculation unit to determine the corresponding second feature; wherein, the number of elements in the first vector is the same as the number of channels in the first model, and the elements are used to characterize the scaling factor of the corresponding channel.

[0136] In this embodiment, the training device can process the first configuration parameters through the adaptive module in the first model, and process each first video frame through the video processing module in the first model. The first configuration parameters can be mapped to a first vector through the adaptive module. The number of elements in the first vector is the same as the number of channels in the video processing module, and each element represents the scaling factor for each corresponding channel. At least one of the luminance and chrominance components in each first video frame can be processed by multiple computing units (ResBlock) in the video processing module to determine the corresponding second feature. The scaling factor can be the opposite of the target bitrate corresponding to the encoder.

[0137] In this embodiment, the first model is mainly used in the preprocessing stage of video encoding, specifically located before the entire encoding loop, thus enabling it to adapt to various encoders conforming to different encoding standards. The network structure of the first model is as follows: Figure 9As shown, the first model may include an adaptive processing module (FS-Adaptive Module) and a video processing module. The video processing module mainly consists of multiple convolution-based residual blocks (ResBlocks) stacked together. The first model can align the chroma and luma components of the first video frame to reduce the computational complexity of the neural network. Finally, the network output undergoes further feature processing to restore the original resolution. Specifically, if the luma and chroma information of the first video frame are input for processing simultaneously, since the video is in YUV420p format, the chroma component is half the size of the luma component, requiring nearest-neighbor upsampling of the chroma information to match the luma component size. The inputs to the video processing module include the luma component Y and chroma components UV of the original video, as well as the target bitrate or quantization parameters used for subsequent compression. The input to the first model can include the following different schemes: 1) Input only luminance Y; 2) Input only chrominance U and V; 3) Input only luminance Y and target bitrate; 4) Input only luminance Y and quantization parameter QP; 5) Input only chrominance U and V and target bitrate; 6) Input only chrominance U and V and quantization parameter QP; 7) Input luminance Y, chrominance U and V and target bitrate; 8) Input luminance Y, chrominance U and V and quantization parameter QP. The difference between the different schemes lies in their complexity and preprocessing effect. The quantization parameter and target bitrate are hyperparameters that adaptively adjust the processing quality of the first model for different application scenarios. For example, if the first model needs to preprocess the luminance of the video, schemes 3) and 4) can be selected: 3) Input only luminance Y and target bitrate; and 4) Input only luminance Y and quantization parameter QP.

[0138] Combination Figure 9 The target bitrate or quantization parameters can be processed quickly using an adaptive model. Multiple fully connected networks and the sigmoid function are used to map these parameters to a first vector, the scale factor. The length of this first vector is the same as the number of channels in the video processing module, and it represents the scaling factor for different channels. This operation allows for adaptive control of the preprocessing intensity of the first model under different target bitrates or quantization parameters. At high bitrates, more texture detail needs to be preserved after compression, so the preprocessing intensity of the preprocessing network should be relatively low. At low bitrates, most texture detail is lost during compression, so the preprocessing intensity can be moderately high to benefit subsequent compression processes.

[0139] S602. The second video frame is determined by multiplying the first vector with the second feature corresponding to the calculation unit and then transforming the vector.

[0140] In this embodiment of the application, the training device can multiply the first vector with the second feature processed by each computing unit in the video processing module to obtain a feature, and then input the feature into the next computing unit until the final feature is output, and perform pixel restoration processing on the feature to obtain the corresponding second video frame.

[0141] For example, in combination Figure 9 The FS-Adaptive Module outputs the first vector. The second feature output in the computation unit (ResBlock) needs to be jointly calculated with the first vector to obtain the corresponding feature, and then the feature is output to the next computation unit.

[0142] In this embodiment, the first configuration parameter is processed into a first vector, and at least one of the luminance and chrominance components in each first video frame is processed by each computing unit to determine the corresponding second feature. The number of elements in the first vector is the same as the number of channels in the first model, and each element represents the scaling factor for each corresponding channel. The first vector is multiplied by the second feature corresponding to each computing unit, and then transformed to determine each second video frame. In this way, the processing intensity during the training of the first model can be controlled using the first configuration parameter, thereby achieving end-to-end joint training of the first model and the encoder. This allows the third model and the encoder to be globally aligned in terms of perceptual enhancement targets and bitrate control, resulting in high-quality video output from the trained third model after processing by the encoder.

[0143] Please see Figure 10 This is a flowchart illustrating the video processing model training method provided in this application embodiment. Figure 5 , will combine Figure 10 The steps shown are explained below:

[0144] S701. Input the first video frame in the first video into the first model, determine the second video frame corresponding to the first video frame, and determine the second video based on the second video frame; wherein, the first model is constructed based on the first configuration parameters by adjusting the processing intensity.

[0145] In this embodiment, the training device can pre-adjust the scaling factor parameter of each channel of the first model based on the target bitrate or quantization parameters, thus, in this embodiment, the first model does not... Figure 9 The adaptive module in the training device can input only the luminance component or chrominance component of each video frame into the first model. The first model performs convolutional residual processing on the first video frame to determine a final feature, and performs pixel restoration processing on the final feature to determine the second video frame.

[0146] In this embodiment, each first video frame from the first video is input into the first model to determine the second video frame corresponding to each first video frame, and the second video is determined based on each second video frame. The first model is constructed by adjusting the channel scaling factor based on the first configuration parameters. Because the first model is constructed by adjusting the channel processing intensity based on the first configuration parameters, end-to-end joint training of the first model and the encoder is indirectly achieved. This allows the third model and the encoder to be globally aligned in terms of perceptual enhancement targets and bitrate control, resulting in high-quality video output from the trained third model after processing by the encoder.

[0147] Please see Figure 11 This is a flowchart illustrating the video processing model training method provided in this application embodiment. Figure 6 , Figure 5 The shown S401 can be implemented by S801 to S802, and will be combined Figure 11 The steps shown are explained below:

[0148] S801. In response to the user's instruction, determine the weights corresponding to the actual bitrate, the encoding quality information, and the pixel similarity, and perform a weighted sum to determine the maximum reward function value; wherein, the weight of the actual bitrate is a negative number; each of the weights is determined by the user based on the magnitude of the actual bitrate, the encoding quality information, and the pixel similarity representation value to determine the maximum reward function value.

[0149] In this embodiment, the processing indicator information includes at least one of the following: true bitrate, encoding quality information, and pixel similarity; pixel similarity is used to characterize the pixel similarity between each first video frame and the corresponding second video frame in the first video; in this embodiment, the training device can process the second video through an encoder to determine the encoded video, and during the processing, collect the true bitrate and encoding quality information (PSNR (Objective Quality) or VMAF (Subjective Quality)), and obtain pixel similarity. After obtaining these three parameters, they can be provided to the user, who determines the weights of each parameter used to determine the maximum reward function value based on the magnitude of the true bitrate, the encoding quality information, and the pixel similarity characterization values. Then, a weighted sum is performed based on the weights corresponding to the true bitrate, the encoding quality information, and the pixel similarity to determine the maximum reward function value. True bitrate

[0150] For example, the training device can determine the maximum reward function value R using formula (1).

[0151] R = α * actual bit rate + β * coding quality information + λ * pixel similarity formula (1)

[0152] Where α is the weight corresponding to the actual bitrate, β is the weight corresponding to the encoding quality information, and λ is the weight corresponding to pixel similarity. The maximum reward function value consists of three terms, where α is negative and β and γ are positive. The weights of these three terms are adjusted according to the actual training situation. The goal is to maximize the reward value obtained by the video after preprocessing by the preprocessing network and compression by the encoder, that is, to have a lower actual bitrate, higher PSNR or VMAF value, and to maintain pixel similarity with the original image to prevent too much content change.

[0153] S802, the parameters of the second model are iteratively updated based on the loss determined by the reward function value and the evaluation information, and the parameters of the first model are iteratively updated based on the loss determined by the negative value of the numerical value represented by the evaluation information.

[0154] In this embodiment, since the evaluation information consists of evaluation values ​​corresponding to multiple first video frames, the training device can combine each evaluation value with the reward function value to determine a set of values, and then use this set of values ​​to determine the parameters of the second model for iterative loss updates. Alternatively, the training device can determine a comprehensive value based on multiple evaluation values, and use the negative value of this comprehensive value to determine the parameters of the first model for iterative loss updates.

[0155] In this embodiment, the optimization objective of the second model is to minimize the mean squared error of its output Q-value (evaluation information) and the maximum reward function value R. The essence of updating the second model network is to enable the second model to more accurately predict the reward value obtained by the first model after processing the current frame, thus becoming a reliable reward predictor. The optimization objective of the first model is to directly use the negative value of the second model's output value information as the loss function; minimizing this loss function is equivalent to maximizing the expected return. The essence of updating the first model is to adjust the policy network parameters along the value growth direction indicated by the value network, i.e., how to improve actions to obtain higher scores.

[0156] In this embodiment, addressing the problems of existing technologies such as the encoder's non-differentiability preventing end-to-end joint training of the preprocessing network and compression process, and the limited computing power at the decoding end hindering the application of neural network tools, a video processing model training method is provided. By inputting luminance / chrominance information and target bitrate / target QP information into the preprocessing network (first model), luminance and chrominance can be preprocessed. The preprocessing result is input into the encoder, resulting in higher subjective and objective quality of the compressed video after preprocessing, while also achieving a lower bitrate, further improving compression efficiency. This method is applied only at the encoding end and does not introduce additional computational complexity at the decoding end. The innovation lies in:

[0157] 1. A first model is proposed, which, based on convolutional operations, achieves real-time preprocessing of YUV video at 4K@50fps (4K resolution at 50 frames per second) by stacking multiple residual blocks. Furthermore, through an adaptive module, the input target bitrate or quantization parameter information is mapped to a first vector, scale factor, through a fully connected layer and a sigmoid function. The vector length is the same as the number of channels in the preprocessing network, and this vector represents the scaling factor for different channels, thus adaptively controlling the preprocessing intensity of the network under different target bitrates or quantization parameters.

[0158] 2. A training method for the first model is proposed. The second video is input into the encoder, and the actual bitrate consumption and decoding PSNR (objective quality) or VMAF (subjective quality) after encoding are received. The maximum reward function value R is calculated based on the actual bitrate consumption and decoding PSNR (objective quality) or VMAF (subjective quality). The original YUV video and the preprocessing results are input into the value network to obtain the evaluation information Q. The second model is updated with the mean square error of the output Q value and the reward function value R as the optimization objective. The output value of the value network is negative as the loss function, and the first model is updated with the optimization objective of minimizing this loss function.

[0159] Compared with the closest existing technology, the present application proposal shows significant technical advantages in terms of method generalization and coding process awareness, and can more effectively improve the preprocessing enhancement effect.

[0160] ① Method generalization: Most existing technologies are traditional manually designed algorithms. Manually designed algorithms have limitations, cannot effectively cope with different types of video content, and cannot achieve image adaptation. There are bottlenecks in algorithm updates and optimizations, and they cannot effectively adapt to different codecs and application scenarios.

[0161] ② Encoding Process Awareness: Existing technologies, whether traditional algorithms or deep learning-based algorithms, cannot be trained end-to-end due to the non-differentiability of the encoder. They can only simulate the degradation process by manually designing degradation operators to approximate the actual encoding environment as closely as possible. In contrast, this method uses reinforcement learning algorithms, treating the encoder as the environment and the first model as the agent. It continuously interacts with the encoder and obtains real-time information on the current video's bitrate, PSNR, VMAF, and other metrics. This enables real-time awareness of the encoding process and adaptive adjustment of the preprocessing network parameters based on the content. This allows the network to adapt to different codecs through simple fine-tuning, achieving efficient and rapid deployment.

[0162] Please see Figure 12 This is a schematic diagram of the structure of the video processing device provided in the embodiments of this application, which will be combined with... Figure 12 Explanation:

[0163] This application embodiment also provides a video processing apparatus 600, including:

[0164] Input unit 601 is used to acquire the third video;

[0165] The first preprocessing unit 602 is used to process the third video and the second configuration parameters of the encoder based on the third model to determine the fourth video; wherein, the process of processing the third video and the second configuration parameters of the encoder based on the third model to determine the fourth video includes:

[0166] The third video frame in the third video and the second configuration parameters are input into the third model to determine the third feature corresponding to the second configuration parameters and the fourth feature corresponding to the calculation unit of the third model; wherein, the third feature is used to characterize the scaling factor of the channel processing intensity of the third model;

[0167] Scaling unit 603 is used to scale the fourth feature based on the third feature, determine the fourth video frame corresponding to the third video frame, and determine the fourth video based on the fourth video frame.

[0168] The first encoding unit 604 is used to process the fourth video based on the encoder.

[0169] In this embodiment of the application, the first preprocessing unit 602 in the video processing device 600 is used to process the second configuration parameters into a second vector, and process at least one of the luminance component and chrominance component in each third video frame through each calculation unit of the third model to determine the corresponding fourth feature; wherein, the number of elements in the second vector is the same as the number of channels in the third model, and each element is used to characterize the scaling factor of each corresponding channel.

[0170] In this embodiment of the application, the first preprocessing unit 602 in the video processing device 600 is used to transform the second vector by multiplying it with the corresponding fourth feature of each of the calculation units, and then determine each fourth video frame.

[0171] It should be noted that, in the embodiments of this application, if the above-described video processing method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a video processing model training device (which may be a personal computer, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a magnetic disk, or an optical disk. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0172] Correspondingly, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a first processor, implements the steps of the method on one side of the video processing apparatus 600.

[0173] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0174] It should be noted that, Figure 13 A schematic diagram of a hardware entity of an electronic device provided in an embodiment of this application, such as... Figure 13 As shown, this application embodiment provides an electronic device 700, including a first memory 702 and a first processor 701. The first memory 702 stores a computer program that can run on the first processor 701. When the processor 701 executes the program, it implements the steps in the above-described method, wherein;

[0175] The first processor 701 typically controls the overall operation of the electronic device 700.

[0176] The first memory 702 is configured to store instructions and applications executable by the first processor 701, and can also cache data to be processed or already processed by the first processor 701 and various modules in the electronic device 700 (e.g., image data, audio data, voice communication data and video communication data), which can be implemented by flash memory or random access memory (RAM).

[0177] Correspondingly, this application embodiment also provides a computer program product, including a computer program that can be executed by a first processor 701 of an electronic device 700 to complete the steps in the method of the video processing device 600.

[0178] Please see Figure 14 This is a schematic diagram of the structure of the video processing model training device provided in the embodiments of this application, which will be combined with Figure 14 Explanation:

[0179] This application provides a video processing model training device 800, including:

[0180] The second preprocessing unit 801 is used to determine the second video based on the first model processing the first video and the first configuration parameters of the encoder; wherein, the first configuration parameters are used to determine the processing intensity during the training of the first model;

[0181] The second encoding unit 802 is used to process the second video based on the encoder and determine processing index information for the second video;

[0182] The update unit 803 is used to iteratively train the first model based on the processing index information and the evaluation information of the first model for the processing quality of the first video, until training stops when a predetermined training condition is reached, and then determine the third model.

[0183] In this embodiment, the update unit 803 in the video processing model training device 800 is used to jointly iteratively train a second model based on the evaluation information and the processing index information, and iteratively train a first model based on the evaluation information; wherein, the second model is used to determine the evaluation information; the second preprocessing unit 801 in the video processing model training device 800 is used to process the next first video based on the trained first model to determine the next second video, and to process the next second video using the encoder to determine the corresponding next processing index information, and to process the next first video and the next second video based on the trained second model to determine the next evaluation information;

[0184] The update unit 803 in the video processing model training device 800 is used to jointly iteratively train the second model based on the next evaluation information and the processing index information, and iteratively train the first model based on the next evaluation information until training stops when a predetermined number of times is reached, and to determine the third model corresponding to the first model.

[0185] In this embodiment of the application, the second preprocessing unit 801 in the video processing model training device 800 is used to input the first video frame and the first configuration parameters in the first video into the first model, and determine the first feature corresponding to the first configuration parameters, and the second feature of the first model calculation unit corresponding to the first video frame; wherein, the first feature is used to characterize the scaling factor of the channel processing intensity of the first model.

[0186] Based on the first feature, the second feature is subjected to intensity scaling processing to determine the second video frame corresponding to the first video frame, and the second video is determined based on the second video frame.

[0187] In this embodiment of the application, the second preprocessing unit 801 in the video processing model training device 800 is used to process the first configuration parameters into a first vector, and to process at least one of the luminance component and chrominance component in the first video frame through the calculation unit to determine the corresponding second feature; wherein, the number of elements in the first vector is the same as the number of channels in the first model, and the elements are used to characterize the scaling factor of the corresponding channel.

[0188] In this embodiment of the application, the second preprocessing unit 801 in the video processing model training device 800 is used to transform the first vector by multiplying it with the second feature corresponding to the calculation unit, and then determine the second video frame.

[0189] In this embodiment of the application, the second preprocessing unit 801 in the video processing model training device 800 is used to align the first video frame of the next first video with the corresponding second video frame of the next second video and then perform quality evaluation through the trained second model to determine the evaluation value of the corresponding first video frame; wherein, the evaluation information includes: the evaluation value of the first video frame.

[0190] In this embodiment of the application, the first video includes: a first type of video and a second type of video with a training order; wherein, the first type of video is a preset sample video; the second type of video is a sample video obtained when the encoder is applied; the first configuration parameter includes: any one of the target bitrate and quantization parameter; the first configuration parameter is different for each iteration of training.

[0191] In this embodiment of the application, the second preprocessing unit 801 in the video processing model training device 800 is used to input the first video frame in the first video into the first model, determine the second video frame corresponding to the first video frame, and determine the second video based on the second video frame; wherein, the first model is constructed based on the first configuration parameters by adjusting the processing intensity.

[0192] In this embodiment, the processing indicator information includes at least one of the following: true bitrate, encoding quality information, and pixel similarity; the pixel similarity is used to characterize the pixel similarity between a first video frame and a corresponding second video frame in the first video; the update unit 801 in the video processing model training device 800 is used to respond to a user's instruction to determine the weights corresponding to the true bitrate, the encoding quality information, and the pixel similarity, and to perform a weighted summation to determine the maximum reward function value; wherein, the weight of the true bitrate is a negative number; each of the weights is a weight determined by the user based on the magnitude of the true bitrate, the encoding quality information, and the pixel similarity characterization values ​​to determine the maximum reward function value;

[0193] The parameters of the second model are updated based on the loss determined by the reward function value and the evaluation information, and the parameters of the first model are iteratively updated based on the loss determined by the negative value of the numerical value represented by the evaluation information.

[0194] Correspondingly, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a second processor, implements the steps of the method on one side of the video processing model training device 800.

[0195] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0196] It should be noted that, Figure 15 A schematic diagram of a hardware entity of an electronic device provided in an embodiment of this application, such as... Figure 15 As shown, this application embodiment provides an electronic device 900, including a second memory 902 and a second processor 901. The second memory 902 stores a computer program that can run on the second processor 901. When the processor 901 executes the program, it implements the steps in the above-described method, wherein;

[0197] The second processor 901 typically controls the overall operation of the electronic device 900.

[0198] The second memory 902 is configured to store instructions and applications executable by the second processor 901, and can also cache data to be processed or already processed by the second processor 901 and various modules in the electronic device 900 (e.g., image data, audio data, voice communication data, and video communication data), which can be implemented by flash memory or random access memory (RAM).

[0199] Correspondingly, this application also provides a computer program product, including a computer program that can be executed by a second processor 901 of an electronic device 900 to complete the steps in the method of the video processing model training device 800.

[0200] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0201] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0202] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the apparatus or units can be electrical, mechanical, or other forms.

[0203] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0204] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0205] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0206] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0207] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A video processing model training method, characterized in that, include: A second video is determined based on the first video processed by the first model and the first configuration parameters of the encoder; wherein, the first configuration parameters are used to determine the processing intensity during the training of the first model; the first model includes a preprocessing model prior to the encoder; the preprocessing model is used to remove redundant information, remove noise, or compress the input video; Based on the encoder, the second video is processed to determine the processing index information for the second video; Based on the processing index information and the evaluation information of the first model for the processing quality of the first video, the first model is iteratively trained until the predetermined training conditions are met, at which point training is stopped, and the third model is determined. The step of determining the second video based on the first model processing of the first video and the first configuration parameters of the encoder includes: The first video frame and the first configuration parameters in the first video are input into the first model to determine the first feature corresponding to the first configuration parameters and the second feature of the first model calculation unit corresponding to the first video frame; wherein, the first feature is used to characterize the scaling factor of the channel processing intensity of the first model. Based on the first feature, the second feature is subjected to intensity scaling processing to determine the second video frame corresponding to the first video frame, and the second video is determined based on the second video frame.

2. The video processing model training method according to claim 1, characterized in that, The step of iteratively training the first model based on the processing index information and the evaluation information of the first model for the processing quality of the first video, until training stops when a predetermined training condition is met, and determining the third model, includes: The second model is jointly and iteratively trained based on the evaluation information and the processing indicator information, and the first model is iteratively trained based on the evaluation information. Based on the trained first model, the next first video is processed to determine the next second video, and the encoder is used to process the next second video to determine the corresponding next processing index information. Based on the trained second model, the next first video and the next second video are processed to determine the next evaluation information. The second model is jointly and iteratively trained based on the evaluation information and the processing index information described below, and the first model is iteratively trained based on the evaluation information described below until a predetermined number of training iterations are reached, at which point training stops, and the third model corresponding to the first model is determined.

3. The video processing model training method according to claim 2, characterized in that, Determining the first feature corresponding to the first configuration parameter and the second feature corresponding to the first model calculation unit of the first video frame includes: The first configuration parameter is processed into a first vector, and at least one of the luminance component and chrominance component in the first video frame is processed by the calculation unit to determine the corresponding second feature; wherein, the number of elements in the first vector is the same as the number of channels in the first model, and the elements are used to characterize the scaling factor of the corresponding channel.

4. The video processing model training method according to claim 3, characterized in that, The step of performing intensity scaling processing on the second feature based on the first feature to determine the second video frame corresponding to the first video frame includes: The second video frame is determined by multiplying the first vector with the second feature corresponding to the computing unit and then transforming the vector.

5. The video processing model training method according to claim 4, characterized in that, The step of processing the next first video and the next second video based on the trained second model to determine the next evaluation information includes: Align the first video frame of the next first video with the corresponding second video frame of the next second video, and then perform quality evaluation using the trained second model to determine the evaluation value of the corresponding first video frame; wherein, the evaluation information includes: the evaluation value of the first video frame.

6. The video processing model training method according to claim 1, characterized in that, The first video includes: a first type of video and a second type of video with a training order; wherein, the first type of video is a preset sample video; the second type of video is a sample video obtained when the encoder is applied; the first configuration parameter includes: either the target bitrate or the quantization parameter; the first configuration parameter is different for each iteration of training.

7. The video processing model training method according to any one of claims 2 to 6, characterized in that, The processing index information includes at least one of the following: true bit rate, encoding quality information, and pixel similarity; The pixel similarity is used to characterize the pixel similarity between the first video frame and the corresponding second video frame in the first video; The step of jointly iteratively training the second model based on the evaluation information and the processing indicator information, and iteratively training the first model based on the evaluation information, includes: In response to user instructions, the weights corresponding to the actual bitrate, the encoding quality information, and the pixel similarity are determined, and a weighted sum is performed to determine the maximum reward function value; wherein, the weight of the actual bitrate is a negative number; each of the weights is determined by the user based on the magnitude of the actual bitrate, the encoding quality information, and the pixel similarity representation value to determine the maximum reward function value; The parameters of the second model are updated based on the loss determined by the reward function value and the evaluation information, and the parameters of the first model are iteratively updated based on the loss determined by the negative value of the numerical value represented by the evaluation information.

8. A video processing method, characterized in that, include: The fourth video is determined based on the second configuration parameters of the encoder and the third video processed by the third model. The step of determining the fourth video based on the second configuration parameters of the encoder and the third model processing includes: The third video frame in the third video and the second configuration parameters are input into the third model to determine the third feature corresponding to the second configuration parameters and the fourth feature corresponding to the calculation unit of the third model; wherein, the third feature is used to characterize the scaling factor of the channel processing intensity of the third model; The fourth feature is scaled based on the third feature to determine the fourth video frame corresponding to the third video frame, and the fourth video is determined based on the fourth video frame. The fourth video is processed based on the encoder; The third model is iteratively trained based on processing index information and the evaluation information of the first model on the processing quality of the first video; the first model includes a pre-processing model before the encoder; the pre-processing model is used to remove redundant information, remove noise, or compress the input video; the processing index information is determined by the encoder processing the second video; the second video is determined by the first model processing the first video and the first configuration parameters of the encoder; determining the second video based on the first model processing the first video and the first configuration parameters of the encoder includes: The first video frame and the first configuration parameters in the first video are input into the first model to determine the first feature corresponding to the first configuration parameters and the second feature of the first model calculation unit corresponding to the first video frame; wherein, the first feature is used to characterize the scaling factor of the channel processing intensity of the first model. Based on the first feature, the second feature is subjected to intensity scaling processing to determine the second video frame corresponding to the first video frame, and the second video is determined based on the second video frame.

9. The processing method according to claim 8, characterized in that, The first configuration parameter is used to determine the processing intensity during the training of the first model; the second configuration parameter is used to determine the processing intensity when the third model is applied.

10. A video processing apparatus, characterized in that, include: Input unit, used to acquire third-party video; The first preprocessing unit is used to determine the fourth video based on the second configuration parameters of the encoder and the third video processed by the third model; wherein, determining the fourth video based on the second configuration parameters of the encoder and the third video processed by the third model includes: The third video frame in the third video and the second configuration parameters are input into the third model to determine the third feature corresponding to the second configuration parameters and the fourth feature corresponding to the calculation unit of the third model; wherein, the third feature is used to characterize the scaling factor of the channel processing intensity of the third model; A scaling unit is used to scale the fourth feature based on the third feature, determine the fourth video frame corresponding to the third video frame, and determine the fourth video based on the fourth video frame. The first encoding unit is used to process the fourth video based on the encoder; The third model is iteratively trained based on processing index information and the evaluation information of the first model on the processing quality of the first video; the first model includes a pre-processing model before the encoder; the pre-processing model is used to remove redundant information, remove noise, or compress the input video; the processing index information is determined by the encoder processing the second video; the second video is determined by the first model processing the first video and the first configuration parameters of the encoder; determining the second video based on the first model processing the first video and the first configuration parameters of the encoder includes: The first video frame and the first configuration parameters in the first video are input into the first model to determine the first feature corresponding to the first configuration parameters and the second feature of the first model calculation unit corresponding to the first video frame; wherein, the first feature is used to characterize the scaling factor of the channel processing intensity of the first model. Based on the first feature, the second feature is subjected to intensity scaling processing to determine the second video frame corresponding to the first video frame, and the second video is determined based on the second video frame.

11. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program that can run on the processor, the processor executing the computer program to implement the steps of the method of any one of claims 1 to 7, or to implement the steps of the method of claim 8 or 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7, or the steps of the method according to claim 8 or 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7, or the steps of the method according to claim 8 or 9.

Images