Video processing model training method, video processing method, electronic equipment, storage medium and computer program product

Abstract

The invention provides a video processing model training method, a video processing method, electronic equipment, a storage medium and a computer program product, and relates to the technical field of video processing, and the method comprises the steps: processing a first video based on a first model and a first configuration parameter of an encoder, and determining a second video; wherein the first configuration parameter is used for determining the processing intensity when the first model is trained; processing the second video based on the encoder, and determining processing index information for the second video; and iteratively training the first model according to the processing index information and evaluation information of the first model for the processing quality of the first video, stopping training until a predetermined training condition is met, and determining a third model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing, and particularly relates to a video processing model training method, a video processing method, an electronic device, a storage medium and a computer program product. BACKGROUND

[0002] In the related art, a traditional pre-processing algorithm model based on manual features, such as Gaussian blur, sharpening filter and Motion Compensated Temporal Filter (MCTF), is difficult to effectively eliminate redundant information in a complex scene, and also causes subjective quality to decrease, thereby making the effect of a video output by the pre-processing algorithm model after processing by an encoder poor. In addition, a pre-processing algorithm model based on deep learning learns statistical rules of a video signal in a data-driven manner, which provides a new paradigm for breaking through the dilemma of an algorithm based on experience and manual design. However, the data-driven method mostly simulates a degradation process in a compression process through a plurality of blur, scaling and noise adding operators designed manually, cannot actually perceive an encoding process, and also causes the effect quality of the video output by the pre-processing algorithm model after processing by the encoder to be poor. SUMMARY

[0003] The present application provides a video processing model training method, a video processing method, an electronic device, a storage medium and a computer program product.

[0004] The technical solution of the present application is implemented as follows:

[0005] The present application provides a video processing model training method, which comprises the following steps.

[0006] processing the first video based on a first model and a first configuration parameter of an encoder to determine a second video; wherein the first configuration parameter is used to determine a processing intensity when the first model is trained;

[0007] processing the second video based on the encoder to determine processing index information for the second video;

[0008] iteratively training the first model according to the processing index information and evaluation information of the first model for the first video processing quality, until a predetermined training condition is reached to stop training, and determining a third model.

[0009] In the above solution, the iteratively training the first model according to the processing index information and evaluation information of the first model for the first video processing quality, until a predetermined training condition is reached to stop training, and determining a third model comprises the following steps.

[0010] train the second model based on the evaluation information and the processing indicator information jointly and iteratively, and train the first model based on the evaluation information iteratively; wherein the second model is used to determine the evaluation information;

[0011] determine the next second video based on the trained first model processing the next first video, and determine the corresponding next processing indicator information by processing the next second video using the encoder, and determine the next evaluation information based on the trained second model processing the next first video and the next second video;

[0012] train the second model based on the next evaluation information and the processing indicator information jointly and iteratively, and train the first model based on the next evaluation information iteratively until the training is stopped when a predetermined number of times is reached, and determine the third model corresponding to the first model.

[0013] In the above scheme, the second video is determined based on the first model processing the first video and the first configuration parameter of the encoder, comprising:

[0014] input the first video frame in the first video and the first configuration parameter into the first model to determine the first feature corresponding to the first configuration parameter and the second feature corresponding to the first video frame calculated by the first model calculation unit; wherein the first feature is used to represent the scaling coefficient of the channel processing intensity of the first model;

[0015] based on the intensity scaling processing of the first feature on the second feature, determine the second video frame corresponding to the first video frame, and determine the second video based on the second video frame.

[0016] In the above scheme, the first feature corresponding to the first configuration parameter and the second feature corresponding to the first video frame calculated by the first model calculation unit are determined, comprising:

[0017] processing the first configuration parameter into a first vector, and determining the corresponding second feature by processing at least one of the luminance component and the chrominance component in the first video frame through the calculation unit; wherein the number of elements of the first vector is the same as the number of channels of the first model, and the element is used to represent the scaling coefficient of the corresponding channel.

[0018] In the above scheme, the intensity scaling processing of the first feature on the second feature is performed to determine the second video frame corresponding to the first video frame, comprising:

[0019] based on the multiplication of the first vector and the second feature corresponding to the calculation unit and then converting to determine the second video frame.

[0020] In the scheme, the processing of the next first video and the next second video based on the trained second model determines the next evaluation information, comprising:

[0021] The first video frame of the next first video is aligned with the corresponding second video frame in the next second video, and the quality evaluation is performed by the trained second model to determine the evaluation value of the first video frame; wherein the evaluation information includes: the evaluation value of the first video frame.

[0022] In the scheme, the first video includes: a first type of video and a second type of video with a training order; wherein the first type of video is a preset sample video; the second type of video is a sample video obtained when the encoder is applied; the first configuration parameter includes: any one of the target code rate and the quantization parameter; the first configuration parameter is different in each iteration training.

[0023] In the scheme, the method further comprises:

[0024] The first video frame in the first video is input into the first model to determine the corresponding second video frame, and the second video is determined based on the second video frame; wherein the first model is constructed based on the adjustment of the processing intensity of the first configuration parameter.

[0025] In the scheme, the processing index information includes at least one of the following: real code rate, encoding quality information and pixel similarity; the pixel similarity is used to represent the pixel similarity between the first video frame in the first video and the corresponding second video frame; the joint iterative training of the second model based on the evaluation information and the processing index information, and the iterative training of the first model based on the evaluation information, comprising:

[0026] In response to the instruction of the user, the weights corresponding to the real code rate, the encoding quality information and the pixel similarity are determined, and the maximum reward function value is determined by weighted summation; wherein the weight of the real code rate is a negative number; each weight is a weight for determining the maximum reward function value, which is determined by the user according to the size of the value represented by the real code rate, the encoding quality information and the pixel similarity;

[0027] The parameters of the second model are updated based on the loss determined by the reward function value and the evaluation information, and the parameters of the first model are iteratively updated based on the loss determined by the negative value of the value represented by the evaluation information.

[0028] The embodiments of the present application also provide a video processing method, comprising:

[0029] determine a fourth video based on the third model processing the third video and second configuration parameters of the encoder;

[0030] The determining the fourth video based on the third model processing the third video and the second configuration parameters of the encoder includes:

[0031] inputting a third video frame in the third video and the second configuration parameters into the third model to determine a third feature corresponding to the second configuration parameters and a fourth feature corresponding to a calculation unit of the third model, wherein the third feature is used to represent a scaling coefficient of channel processing intensity of the third model;

[0032] scaling the fourth feature based on the third feature to determine a fourth video frame corresponding to the third video frame, and determining the fourth video based on the fourth video frame;

[0033] processing the fourth video based on the encoder.

[0034] In the scheme, the third model is iteratively trained according to processing index information and evaluation information of the first model for a first video processing quality; the processing index information is determined by the encoder processing a second video; the second video is determined by the first model processing the first video and first configuration parameters of the encoder; the first configuration parameters are used to determine processing intensity when the first model is trained; and the second configuration parameters are used to determine processing intensity when the third model is applied.

[0035] In the scheme, the determining the third feature corresponding to the second configuration parameters and the fourth feature corresponding to the third video frame and the calculation unit of the third model includes:

[0036] processing the second configuration parameters into a second vector, and processing at least one of a luminance component and a chroma component in the third video frame through the calculation unit of the third model to determine the corresponding fourth feature; wherein the number of elements of the second vector is the same as the number of channels of the third model, and the elements are used to represent the scaling coefficient of the corresponding channel.

[0037] In the scheme, the scaling the fourth feature based on the third feature to determine the fourth video frame corresponding to the third video frame includes:

[0038] determining the fourth video frame based on a transformation after the second vector is multiplied by the corresponding fourth feature of the calculation unit.

[0039] Embodiments of the present application further provide a video processing device, which includes:

[0040] The input unit is configured to obtain a third video;

[0041] The first preprocessing unit is configured to process the third video and the second configuration parameter of the encoder based on a third model to determine a fourth video.

[0042] The third video frame in the third video and the second configuration parameter are input into the third model to determine a third feature corresponding to the second configuration parameter and a fourth feature corresponding to a calculation unit of the third model, where the third feature is used to represent a scaling coefficient of a channel processing intensity of the third model.

[0043] The scaling unit is configured to perform scaling processing on the fourth feature based on the third feature to determine a fourth video frame corresponding to the third video frame, and determine the fourth video based on the fourth video frame.

[0044] The first encoding unit is configured to process the fourth video based on the encoder.

[0045] The embodiments of the present application also provide an electronic device, including a first memory and a first processor, the first memory stores a computer program capable of running on the first processor, and the first processor implements the steps in the video processing model training method when executing the computer program.

[0046] The embodiments of the present application also provide an electronic device, including a second memory and a second processor, the second memory stores a computer program capable of running on the second processor, and the second processor implements the steps in the video processing method when executing the computer program.

[0047] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program, and the computer program implements the steps in the video processing model training method when executed by a first processor.

[0048] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program, and the computer program implements the steps in the video processing method when executed by a second processor.

[0049] The embodiments of the present application also provide a computer program product, which includes a computer program, and the computer program implements the steps in the video processing model training method when executed by a first processor.

[0050] The embodiments of the present application also provide a computer program product, which includes a computer program, and the computer program implements the steps in the video processing method when executed by a second processor.

[0051] In the embodiments of the present application, the first video and the first configuration parameter of the encoder are processed based on the first model to determine a second video; wherein the first configuration parameter is used to determine the processing intensity during training of the first model; the second video is processed based on the encoder to determine processing index information for the second video; and the first model is iteratively trained according to the processing index information and evaluation information of the first model for the processing quality of the first video, until the training is stopped when a predetermined training condition is reached, to determine a third model. In this way, the first configuration parameter of the encoder and the corresponding processing index information are considered in the training process of the third model, so that the encoding process of the encoder can be fully perceived in the training process of the third model. Moreover, the scheme of training the third model by combining the corresponding first configuration parameter and processing index information of the encoder can achieve an end-to-end joint training effect of the third model and the encoder, so that the third model and the encoder can be globally aligned in terms of perceptual enhancement target and code rate control, and then the quality of the video output by the trained third model after being processed by the encoder is high. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 Flowchart of the video processing method provided by the embodiments of the present application Figure One ;

[0053] Figure 2 Flowchart of the video processing method provided by the embodiments of the present application Figure Two ;

[0054] Figure 3 Flowchart of the video processing model training method provided by the embodiments of the present application Figure One ;

[0055] Figure 4 Effect diagram of the video processing model training method provided by the embodiments of the present application Figure One ;

[0056] Figure 5 Flowchart of the video processing model training method provided by the embodiments of the present application Figure Two ;

[0057] Figure 6 Effect diagram of the video processing model training method provided by the embodiments of the present application Figure Two ;

[0058] Figure 7 Effect diagram of the video processing model training method provided by the embodiments of the present application Figure Three ;

[0059] Figure 8 Flowchart of the video processing model training method provided by the embodiments of the present application Figure Four ;

[0060] Figure 9 Effect diagram of the video processing model training method provided by the embodiment of the present application Figure Three ;

[0061] Figure 10 Flow diagram of the video processing model training method provided by the embodiment of the present application Figure Five ;

[0062] Figure 11 Flow diagram of the video processing model training method provided by the embodiment of the present application Figure Six ;

[0063] Figure 12 Structure diagram of the video processing device provided by the embodiment of the present application

[0064] Figure 13 Hardware entity diagram of the electronic device provided by the embodiment of the present application

[0065] Figure 14 Structure diagram of the video processing model training device provided by the embodiment of the present application

[0066] Figure 15 Hardware entity diagram of the electronic device provided by the embodiment of the present application DETAILED DESCRIPTION

[0067] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application are further described in detail below in conjunction with the drawings and embodiments, and the described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by those of ordinary skill in the art without making any creative labor fall within the scope of protection of the present application.

[0068] In the following description, “some embodiments” are described, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” can be the same subset or different subset of all possible embodiments, and can be combined with each other without conflict.

[0069] If similar descriptions of “first / second” appear in the application file, the following description is added, in the following description, the terms “first\second\third” involved only distinguish similar objects, and do not represent the specific order of the objects, and it can be understood that “first\second\third” can be interchanged with specific order or sequence as allowed, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0070] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description herein is for describing particular embodiments only and is not intended to be limiting of the application.

[0071] 1. Traditional algorithm.

[0072] In the field of video compression, traditional pre-processing algorithms based on manual features, such as Gaussian blur, sharpening filter, and temporal filter, not only fail to effectively eliminate redundant information in complex scenes, but also lead to a decline in subjective quality, making it difficult to improve compression efficiency. The core idea of this technology is to directly process the original frame, using motion estimation and motion compensation to weight average all the best matching blocks in the forward 4 frames and backward 4 frames of the current frame with the current block, achieving the effect of noise reduction and improving coding efficiency. The disadvantage of this method is that the fixed rule-based weighting coefficient designed by hand has generalization problems, and as the demand for improving coding efficiency increases, the difficulty of further optimization becomes greater. In the face of complex motion scenes and rich detail textures, the lack of adaptive ability to complex image content makes it difficult to achieve optimal results in different scenes, leading to a decline in subjective quality and the inability to effectively maintain object edge information, which is a common problem with hand-designed algorithms. This leads to poor coding results.

[0073] 2. Deep learning method.

[0074] Pre-processing technology based on deep learning learns the statistical rules of video signals through data-driven methods, providing a new paradigm for breaking through the difficulties of experience-based hand-designed algorithms. However, deep learning-based methods face the following problems in practical applications: 1) Bottleneck of neural network coding tools: Due to the limited computing power of current terminal decoding devices, it is difficult to support the application of neural network tools. 2) Optimization fragmentation caused by non-differentiable encoder and decoder: The non-differentiability of traditional encoders makes it impossible for pre-processing networks to be trained end-to-end with the compression process, making it difficult to align the perceptual enhancement target with the rate control demand globally. And video encoding needs to strictly follow the standard, modifying the code stream or decoder will destroy the compatibility, it is difficult to land in the existing industrial ecology. 3) Limitations of existing deep learning pre-processing methods: Most existing data-driven methods simulate the degradation process in the compression process through hand-designed multiple blur, scaling, and noise adding operators, which cannot actually perceive the encoding process.

[0075] Please refer to Figure 1 , the flowchart of the video processing method provided by the embodiments of the present application Figure One , the steps shown in Figure 1 will be described:

[0076] In S101, the third video and the second configuration parameter of the encoder are processed based on the third model to determine a fourth video. The processing of the third video and the second configuration parameter of the encoder based on the third model to determine the fourth video includes: inputting a third video frame in the third video and the second configuration parameter into the third model, determining a third feature corresponding to the second configuration parameter, and a fourth feature corresponding to a calculation unit of the third model; and the third feature is used to represent a scaling coefficient of a channel processing intensity of the third model.

[0077] In the embodiments of the present application, the third model is iteratively trained according to processing index information and evaluation information of the first video processing quality of the first model; the processing index information is determined by the encoder processing the second video; the second video is determined by the first model processing the first video and the first configuration parameter of the encoder; the first configuration parameter is used to determine the processing intensity when the first model is trained; and the second configuration parameter is used to determine the processing intensity when the third model is applied. In the embodiments of the present application, the video processing device can obtain the third model and the second configuration parameter of the corresponding encoder, and determine the fourth video by processing the third video and the second configuration parameter by using the third model. The video processing device can input each third video frame in the third video and the corresponding second configuration parameter into the third model, determine the third feature corresponding to the second configuration parameter, and the fourth feature corresponding to the calculation unit of the third model. The third feature is used to represent the scaling coefficient of each channel of the third model.

[0078] In the embodiments of the present application, the video processing device can be a model training device, or other devices connected to the model training device.

[0079] The first model can include a pre-processing model before the encoder. The pre-processing model can include a model based on Gaussian blur, sharpening filter, and Motion Compensated Temporal Filter (MCTF) algorithm. The pre-processing model is used to remove redundant information, remove noise, or compress the input video. The model category of the third model is the same as that of the first model.

[0080] The first configuration parameter is a first configuration parameter set by the user for the encoder when facing the training of the first model. The first configuration parameter can include a parameter expected by the user to represent the code rate or the encoding quality. When training for the first model, the first configuration parameter can be trained through multiple numerical values to cover the configuration parameter range that the encoder can configure as much as possible. The second configuration parameter is a configuration parameter set by the user for the encoder when facing the application scenario of the encoder. The second configuration parameter can include a parameter expected by the user to represent the code rate or the encoding quality. The second configuration parameter can be the same as or different from the first configuration parameter.

[0081] The first video can include a video prepared in advance for training, and can include an animation video, a live video, or other types of videos.

[0082] In the embodiment of the application, the third model is located before the entire encoding loop in the traditional video encoding framework, and first performs pre-processing enhancement on each frame of the video to be encoded. The video after the pre-processing enhancement is directly input into the encoder for subsequent encoding process. The method proposed in the application complies with the code stream specification of the original encoder and does not change the original code stream. The model fine-tuning can be used to adapt to the codec of different video encoding standards.

[0083] In S102, the fourth feature is scaled based on the third feature to determine a fourth video frame corresponding to the third video frame, and the fourth video is determined based on the fourth video frame.

[0084] In the embodiment of the application, the processing device can determine the final feature by multiplying the third feature and the output feature of each network processing module of the third model for the third video frame, and then perform pixel restoration processing on the final feature to determine the fourth video frame.

[0085] In the embodiment of the application, the processing device can determine the final feature by multiplying the third feature and the output feature of each network processing module of the third model for the third video frame, and then perform pixel restoration processing on the final feature to determine the fourth video frame.

[0086] In S103, the fourth video is processed based on the encoder.

[0087] In the embodiment of the application, the video processing device can determine a fifth video by encoding and compressing the fourth video based on the encoder.

[0088] In the embodiment of the present application, the first configuration parameter of the encoder and the corresponding processing index information are considered in the training process of the third model, so that the encoding process of the encoder can be fully perceived in the training process of the third model. Moreover, by combining the first configuration parameter and the processing index information of the encoder with the scheme of training the third model, the end-to-end joint training effect of the third model and the encoder can be achieved, the global alignment of the third model and the encoder in the perceptual enhancement target and the rate control can be achieved, and then the quality of the effect after the video output by the trained third model is processed by the encoder is high.

[0089] Please refer to Figure 2 The flowchart of the video processing method provided in the embodiment of the present application is shown in Figure Two , Figure 1 S101 in the embodiment of the present application can also be implemented by S201 to S202, which will be described in combination with the steps shown in Figure 2

[0090] S201, processing the second configuration parameter into a second vector by the third model, and determining the corresponding fourth feature by processing at least one of the luminance component and the chroma component in the third video frame by the calculation unit of the third model; wherein the number of elements of the second vector is the same as the number of channels of the third model, and each element is used to represent the scaling coefficient of the corresponding channel.

[0091] In the embodiment of the present application, the video processing device can input each third video frame and the second configuration parameter into the third model in the order of the video frames in the third video, and process the third video frame and the second configuration parameter respectively. One path of the third model can map the second configuration parameter into a second vector. The third feature represents the scaling coefficient of the processing intensity of each channel in the path of the third model processing the third video frame. The third model can process the third video frame through the convolution layer and each calculation unit (ResBlock) to determine the fourth feature when each calculation unit processes. The video processing device can process the second configuration parameter through the adaptive module in the third model, and process each third video frame through the video processing module in the third model. The second configuration parameter can be mapped into a second vector through the adaptive module. The number of elements of the second vector is the same as the number of channels of the video processing module, and each element is used to represent the scaling coefficient of each corresponding channel. At least one of the luminance component and the chroma component in each first video frame can be processed by the multiple calculation units (ResBlock) in the video processing module to determine the corresponding fourth feature. The scaling coefficient can be opposite to the size of the target code rate corresponding to the encoder.

[0092] ​In the embodiments of the present application, the second configuration parameter can include any one of a quantization parameter and a target code rate.

[0093] The input scheme of each third video frame of the third video is consistent with the input scheme during training of the first model, and can include the following different schemes: 1) only inputting luminance Y; 2) only inputting chrominance U and V; 3) only inputting luminance Y and target code rate Bitrate; 4) only inputting luminance Y and quantization parameter QP; 5) only inputting chrominance U and V and target code rate Bitrate; 6) only inputting chrominance U and V and quantization parameter QP; 7) inputting luminance Y, chrominance U and V, and target code rate Bitrate; and 8) inputting luminance Y, chrominance U and V, and quantization parameter QP. The different schemes differ in complexity and pre-processing effects, and the quantization parameter and the target code rate are super-parameters for adaptive adjustment of processing quality of the third model in different application scenarios. For example, if the third model needs to perform pre-processing on the luminance of the video, schemes 3) and 4) can be selected: 3) only inputting luminance Y and target code rate Bitrate; and 4) only inputting luminance Y and quantization parameter QP.

[0094] In S202, each fourth video frame is determined based on multiplication of the second vector and the corresponding fourth feature of each calculation unit and then pixel restoration.

[0095] In the embodiments of the present application, the video processing device can perform intensity scaling processing on the fourth feature in each calculation unit based on the second vector to determine a final feature, and then perform pixel restoration processing on the final feature to determine a fourth video frame. Then, the fourth video frames are combined in sequence to determine a fourth video. In this case, the video processing device can determine each fourth video frame based on multiplication of the second vector and the corresponding fourth feature of each calculation unit and then pixel restoration.

[0096] In the embodiments of the present application, if the third model is a model whose parameters are pre-set by the training device for the first configuration parameter, and only the video frame trained during training is needed. Then, the video processing device can process each third video frame through the third model to determine the corresponding fourth video frame.

[0097] In the embodiment of the present application, the second configuration parameter is processed into a second vector through a third model, and at least one of the luminance component and the chroma component in the third video frame is processed through the calculation unit of the third model to determine the corresponding fourth feature; wherein the number of elements of the second vector is the same as the number of channels of the third model, and the elements are used to represent the scaling coefficients of the corresponding channels. After the second vector is multiplied by the corresponding fourth feature of each calculation unit and then converted, each fourth video frame is determined. In this way, the second configuration parameter of the encoder is considered in the process of the third model processing the video, so that the encoding process of the encoder can be fully perceived in the processing process of the third model, so that the end-to-end joint processing formed by the third model and the encoder is improved, and the quality of the third model output video processed by the encoder is improved.

[0098] The embodiment of the present application provides a video processing model training method, please refer to Figure 3 The flowchart of the video processing model training method provided by the embodiment of the present application is shown in Figure One The steps shown in Figure 3 will be described:

[0099] S301, determining a second video based on a first model processing a first video and a first configuration parameter of an encoder; wherein the first configuration parameter is used to determine the processing intensity when the first model is trained.

[0100] In the embodiment of the present application, the video processing model training device can obtain the first configuration parameter of the encoder in advance. The training device can obtain the first video (sample video), input each frame of the first video and the first configuration parameter into the first model for processing, determine the corresponding video frame of each output frame, and then combine the second video. Wherein, the first model processes the first configuration parameter to obtain the corresponding feature, which can be used to constrain or control the processing intensity of the first model for each frame, and obtain the second video after the first video is processed.

[0101] Wherein, the first model can include a pre-processing model before the encoder. The pre-processing model can include a model based on Gaussian blur, sharpening filter and motion compensated temporal filter (MCTF) algorithm. The pre-processing model is used to remove redundant information, remove noise or compress the input video.

[0102] The first configuration parameter is a first configuration parameter set by the user for the encoder when facing the training of the first model. The first configuration parameter can include a parameter expected by the user for characterizing the code rate or the encoding quality. When training for the first model, the first configuration parameter can be trained through multiple numerical values to cover the configuration parameter range as much as possible that can be configured by the encoder.

[0103] The first video can include a pre-prepared video for training, and the first video can include an animation video, a live video, or other types of videos.

[0104] In the embodiments of the present application, the second video frame corresponding to each first video frame of the first video is obtained after the first model processes the first video frame. The training device can evaluate the processing quality of the first video frame and the second video frame by using the second model, and determine the evaluation information.

[0105] S302, determining processing index information for the second video based on the processing of the second video by the encoder.

[0106] In the embodiments of the present application, the training device can input the processed second video into the encoder to encode and output a corresponding video, and the processing index information for the second video.

[0107] The processing index information is used to represent the information of the real code rate or the encoding quality of the encoder for the second video.

[0108] The processing index information can also include information for representing the similarity between the first video and the second video.

[0109] S303, iteratively training the first model according to the processing index information and the evaluation information of the first model for the processing quality of the first video, stopping the training when a predetermined training condition is reached, and determining a third model.

[0110] In the embodiments of the present application, the training device can determine the loss of the first model by combining the processing index information and the evaluation information of the first model for the processing quality of the first video, and then use the loss to strengthen the iterative training of the first model, and stop the training when a predetermined training number or loss convergence is reached, to determine the third model corresponding to the first model.

[0111] In the embodiments of the present application, the training device can determine the loss of the second model by combining the processing index information and the evaluation information, determine the loss of the first model based on the evaluation information, and then iteratively train the first model and the second model synchronously, and use the second model trained each time to determine the evaluation information next time, until the training is stopped when a predetermined training condition is reached, to determine the third model corresponding to the first model.

[0112] Exemplarily, in combination with Figure 4 The trained first model is regarded as an agent, the pre-processed second video is regarded as an action a t The open-source encoder is regarded as an environment, the agent interacts with the environment through the action, and the output compressed video is regarded as a state s t The processing index information can be determined through the compressed video, and the processing index information includes a real bitrate Bitrate output by the encoder and an encoding quality: an objective quality or a subjective quality. The objective quality is a peak signal-to-noise ratio (PSNR), and the subjective quality is a video multi-method assessment fusion (VMAF). The training device can construct a multi-objective reward function according to the real bitrate and the encoding quality to determine a reward value R. The reward value and the evaluation information are used to realize end-to-end optimization training of the first model in a codec black box scenario. However, only the first model is trained in the training process, and the encoder is not trained.

[0113] In the embodiment of the application, the second video is determined based on the first model processing the first video and the first configuration parameter of the encoder; the first configuration parameter is used to determine the processing intensity of the first model during training; the processing index information for the second video is determined based on the encoder processing the second video; the processing index information and the evaluation information of the first model for the processing quality of the first video are combined to strengthen and iteratively train the first model, and the training is stopped when a predetermined training condition is reached to determine a third model. In this way, the first configuration parameter of the encoder and the corresponding processing index information are considered in the training process of the third model, so that the encoding process of the encoder can be fully perceived in the training process of the third model. Moreover, the scheme of training the third model by combining the corresponding first configuration parameter and the processing index information of the encoder can achieve end-to-end joint training effect of the third model and the encoder, so that the third model and the encoder can be globally aligned in terms of perceptual enhancement target and rate control, and the quality of the video output by the trained third model after being processed by the encoder is high.

[0114] Please refer to Figure 5 The flowchart of the video processing model training method provided in the embodiment of the application is shown in Figure Two , Figure 3 S303 in the embodiment of the application can also be implemented by S401 to S403, which will be described in combination with the steps shown in Figure 5

[0115] S401, iteratively training a second model based on the evaluation information and the processing index information, and iteratively training the first model based on the evaluation information.

[0116] ​In the embodiments of the present application, the training device can determine the maximum reward function value based on the processing index information, and update the parameters of the second model to obtain a first trained second model in combination with the evaluation information and the maximum reward function value. The parameters of the first model can also be updated according to the evaluation information to obtain a first trained first model.

[0117] In the embodiments of the present application, the first model and the second model both need to be trained through the first video (sample video) for multiple rounds until the predetermined training condition is reached to stop training. The second model can include a value network model, and the network structure of the value network model is as shown in Figure 6 The value network model only exists in the training link. The value network model in Deep Deterministic Policy Gradient (DDPG) is a deep reinforcement learning algorithm for continuous action control, and its core framework is composed of the following key components. The environment evaluator plays a key role in it, and its core role is reflected in that it predicts the expected reward value that can be obtained in the future after performing an action in the current state. That is, the luminance components (Input first video frame Y and Input second video frame Y) and the chrominance components (Input first video frame UV and Input second video frame UV) corresponding to the first video frame and the second video frame processed by the pre-processing network are input respectively, to obtain the value Q, and the evaluation information is determined by the Q value to provide an optimization direction for the first model.

[0118] S402, based on the first trained model processing the next first video to determine the next second video, and using the encoder to process the next second video to determine the corresponding next processing index information, based on the second trained model processing the next first video and the next second video to determine the next evaluation information.

[0119] In the embodiments of the present application, the training device can determine the maximum reward function value based on the processing index information, and update the parameters of the second model to obtain a first trained second model in combination with the evaluation information and the maximum reward function value. The parameters of the first model can also be updated according to the evaluation information to obtain a first trained first model.

[0120] In the embodiments of the present application, the training device aligns each first video frame of the next first video and each corresponding second video frame in the next second video, and then performs quality evaluation on the aligned frames by using the trained second model to determine an evaluation value corresponding to each first video frame; wherein the evaluation information includes the evaluation value of each first video frame. The quality evaluation can be a comparison of the pixel similarity between the first video frame and the second video frame, or a comparison of the similarity of the picture content between the first video frame and the second video frame.

[0121] In the embodiments of the present application, the training device can determine a maximum reward function value according to the processing index information, determine a loss of the second model based on the maximum reward function value and the evaluation information, determine a loss of the first model based on the evaluation information, further update the parameters of the second model according to the loss gradient of the second model, update the parameters of the first model according to the loss gradient of the first model, and then synchronously iteratively train the first model and the second model. The second model after each training is used to determine the evaluation information of the next time, and the training is stopped when a predetermined number of training times is reached to determine the third model corresponding to the first model.

[0122] In the embodiments of the present application, the training device can determine a maximum reward function value according to the processing index information, determine a loss of the second model based on the maximum reward function value and the evaluation information, determine a loss of the first model based on the evaluation information, further update the parameters of the second model according to the loss gradient of the second model, update the parameters of the first model according to the loss gradient of the first model, and then synchronously iteratively train the first model and the second model. The second model after each training is used to determine the evaluation information of the next time, and the training is stopped when a predetermined number of training times is reached to determine the third model corresponding to the first model.

[0123] In the embodiments of the present application, the second model is trained based on the evaluation information and the processing index information, and the first model is trained based on the evaluation information; wherein the second model is used to determine the evaluation information; the next second video is determined based on the trained first model processing the next first video, and the corresponding next processing index information is determined by using the encoder to process the next second video, the next evaluation information is determined based on the trained second model processing the next first video and the next second video; the second model is iteratively trained based on the next evaluation information and the processing index information, and the first model is iteratively trained based on the next evaluation information, and the training is stopped when a predetermined number of times is reached to determine the third model corresponding to the first model. In this way, since the evaluation information is the evaluation information of the processing quality of the first video by the first model under the influence of the first configuration parameter, when training the first model and the second model, the first configuration parameter of the encoder and the corresponding processing index information are considered. This training scheme of the third model combined with the first configuration parameter of the encoder and the processing index information can achieve the end-to-end joint training effect of the third model and the encoder, and can make the third model and the encoder globally aligned in the perception enhancement target and the code rate control, and then the quality of the video output by the trained third model after being processed by the encoder is high.

[0124] Please refer to Figure 7 , the flowchart of the video processing model training method provided by the embodiment of the application is shown Figure Three , Figure 3 S301 in the embodiment of the application can also be implemented through S501 to S502, which will be described in combination with the steps shown in Figure 7 :

[0125] S501, input the first video frame in the first video and the first configuration parameter into the first model, determine the first feature corresponding to the first configuration parameter, and the second feature corresponding to the first video frame corresponding to the calculation unit of the first model; wherein the first feature is used to represent the scaling coefficient of the channel processing intensity of the first model.

[0126] In the embodiment of the application, the first model can include two processing paths, one path processes the first configuration parameter, and the other path processes each first video frame. The training device can input each first video frame and first configuration parameter into the first model in the order of video frames in the first video, and process the first video frame and the first configuration parameter by using the two paths. One path of the first model can map the first configuration parameter to the first feature, and the first feature represents the scaling coefficient of the processing intensity of each channel in the path of processing the first video frame of the first model. The first model can process the first video frame through the convolution layer and each calculation unit (ResBlock) to determine the second feature when each calculation unit processes.

[0127] In the embodiment of the application, the first video includes: a first type of video and a second type of video with a training order; wherein the first type of video is a preset sample video; the second type of video is a sample video obtained when the encoder is applied; the first configuration parameter includes: any one of the target code rate and the quantization parameter (Quantization Parameter, QP). The first configuration parameter in each iteration training is different.

[0128] In the embodiment of the application, the training process of the training device for the first model can include two stages, the first stage uses a plurality of first type of videos for training, and the second stage uses a plurality of second type of videos for training. One first type of video or one second type of video in each training stage can correspond to one Batch. In each Batch, the video frames and the first configuration parameter (target code rate or quantization parameter) of the corresponding video can be input into the first model for training. Wherein, the selection of the first configuration parameter (target code rate or quantization parameter) during training is affected by the application scene of the encoder.

[0129] In the embodiments of the present application, the training device can also pre-set the parameters of the first model for the first configuration parameters, and only needs to input the video frames to complete the training.

[0130] In the embodiments of the present application, the first type of video can include a new training data set for deep video compression optimization (ANew Training Dataset for Deep Video Compression Optimization, BVI-AOM) and a Tencent video data set (Tencent Video Dataset, TVD), and the second type of video can include a live video data set in an actual application scenario of an encoder. The training device can first convert the BVI-AOM and the TVD video data set into YUV420p format as a training set. Taking the deep deterministic policy gradient algorithm as a framework, the training set video is input into the first model frame by frame, and the enhanced video output by the first model is input into a non-differentiable traditional encoder (any encoder conforming to any video coding standard can be used). The encoding configuration during encoding is the first configuration parameter of the specific actual live application scenario, and finally the real code rate consumption and the decoding PSNR (objective quality) or VMAF (subjective quality) are received as a reward signal (Reward). The parameter update direction (Policy Gradient) of the first model is directly optimized to realize the closed-loop optimization of coding and decoding. The first model is trained based on the PyTorch (Python-based Torch) framework. There are two training stages in total, and each stage contains 100 epochs. The first 100 epochs are trained using the BVI-AOM and the TVD video data set, and the last 100 epochs are used to fine-tune the model using the live data set in the actual application scenario, for a total of 200 epochs. During training, each video is regarded as a Batch, and the batch size (Batch Size) is 1.

[0131] S502, based on the first feature, the intensity scaling processing of the second feature is performed, the second video frame corresponding to the first video frame is determined, and the second video is determined based on the second video frame.

[0132] In the embodiments of the present application, because the first feature is used to represent the scaling coefficient of the processing intensity of each channel of the first model, the coefficient (that is, the first feature) can be combined with the second feature of each calculation unit of the first model to determine a final feature, and then the final feature is converted into an image to determine the corresponding second video frame. The second video is determined by combining the second video frames in sequence.

[0133] In the embodiment of the present application, each first video frame in the first video and the first configuration parameter are input into the first model, the first feature corresponding to the first configuration parameter and the second feature corresponding to each calculation unit of the first model of each first video frame are determined; wherein the first feature is used to represent the scaling factor of the processing intensity of each channel of the first model; each second feature is intensity scaled based on the first feature, the second video frame corresponding to each first video frame is determined, and the second video is determined based on each second video frame. In this way, the first configuration parameter of the encoder is considered in the process of processing the video by the first model, so that the encoding process of the encoder can be fully perceived in the training process of the first model, so that the end-to-end joint training of the first model and the encoder is formed, and the performance of the third model corresponding to the first model is improved, and the quality of the video output by the third model after being processed by the encoder is improved.

[0134] Please refer to Figure 8 The flowchart of the video processing model training method provided in the embodiment of the present application is shown in Figure Four , Figure 7 S501 to S502 in the embodiment of the present application can also be implemented by S601 to S602, which will be described in combination with the steps shown in Figure 8

[0135] S601, the first configuration parameter is processed into a first vector, and the second feature corresponding to at least one of the luminance component and the chroma component in the first video frame is determined by processing the first video frame through the calculation unit; wherein the number of elements of the first vector is the same as the number of channels of the first model, and the element is used to represent the scaling factor of the corresponding channel.

[0136] In the embodiment of the present application, the training device can process the first configuration parameter through the adaptive module in the first model, and process each first video frame through the video processing module in the first model. The first configuration parameter can be mapped into a first vector through the adaptive module. Wherein the number of elements of the first vector is the same as the number of channels of the video processing module, and each element is used to represent the scaling factor of each corresponding channel. At least one of the luminance component and the chroma component in each first video frame can be processed through a plurality of calculation units (ResBlock) in the video processing module to determine the corresponding second feature. Wherein the scaling factor can be opposite to the size of the target code rate corresponding to the encoder.

[0137] In the embodiment of the present application, the first model is mainly used for the pre-processing link in the video encoding process, and is specifically located before the entire encoding loop, so it can adapt to various encoders conforming to different encoding standards. The network structure of the first model is as shown in Figure 9 ​As shown, the first model can include an adaptive processing module (FS-Adaptive Module) and a video processing module. The video processing module is mainly composed of multiple residual blocks (ResBlock) based on convolution operation. The first model can align the chrominance component and the luminance component of the first video frame to reduce the neural network calculation complexity. The output of the final network is processed to restore the original resolution. If the luminance and chrominance information of the first video frame is input at the same time, since the video is in YUV420p format, the chrominance component is half the size of the luminance component, and the chrominance information needs to be nearest neighbor up-sampled to the same size as the luminance component. The input of the video processing module includes the luminance component Y and the chrominance component UV of the original video, and the target bit rate Bitrate or the quantization parameter used in subsequent compression. The input of the first model can include the following different schemes: 1) only input luminance Y; 2) only input chrominance U, V; 3) only input luminance Y and target bit rate Bitrate; 4) only input luminance Y and quantization parameter QP; 5) only input chrominance U, V and target bit rate Bitrate; 6) only input chrominance U, V and quantization parameter QP; 7) input luminance Y, chrominance U, V and target bit rate Bitrate; 8) input luminance Y, chrominance U, V and quantization parameter QP. The difference between different schemes is the complexity and the effect of pre-processing, and the quantization parameter and the target bit rate are super-parameters for adaptive adjustment of the processing quality of the first model in different application scenarios. For example, if the first model needs to pre-process the luminance of the video, schemes 3) and 4) can be selected: 3) only input luminance Y and target bit rate Bitrate; and scheme 4) only input luminance Y and quantization parameter QP.

[0138] In combination Figure 9 The target bit rate or the quantization parameter input by the adaptive model can be processed to map it to a first vector Scale factor using multiple fully connected networks and a Sigmoid function. The length of the first vector is the same as the number of channels of the video processing module, and the first vector represents the scaling coefficients of different channels. Through this operation, the pre-processing strength of the first model under different target bit rates or different quantization parameters can be adaptively controlled. Because at high bit rates, more texture details need to be preserved after compression, the pre-processing strength of the pre-processing network should be relatively low at this time. At low bit rates, most texture details will be lost during compression, so the pre-processing strength of the pre-processing network can be moderately high, which can benefit the subsequent compression process.

[0139] S602, based on the multiplication of the first vector corresponding to the second feature of the calculation unit and the conversion, determine the second video frame.

[0140] In the embodiment of the present application, the training device can multiply the first vector and the second feature processed by each calculation unit in the video processing module to obtain a feature, and then input the feature into the next calculation unit until the final feature is output, and then perform pixel restoration processing on the feature to obtain the corresponding second video frame.

[0141] For example, in combination with Figure 9 , the FS-Adaptive Module outputs the first vector, and the output second feature in the calculation unit (ResBlock) needs to be calculated in combination with the first vector to obtain the corresponding feature, and then output the feature to the next calculation unit.

[0142] In the embodiment of the present application, the first configuration parameter is processed into a first vector, and at least one of the luminance component and the chrominance component in each first video frame is processed by each calculation unit to determine the corresponding second feature; wherein the number of elements of the first vector is the same as the number of channels of the first model, and each element is used to represent the scaling coefficient of each corresponding channel. Based on the multiplication of the first vector and the corresponding second feature of each calculation unit, the transformation is performed to determine each second video frame. In this way, the processing intensity of the first model during training can be controlled by using the first configuration parameter, and then the end-to-end joint training of the first model and the encoder can be achieved, which can make the third model and the encoder globally aligned in terms of perceptual enhancement target and code rate control, and then the quality of the video output by the trained third model after being processed by the encoder is high.

[0143] Please refer to Figure 10 , the flowchart of the video processing model training method provided in the embodiment of the present application Figure Five , the steps shown in Figure 10 will be described in combination:

[0144] S701, input the first video frame in the first video into the first model, determine the corresponding second video frame of the first video frame, and determine the second video based on the second video frame; wherein the first model is constructed by adjusting the processing intensity based on the first configuration parameter.

[0145] In the embodiment of the present application, the training device can pre-adjust the scaling coefficient parameter of each channel of the first model based on the target code rate or the quantization parameter, and then the first model in the embodiment does not Figure 9 adaptive module in the adaptive module. The training device can only input at least one of the luminance component or the chrominance component of each video frame to the first model, and the first model performs convolution residual processing on the first video frame to determine a final feature, and then performs pixel restoration processing on the final feature to determine the second video frame.

[0146] In the embodiment of the present application, each first video frame in the first video is input into the first model, a corresponding second video frame of each first video frame is determined, and the second video is determined based on each second video frame; wherein the first model is constructed based on the first configuration parameter adjusting the channel scaling factor. In this way, because the first model is constructed based on the first configuration parameter adjusting the channel processing intensity, the end-to-end joint training of the first model and the encoder is indirectly achieved, which can globally align the third model and the encoder in terms of the perceptual enhancement target and the code rate control, and thus the quality of the video output by the trained third model after being processed by the encoder is high.

[0147] Please refer to Figure 11 The flowchart of the video processing model training method provided in the embodiment of the present application is shown in Figure Six , Figure 5 S401 shown can be implemented through S801 to S802, which will be described in combination with Figure 11 The steps shown will be described:

[0148] S801, in response to the instruction of the user, determining the weights corresponding to the real code rate, the encoding quality information and the pixel similarity respectively, and performing weighted summation to determine the maximum reward function value; wherein the weight of the real code rate is a negative number; each weight is a weight for determining the maximum reward function value determined by the user according to the size of the real code rate, the encoding quality information and the pixel similarity.

[0149] In the embodiment of the present application, the processing index information includes at least one of the following: real code rate, encoding quality information and pixel similarity; the pixel similarity is used to represent the pixel similarity between each first video frame in the first video and the corresponding second video frame; in the embodiment of the present application, the training device can determine the encoded video by processing the second video through the encoder, collect the real code rate and the encoding quality information (PSNR (objective quality) or VMAF (subjective quality)) in the processing process, and obtain the pixel similarity. After obtaining the three parameters, they can be provided to the user, and the user determines the weight of each parameter for determining the maximum reward function value according to the size of the real code rate, the encoding quality information and the pixel similarity. Then, the maximum reward function value is determined by weighted summation based on the weights corresponding to the real code rate, the encoding quality information and the pixel similarity respectively. Real code rate

[0150] For example, the training device can determine the maximum reward function value R through formula (1).

[0151] R = a * real code rate + b * encoding quality information + l * pixel similarity formula (1)

[0152] wherein, a is the weight corresponding to the actual code rate, β is the weight corresponding to the encoding quality information, and λ is the weight corresponding to the pixel similarity. The maximum reward function value is composed of three parts, wherein a is negative, β and γ are positive, and the weights of the three are adjusted according to the actual training situation, and the goal is to make the video processed by the pre-processing network have the maximum reward value after being compressed by the encoder, that is, the actual code rate is lower, the PSNR or VMAF value is higher, and the pixel similarity with the original picture can be maintained to prevent the content from changing too much.

[0153] S802, based on the reward function value and the evaluation information, the loss is iteratively updated to update the parameters of the second model, and based on the negative value of the evaluation information, the loss is iteratively updated to update the parameters of the first model.

[0154] In the embodiments of the application, since the evaluation information is the evaluation value corresponding to each of the plurality of first video frames, the training device can determine a group of values by combining each evaluation value with the reward function value, and update the parameters of the second model based on the group of values. The training device can determine a comprehensive value based on the plurality of evaluation values, and update the parameters of the first model based on the negative value of the comprehensive value.

[0155] In the embodiments of the application, the optimization goal of the second model is to minimize the mean square error of the Q value (evaluation information) output by the second model and the maximum reward function value R. The essence of the second model network update is to make the second model more accurately predict the reward value obtained after the first model processes the current frame, and become a reliable reward predictor. The optimization goal of the first model is to use the output value information of the second model as a loss function by taking the negative value, and to minimize the loss function, which is equivalent to maximizing the expected return. The essence of the first model update is to adjust the strategy network parameters in the direction of value growth indicated by the value network, that is, how to improve the action to obtain a higher score.

[0156] In the embodiments of the application, in order to solve the problems in the prior art that the encoder is not differentiable, the pre-processing network and the compression process cannot be jointly trained end-to-end, and the decoding end computing power is limited and difficult to support the application of neural network tools, a video processing model training is provided. By inputting the luminance information / chrominance information and the target code rate / target QP information into the pre-processing network (first model), the luminance and chrominance can be pre-processed, and the pre-processing result is input into the encoder. As a result, the pre-processed video has higher subjective and objective quality after compression, and the code rate is lower, further improving the compression efficiency. The application is only applied to the encoding end and does not introduce additional computational complexity in the decoding end. The innovation lies in that:

[0157] 1. A first model is proposed, which, based on convolutional operations, achieves real-time preprocessing of YUV video at 4K@50fps (4K resolution at 50 frames per second) by stacking multiple residual blocks. Furthermore, through an adaptive module, the input target bitrate or quantization parameter information is mapped to a first vector, scale factor, through a fully connected layer and a sigmoid function. The vector length is the same as the number of channels in the preprocessing network, and this vector represents the scaling factor for different channels, thus adaptively controlling the preprocessing intensity of the network under different target bitrates or quantization parameters.

[0158] 2. A training method for the first model is proposed. The second video is input into the encoder, and the actual bitrate consumption and decoding PSNR (objective quality) or VMAF (subjective quality) after encoding are received. The maximum reward function value R is calculated based on the actual bitrate consumption and decoding PSNR (objective quality) or VMAF (subjective quality). The original YUV video and the preprocessing results are input into the value network to obtain the evaluation information Q. The second model is updated with the mean square error of the output Q value and the reward function value R as the optimization objective. The output value of the value network is negative as the loss function, and the first model is updated with the optimization objective of minimizing this loss function.

[0159] Compared with the closest existing technology, the present application proposal shows significant technical advantages in terms of method generalization and coding process awareness, and can more effectively improve the preprocessing enhancement effect.

[0160] ① Method generalization: Most existing technologies are traditional manually designed algorithms. Manually designed algorithms have limitations, cannot effectively cope with different types of video content, and cannot achieve image adaptation. There are bottlenecks in algorithm updates and optimizations, and they cannot effectively adapt to different codecs and application scenarios.

[0161] ② Encoding Process Awareness: Existing technologies, whether traditional algorithms or deep learning-based algorithms, cannot be trained end-to-end due to the non-differentiability of the encoder. They can only simulate the degradation process by manually designing degradation operators to approximate the actual encoding environment as closely as possible. In contrast, this method uses reinforcement learning algorithms, treating the encoder as the environment and the first model as the agent. It continuously interacts with the encoder and obtains real-time information on the current video's bitrate, PSNR, VMAF, and other metrics. This enables real-time awareness of the encoding process and adaptive adjustment of the preprocessing network parameters based on the content. This allows the network to adapt to different codecs through simple fine-tuning, achieving efficient and rapid deployment.

[0162] Please see Figure 12 This is a schematic diagram of the structure of the video processing device provided in the embodiments of this application, which will be combined with... Figure 12 Explanation:

[0163] This application embodiment also provides a video processing apparatus 600, including:

[0164] Input unit 601 is used to acquire the third video;

[0165] The first preprocessing unit 602 is configured to determine a fourth video by processing the third video and the second configuration parameters of the encoder based on the third model; wherein, determining the fourth video by processing the third video and the second configuration parameters of the encoder based on the third model includes:

[0166] The third video frame in the third video and the second configuration parameters are input into the third model to determine the third feature corresponding to the second configuration parameters and the fourth feature corresponding to the calculation unit of the third model; wherein, the third feature is used to characterize the scaling factor of the channel processing intensity of the third model;

[0167] Scaling unit 603 is used to scale the fourth feature based on the third feature, determine the fourth video frame corresponding to the third video frame, and determine the fourth video based on the fourth video frame.

[0168] The first encoding unit 604 is used to process the fourth video based on the encoder.

[0169] In this embodiment of the application, the first preprocessing unit 602 in the video processing device 600 is used to process the second configuration parameters into a second vector, and process at least one of the luminance component and chrominance component in each third video frame through each calculation unit of the third model to determine the corresponding fourth feature; wherein, the number of elements in the second vector is the same as the number of channels in the third model, and each element is used to characterize the scaling factor of each corresponding channel.

[0170] In this embodiment of the application, the first preprocessing unit 602 in the video processing device 600 is used to transform the second vector by multiplying it with the corresponding fourth feature of each of the calculation units, and then determine each fourth video frame.

[0171] It should be noted that, in the embodiments of this application, if the above-described video processing method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a video processing model training device (which may be a personal computer, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a magnetic disk, or an optical disk. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0172] Correspondingly, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a first processor, implements the steps of the method on one side of the video processing apparatus 600.

[0173] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0174] It should be noted that, Figure 13 A schematic diagram of a hardware entity of an electronic device provided in an embodiment of this application, such as... Figure 13 As shown, this application embodiment provides an electronic device 700, including a first memory 702 and a first processor 701. The first memory 702 stores a computer program that can run on the first processor 701. When the processor 701 executes the program, it implements the steps in the above-described method, wherein;

[0175] The first processor 701 typically controls the overall operation of the electronic device 700.

[0176] The first memory 702 is configured to store instructions and applications executable by the first processor 701, and can also cache data to be processed or already processed by the first processor 701 and various modules in the electronic device 700 (e.g., image data, audio data, voice communication data and video communication data), which can be implemented by flash memory or random access memory (RAM).

[0177] Correspondingly, this application embodiment also provides a computer program product, including a computer program that can be executed by a first processor 701 of an electronic device 700 to complete the steps in the method of the video processing device 600.

[0178] Please see Figure 14 This is a schematic diagram of the structure of the video processing model training device provided in the embodiments of this application, which will be combined with Figure 14 Explanation:

[0179] This application provides a video processing model training device 800, including:

[0180] The second preprocessing unit 801 is used to determine the second video based on the first model processing the first video and the first configuration parameters of the encoder; wherein, the first configuration parameters are used to determine the processing intensity during the training of the first model;

[0181] The second encoding unit 802 is used to process the second video based on the encoder and determine processing index information for the second video;

[0182] The update unit 803 is used to iteratively train the first model based on the processing index information and the evaluation information of the first model for the processing quality of the first video, until training stops when a predetermined training condition is reached, and then determine the third model.

[0183] In this embodiment, the update unit 803 in the video processing model training device 800 is used to jointly iteratively train a second model based on the evaluation information and the processing index information, and iteratively train a first model based on the evaluation information; wherein, the second model is used to determine the evaluation information; the second preprocessing unit 801 in the video processing model training device 800 is used to process the next first video based on the trained first model to determine the next second video, and to process the next second video using the encoder to determine the corresponding next processing index information, and to process the next first video and the next second video based on the trained second model to determine the next evaluation information;

[0184] The update unit 803 in the video processing model training device 800 is used to jointly iteratively train the second model based on the next evaluation information and the processing index information, and iteratively train the first model based on the next evaluation information until training stops when a predetermined number of times is reached, and to determine the third model corresponding to the first model.

[0185] In this embodiment of the application, the second preprocessing unit 801 in the video processing model training device 800 is used to input the first video frame and the first configuration parameters in the first video into the first model, and determine the first feature corresponding to the first configuration parameters, and the second feature of the first model calculation unit corresponding to the first video frame; wherein, the first feature is used to characterize the scaling factor of the channel processing intensity of the first model.

[0186] Based on the first feature, the second feature is subjected to intensity scaling processing to determine the second video frame corresponding to the first video frame, and the second video is determined based on the second video frame.

[0187] In this embodiment of the application, the second preprocessing unit 801 in the video processing model training device 800 is used to process the first configuration parameters into a first vector, and to process at least one of the luminance component and chrominance component in the first video frame through the calculation unit to determine the corresponding second feature; wherein, the number of elements in the first vector is the same as the number of channels in the first model, and the elements are used to characterize the scaling factor of the corresponding channel.

[0188] In this embodiment of the application, the second preprocessing unit 801 in the video processing model training device 800 is used to transform the first vector by multiplying it with the second feature corresponding to the calculation unit, and then determine the second video frame.

[0189] In this embodiment of the application, the second preprocessing unit 801 in the video processing model training device 800 is used to align the first video frame of the next first video with the corresponding second video frame of the next second video and then perform quality evaluation through the trained second model to determine the evaluation value of the corresponding first video frame; wherein, the evaluation information includes: the evaluation value of the first video frame.

[0190] In this embodiment of the application, the first video includes: a first type of video and a second type of video with a training order; wherein, the first type of video is a preset sample video; the second type of video is a sample video obtained when the encoder is applied; the first configuration parameter includes: any one of the target bitrate and quantization parameter; the first configuration parameter is different for each iteration of training.

[0191] In this embodiment of the application, the second preprocessing unit 801 in the video processing model training device 800 is used to input the first video frame in the first video into the first model, determine the second video frame corresponding to the first video frame, and determine the second video based on the second video frame; wherein, the first model is constructed based on the first configuration parameters by adjusting the processing intensity.

[0192] In this embodiment, the processing indicator information includes at least one of the following: true bitrate, encoding quality information, and pixel similarity; the pixel similarity is used to characterize the pixel similarity between a first video frame and a corresponding second video frame in the first video; the update unit 801 in the video processing model training device 800 is used to respond to a user's instruction to determine the weights corresponding to the true bitrate, the encoding quality information, and the pixel similarity, and to perform a weighted summation to determine the maximum reward function value; wherein, the weight of the true bitrate is a negative number; each of the weights is a weight determined by the user based on the magnitude of the true bitrate, the encoding quality information, and the pixel similarity characterization values ​​to determine the maximum reward function value;

[0193] The parameters of the second model are updated based on the loss determined by the reward function value and the evaluation information, and the parameters of the first model are iteratively updated based on the loss determined by the negative value of the numerical value represented by the evaluation information.

[0194] Correspondingly, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a second processor, implements the steps of the method on one side of the video processing model training device 800.

[0195] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0196] It should be noted that, Figure 15 A schematic diagram of a hardware entity of an electronic device provided in an embodiment of this application, such as... Figure 15 As shown, this application embodiment provides an electronic device 900, including a second memory 902 and a second processor 901. The second memory 902 stores a computer program that can run on the second processor 901. When the processor 901 executes the program, it implements the steps in the above-described method, wherein;

[0197] The second processor 901 typically controls the overall operation of the electronic device 900.

[0198] The second memory 902 is configured to store instructions and applications executable by the second processor 901, and can also cache data to be processed or already processed by the second processor 901 and various modules in the electronic device 900 (e.g., image data, audio data, voice communication data, and video communication data), which can be implemented by flash memory or random access memory (RAM).

[0199] Correspondingly, this application also provides a computer program product, including a computer program that can be executed by a second processor 901 of an electronic device 900 to complete the steps in the method of the video processing model training device 800.

[0200] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0201] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0202] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the apparatus or units can be electrical, mechanical, or other forms.

[0203] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0204] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0205] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0206] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0207] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A video processing model training method, characterized in that, include: A second video is determined based on the first model processing the first video and the first configuration parameters of the encoder; wherein, the first configuration parameters are used to determine the processing intensity during the training of the first model; Based on the encoder, the second video is processed to determine the processing index information for the second video; Based on the processing index information and the evaluation information of the first model on the processing quality of the first video, the first model is iteratively trained until the predetermined training conditions are met, at which point training stops, and the third model is determined.

2. The video processing model training method according to claim 1, characterized in that, The step of iteratively training the first model based on the processing index information and the evaluation information of the first model for the processing quality of the first video, until training stops when a predetermined training condition is met, and determining the third model, includes: The second model is jointly and iteratively trained based on the evaluation information and the processing indicator information, and the first model is iteratively trained based on the evaluation information. Based on the trained first model, the next first video is processed to determine the next second video, and the encoder is used to process the next second video to determine the corresponding next processing index information. Based on the trained second model, the next first video and the next second video are processed to determine the next evaluation information. The second model is jointly and iteratively trained based on the evaluation information and the processing index information described below, and the first model is iteratively trained based on the evaluation information described below until a predetermined number of training iterations are reached, at which point training stops, and the third model corresponding to the first model is determined.

3. The video processing model training method according to claim 2, characterized in that, The step of determining the second video based on the first model processing of the first video and the first configuration parameters of the encoder includes: The first video frame and the first configuration parameters in the first video are input into the first model to determine the first feature corresponding to the first configuration parameters and the second feature of the first model calculation unit corresponding to the first video frame; wherein, the first feature is used to characterize the scaling factor of the channel processing intensity of the first model. The second feature is subjected to intensity scaling based on the first feature to determine the second video frame corresponding to the first video frame, and the second video is determined based on the second video frame.

4. The video processing model training method according to claim 3, characterized in that, Determining the first feature corresponding to the first configuration parameter and the second feature corresponding to the first model calculation unit of the first video frame includes: The first configuration parameter is processed into a first vector, and at least one of the luminance component and chrominance component in the first video frame is processed by the calculation unit to determine the corresponding second feature; wherein, the number of elements in the first vector is the same as the number of channels in the first model, and the elements are used to characterize the scaling factor of the corresponding channel.

5. The video processing model training method according to claim 4, characterized in that, The step of performing intensity scaling processing on the second feature based on the first feature to determine the second video frame corresponding to the first video frame includes: The second video frame is determined by multiplying the first vector with the second feature corresponding to the computing unit and then transforming the vector.

6. The video processing model training method according to claim 5, characterized in that, The step of processing the next first video and the next second video based on the trained second model to determine the next evaluation information includes: Align the first video frame of the next first video with the corresponding second video frame of the next second video, and then perform quality evaluation using the trained second model to determine the evaluation value of the corresponding first video frame; wherein, the evaluation information includes: the evaluation value of the first video frame.

7. The video processing model training method according to claim 1, characterized in that, The first video includes: a first type of video and a second type of video with a training order; wherein, the first type of video is a preset sample video; the second type of video is a sample video obtained when the encoder is applied; the first configuration parameter includes: either the target bitrate or the quantization parameter; the first configuration parameter is different for each iteration of training.

8. The video model training method according to any one of claims 2 to 7, characterized in that, The processing index information includes at least one of the following: true bit rate, encoding quality information, and pixel similarity; The pixel similarity is used to characterize the pixel similarity between the first video frame and the corresponding second video frame in the first video; The step of jointly iteratively training the second model based on the evaluation information and the processing indicator information, and iteratively training the first model based on the evaluation information, includes: In response to user instructions, the weights corresponding to the actual bitrate, the encoding quality information, and the pixel similarity are determined, and a weighted sum is performed to determine the maximum reward function value; wherein, the weight of the actual bitrate is a negative number; each of the weights is determined by the user based on the magnitude of the actual bitrate, the encoding quality information, and the pixel similarity representation value to determine the maximum reward function value; The parameters of the second model are updated based on the loss determined by the reward function value and the evaluation information, and the parameters of the first model are iteratively updated based on the loss determined by the negative value of the numerical value represented by the evaluation information.

9. A video processing method, characterized in that, include: The fourth video is determined based on the second configuration parameters of the encoder and the third video processed by the third model. The step of determining the fourth video based on the second configuration parameters of the encoder and the third model processing includes: The third video frame in the third video and the second configuration parameters are input into the third model to determine the third feature corresponding to the second configuration parameters and the fourth feature corresponding to the calculation unit of the third model; wherein, the third feature is used to characterize the scaling factor of the channel processing intensity of the third model; The fourth feature is scaled based on the third feature to determine the fourth video frame corresponding to the third video frame, and the fourth video is determined based on the fourth video frame. The fourth video is processed based on the encoder.

10. The processing method according to claim 9, characterized in that, The third model is trained iteratively based on the processing index information and the evaluation information of the first model on the processing quality of the first video; the processing index information is determined by the encoder processing the second video. The second video is determined by the first model processing the first video and the first configuration parameters of the encoder; the first configuration parameters are used to determine the processing intensity during the training of the first model; The second configuration parameter is used to determine the processing intensity when the third model is applied.

11. A video processing apparatus, characterized in that, include: Input unit, used to acquire third-party video; The first preprocessing unit is used to determine the fourth video based on the second configuration parameters of the encoder and the third video processed by the third model; wherein, determining the fourth video based on the second configuration parameters of the encoder and the third video processed by the third model includes: The third video frame in the third video and the second configuration parameters are input into the third model to determine the third feature corresponding to the second configuration parameters and the fourth feature corresponding to the calculation unit of the third model; wherein, the third feature is used to characterize the scaling factor of the channel processing intensity of the third model; A scaling unit is used to scale the fourth feature based on the third feature, determine the fourth video frame corresponding to the third video frame, and determine the fourth video based on the fourth video frame. The first encoding unit is used to process the fourth video based on the encoder.

12. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program that can run on the processor, the processor executing the computer program to implement the steps of the method of any one of claims 1 to 8, or to implement the steps of the method of claim 9 or 10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8, or the steps of the method according to claim 9 or 10.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8, or the steps of the method according to claim 9 or 10.

Images