Sharpening parameter determination method, model training method and video processing method

Abstract

The embodiment of the invention provides a sharpening parameter determination method, a model training method and a video processing method, and the sharpening parameter determination method comprises the steps: obtaining a to-be-processed video, carrying out the coding processing of the to-be-processed video for at least two times according to a preset coding strategy, and constructing a coding video set according to the coding processing result; determining coding information of each coded video in the coded video set, and determining a rate distortion cost relationship of the coded video set under a video quality constraint condition according to the coding information of each coded video; and determining a sharpening parameter corresponding to the to-be-processed video according to the rate distortion cost relationship.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of machine learning technology, and in particular to a method for determining sharpening parameters, a model training method, and a video processing method. Background Art

[0002] With the development of computer and Internet technology, users have higher and higher requirements for video quality, and with the improvement of video quality, the consumed resources have increased significantly. In video transcoding, under the condition of fixed encoding parameters, videos with different degrees of texture can be obtained by transcoding according to different sharpening intensities; and for videos with inconsistent video content complexity, the corresponding bit rate and quality score are also inconsistent, and the performance difference between the bit rate and video quality evaluation results is also inconsistent. It can be seen that selecting different sharpening parameters for different videos can consume less resources while ensuring video quality. However, when selecting sharpening parameters before video encoding, most of the existing technologies use set standards and will not adjust according to video characteristics. Although the stability of video processing can be guaranteed, resource consumption cannot be guaranteed. If resource consumption is limited, video quality cannot be guaranteed. Therefore, an effective solution is urgently needed to solve the above problems. Summary of the invention

[0003] In view of this, the embodiments of this specification provide a method for determining sharpening parameters. One or more embodiments of this specification also relate to a model training method, a video processing method, a sharpening parameter determination device, a model training device, a video processing device, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defects existing in the prior art.

[0004] According to a first aspect of an embodiment of this specification, a method for determining a sharpening parameter is provided, comprising: Obtaining a video to be processed, and performing encoding processing on the video to be processed at least twice according to a preset encoding strategy, and constructing an encoded video set according to the encoding processing results; Determining encoding information of each encoded video in the encoded video set, and determining a rate-distortion cost relationship of the encoded video set under a video quality constraint according to the encoding information of each encoded video; A sharpening parameter corresponding to the video to be processed is determined according to the rate-distortion cost relationship.

[0005] According to a second aspect of an embodiment of this specification, a model training method is provided, including: Obtaining a sample video, and performing encoding processing on the sample video at least twice according to a preset encoding strategy, and constructing an encoded video set according to the encoding processing results; Determining encoding information of each encoded video in the encoded video set, and determining a rate-distortion cost relationship of the encoded video set under a video quality constraint according to the encoding information of each encoded video; Determining a sample pre-processing parameter corresponding to the sample video according to the rate-distortion cost relationship; The parameter prediction model is trained based on the sample video and the sample pre-processing parameters to obtain a target parameter prediction model.

[0006] According to a third aspect of the embodiments of this specification, a video processing method is provided, including: Acquire a target video, and determine a target video feature corresponding to the target video; Inputting the target video features into a target parameter prediction model for processing to obtain target pre-processing parameters, wherein the target parameter prediction model is trained by the above method; The target video is processed according to the target pre-processing parameters.

[0007] According to a fourth aspect of the embodiments of this specification, there is provided a sharpening parameter determination device, comprising: The video acquisition module is configured to acquire a video to be processed, and perform encoding processing on the video to be processed at least twice according to a preset encoding strategy, and construct an encoded video set according to the encoding processing results; An information determination module is configured to determine the encoding information of each encoded video in the encoded video set, and determine the rate-distortion cost relationship of the encoded video set under the video quality constraint condition according to the encoding information of each encoded video; The parameter determination module is configured to determine the sharpening parameters corresponding to the video to be processed according to the rate-distortion cost relationship.

[0008] According to a fifth aspect of the embodiments of this specification, a model training device is provided, including: A video acquisition module is configured to acquire a sample video, perform encoding processing on the sample video at least twice according to a preset encoding strategy, and construct an encoded video set according to the encoding processing results; An information determination module is configured to determine the encoding information of each encoded video in the encoded video set, and determine the rate-distortion cost relationship of the encoded video set under the video quality constraint condition according to the encoding information of each encoded video; A parameter determination module, configured to determine a sample pre-processing parameter corresponding to the sample video according to the rate-distortion cost relationship; The model training module is configured to train the parameter prediction model based on the sample video and the sample pre-processing parameters to obtain a target parameter prediction model.

[0009] According to a sixth aspect of the embodiments of this specification, there is provided a video processing device, including: A video acquisition module is configured to acquire a target video and determine a target video feature corresponding to the target video; A model prediction module is configured to input the target video features into a target parameter prediction model for processing to obtain target pre-processing parameters, wherein the target parameter prediction model is obtained by training using the above method; The video processing module is configured to process the target video according to the target pre-processing parameters.

[0010] According to a seventh aspect of an embodiment of this specification, there is provided a computing device, including: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above-mentioned sharpening parameter determination method, model training method or video processing method are implemented.

[0011] According to an eighth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned sharpening parameter determination method or model training method or video processing method.

[0012] According to the ninth aspect of the embodiments of this specification, a computer program product is provided, including a computer program or instructions, which, when executed by a processor, implement the steps of the above-mentioned sharpening parameter determination method or model training method or video processing method.

[0013] The sharpening parameter determination method provided in this embodiment can accurately determine the sharpening parameters of the video to be processed, so as to ensure that the video can have higher quality and consume less resources when being processed. After obtaining the processed video, at least two encoding processes can be performed according to the preset encoding strategy, so as to achieve the construction of an encoded video set composed of encoded videos according to the encoding process results; and the encoded videos contained in the encoded video set are videos obtained by encoding the processed video multiple times. On this basis, the encoding information of each encoded video in the encoded video set can be determined, and then the rate-distortion cost relationship of the encoded video set under the video quality constraint condition can be constructed according to the encoding information of each encoded video; the rate-distortion cost relationship can reflect the intrinsic correlation between the encoding information corresponding to each encoded video and the rate-distortion cost corresponding to the video, and then the optimal solution of the encoding information can be determined according to the relationship, so as to determine the sharpening parameters corresponding to the optimal solution of the encoding information as the sharpening parameters corresponding to the video to be processed; so as to achieve the subsequent use of the sharpening parameters to process the video, it can be ensured that the video saves more bit rate under the same picture quality, or improves the video quality under the same bit rate, so as to facilitate the use of downstream services. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a schematic diagram of a method for determining sharpening parameters provided by an embodiment of this specification; Figure 2 is a flow chart of a method for determining sharpening parameters provided by an embodiment of this specification; Figure 3 is a schematic diagram of a model structure in a method for determining sharpening parameters provided by an embodiment of this specification; Figure 4 is a flow chart of a model training method provided by an embodiment of this specification; Figure 5 is a processing flow chart of a video processing method provided by an embodiment of this specification; Figure 6 is a flow chart of a video processing method provided by an embodiment of this specification; Figure 7 is a structural schematic diagram of a sharpening parameter determination device provided by an embodiment of this specification; Figure 8 It is a structural schematic diagram of a model training device provided by an embodiment of this specification; Fig. 9 is a structural schematic diagram of a video processing device provided by an embodiment of this specification; Fig.10 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0015] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0016] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0017] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0018] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0019] First, the terms involved in one or more embodiments of this specification are explained.

[0020] Pre-processing parameters are mainly used in the processing stage before video encoding. The purpose is to optimize and pre-process the original video data to improve video quality, reduce encoding complexity, and improve encoding efficiency.

[0021] FFmpeg: is an open source computer program that is mainly used to record, convert digital audio and video, and convert them into streams. It provides a complete solution for recording, converting and streaming audio and video, and includes a very advanced audio / video codec library libavcodec. In FFmpeg, one of the most common ways to sharpen is to use the unsharp filter. The unsharp filter achieves a sharpening effect by increasing the contrast of the edges in the image, making the image look clearer. This filter can control the degree and effect of sharpening by adjusting different parameters.

[0022] SAD (Sum of Absolute Differences): Calculates the sum of the absolute differences between each pixel in the prediction block and the source image block.

[0023] SATD (Sum of Absolute Transformed Differences): First, the residuals are transformed (such as Hadamard transform), and then the sum of the absolute values ​​of the transformed coefficients is calculated.

[0024] The fully connected layer (Fully Connected Layer), also known as the densely connected layer (Dense Layer), is a common layer type in neural networks, especially playing an important role in convolutional neural networks (CNN). Each neuron in the fully connected layer is connected to all neurons in the previous layer, and is used to comprehensively process the output features of the previous layer.

[0025] Hidden layers: These are the layers between the input and output layers of a neural network. These layers are invisible to the outside world (i.e., the user or designer of the network) because they do not directly receive input data nor directly produce output. The main role of hidden layers is to learn representations of input data that are useful for solving specific tasks such as classification, regression, etc. Hidden layers can contain different types of layers such as fully connected layers, convolutional layers, recurrent layers, etc., depending on the architecture of the network and the problem to be solved.

[0026] In this specification, a model training method is provided. One or more embodiments of this specification also relate to a sharpening parameter determination method. One or more embodiments of this specification also relate to a model training method, a video processing method, a sharpening parameter determination device, a model training device, a video processing device, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail one by one in the following embodiments.

[0027] See also Figure 1As shown in the schematic diagram, the sharpening parameter determination method provided by the present embodiment can accurately determine the sharpening parameters of the video to be processed, so as to ensure that the video can have higher quality and consume less resources when being processed. After obtaining the processed video, at least two encoding processes can be performed according to the preset encoding strategy, so as to realize the construction of an encoded video set composed of encoded videos according to the encoding process results; and the encoded videos contained in the encoded video set are videos obtained by encoding the processed video multiple times. On this basis, the encoding information of each encoded video in the encoded video set can be determined, and then the rate-distortion cost relationship of the encoded video set under the video quality constraint condition can be constructed according to the encoding information of each encoded video; the rate-distortion cost relationship can reflect the intrinsic correlation between the encoding information corresponding to each encoded video and the rate-distortion cost corresponding to the video, and then the optimal solution of the encoding information can be determined according to the relationship, so as to determine the sharpening parameters corresponding to the optimal solution of the encoding information as the sharpening parameters corresponding to the video to be processed; so as to realize that when the sharpening parameters are used to process the video in the future, it can be ensured that the video saves more bit rate under the same picture quality, or improves the video quality under the same bit rate, so as to facilitate the use of downstream services.

[0028] See also Figure 2 , Figure 2 A flowchart of a method for determining sharpening parameters provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0029] Step S202: obtaining a video to be processed, and performing encoding processing on the video to be processed at least twice according to a preset encoding strategy, and constructing an encoded video set according to the encoding processing results.

[0030] The target parameter prediction model trained by the model training method provided in this embodiment can be applied to the video optimization and processing stage before video encoding in any scenario, such as live broadcast, on-demand, transcoding and other application scenarios. Before encoding the live video stream, on-demand video or to-be-transcoded video, the sharpening parameter determination method provided in this embodiment can be used to calculate the sharpening parameters required before video encoding, so as to process the video with the sharpening parameters before performing encoding processing operations. This can save more bitrate at the same image quality, or improve video quality at the same bitrate for subsequent use.

[0031] This embodiment takes the parameter prediction model required for training video on demand scenarios as an example to describe the model training method provided in this embodiment. The model training process required for other scenarios can refer to the same or corresponding description in this embodiment, and this embodiment will not be elaborated in detail here.

[0032] Specifically, the video to be processed specifically refers to the video clip that needs to be processed at the current stage, wherein the video length corresponding to the video to be processed can be set according to actual needs, and this embodiment does not make any limitation here. Correspondingly, the encoding strategy specifically refers to the strategy for encoding the video to be processed, and the strategy is used to record different encoding parameters so that after encoding the videos to be processed respectively with different encoding parameters, the encoded videos corresponding to each encoding parameter can be obtained, thereby forming an encoded video set, which is convenient for the subsequent determination and use of the optimal sharpening parameters corresponding to the video to be processed. Among them, the encoded videos contained in the encoded video set are all obtained after encoding the video to be processed with different encoding parameters, so that the encoded videos contained in the encoded video set can be used to subsequently deduce the optimal desharpening parameters corresponding to the video to be processed, so that it can be used as a label to be processed for model training.

[0033] Based on this, in order to accurately determine the sharpening parameters of the video to be processed, so as to ensure that the video can have higher quality and consume less resources when being processed, after obtaining the processed video, at least two encoding processes can be performed according to the preset encoding strategy, so as to realize the construction of an encoded video set composed of encoded videos according to the encoding processing results; and the encoded videos contained in the encoded video set are videos obtained by encoding the processed video multiple times. On this basis, the encoding information of each encoded video in the encoded video set can be determined, and then the rate-distortion cost relationship of the encoded video set under the video quality constraint can be constructed according to the encoding information of each encoded video; the rate-distortion cost relationship can reflect the intrinsic correlation between the encoding information corresponding to each encoded video and the rate-distortion cost corresponding to the video, and then the optimal solution of the encoding information can be determined according to the relationship, so as to determine the sharpening parameters corresponding to the optimal solution of the encoding information as the sharpening parameters corresponding to the video to be processed; so as to realize the subsequent processing of the video using the sharpening parameters.

[0034] Furthermore, when the encoding strategy is used to encode the processed video multiple times, in order to ensure that the encoded videos included in the constructed encoded video set are more evenly distributed, the encoding parameters can be selected according to the set parameter interval and step size, thereby completing the construction of the encoded video set. In this embodiment, the specific implementation method is as follows: Determine a preset encoding strategy, and determine encoding parameter interval information and step information according to the encoding strategy; generate at least two target encoding parameters based on the encoding parameter interval information and the step information, and use the at least two target encoding parameters to encode the video to be processed respectively; generate an encoded video corresponding to each target encoding parameter according to the encoding processing result, and construct an encoded video set based on the encoded video corresponding to each target encoding parameter.

[0035] Specifically, the encoding parameter interval information specifically refers to the interval of the parameter value that affects the strength of video sharpening when encoding the processed video. Correspondingly, the step information specifically refers to the step of the parameter value that affects the strength of video sharpening in the encoding parameter interval information. In practical applications, the encoding parameter interval information and the step information can be set according to actual needs, and this embodiment does not make any restrictions here. Correspondingly, the target encoding parameter is the encoding parameter composed of different parameters that affect the strength of video sharpening selected in the encoding parameter interval information according to the step information.

[0036] Based on this, when constructing a set of encoded videos corresponding to the video to be processed, in order to ensure the uniformity of video distribution, a preset encoding strategy can be determined first. At this time, the encoding parameter interval information and the step information can be determined according to the encoding strategy; then different encoding parameters can be selected in the encoding parameter interval information according to the step information, and then at least two target encoding parameters can be generated in combination with the encoding strategy. Thereafter, the at least two target encoding parameters can be used to encode the video to be processed respectively; so as to generate the encoded video corresponding to each target encoding parameter according to the encoding processing result. On this basis, a set of encoded videos can be constructed based on the encoded video corresponding to each target encoding parameter, so that the encoded video set can be used to infer the optimal desharpening parameters for the video to be processed later.

[0037] In practical applications, when encoding the video to be processed, the sharpening method unsharp in ffmpeg can be used to perform multiple transcoding processes on the video to be processed, and multiple transcoded videos can be generated by adjusting the weight (sharpening strength parameter) of unsharp. In specific implementation, the configuration interval of the weight can be set according to actual needs, such as [0.25, 0.45], and the step size can be 0.1. On this basis, by transcoding the video to be processed three times, three transcoded videos can be obtained for subsequent use.

[0038] For example, in order to ensure that more bitrate can be saved under the premise of the same video quality in the video on demand scenario, it is necessary to make reasonable sharpening parameter selection before encoding the video on demand. In order to be able to adaptively adjust the sharpening parameters of the video, a model for predicting the sharpening parameters of the video on demand can be trained. Specifically, first obtain the video to be processed for model training. After that, the sharpening method unsharp in ffmpeg can be used, and the weight of the sharpening method unsharp is in the interval [0.25, 0.45] with a step size of 0.1. In this way, the video to be processed is transcoded three times. At this time, transcoded video 1, transcoded video 2 and transcoded video 3 can be obtained, so that the optimal sharpening parameters of the video to be processed can be determined in combination with transcoded videos 1 to 3, and it can be used as a sample label for model training.

[0039] In summary, by constructing the target coding parameters according to the set coding parameter interval information and step information, it can be ensured that the target coding parameters are distributed more evenly, so the encoded video constructed as the basis is also more uniform, which facilitates the subsequent accurate calculation of sharpening parameters.

[0040] Step S204: determine the encoding information of each encoded video in the encoded video set, and determine the rate-distortion cost relationship of the encoded video set under the video quality constraint condition according to the encoding information of each encoded video.

[0041] Specifically, after determining the video to be processed and its corresponding encoded video set as mentioned above, in order to be able to infer the optimal solution sharpening parameters in combination with the encoded videos contained in the encoded video set, the encoding information of each encoded video in the encoded video set can be determined first, and on this basis, the rate-distortion cost relationship of the encoded video set under the video quality constraint condition can be mined, so that the optimal solution sharpening parameters can be calculated in combination with the rate-distortion cost relationship.

[0042] In other words, the coding information can characterize the quality of each coded video, and the video quality has a direct correlation with the rate-distortion cost of the video. The rate-distortion cost is an indicator to measure the degree of distortion generated during the video coding process at a given bit rate. Therefore, the rate-distortion cost relationship of the coded video set under the video quality constraint can be mined in combination with the coding information corresponding to each coded video. Based on the rate-distortion cost relationship, the quality of the coded video under different coding information can be reflected. Therefore, it is convenient to filter out the coded video with the highest quality and the lowest rate-distortion cost by combining the rate-distortion cost relationship and the coding information of each coded video, and use the corresponding sharpening parameters as the optimal desharpening parameters corresponding to the video to be processed, so that the parameter prediction model can be effectively trained.

[0043] Among them, the coding information specifically refers to the information recording the bit rate and video quality assessment results corresponding to the coded video. Correspondingly, the rate-distortion cost relationship specifically refers to the calculation relationship established for the coding information and the rate-distortion cost. Based on this relationship, the rate-distortion cost of the coded video can be calculated after determining the coding information corresponding to the coded video, and then different optimal rate-distortion costs can be calculated for the video based on different coding information in the subsequent process, so as to select the optimal sharpening parameters that match the video to be processed.

[0044] Furthermore, in order to accurately construct the rate-distortion cost relationship, the coding information can be constructed by combining the bit rate information and the video quality assessment information, so that the rate-distortion cost relationship can be constructed by combining the above information. In this embodiment, the specific implementation method is as follows: Determine the bit rate information and video quality assessment information corresponding to the encoded video, and construct the encoding information of the encoded video based on the bit rate information and the video quality assessment information; wherein, determining the rate-distortion cost relationship of the encoded video set under the video quality constraint condition according to the encoding information of each encoded video includes: determining the sub-rate-distortion cost relationship of each encoded video under the video quality constraint condition according to the bit rate information and video quality assessment information of each encoded video; integrating the sub-rate-distortion cost relationship corresponding to each encoded video to obtain the rate-distortion cost relationship corresponding to the encoded video set.

[0045] Specifically, the bitrate information refers to the bitrate corresponding to the encoded video, and the video quality assessment information refers to the score representing the quality of the video, namely the VMAF (Video Multimethod Assessment Fusion) value. Correspondingly, the sub-rate distortion cost relationship refers to the relationship between the bitrate corresponding to each encoded video and the video quality assessment information and the video rate distortion cost.

[0046] Based on this, in order to calculate the optimal desharpening parameters based on the video to be processed, the rate-distortion cost relationship can be calculated in combination with the bit rate and video quality assessment information of the encoded video. In this process, for any encoded video, the bit rate information and video quality assessment information corresponding to the encoded video can be combined to construct the encoding information. After obtaining the encoding information corresponding to each encoded video, the sub-rate-distortion cost relationship of each encoded video under the video quality constraint can be determined according to the bit rate information and video quality assessment information of each encoded video; on this basis, considering that each encoded video is obtained by encoding the video to be processed, the sub-rate-distortion cost relationship corresponding to each encoded video can be integrated to achieve the construction of the rate-distortion cost relationship corresponding to the encoded video set, so that the sharpening parameters corresponding to the video to be processed can be determined in combination with the rate-distortion cost relationship later.

[0047] In specific implementation, when establishing the rate-distortion cost relationship corresponding to the set of coded videos, in order to reflect the relationship between the coding information and the rate-distortion cost by establishing a good rate-distortion cost relationship, and then determine the optimal coding information under the premise of the optimal rate-distortion cost, and then determine the corresponding sharpening parameters, the Lagrangian algorithm can be used to calculate the optimal solution. Specifically, after obtaining the bit rate and video quality assessment information (vamf) corresponding to each coded video, the Lagrangian algorithm can be used to calculate the optimal solution of each current coded video under the premise of limiting the bit rate, as shown in the following formula (1): max: Q st.R<Rc(1) Where Q represents the video quality assessment information, R represents the bit rate, and formula (1) represents the optimal solution max: Q for the average video quality of the coded video set under the premise of limiting the bit rate R to be less than the bit rate threshold Rc. The Lagrangian algorithm can be used to solve formula (1), and formula (1) can be converted into formula (2): min: J = -Q + λ*R (2) Among them, min:J represents the minimum rate-distortion cost. That is, after converting formula (1) into formula (2), the minimum rate-distortion cost of the video can be represented. On this basis, it can be deduced that the minimum rate-distortion cost of the coded video set can be obtained by formula (3): min: J = - +λ*- (3) By transforming formula (3), we can get the following formula (4): min:J= -Qi+λ*Ri](4) Where i represents any video in the coded video set. At this point, it can be determined that the minimum rate-distortion cost of the coded video set is the sum of the minimum rate-distortion costs corresponding to all coded videos. Based on this relationship, it can be adjusted to calculate the minimum rate-distortion cost of the coded video set under the premise of limiting the Lagrangian algorithm. The relationship is as shown in formula (5): min: Ji = -Qi + λ*Ri (5) Based on formula (5), it can be clearly seen that it establishes the relationship between the bit rate corresponding to the encoded video, the video quality assessment information and the minimum rate distortion cost. After that, when determining the sharpening parameters for the video to be processed, the above formula (5) can be used to solve the optimal solution based on the minimum rate distortion cost of the video to be processed under different sharpening parameters, and then it can be achieved by comparing different optimal solutions under different sharpening parameters to obtain the sharpening parameters matching the video to be processed for model training.

[0048] Using the above example, after transcoding the video to be processed to obtain transcoded video 1, transcoded video 2 and transcoded video 3, the bit rate and vamf value corresponding to each transcoded video can be determined first. At this time, the bit rate and vamf value corresponding to transcoded video 1 are determined to be {bitrate1, vmaf1}; the bit rate and vamf value corresponding to transcoded video 2 are {bitrate2, vmaf2}; the bit rate and vamf value corresponding to transcoded video 2 are {bitrate3, vmaf3}; on this basis, the Lagrangian algorithm and the above-mentioned Formulas (1) to (5) calculate the optimal solutions of the three sets of parameters under the premise of limiting the bit rate, so that the rate-distortion cost J1 corresponding to {bitrate1, vmaf1} of transcoded video 1, the rate-distortion cost J2 corresponding to {bitrate2, vmaf2} of transcoded video 2, and the rate-distortion cost J3 corresponding to {bitrate3, vmaf3} of transcoded video 3 can be obtained, so that the sharpening parameters of the video to be processed can be determined in combination with the rate-distortion costs corresponding to each transcoded video.

[0049] In summary, by using the Lagrangian algorithm to establish the rate-distortion cost relationship, the relationship between the bit rate and video quality assessment information and the rate-distortion cost can be established, so that after obtaining the encoding information corresponding to the encoded video, the rate-distortion cost relationship corresponding to the encoded video can be directly determined, and the rate-distortion cost relationship is directly related to the video quality. The sharpening parameters are parameters used to process and optimize the video before encoding. Therefore, combined with the relationship between the above information, it is convenient to complete the sharpening parameter selection according to the rate-distortion cost when selecting the sharpening parameters, so that it can be used for video processing or model training.

[0050] Step S206: determining a sharpening parameter corresponding to the video to be processed according to the rate-distortion cost relationship.

[0051] Specifically, after determining the rate-distortion cost relationship corresponding to the set of coded videos, considering that the relationship can reflect the mapping relationship between coding information and rate-distortion cost, the rate-distortion cost corresponding to each coded video can be calculated in combination with the coding information corresponding to each coded video, and the smaller the rate-distortion cost, the higher the video quality. Therefore, the sharpening parameters corresponding to the coded video with the smallest rate-distortion cost can be selected as the sharpening parameters corresponding to the video to be processed, so as to determine the real sharpening parameters corresponding to the video to be processed, so as to facilitate the subsequent model training or processing of the video to be processed. Among them, the sharpening parameters specifically refer to the optimal desharpening parameters selected for the video to be processed.

[0052] Furthermore, when determining the sharpening parameters corresponding to the video to be processed, in order to select the optimal solution, it can be screened by comparing the rate-distortion cost. In this embodiment, the specific implementation is as follows: Select multiple candidate sharpening parameters for the video to be processed, and determine the candidate encoding information corresponding to each candidate sharpening parameter; calculate the rate-distortion cost of the video to be processed under each candidate sharpening parameter based on the candidate encoding information corresponding to each candidate sharpening parameter and the rate-distortion cost relationship; compare the rate-distortion cost of each candidate sharpening parameter, and select the sharpening parameter corresponding to the video to be processed from the multiple candidate sharpening parameters according to the comparison result.

[0053] Specifically, the candidate sharpening parameters specifically refer to the sharpening parameters corresponding to the encoding information of each encoded video, and the sharpening parameters are selected from multiple candidate sharpening parameters by calculation. Accordingly, the candidate encoding information is the encoding information corresponding to the encoded video. Accordingly, the rate-distortion cost comparison specifically refers to the comparison of the rate-distortion cost corresponding to each candidate encoding information, so as to select the sharpening parameters of the encoding information corresponding to the minimum rate-distortion cost as the sharpening parameters of the video to be processed.

[0054] Based on this, when determining the sharpening parameters, multiple candidate sharpening parameters can be selected for the video to be processed first, and then the candidate encoding information corresponding to each candidate sharpening parameter can be determined; on this basis, the rate-distortion cost corresponding to each candidate sharpening parameter of the video to be processed can be calculated based on the candidate encoding information corresponding to each candidate sharpening parameter and the rate-distortion cost relationship; the smaller the rate-distortion cost, the higher the video quality, and thus the corresponding sharpening parameters are more reasonable. Therefore, the rate-distortion cost of each candidate sharpening parameter can be compared, and the candidate sharpening parameter corresponding to the smallest rate-distortion cost can be selected according to the comparison result as the sharpening parameter corresponding to the video to be processed, so as to be used for model training or video processing.

[0055] In specific implementation, the above formula (5) can be combined with the minimum rate-distortion cost corresponding to the encoding information of each encoded video under the premise of limiting the bit rate, and then the sharpening parameter corresponding to the minimum rate-distortion cost can be selected as the sharpening parameter corresponding to the video to be processed and can be used as the sample label for model training.

[0056] Continuing with the above example, after obtaining the rate-distortion cost J1 corresponding to {bitrate1, vmaf1} of transcoded video 1, the rate-distortion cost J2 corresponding to {bitrate2, vmaf2} of transcoded video 2, and the rate-distortion cost J3 corresponding to {bitrate3, vmaf3} of transcoded video 3, the rate-distortion costs J1, J2, and J3 can be compared to determine that J1<J3<J2. Therefore, it is determined that the rate-distortion cost corresponding to transcoded video 1 is the smallest under the premise of limiting the bit rate. At this time, the sharpening parameters corresponding to transcoded video 1 can be determined and used as the sharpening parameters of the video to be processed for training the model.

[0057] In summary, by selecting the sharpening parameters by comparing the rate-distortion cost, it is possible to map the relationship between the encoding information, the rate-distortion cost and the sharpening parameters, thereby ensuring that the selected sharpening parameters better match the video to be processed, thereby improving the model prediction accuracy during the model training phase, or after processing the video to be processed, high-quality video can be obtained for use by downstream businesses.

[0058] Furthermore, on the one hand, after obtaining the sharpening parameters corresponding to the video to be processed, the video to be processed can be processed using the sharpening parameters to obtain the target video, so that the video can be adjusted by more reasonable sharpening parameters before encoding the video to be processed, so that after the video is encoded, a target video with higher quality and less computing resources can be obtained for use in downstream businesses.

[0059] On the other hand, the parameter prediction model may also be trained based on the video to be processed and the sharpening parameters to obtain a target parameter prediction model.

[0060] That is to say, after obtaining the video to be processed and the sharpening parameters as mentioned above, the video to be processed can be used as the model input, and the sharpening parameters can be used as labels. In the model training stage, after the model output results are obtained, the model parameters can be adjusted according to the model output results and the labels. After the parameter adjustment is completed, new sample pairs are selected for further training, so as to obtain a target parameter prediction model that meets the training stop conditions. After the target parameter prediction model is deployed in the business scenario, the model can be used to predict the corresponding sharpening parameters before encoding the video stream or video clip. After the video is optimized and processed by the sharpening parameters, such as improving video quality (color correction, denoising), optimizing encoding efficiency (scene recognition, dynamic bit rate allocation), reducing encoding complexity (image scaling, frame rate adjustment), etc., the video can be encoded after the optimization and processing are completed, so as to save bit rate under the same video quality, or improve video quality under the same bit rate conditions.

[0061] Among them, the parameter prediction model specifically refers to a model that can predict the sharpening parameters needed for any video clip. Correspondingly, the target parameter prediction model is a prediction model obtained after training the parameter prediction model. The prediction model can use four fully connected layers to predict the sharpening parameters. In this case, the target parameter prediction model can include an input layer, a first hidden layer, a second hidden layer, a third hidden layer and an output layer; wherein the first hidden layer, the second hidden layer and the third hidden layer have different corresponding network parameters, and the output layer is a multi-classification network structure; the target parameter prediction model is used to predict the sharpening parameters of the target video.

[0062] Furthermore, in the model training stage, in order to improve the prediction ability of the model, the features of the video can be extracted as the model input, and then trained in combination with the sample labels. In this embodiment, the specific implementation is as follows: Feature extraction is performed on the video to be processed to obtain video features; the video features are input into a parameter prediction model for processing to obtain predicted sharpening parameters; the parameter prediction model is trained based on the predicted sharpening parameters and the sharpening parameters until a target parameter prediction model that meets the training stop condition is obtained.

[0063] Specifically, the video feature refers to a vector expression that meets the model input after feature extraction of the video to be processed. In practical applications, the video feature can be constructed by a variety of video complexity features, such as sad (Sum of Absolute Difference) and satd (Sum of Absolute Transformed Difference) at the video frame level. Complexity information such as the prediction result obtained after the parameter prediction model predicts the sharpening parameters for the sample video. Correspondingly, the training stop condition refers to the condition used to stop training the parameter prediction model, including but not limited to the number of iterations, the loss value comparison condition, etc. This embodiment does not make any restrictions here.

[0064] Based on this, after obtaining the video to be processed and its corresponding sharpening parameters, feature extraction can be performed on the video to be processed to obtain video features; thereafter, the video features can be input into the parameter prediction model for processing, so as to obtain predicted sharpening parameters after processing by the model; on this basis, the parameter prediction model is trained according to the predicted sharpening parameters and sharpening parameters until a target parameter prediction model that meets the training stop conditions is obtained.

[0065] In practical applications, the model input features can be constructed based on the complexity information of the frame-level sad and satd corresponding to the video to be processed. Among them, sad is determined by calculating the sum of the absolute values ​​of the difference between each pixel of the prediction block and the source image block. Satd can be understood as first transforming the residual (such as Hadamard transform), and then calculating the sum of the absolute values ​​of the transformed coefficients. By integrating various types of information, video features can be obtained. Then a four-layer fully connected network can be selected as the parameter prediction model, and the cross entropy loss function is used to calculate the loss value, and then a model that meets the usage requirements is trained and deployed in business scenarios.

[0066] In specific implementation, in addition to sad and satd information, the input features of the model can also contain features of other dimensions to form a 1*8 vector expression. And the parameter prediction model can set three hidden layers, the first hidden layer can be 8*32 network parameters, the second hidden layer can be 32*32 network parameters, and the third hidden layer can be 32*5 network parameters. Finally, an n-classified network parameter structure is set to achieve the prediction of the sharpening parameters corresponding to the output of the input features.

[0067] In summary, by constructing video features corresponding to the video for model input training, the intrinsic correlation between model learning features and sharpening parameters can be realized, and then the sharpening parameters of any video can be accurately predicted for use.

[0068] On this basis, the model optimization stage can complete the optimization operation by calculating the loss value. In this embodiment, the specific implementation method is as follows: A loss value is calculated according to the predicted sharpening parameter and the sharpening parameter, and the parameter prediction model is optimized according to the loss value; whether the optimized parameter prediction model satisfies the training stop condition is detected; if not, a candidate video to be processed is obtained, the candidate video to be processed is used as the video to be processed, and the step of encoding the video to be processed at least twice according to a preset encoding strategy is executed; if so, the optimized parameter prediction model is used as the target parameter prediction model.

[0069] Specifically, the loss value specifically refers to the loss value calculated by combining the predicted result and the actual result. The loss value can be calculated by a cross entropy loss function or by other loss functions. This embodiment does not make any limitation here.

[0070] Based on this, after the model outputs the predicted sharpening parameters, the loss value can be calculated based on the predicted sharpening parameters and the sharpening parameters, and then the parameter prediction model can be optimized based on the loss value; at this time, it can be detected whether the optimized parameter prediction model meets the training stop condition; if not, it means that the model still needs to continue training, so the candidate video to be processed can be obtained, and the candidate video to be processed can be used as the video to be processed, and the step of encoding the video to be processed at least twice according to the preset encoding strategy is executed; until a certain moment the optimized model meets the training stop condition, the optimized parameter prediction model can be used as the target parameter prediction model.

[0071] Following the above example, after obtaining the sharpening parameters corresponding to the video to be processed, the parameters such as sad, satd, sad_min at the corresponding frame level of the video to be processed can be extracted to construct the video features corresponding to the video to be processed. At this time, the video features can be input into Figure 3 The parameter prediction model shown in FIG. 1 includes an input layer, a first hidden layer, a second hidden layer, a third hidden layer and an output layer. After being processed by the model, the predicted sharpening parameters corresponding to the video to be processed are obtained. At this time, the sharpening parameters and the predicted sharpening parameters can be combined with the cross entropy loss function to calculate the loss value and used to adjust the parameter prediction model. Similarly, after m iterations, the target parameter prediction model can be obtained. Furthermore, after the target parameter prediction model is deployed to the business scenario, the target parameter prediction model can be used to predict the sharpening parameters of the on-demand video before the video on demand is transcoded, and the sharpening parameters matching the on-demand video can be determined according to the prediction results. At this time, the sharpening parameters can be used to improve the video quality (color correction, denoising), optimize the encoding efficiency (scene recognition, dynamic bit rate allocation), reduce the encoding complexity (image scaling, frame rate adjustment) and other processing of the on-demand video, and then used for subsequent encoding, so as to send high-quality on-demand video to the client of the on-demand video.

[0072] In summary, by using an iterative training model to generate a target parameter prediction model, the model can have a strong sharpening parameter prediction capability, so that after being deployed in a business scenario, the sharpening parameters used in the video can be adaptively adjusted, thereby saving bit rate.

[0073] In addition, considering that the training stop condition of the model may be the number of iterations, whether the trained model meets the requirements is still unknown at this time, so the model can also be verified. In this embodiment, the specific implementation method is as follows: Obtain a target verification video and target verification sharpening parameters; input the target verification video into the target parameter prediction model for processing to obtain verification sharpening parameters, and compare the verification sharpening parameters with the target verification sharpening parameters; if it is determined based on the comparison result that the target parameter prediction model does not meet the model deployment conditions, optimize the target parameter prediction model.

[0074] Specifically, the target verification video specifically refers to a video for verifying the prediction ability of the target parameter prediction model, and the target verification sharpening parameters specifically refer to the optimal desharpening parameters corresponding to the target verification video, which can be obtained by the same processing method as the sharpening parameters corresponding to the video to be processed. This embodiment does not make any restrictions here. Correspondingly, the verification sharpening parameters specifically refer to the sharpening parameters output by the target parameter prediction model. Correspondingly, the model deployment conditions specifically refer to the conditions set for model deployment in business scenarios, such as model prediction accuracy and other conditions. This embodiment does not make any restrictions here.

[0075] Based on this, considering that the training stop condition of the model may be the number of iterations, the target prediction model obtained at this time may not necessarily meet the deployment requirements, that is, the prediction accuracy may need to be improved. Therefore, the model can be verified before deployment, and the target verification video and target verification sharpening parameters can be obtained at this time; then, by inputting the target verification video into the target parameter prediction model for processing, the verification sharpening parameters can be obtained, and then the verification sharpening parameters and the target verification sharpening parameters are compared; if the target parameter prediction model is determined to meet the model deployment conditions according to the comparison results, the target parameter prediction model can be deployed for use in business scenarios. If the target parameter prediction model is determined not to meet the model deployment conditions according to the comparison results, the model needs to be further optimized to obtain a model that meets the deployment conditions for use.

[0076] In summary, by using model verification as a test before model deployment, we can avoid the situation where the trained model fails to achieve the expected effect after deployment, and thus provide a more stable model prediction service.

[0077] In the process of model verification, in order to ensure the accuracy of the verification results, the obtained sharpening parameters can be used to verify video processing and optimization, and then determine whether optimization is needed by comparing the video. In this embodiment, the specific implementation method is as follows: The target verification video is processed using the verification sharpening parameters to obtain a first video, and the target verification video is processed using the target verification sharpening parameters to obtain a second video; the first video and the second video are compared for video information, and when it is determined based on the comparison result that the target parameter prediction model does not meet the model deployment conditions, the step of optimizing the target parameter prediction model is performed.

[0078] Specifically, the first video specifically refers to a video obtained by optimizing and processing the target verification video using the verification sharpening parameters, and the second video specifically refers to a video obtained by optimizing and processing the target verification video using the target verification sharpening parameters. By comparing the first video and the second video, the advantages and disadvantages of the two sharpening parameters can be more intuitively determined, and then whether the model meets the model deployment conditions.

[0079] Based on this, when testing whether the model needs to be further optimized, in order to be able to intuitively feedback the video quality effect, the target verification video can be processed using the verification sharpening parameters to obtain the first video, and the target verification video can be processed using the target verification sharpening parameters to obtain the second video; on this basis, the first video and the second video can be compared for video information. If the comparison results of the two video information are highly consistent, it means that no optimization is required. If the comparison results of the two video information are inconsistent, it means that the target parameter prediction model does not meet the model deployment conditions, and then the step of optimizing the target parameter prediction model can be performed.

[0080] The sharpening parameter determination method provided in this embodiment can accurately determine the sharpening parameters of the video to be processed, so as to ensure that the video can have higher quality and consume less resources when being processed. After obtaining the processed video, at least two encoding processes can be performed according to the preset encoding strategy, so as to achieve the construction of an encoded video set composed of encoded videos according to the encoding process results; and the encoded videos contained in the encoded video set are videos obtained by encoding the processed video multiple times. On this basis, the encoding information of each encoded video in the encoded video set can be determined, and then the rate-distortion cost relationship of the encoded video set under the video quality constraint condition can be constructed according to the encoding information of each encoded video; the rate-distortion cost relationship can reflect the intrinsic correlation between the encoding information corresponding to each encoded video and the rate-distortion cost corresponding to the video, and then the optimal solution of the encoding information can be determined according to the relationship, so as to determine the sharpening parameters corresponding to the optimal solution of the encoding information as the sharpening parameters corresponding to the video to be processed; so as to achieve the subsequent use of the sharpening parameters to process the video, it can be ensured that the video saves more bit rate under the same picture quality, or improves the video quality under the same bit rate, so as to facilitate the use of downstream services.

[0081] See also Figure 4 , Figure 4 A flowchart of a model training method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0082] Step S402, obtaining a sample video, and performing encoding processing on the sample video at least twice according to a preset encoding strategy, and constructing an encoded video set according to the encoding processing results; Step S404, determining the encoding information of each encoded video in the encoded video set, and determining the rate-distortion cost relationship of the encoded video set under the video quality constraint condition according to the encoding information of each encoded video; Step S406, determining a sample pre-processing parameter corresponding to the sample video according to the rate-distortion cost relationship; Step S408: training a parameter prediction model based on the sample video and the sample pre-processing parameters to obtain a target parameter prediction model.

[0083] The model training method provided in this embodiment is similar to the above-mentioned sharpening parameter determination method, and any content not described in detail can be referred to the above-mentioned embodiment. This embodiment will not be described in detail here.

[0084] The above is a schematic scheme of a model training method of this embodiment. It should be noted that the technical scheme of the model training method and the technical scheme of the above-mentioned sharpening parameter determination method belong to the same concept, and the details not described in detail in the technical scheme of the model training method can be referred to the description of the technical scheme of the above-mentioned sharpening parameter determination method.

[0085] The following combination Figure 5 , taking the application of the video processing method provided in this specification in a live broadcast scenario as an example, the video processing method is further described. Figure 5 A processing flow chart of a video processing method provided by an embodiment of the present specification is shown, which specifically includes the following steps.

[0086] Step S502: Obtain a sample video, determine a preset transcoding strategy, and determine transcoding parameter interval information and step information according to the transcoding strategy.

[0087] Step S504: generating at least two target transcoding parameters based on the transcoding parameter interval information and the step length information, and performing transcoding processing on the sample videos respectively using the at least two target transcoding parameters.

[0088] Step S506: Generate a transcoded video corresponding to each target transcoding parameter according to the transcoding processing result, and construct a transcoded video set based on the transcoded video corresponding to each target transcoding parameter.

[0089] Step S508: determine the transcoding information of each transcoded video in the transcoded video set, and determine the sub-rate distortion cost relationship of each transcoded video under the video quality constraint condition according to the bit rate information and video quality assessment information of each transcoded video.

[0090] Step S510: Integrate the sub-rate-distortion cost relationships corresponding to each transcoded video to obtain a rate-distortion cost relationship corresponding to the transcoded video set.

[0091] Step S512: select multiple candidate sharpening parameters for the sample video, and determine candidate transcoding information corresponding to each candidate sharpening parameter.

[0092] Step S514: based on the candidate transcoding information corresponding to each candidate sharpening parameter and the rate-distortion cost relationship, the rate-distortion cost corresponding to the sample video under each candidate sharpening parameter is calculated.

[0093] Step S516: compare the rate-distortion cost of each candidate sharpening parameter, and select a sharpening parameter corresponding to the sample video from a plurality of candidate sharpening parameters according to the comparison result.

[0094] Step S518, extracting features from the sample video to obtain video features, inputting the video features into a parameter prediction model for processing, and obtaining predicted sharpening parameters.

[0095] Step S520: training the parameter prediction model based on the predicted sharpening parameters and the sharpening parameters until a target parameter prediction model that meets the training stop condition is obtained, and deploying the target parameter prediction model on the streaming media server.

[0096] Step S522: When the streaming media server receives the live video stream, the target parameter prediction model is used to predict the live video stream sharpening parameters corresponding to the live video stream.

[0097] Step S524, optimizing the live video stream based on the live video stream sharpening parameters, and encoding the optimized live video stream.

[0098] The model training method provided in this embodiment, in order to train a model for accurately predicting the sharpening parameters used for video processing and optimization, can perform at least two transcoding processes on the sample videos obtained according to the preset transcoding strategy in the sample preparation stage before the model training stage, so as to build a transcoded video set composed of transcoded videos according to the transcoding processing results; and the transcoded videos included in the transcoded video set are videos obtained by transcoding the sample videos multiple times. On this basis, the transcoding information of each transcoded video in the transcoded video set can be determined, and then the rate-distortion cost relationship of the transcoded video set under the video quality constraint can be constructed according to the transcoding information of each transcoded video; the rate-distortion cost relationship can reflect the intrinsic correlation between the transcoding information corresponding to each transcoded video and the rate-distortion cost corresponding to the video, and then the optimal solution of the transcoding information can be determined according to the relationship, so as to determine the sharpening parameters corresponding to the optimal solution of the transcoding information as the sharpening parameters corresponding to the sample video; so as to build a label for model training for the sample video. After that, the parameter prediction model can be trained based on the sample video and the sharpening parameters until the target parameter prediction model that meets the training stop conditions is obtained. In the model application stage, the model can be used to accurately predict the corresponding sharpening parameters for any video to ensure that the video saves more bit rate at the same picture quality, or improves the video quality at the same bit rate to facilitate downstream business use.

[0099] See also Figure 6 , Figure 6 A flow chart of a video processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0100] Step S602, obtaining a target video, and determining a target video feature corresponding to the target video; Step S604, inputting the target video features into a target parameter prediction model for processing to obtain target pre-processing parameters, wherein the target parameter prediction model is obtained by training using the above method; Step S606: Process the target video according to the target pre-processing parameters.

[0101] The video processing method provided in this embodiment can be applied to any processing scenario before video encoding, such as live broadcast, on-demand, transcoding and other application scenarios. Before encoding the live video stream, on-demand video or to-be-transcoded video, the target parameter prediction model trained by the above method can be used to predict the pre-processing parameters required before video encoding, so as to optimize and process the video using the pre-processing parameters before performing encoding processing operations. This can save more bitrate at the same image quality, or improve video quality at the same bitrate for subsequent use.

[0102] Based on this, after obtaining the target video, the target video features corresponding to the target video can be determined first. On this basis, the target video features can be input into the target parameter prediction model obtained by training the above method, and then the target pre-processing parameters corresponding to the target video can be obtained, and then the target video can be processed according to the target pre-processing parameters. For example, after optimizing and processing the video, such as improving video quality (color correction, denoising), optimizing coding efficiency (scene recognition, dynamic bit rate allocation), reducing coding complexity (image scaling, frame rate adjustment), etc., the video can be encoded after the optimization and processing are completed, so as to save bit rate under the same video quality, or improve video quality under the same bit rate conditions.

[0103] In specific implementation, the target video may be a video requested by a user in a video-on-demand scenario, such as a movie or TV series. It may also be a live video stream watched by a user in a live broadcast scenario, such as a live game video or a live emotion analysis video. It may also be a video to be edited that a user needs to transcode in a transcoding application, and this embodiment does not make any limitation here.

[0104] For example, in a video on demand scenario, before the video requested by the user is transcoded, the target parameter prediction model can be used to predict the pre-processing parameters of the on-demand video, so that the pre-processing parameters that match the on-demand video can be determined based on the prediction results. At this time, the pre-processing parameters can be used to improve the video quality (color correction, denoising), optimize the encoding efficiency (scene recognition, dynamic bit rate allocation), and reduce the encoding complexity (image scaling, frame rate adjustment) of the on-demand video, and then used for subsequent encoding, thereby sending high-quality on-demand video to the on-demand video client.

[0105] Alternatively, in a live broadcast scenario, after receiving the live video stream submitted by the live broadcast end, the streaming media server needs to encode it according to the needs of the viewing users. Before encoding, the target parameter prediction model can be used to predict the pre-processing parameters of the live video stream, and the pre-processing parameters that match the live video stream can be determined according to the prediction results. At this time, the pre-processing parameters can be used to improve the video quality (color correction, denoising), optimize the encoding efficiency (scene recognition, dynamic bit rate allocation), and reduce the encoding complexity (image scaling, frame rate adjustment) of the live video stream, and then used for subsequent encoding, thereby sending a high-quality live video stream to the audience client.

[0106] Alternatively, in a scenario where a user is editing a video, after the edited video is soft-loaded into a video clip submitted by the user, the target parameter prediction model can be used to predict the pre-processing parameters of the video clip, so that the pre-processing parameters that match the video clip can be determined based on the prediction results. At this time, the pre-processing parameters can be used to improve the video quality (color correction, denoising), optimize the encoding efficiency (scene recognition, dynamic bit rate allocation), reduce the encoding complexity (image scaling, frame rate adjustment), and then used for subsequent encoding, thereby providing users with high-quality videos for editing, so that users can publish and perform other operations based on the edited videos.

[0107] In summary, by using the target parameter prediction model trained by the above method to predict the pre-processing parameters of the video in the business scenario, it is possible to save bit rate under the same video quality, or improve video quality under the same bit rate conditions.

[0108] The above is a schematic scheme of a video processing method of this embodiment. It should be noted that the technical scheme of the video processing method and the technical scheme of the above-mentioned model training method belong to the same concept, and the details not described in detail in the technical scheme of the other model training method can be referred to the description of the technical scheme of the above-mentioned model training method.

[0109] Corresponding to the above method embodiment, this specification also provides a sharpening parameter determination device embodiment, Figure 7 FIG. 2 shows a schematic diagram of a sharpening parameter determination device provided by an embodiment of the present specification. Figure 7 As shown, the device comprises: The video acquisition module 702 is configured to acquire a video to be processed, and perform encoding processing on the video to be processed at least twice according to a preset encoding strategy, and construct an encoded video set according to the encoding processing results; The information determination module 704 is configured to determine the encoding information of each encoded video in the encoded video set, and determine the rate-distortion cost relationship of the encoded video set under the video quality constraint condition according to the encoding information of each encoded video; The parameter determination module 706 is configured to determine the sharpening parameters corresponding to the video to be processed according to the rate-distortion cost relationship.

[0110] In an optional embodiment, the video acquisition module 702 is further configured to: Determine a preset encoding strategy, and determine encoding parameter interval information and step information according to the encoding strategy; generate at least two target encoding parameters based on the encoding parameter interval information and the step information, and use the at least two target encoding parameters to encode the video to be processed respectively; generate an encoded video corresponding to each target encoding parameter according to the encoding processing result, and construct an encoded video set based on the encoded video corresponding to each target encoding parameter.

[0111] In an optional embodiment, determining the encoding information of any one of the encoded videos in the encoded video set includes: Determine the bit rate information and video quality assessment information corresponding to the encoded video, and construct the encoding information of the encoded video based on the bit rate information and the video quality assessment information; wherein, determining the rate-distortion cost relationship of the encoded video set under the video quality constraint condition according to the encoding information of each encoded video includes: determining the sub-rate-distortion cost relationship of each encoded video under the video quality constraint condition according to the bit rate information and video quality assessment information of each encoded video; integrating the sub-rate-distortion cost relationship corresponding to each encoded video to obtain the rate-distortion cost relationship corresponding to the encoded video set.

[0112] In an optional embodiment, the parameter determination module 706 is further configured to: Select multiple candidate sharpening parameters for the video to be processed, and determine the candidate encoding information corresponding to each candidate sharpening parameter; calculate the rate-distortion cost of the video to be processed under each candidate sharpening parameter based on the candidate encoding information corresponding to each candidate sharpening parameter and the rate-distortion cost relationship; compare the rate-distortion cost of each candidate sharpening parameter, and select the sharpening parameter corresponding to the video to be processed from the multiple candidate sharpening parameters according to the comparison result.

[0113] In an optional embodiment, the device further comprises: A model training module is configured to train a parameter prediction model based on the video to be processed and the sharpening parameters to obtain a target parameter prediction model; or, The video processing module is configured to process the video to be processed using the sharpening parameters to obtain a target video.

[0114] In an optional embodiment, the model training module is further configured to: Feature extraction is performed on the video to be processed to obtain video features; the video features are input into a parameter prediction model for processing to obtain predicted sharpening parameters; the parameter prediction model is trained based on the predicted sharpening parameters and the sharpening parameters until a target parameter prediction model that meets the training stop condition is obtained.

[0115] In an optional embodiment, the model training module is further configured to: A loss value is calculated according to the predicted sharpening parameter and the sharpening parameter, and the parameter prediction model is optimized according to the loss value; whether the optimized parameter prediction model satisfies the training stop condition is detected; if not, a candidate video to be processed is obtained, the candidate video to be processed is used as the video to be processed, and the step of encoding the video to be processed at least twice according to a preset encoding strategy is executed; if so, the optimized parameter prediction model is used as the target parameter prediction model.

[0116] In an optional embodiment, the target parameter prediction model includes an input layer, a first hidden layer, a second hidden layer, a third hidden layer and an output layer; wherein the first hidden layer, the second hidden layer and the third hidden layer have different corresponding network parameters, and the output layer is a multi-classification network structure; the target parameter prediction model is used to predict the sharpening parameters of the target video.

[0117] In an optional embodiment, the device further comprises: The verification module is configured to obtain a target verification video and a target verification sharpening parameter; input the target verification video into the target parameter prediction model for processing to obtain a verification sharpening parameter; and compare the verification sharpening parameter with the target verification sharpening parameter; and optimize the target parameter prediction model when it is determined according to the comparison result that the target parameter prediction model does not meet the model deployment conditions.

[0118] In an optional embodiment, the verification module is further configured to: The target verification video is processed using the verification sharpening parameters to obtain a first video, and the target verification video is processed using the target verification sharpening parameters to obtain a second video; the first video and the second video are compared for video information, and when it is determined based on the comparison result that the target parameter prediction model does not meet the model deployment conditions, the step of optimizing the target parameter prediction model is performed.

[0119] The sharpening parameter determination device provided in this embodiment can accurately determine the sharpening parameters of the video to be processed, so as to ensure that the video can have higher quality and consume less resources when being processed. After obtaining the processed video, at least two encoding processes can be performed according to the preset encoding strategy, so as to achieve the construction of an encoded video set composed of encoded videos according to the encoding process results; and the encoded videos contained in the encoded video set are videos obtained by encoding the processed video multiple times. On this basis, the encoding information of each encoded video in the encoded video set can be determined, and then the rate-distortion cost relationship of the encoded video set under the video quality constraint condition can be constructed according to the encoding information of each encoded video; the rate-distortion cost relationship can reflect the intrinsic correlation between the encoding information corresponding to each encoded video and the rate-distortion cost corresponding to the video, and then the optimal solution of the encoding information can be determined according to the relationship, so as to determine the sharpening parameters corresponding to the optimal solution of the encoding information as the sharpening parameters corresponding to the video to be processed; so as to achieve the subsequent use of the sharpening parameters to process the video, it can ensure that the video saves more bit rate under the same picture quality, or improve the video quality under the same bit rate, so as to facilitate the use of downstream services.

[0120] The above is a schematic scheme of a sharpening parameter determination device of this embodiment. It should be noted that the technical scheme of the sharpening parameter determination device and the technical scheme of the sharpening parameter determination method described above belong to the same concept, and the details of the technical scheme of the sharpening parameter determination device that are not described in detail can all be referred to the description of the technical scheme of the sharpening parameter determination method described above.

[0121] Corresponding to the above method embodiment, this specification also provides a model training device embodiment, Figure 8 FIG. 1 shows a schematic diagram of a model training device provided by an embodiment of the present specification. Figure 8 As shown, the device comprises: The video acquisition module 802 is configured to acquire a sample video, perform encoding processing on the sample video at least twice according to a preset encoding strategy, and construct an encoded video set according to the encoding processing results; The information determination module 804 is configured to determine the encoding information of each encoded video in the encoded video set, and determine the rate-distortion cost relationship of the encoded video set under the video quality constraint condition according to the encoding information of each encoded video; A parameter determination module 806 is configured to determine a sample pre-processing parameter corresponding to the sample video according to the rate-distortion cost relationship; The model training module 808 is configured to train the parameter prediction model based on the sample video and the sample pre-processing parameters to obtain a target parameter prediction model.

[0122] In summary, by using the target parameter prediction model obtained through the above training to predict the pre-processing parameters of the video in the business scenario, it is possible to save bit rate under the condition of the same video quality, or improve the video quality under the condition of the same bit rate.

[0123] The above is a schematic scheme of a model training device of this embodiment. It should be noted that the technical scheme of the model training device and the technical scheme of the above-mentioned model training method belong to the same concept, and the details not described in detail in the technical scheme of the model training device can be referred to the description of the technical scheme of the above-mentioned model training method.

[0124] Corresponding to the above method embodiment, this specification also provides a video processing device embodiment, Fig. 9 FIG. 2 shows a schematic diagram of the structure of a video processing device provided by an embodiment of the present specification. Fig. 9 As shown, the device comprises: The video acquisition module 902 is configured to acquire a target video and determine a target video feature corresponding to the target video; The model prediction module 904 is configured to input the target video features into a target parameter prediction model for processing to obtain target pre-processing parameters, wherein the target parameter prediction model is obtained by training using the above method; The video processing module 906 is configured to process the target video according to the target pre-processing parameters.

[0125] The above is a schematic scheme of a video processing device of this embodiment. It should be noted that the technical scheme of this video processing belongs to the same concept as the technical scheme of the above video processing method, and the details not described in detail in the technical scheme of the video processing device can be referred to the description of the technical scheme of the above video processing method.

[0126] Fig.10 The block diagram of a computing device 1000 according to an embodiment of the present specification is shown. The components of the computing device 1000 include but are not limited to a memory 1010 and a processor 1020. The processor 1020 is connected to the memory 1010 via a bus 1030, and the database 1050 is used to store data.

[0127] The computing device 1000 also includes an access device 1040 that enables the computing device 1000 to communicate via one or more networks 1060. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1040 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0128] In one embodiment of the present specification, the above components of the computing device 1000 and Fig.10 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig.10 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0129] The computing device 1000 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 1000 may also be a mobile or stationary server.

[0130] Among them, the processor 1020 is used to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-mentioned model training method or video processing method.

[0131] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned sharpening parameter determination method or model training method or video processing method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above-mentioned sharpening parameter determination method or model training method or video processing method.

[0132] An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, can implement the steps of the above-mentioned sharpening parameter determination method or model training method or video processing method.

[0133] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned sharpening parameter determination method or model training method or video processing method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned sharpening parameter determination method or model training method or video processing method.

[0134] An embodiment of the present specification also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the steps of the above-mentioned sharpening parameter determination method or model training method or video processing method.

[0135] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the above-mentioned sharpening parameter determination method or model training method or video processing method belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be referred to the description of the technical scheme of the above-mentioned sharpening parameter determination method or model training method or video processing method.

[0136] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0137] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0138] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0139] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0140] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can understand and use this specification well.

Claims

1. A method for determining sharpening parameters, characterized in that: include: Obtaining a video to be processed, and performing encoding processing on the video to be processed at least twice according to a preset encoding strategy, and constructing an encoded video set according to the encoding processing results; Determining encoding information of each encoded video in the encoded video set, and determining a rate-distortion cost relationship of the encoded video set under a video quality constraint according to the encoding information of each encoded video; A sharpening parameter corresponding to the video to be processed is determined according to the rate-distortion cost relationship.

2. The method for determining sharpening parameters according to claim 1, characterized in that: The encoding process is performed at least twice on the video to be processed according to a preset encoding strategy, and a set of encoded videos is constructed according to the encoding process results, including: Determine a preset encoding strategy, and determine encoding parameter interval information and step size information according to the encoding strategy; Generate at least two target encoding parameters based on the encoding parameter interval information and the step size information, and use the at least two target encoding parameters to encode the video to be processed respectively; An encoded video corresponding to each target encoding parameter is generated according to the encoding processing result, and an encoded video set is constructed based on the encoded video corresponding to each target encoding parameter.

3. The method for determining sharpening parameters according to claim 1, characterized in that: Determining the encoding information of any one of the encoded videos in the encoded video set includes: Determine bit rate information and video quality assessment information corresponding to the encoded video, and construct encoding information of the encoded video based on the bit rate information and the video quality assessment information; The step of determining the rate-distortion cost relationship of the coded video set under the video quality constraint condition according to the coding information of each coded video includes: Determine a sub-rate distortion cost relationship of each encoded video under a video quality constraint condition according to the bit rate information and the video quality evaluation information of each encoded video; The sub-rate-distortion cost relations corresponding to each coded video are integrated to obtain the rate-distortion cost relation corresponding to the coded video set.

4. The method for determining sharpening parameters according to claim 1, characterized in that: The determining, according to the rate-distortion cost relationship, a sharpening parameter corresponding to the video to be processed includes: Selecting a plurality of candidate sharpening parameters for the video to be processed, and determining candidate encoding information corresponding to each candidate sharpening parameter; Based on the candidate encoding information corresponding to each candidate sharpening parameter and the rate-distortion cost relationship, calculating the rate-distortion cost corresponding to each candidate sharpening parameter of the video to be processed; The rate-distortion cost of each candidate sharpening parameter is compared, and the sharpening parameter corresponding to the to-be-processed video is selected from the plurality of candidate sharpening parameters according to the comparison result.

5. The method for determining sharpening parameters according to claim 1, characterized in that: After the step of determining the sharpening parameter corresponding to the video to be processed according to the rate-distortion cost relationship is performed, the method further includes: Training a parameter prediction model based on the video to be processed and the sharpening parameters to obtain a target parameter prediction model; or, The video to be processed is processed using the sharpening parameters to obtain a target video.

6. The method for determining sharpening parameters according to claim 5, characterized in that: The step of training the parameter prediction model based on the video to be processed and the sharpening parameter to obtain a target parameter prediction model includes: Performing feature extraction on the video to be processed to obtain video features; Inputting the video features into a parameter prediction model for processing to obtain predicted sharpening parameters; The parameter prediction model is trained based on the predicted sharpening parameter and the sharpening parameter until a target parameter prediction model that meets a training stop condition is obtained.

7. The method for determining sharpening parameters according to claim 6, characterized in that: The step of training the parameter prediction model based on the predicted sharpening parameter and the sharpening parameter until a target parameter prediction model that satisfies a training stop condition is obtained includes: Calculating a loss value according to the predicted sharpening parameter and the sharpening parameter, and optimizing the parameter prediction model according to the loss value; Check whether the optimized parameter prediction model meets the training stop condition; If not, obtaining a candidate video to be processed, taking the candidate video to be processed as the video to be processed, and performing the step of encoding the video to be processed at least twice according to a preset encoding strategy; If so, the optimized parameter prediction model is used as the target parameter prediction model.

8. The method for determining sharpening parameters according to claim 6 or 7, characterized in that: The target parameter prediction model comprises an input layer, a first hidden layer, a second hidden layer, a third hidden layer and an output layer; wherein the first hidden layer, the second hidden layer and the third hidden layer have different corresponding network parameters, and the output layer is a multi-classification network structure; the target parameter prediction model is used to predict the sharpening parameters of the target video.

9. The method for determining sharpening parameters according to claim 6 or 7, characterized in that: After the step of training the parameter prediction model based on the predicted sharpening parameter and the sharpening parameter until a target parameter prediction model that satisfies the training stop condition is obtained is executed, the step further includes: Get target verification video and target verification sharpening parameters; Inputting the target verification video into the target parameter prediction model for processing, obtaining verification sharpening parameters, and comparing the verification sharpening parameters with the target verification sharpening parameters; When it is determined according to the comparison result that the target parameter prediction model does not meet the model deployment conditions, the target parameter prediction model is optimized.

10. The method for determining sharpening parameters according to claim 9, characterized in that: The comparing the verification sharpening parameter with the target verification sharpening parameter includes: Processing the target verification video using the verification sharpening parameters to obtain a first video, and processing the target verification video using the target verification sharpening parameters to obtain a second video; The first video and the second video are compared for video information, and when it is determined according to the comparison result that the target parameter prediction model does not meet the model deployment conditions, a step of optimizing the target parameter prediction model is performed.

11. A model training method, characterized in that: include: Obtaining a sample video, and performing encoding processing on the sample video at least twice according to a preset encoding strategy, and constructing an encoded video set according to the encoding processing results; Determining encoding information of each encoded video in the encoded video set, and determining a rate-distortion cost relationship of the encoded video set under a video quality constraint according to the encoding information of each encoded video; Determining a sample pre-processing parameter corresponding to the sample video according to the rate-distortion cost relationship; The parameter prediction model is trained based on the sample video and the sample pre-processing parameters to obtain a target parameter prediction model.

12. A video processing method, characterized in that: include: Acquire a target video, and determine a target video feature corresponding to the target video; Inputting the target video features into a target parameter prediction model for processing to obtain target pre-processing parameters, wherein the target parameter prediction model is trained by the method of claim 11; The target video is processed according to the target pre-processing parameters.

13. A sharpening parameter determination device, characterized in that: include: The video acquisition module is configured to acquire a video to be processed, and perform encoding processing on the video to be processed at least twice according to a preset encoding strategy, and construct an encoded video set according to the encoding processing results; An information determination module is configured to determine the encoding information of each encoded video in the encoded video set, and determine the rate-distortion cost relationship of the encoded video set under the video quality constraint condition according to the encoding information of each encoded video; The parameter determination module is configured to determine the sharpening parameters corresponding to the video to be processed according to the rate-distortion cost relationship.

14. A model training device, characterized in that: include: A video acquisition module is configured to acquire a sample video, perform encoding processing on the sample video at least twice according to a preset encoding strategy, and construct an encoded video set according to the encoding processing results; An information determination module is configured to determine the encoding information of each encoded video in the encoded video set, and determine the rate-distortion cost relationship of the encoded video set under the video quality constraint condition according to the encoding information of each encoded video; A parameter determination module, configured to determine a sample pre-processing parameter corresponding to the sample video according to the rate-distortion cost relationship; The model training module is configured to train the parameter prediction model based on the sample video and the sample pre-processing parameters to obtain a target parameter prediction model.

15. A video processing device, characterized in that: include: A video acquisition module is configured to acquire a target video and determine a target video feature corresponding to the target video; A model prediction module, configured to input the target video features into a target parameter prediction model for processing to obtain target pre-processing parameters, wherein the target parameter prediction model is obtained by training the method according to claim 11; The video processing module is configured to process the target video according to the target pre-processing parameters.

16. A computing device, characterized in that include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method described in any one of claims 1 to 12 are implemented.

17. A computer-readable storage medium, characterized in that: It stores computer executable instructions, which, when executed by a processor, implement the steps of the method described in any one of claims 1 to 12.

18. A computer program product, characterized in that The method comprises a computer program or an instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 12.

Images