Preprocessor training method and device, electronic equipment and storage medium
By introducing a guideable preprocessor and target encoding model in video encoding, a guideable link is built, which solves the problem of collaborative work caused by the unresponsiveness of traditional encoders, and improves high-quality video compression and encoding performance.
Patent Information
- Application Number
- CN202411309916.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-19
- Publication Date
- 2024-12-31
AI Technical Summary
The unconductivity of traditional encoders makes it impossible to work effectively with the preprocessor, making it difficult to achieve the compression effect of high-quality video, and upgrading traditional encoders requires a lot of work and time.
By introducing guideable preprocessors and target encoding models, guideable links are built, and the preprocessors are optimized to adapt to traditional encoders and improve encoding performance.
It achieves better video compression effects and more efficient encoding performance, meets diverse encoding needs and avoids the need for updates to the decoder.
Smart Images
Figure CN119232948A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of video coding and decoding technology, and in particular to a preprocessor training method, device, electronic device and storage medium. Background Art
[0002] Digital video technology has been widely used in various devices, including digital TVs, smart phones, personal computers, e-readers, and professional video players. With the continuous evolution of video technology, the amount of data generated has increased dramatically, which has put forward higher requirements for data transmission and storage. To meet this challenge, the commonly adopted strategy is to compress the video, which is usually performed by the encoder to improve the transmission efficiency of video data and the economic efficiency of storage.
[0003] Although traditional encoder technology is relatively mature and can efficiently compress videos to a considerable extent to meet basic transmission and storage requirements, it is still difficult to meet the compression requirements of high-quality videos. One solution to this problem is to use a preprocessor with a traditional encoder. However, if the existing preprocessor is directly used with a traditional encoder, the final encoding effect is not ideal due to the poor synergy between the two. Therefore, it is necessary to improve the adaptability of the two. Given the non-differentiable characteristics of traditional encoders, it is impossible to directly use the optimization ability of neural network learning to improve the adaptability of the preprocessor and the traditional encoder. Summary of the invention
[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a preprocessor training method, device, electronic device and storage medium.
[0005] In a first aspect, the present disclosure provides a preprocessor training method, wherein the preprocessor is used to be deployed before a target encoder to eliminate interference factors in a video to be encoded; the method comprises:
[0006] Get sample video;
[0007] Inputting the sample video into the preprocessor to obtain a preprocessing result;
[0008] Inputting the preprocessing result into a target coding model to obtain a first coding result; inputting the preprocessing result into a target encoder to obtain a second coding result;
[0009] Adjusting parameters in the preprocessor based on the first encoding result and the second encoding result;
[0010] Wherein, both the preprocessor and the target coding model are derivative.
[0011] In a second aspect, the present disclosure further provides a video encoding method, comprising:
[0012] Get the video to be encoded;
[0013] Inputting the video to be encoded into the trained preprocessor to obtain a preprocessing result; the preprocessor is trained according to the preprocessor training method;
[0014] The preprocessing result is input into a target encoder to obtain a target encoding result.
[0015] In a third aspect, the present disclosure further provides a preprocessor training device, wherein the preprocessor is used to be deployed before a target encoder to eliminate interference factors in a video to be encoded; the device comprises:
[0016] A first acquisition module, used to acquire a sample video;
[0017] A first preprocessing module, used for inputting the sample video into the preprocessor to obtain a preprocessing result;
[0018] A first encoding module, used for inputting the preprocessing result into a target encoding model to obtain a first encoding result; and inputting the preprocessing result into a target encoder to obtain a second encoding result;
[0019] An adjustment module, configured to adjust parameters in the preprocessor based on the first encoding result and the second encoding result;
[0020] Wherein, both the preprocessor and the target coding model are derivative.
[0021] In a fourth aspect, the present disclosure further provides a video encoding device, including:
[0022] A second data acquisition module, used to acquire the video to be encoded;
[0023] A second preprocessing module is used to input the video to be encoded into the trained preprocessor to obtain a preprocessing result; the preprocessor is trained according to the preprocessor training method as described above;
[0024] The second encoding module is used to input the preprocessing result into a target encoder to obtain a target encoding result.
[0025] In a fifth aspect, the present disclosure further provides an electronic device, the electronic device comprising:
[0026] one or more processors;
[0027] A storage device for storing one or more programs;
[0028] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described above.
[0029] In a sixth aspect, the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which implements the method described above when executed by a processor.
[0030] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:
[0031] The technical solution provided by the embodiment of the present disclosure is configured to input a sample video into a preprocessor to obtain a preprocessing result; input the preprocessing result into a target coding model to obtain a first coding result; input the preprocessing result into a target encoder to obtain a second coding result; based on the first coding result and the second coding result, adjust the parameters in the preprocessor; wherein both the preprocessor and the target coding model are differentiable. The core of the technical solution provided by the embodiment of the present disclosure is to use a differentiable target coding model to replace a non-differentiable target encoder (i.e., a traditional encoder) to construct a complete differentiable link. This differentiable link is composed of a preprocessor and a target coding model, and a preprocessor adapted to the target encoder can be trained using the learning optimization capability based on back propagation. The preprocessor can be deployed before the target encoder to eliminate interference factors in the video to be encoded to improve the encoding performance of the encoder, thereby achieving a better quality video compression effect and meeting more diverse encoding requirements. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0034] Figure 1 A flowchart of a preprocessor training method provided in an embodiment of the present disclosure;
[0035] Figure 2 A schematic diagram of a preprocessor training method provided in an embodiment of the present disclosure;
[0036] Figure 3 A flowchart of a video encoding method provided by an embodiment of the present disclosure;
[0037] Figure 4A schematic diagram of a video encoding method provided by an embodiment of the present disclosure;
[0038] Figure 5 It is a structural schematic diagram of a preprocessor training device in an embodiment of the present disclosure;
[0039] Figure 6 is a structural schematic diagram of a video encoding device in an embodiment of the present disclosure;
[0040] Figure 7 It is a structural schematic diagram of an electronic device in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0041] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.
[0042] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.
[0043] As described in the background technology, although the traditional encoder technology is relatively mature and can efficiently compress videos to a considerable extent to meet the basic transmission and storage requirements, it is still difficult to meet the compression requirements of high-quality videos. In addition, traditional encoders are usually used in conjunction with decoders. If a traditional encoder is modified, the corresponding decoder needs to be modified. Therefore, upgrading a traditional encoder requires a large workload and takes a long time.
[0044] In addition, the traditional encoder is not differentiable. Because it is not differentiable, it is impossible to use the neural network training method to train patches for it to achieve the purpose of upgrading the encoder. If the patch is not trained, the preprocessor with improved video quality is directly used with the traditional encoder. Due to the poor synergy between the two, the final encoding effect is usually not ideal.
[0045] To address this issue, the present disclosure provides a preprocessor training method. Figure 1A flowchart of a preprocessor training method provided in an embodiment of the present disclosure, this embodiment can be applied to the case where a preprocessor is made for a target encoder in a client for use with it, the method can be executed by a preprocessor training device, the device can be implemented in software and / or hardware, the device can be configured in an electronic device, such as a terminal, specifically including but not limited to smart phones, PDAs, tablet computers, wearable devices with display screens, desktop computers, laptop computers, all-in-one machines, smart home devices, etc. Alternatively, this embodiment can be applied to the case where a preprocessor is made for a target encoder in a server for use with it, the method can be executed by a preprocessor training device, the device can be implemented in software and / or hardware, the device can be configured in an electronic device, such as a server.
[0046] In the present application, the target encoder, that is, the traditional encoder, can be, for example, an existing mature encoder that does not have learning capability and is not differentiable. The present application does not limit how the target encoder is specifically encoded.
[0047] One reason why the target encoder does not have the ability to learn is that it is not differentiable. Non-differentiability is a property of the target encoder itself. The target encoder is not differentiable because it contains some non-continuous or non-linear operations. These operations are mathematically non-differentiable. For example, the target encoder may include mapping continuous values to a finite number of discrete values, which is a non-continuous operation and therefore non-differentiable. For another example, the target encoder may involve a quantization process, such as continuous values being truncated to the nearest integer value, and this truncation process is not continuous and therefore non-differentiable. The target encoder may also involve non-linear transformations, such as thresholding, integer operations, etc. In the thresholding process, all values less than a certain threshold are set to zero, which is a non-continuous process and is non-differentiable. For another example, a traditional encoder may also interact directly with hardware, such as memory access instructions or hardware acceleration instructions. These interactive operations are usually not continuous mathematical functions, which are mathematically non-differentiable. For another example, the encoder may rely on external data, such as reference frames in motion estimation. These dependencies may involve complex search and matching processes, which are also non-continuous and therefore non-differentiable.
[0048] The preprocessor may be, for example, a model used to be deployed before the target encoder and used before the target encoder, and the model is differentiable and has learning capabilities. Exemplarily, the preprocessor belongs to a neural network model.
[0049] Taking the neural network model as an example, differentiability is a property of the neural network model itself. The reason why the neural network is differentiable is that it is composed of a series of continuously differentiable mathematical functions, including linear combinations, activation functions, and loss functions. Because the neural network is differentiable, the gradient of the loss function with respect to each layer of weights can be calculated by applying the chain rule, so that the neural network can be learned and optimized end-to-end through optimization algorithms such as gradient descent. In other words, it is because the neural network is differentiable that it has the ability to learn.
[0050] In some scenarios, the preprocessor can be regarded as a patch for the target encoder. The preprocessor is used to eliminate interference factors in the video to be encoded and assist the target encoder to complete the encoding of the video. Compared with the case where only the target encoder is used without the preprocessor, the use of the preprocessor and the target encoder at the same time will make the obtained encoding effect better. The combination of the preprocessor and the target encoder can be understood as the result of optimizing the target encoder. After optimization, the encoding performance is improved and the functions are more complete.
[0051] The present application does not limit the specific functions of preprocessing. Exemplarily, the preprocessor is used to remove noise in the video to be encoded, adjust the resolution of the image, color space conversion, picture cropping and scaling, texture analysis and enhancement, filtering and smoothing, etc.
[0052] The color space conversion may, for example, convert an image in a video to be encoded from one color space (such as RGB) to another (such as YUV).
[0053] Texture analysis and enhancement, for example, can be performed by analyzing the image texture in the video to be encoded to enhance the image details.
[0054] Filtering and smoothing, for example, may be the application of various filters, such as Gaussian filtering, median filtering, etc., to smooth irregular pixels in the image in the video to be encoded, reduce aliasing, and improve the visual coherence of the encoded content.
[0055] like Figure 1 As shown, the method may specifically include:
[0056] S110: Obtain sample video.
[0057] The sample video may be, for example, training data required to be used in the process of training the preprocessor.
[0058] S120: Input the sample video into a preprocessor to obtain a preprocessing result.
[0059] The present application does not limit the specific neural network model used as the preprocessor. In practice, when selecting the model used as the preprocessor, it can be determined based on the specific type of sample video and the type of interference factors that are desired to be eliminated.
[0060] It should be noted that before the technical solution provided in this application is implemented, the preprocessor already has certain functions (such as removing noise in the video to be encoded, adjusting the resolution of the image, color space conversion, image cropping and scaling, texture analysis and enhancement or filtering and smoothing, etc.).
[0061] S130, input the preprocessing result into a target coding model to obtain a first coding result; input the preprocessing result into a target encoder to obtain a second coding result.
[0062] The target coding model may be, for example, a model that has the function of encoding a video. The present application does not limit the specific model of the target coding model. However, the target coding model is required to be derivable. Exemplarily, the target coding model is a neural network model.
[0063] In this application, the object of training is the preprocessor, not the target coding model. Therefore, during the whole process of preprocessor training, the parameters in the target coding model are fixed.
[0064] In practice, when selecting which model with encoding function to use as the target encoding model, it can be determined according to the characteristics of the target encoder. The characteristics of the target encoder may include, for example, the encoding effect of the target encoder.
[0065] S140: Adjust parameters in a preprocessor based on the first encoding result and the second encoding result.
[0066] The essence of this step is to calculate the loss function based on the first encoding result and the second encoding result, and then calculate the gradient of the loss function with respect to the parameters in the preprocessor by applying the chain rule, and adjust the parameters in the preprocessor.
[0067] The core idea of the technical solution provided by the present disclosure is: using the method of training the model, a patch is made for the target encoder (i.e., the traditional encoder), and this patch is the preprocessor. Before training, this patch as a preprocessor is itself a model with a certain interference elimination function. The purpose of training is not to make this patch have the ability to eliminate interference, but to make this patch compatible with the target encoder. If it is not compatible, directly using the patch and the target encoder together may result in the undesirable phenomenon that the image quality after encoding is reduced and the bit rate is increased.
[0068] It should be noted that if the target coding model is not introduced, the training model method is directly used to make patches for the target encoder (i.e., the traditional encoder). According to the normal training model steps, the sample multimedia data is input into the preprocessor to obtain the preprocessing result; the preprocessing result is input into the target encoder to obtain the encoding result, and the loss function is obtained based on the encoding result and the sample multimedia data. Next, the loss function needs to be reversed to the parameters in the preprocessor to obtain the gradient of each parameter. This gradient indicates how the parameters should be modified. However, since the target encoder is not differentiable, it is impossible to reverse the gradient of the parameters from the loss function, and it is also unknown how to adjust the parameters. Therefore, if the target coding model is not introduced, it is impossible to use the training model method to make patches for the target encoder (i.e., the traditional encoder).
[0069] The target encoding model is introduced in this scheme, which is equivalent to replacing the target encoder (i.e., the traditional encoder) with the target encoding model, and establishes a path for inferring the loss function back to the gradient of the parameters.
[0070] Therefore, the technical solution provided by the embodiment of the present disclosure is to input the sample video into a preprocessor to obtain a preprocessing result; input the preprocessing result into a target coding model to obtain a first coding result; input the preprocessing result into a target encoder to obtain a second coding result; based on the first coding result and the second coding result, adjust the parameters in the preprocessor; wherein both the preprocessor and the target coding model are differentiable. Its essence is to use a differentiable target coding model to replace a non-differentiable target encoder (i.e., a traditional encoder) to construct a complete differentiable link. This differentiable link is composed of a preprocessor and a target coding model, and a preprocessor adapted to the target encoder can be trained using the learning optimization capability based on back propagation. The preprocessor can be deployed before the target encoder to eliminate interference factors in the video to be encoded to improve the encoding performance of the encoder, thereby achieving better video compression effects and meeting more diverse encoding requirements.
[0071] It should also be noted that the above technical solution does not adjust the target encoder (i.e., the traditional encoder). This makes the decoder originally adapted to the target encoder still adapted to the whole composed of the preprocessor and the target encoder. In other words, by adopting the above technical solution, there is no need to update the decoder corresponding to the target encoder (i.e., the traditional encoder) later.
[0072] In the above technical solution, there are many methods for implementing S140, which are not limited in this application. Figure 2The implementation method of S140 includes: decoding the first encoding result to obtain a first video; obtaining a first image quality loss function based on the difference between the image quality of the first video and the image quality of the sample video; decoding the second encoding result to obtain a second video; obtaining a second image quality loss function based on the difference between the image quality of the second video and the image quality of the sample video; adjusting the parameters in the preprocessor based on the first image quality loss function and the second image quality loss function.
[0073] Image quality is used to characterize the visual effects and information integrity of an image. Specifically, image quality may include at least one of the following aspects: visual clarity, color accuracy, information integrity, and dynamic range. Among them, visual clarity refers to, for example, whether the edges of people or objects in the image are sharp, whether the texture can be clearly distinguished, etc. Color accuracy refers to, for example, the color performance of the image, such as color saturation, hue, etc. Information integrity refers to, for example, whether some scene details are missing in the image. Dynamic range refers to, for example, the image's performance at different brightness levels.
[0074] The first image quality loss function and the second image quality loss function may be characterized by a peak signal-to-noise ratio (PSNR) or a structural similarity (SSIM).
[0075] "Based on the first image quality loss function and the second image quality loss function, adjusting the parameters in the preprocessor" is essentially taking into account not only the coding characteristics of the target coding model but also the coding characteristics of the target encoder when calculating the gradient of the loss function with respect to the parameters in the preprocessor. That is, the coding characteristics of the target encoder are injected into the preprocessor, which can make the preprocessor and the target encoder more adaptable.
[0076] Furthermore, before S140, the method may further include: obtaining a rate loss function based on the difference between the bit rate of the first encoding result and the bit rate of the sample video. S140 includes: adjusting parameters in the preprocessor based on the first image quality loss function, the second image quality loss function and the rate loss function.
[0077] The purpose of adjusting the parameters in the neural network based on the loss function is to gradually minimize the loss function. Those skilled in the art will understand that in actual situations, over-emphasizing the minimization of the image quality loss function may lead to a surge in bit rate (reduced data compression efficiency), thereby causing adverse consequences. By setting the parameters in the preprocessor based on the first image loss, the second image loss and the bit rate loss, the purpose is to seek the ideal balance between image quality and compression bit rate, ensuring that the optimization process not only improves the visual performance, but also does not consume excessive bandwidth resources.
[0078] Further, S140 may include: taking the sum of the first image quality loss function and the second image quality loss function as the third image quality loss function; back-propagating the third image quality loss function and the bit rate loss function to obtain gradient values corresponding to the parameters in the preprocessor; and adjusting the parameters in the preprocessor based on the gradient values. This setting enables the preprocessor to learn how to compress data more efficiently while maintaining image quality by back-propagating the third image quality loss function and the bit rate loss function together.
[0079] Based on the above technical solution, optionally, before S130, the method may also include: obtaining multiple candidate coding models; determining the similarity in coding effects between the candidate coding models and the target encoder; and determining the target coding model from multiple candidate coding models based on the similarity in coding effects between the candidate coding models and the target encoder.
[0080] The candidate coding model is a pre-specified coding model, and the candidate coding model has the same type of video that can be encoded as the target encoder, but the encoding effect is different. Here, the encoding effect may include, for example, at least one of the following: video quality (such as clarity, color fidelity, detail retention, etc.), the size of the file obtained after encoding, the overall encoding time, the compatibility of the file obtained after encoding, the encoder's dependence on hardware performance, etc.
[0081] There are many specific implementation methods for "determining the similarity between the candidate coding model and the target encoder in coding effect". Exemplarily, in one embodiment, one or more evaluation items can be determined, and corresponding evaluation criteria can be determined for each evaluation item. Evaluation items can be, for example, video quality (such as clarity, color fidelity, detail retention, etc.), file size obtained after encoding, overall encoding time consumption, compatibility of files obtained after encoding, and the degree of dependence of the encoder on hardware performance. The test video is input into each candidate coding model and the target encoder respectively, and the encoding results corresponding to the candidate coding model and the encoding results corresponding to the target encoder are obtained respectively. Using the above evaluation criteria, the encoding results corresponding to the candidate coding model are evaluated to obtain the evaluation results of the candidate coding model. Using the above evaluation criteria, the encoding results corresponding to the target encoder are evaluated to obtain the evaluation results of the target encoder. Based on the evaluation results of the candidate coding model and the evaluation results of the target encoder, the similarity between the candidate coding model and the target encoder in coding effect is determined.
[0082] In another embodiment, optionally, determining the similarity between the candidate coding model and the target encoder in coding effect includes: determining the similarity value between the rate-distortion curve of each candidate coding model and the rate-distortion curve of the target encoder. Based on the similarity between the candidate coding model and the target encoder in coding effect, determining the target coding model from multiple candidate coding models includes: determining the candidate coding model with a similarity value greater than a set threshold as the target coding model.
[0083] The rate-distortion curve, also known as the rate-distortion function curve, is a curve used to measure the difference between a video after encoding and the original video. It reflects the relationship between the bit rate and the distortion.
[0084] The threshold is a pre-specified value used to measure whether a candidate coding model can be used as a target coding model.
[0085] The similarity value between the rate-distortion curve of each candidate coding model and the rate-distortion curve of the target encoder is determined by setting; the candidate coding model with a similarity value greater than the set threshold is determined as the target coding model, which means that during the preprocessor training process, the encoding effect of the target coding model and the target encoder it replaces is very similar, which is conducive to improving the collaborative working efficiency of the preprocessor and the target encoder, ensuring that the trained preprocessor can cooperate with the target encoder more efficiently, thereby significantly improving the encoding quality and / or efficiency of the video.
[0086] In yet another embodiment, based on the above technical solution, optionally, before S120, the method further includes: acquiring parameter values of configuration items in the target encoder; and inputting the parameter values of the configuration items into a preprocessor.
[0087] The configuration items in the target encoder may be, for example, options set to meet the encoding requirements in specific application scenarios (such as online streaming, video storage, real-time communication, etc.). By configuring these configuration items, one or more of the encoding efficiency, video quality, and encoding speed of the target encoder may be adjusted. In some scenarios, the configuration items in the target encoder may be open to the user, that is, the user may be allowed to configure according to his or her own needs.
[0088] The parameter value of the configuration item in the target encoder may include, for example, a specific value used to configure the configuration item in the target encoder.
[0089] Different parameter values of the configuration items in the target encoder mean different encoding performance of the target encoder. Different parameter values reflect the encoding characteristics of the target encoder under different encoding performance. By setting the parameter values of the configuration items to be input into the preprocessor, in essence, in the process of training the preprocessor, the parameter values of the configuration items are used to affect the preprocessing results of the sample video, affect the first image quality loss function, and then affect the parameter adjustment results in the preprocessor. It is equivalent to injecting the encoding performance characteristics of the target encoder into the preprocessor, which can make the preprocessor and the target encoder more compatible and more conducive to the collaborative work of the two.
[0090] Figure 3 A flowchart of a video encoding method provided by an embodiment of the present disclosure. Figure 4 A schematic diagram of a video encoding method provided by an embodiment of the present disclosure. Figure 3 and Figure 4 , the video encoding method comprises:
[0091] S210: Obtain a video to be encoded.
[0092] The video to be encoded may be, for example, a video that needs to be encoded in the preprocessor use phase (ie, the inference phase after training is completed).
[0093] S220, inputting the video to be encoded into a trained preprocessor to obtain a preprocessing result; the preprocessor is trained according to the preprocessor training method mentioned in the present application.
[0094] S230, inputting the preprocessing result into a target encoder to obtain a target encoding result.
[0095] The target encoding result may be, for example, an encoded video corresponding to the video to be encoded.
[0096] Since the above technical solution uses the preprocessor trained by the preprocessor training method mentioned in the present application, it has the same or corresponding beneficial effects as the preprocessor training method, which will not be repeated here.
[0097] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.
[0098] Figure 5 Schematic diagram of the structure of a preprocessor training device in an embodiment of the present disclosure. The preprocessor training device provided in the embodiment of the present disclosure can be configured in a client, or can be configured in a server. The preprocessor is used to encode the video in cooperation with the target encoder. Figure 5 , the preprocessor training device specifically comprises:
[0099] A first acquisition module 310, used to acquire a sample video;
[0100] A first preprocessing module 320, configured to input the sample video into the preprocessor to obtain a preprocessing result;
[0101] The first encoding module 330 is used to input the preprocessing result into a target encoding model to obtain a first encoding result; and input the preprocessing result into a target encoder to obtain a second encoding result;
[0102] An adjustment module 340, configured to adjust parameters in the preprocessor based on the first encoding result and the second encoding result;
[0103] Wherein, both the preprocessor and the target coding model are derivative.
[0104] Furthermore, the adjustment module is used to:
[0105] Decoding the first encoding result to obtain a first video;
[0106] Based on the difference between the image quality of the first video and the image quality of the sample video, a first image quality loss function is obtained; the image quality is information used to characterize the visual effect and / or information integrity of the image;
[0107] Decoding the second encoding result to obtain a second video;
[0108] Based on the difference between the image quality of the second video and the image quality of the sample video, obtaining a second image quality loss function;
[0109] Based on the first image quality loss function and the second image quality loss function, parameters in the preprocessor are adjusted.
[0110] Furthermore, the first encoding module is further used to obtain a bit rate loss function based on the difference between the bit rate of the first encoding result and the bit rate of the sample video;
[0111] Adjustment modules for:
[0112] Based on the first image quality loss function, the second image quality loss function and the bit rate loss function, parameters in the preprocessor are adjusted.
[0113] Furthermore, the adjustment module is used to:
[0114] Obtaining a third image loss based on the first image loss and the second image loss;
[0115] Back-propagating the third image loss and the bit rate loss to obtain gradient values corresponding to the parameters in the preprocessor;
[0116] Parameters in the preprocessor are adjusted based on the gradient value.
[0117] Furthermore, the device also includes an input module, which is used to:
[0118] Inputting the sample video into the preprocessor, and obtaining a parameter value of a configuration item in the target encoder before obtaining a preprocessing result;
[0119] The parameter value of the configuration item is input into the preprocessor.
[0120] Furthermore, the device also includes a selection module, which is used to:
[0121] Inputting the preprocessing result into a target coding model to obtain a plurality of candidate coding models before obtaining a first coding result;
[0122] Determining a similarity value between a rate-distortion curve of each of the candidate coding models and a rate-distortion curve of the target encoder;
[0123] The candidate coding model whose similarity value is greater than a set threshold is determined as the target coding model.
[0124] The preprocessor training device provided in the embodiment of the present disclosure can execute the steps executed by the client or server in the preprocessor training method provided in the embodiment of the method of the present disclosure, and has execution steps and beneficial effects, which will not be repeated here.
[0125] Figure 6Schematic diagram of the structure of a video encoding device in an embodiment of the present disclosure. The video encoding device provided in the embodiment of the present disclosure can be configured in a client or in a server. Figure 6 , the video encoding device specifically comprises:
[0126] The second data acquisition module 410 is used to acquire the video to be encoded;
[0127] The second preprocessing module 420 is used to input the video to be encoded into the trained preprocessor to obtain a preprocessing result; the preprocessor is trained according to the preprocessor training method;
[0128] The second encoding module 430 is used to input the preprocessing result into a target encoder to obtain a target encoding result.
[0129] The video encoding device provided in the embodiment of the present disclosure can execute the steps executed by the client or server in the video encoding device provided in the embodiment of the method of the present disclosure, and has the execution steps and beneficial effects, which will not be repeated here.
[0130] Figure 7 Schematic diagram of the structure of an electronic device in the embodiment of the present disclosure. Figure 7 , which shows a schematic diagram of the structure of an electronic device 1000 suitable for implementing the embodiment of the present disclosure. The electronic device 1000 in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), wearable electronic devices, etc., and fixed terminals such as digital TVs, desktop computers, smart home devices, etc. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0131] like Figure 7 As shown, the electronic device 1000 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1008 to a random access memory (RAM) 1003 to implement the preprocessor training or video encoding method of the embodiment described in the present disclosure. In the RAM 1003, various programs and information required for the operation of the electronic device 1000 are also stored. The processing device 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.
[0132] Typically, the following devices may be connected to the I / O interface 1005: an input device 1006 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 1007 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1008 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 may allow the electronic device 1000 to communicate with other devices wirelessly or by wire to exchange information. Although Figure 7 The electronic device 1000 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.
[0133] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart, thereby implementing the preprocessor training or video encoding method as described above. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 1009, or installed from the storage device 1008, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.
[0134] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include an information signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated information signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0135] In some embodiments, the client and the server may communicate using any known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital information communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any known or future developed network.
[0136] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.
[0137] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:
[0138] Get sample video;
[0139] Inputting the sample video into the preprocessor to obtain a preprocessing result;
[0140] Inputting the preprocessing result into a coding model to obtain a first coding result; inputting the preprocessing result into a target encoder to obtain a second coding result;
[0141] Adjusting parameters in the preprocessor based on the first encoding result and the second encoding result;
[0142] Wherein, both the preprocessor and the target coding model are derivative.
[0143] Alternatively, the computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device:
[0144] Get the video to be encoded;
[0145] Inputting the video to be encoded into the trained preprocessor to obtain a preprocessing result; the preprocessor is trained according to the preprocessor training method as described above;
[0146] The preprocessing result is input into a target encoder to obtain a target encoding result.
[0147] Optionally, when the above one or more programs are executed by the electronic device, the electronic device may also execute other steps described in the above embodiments.
[0148] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).
[0149] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0150] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.
[0151] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.
[0152] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0153] According to one or more embodiments of the present disclosure, the present disclosure provides an electronic device, including:
[0154] one or more processors;
[0155] A memory for storing one or more programs;
[0156] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the pre-processor training or video encoding methods provided in the present disclosure.
[0157] According to one or more embodiments of the present disclosure, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the preprocessor training or video encoding methods provided by the present disclosure.
[0158] The embodiments of the present disclosure also provide a computer program product, which includes a computer program or instructions, and when the computer program or instructions are executed by a processor, the preprocessor training or video encoding method as described above is implemented.
[0159] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0160] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A preprocessor training method, characterized in that: The preprocessor is used to be deployed before the target encoder to eliminate interference factors in the video to be encoded; the method includes: Get sample video; Inputting the sample video into the preprocessor to obtain a preprocessing result; Inputting the preprocessing result into a target coding model to obtain a first coding result; inputting the preprocessing result into a target encoder to obtain a second coding result; Adjusting parameters in the preprocessor based on the first encoding result and the second encoding result; Wherein, both the preprocessor and the target coding model are derivative.
2. The method according to claim 1, characterized in that The adjusting the parameters in the preprocessor based on the first encoding result and the second encoding result includes: Decoding the first encoding result to obtain a first video; Based on the difference between the image quality of the first video and the image quality of the sample video, a first image quality loss function is obtained; the image quality is information used to characterize the visual effect and / or information integrity of the image; Decoding the second encoding result to obtain a second video; Based on the difference between the image quality of the second video and the image quality of the sample video, obtaining a second image quality loss function; Based on the first image quality loss function and the second image quality loss function, parameters in the preprocessor are adjusted.
3. The method according to claim 2, characterized in that Also includes: Obtaining a bit rate loss function based on a difference between a bit rate of the first encoding result and a bit rate of the sample video; The adjusting the parameters in the preprocessor based on the first image quality loss function and the second image quality loss function includes: Based on the first image quality loss function, the second image quality loss function and the bit rate loss function, parameters in the preprocessor are adjusted.
4. The method according to claim 1, characterized in that: Before inputting the sample video into the preprocessor to obtain the preprocessing result, the method includes: Obtaining parameter values of configuration items in the target encoder; The parameter value of the configuration item is input into the preprocessor.
5. The method according to claim 1, characterized in that Before inputting the preprocessing result into the target coding model to obtain the first coding result, the method further includes: Obtain multiple candidate encoding models; Determining a similarity value between a rate-distortion curve of each of the candidate coding models and a rate-distortion curve of the target encoder; The candidate coding model whose similarity value is greater than a set threshold is determined as the target coding model.
6. A video encoding method, characterized in that: include: Get the video to be encoded; Inputting the video to be encoded into the trained preprocessor to obtain a preprocessing result; The preprocessor is trained by the preprocessor training method according to any one of claims 1 to 5; The preprocessing result is input into a target encoder to obtain a target encoding result.
7. A preprocessor training device, characterized in that: The preprocessor is used to be deployed before the target encoder to eliminate interference factors in the video to be encoded; the device includes: A first acquisition module, used to acquire a sample video; A first preprocessing module, used for inputting the sample video into the preprocessor to obtain a preprocessing result; A first encoding module, used for inputting the preprocessing result into a target encoding model to obtain a first encoding result; and inputting the preprocessing result into a target encoder to obtain a second encoding result; An adjustment module, configured to adjust parameters in the preprocessor based on the first encoding result and the second encoding result; Wherein, both the preprocessor and the target coding model are derivative.
8. A video encoding device, characterized in that: include: A second data acquisition module, used to acquire the video to be encoded; A second preprocessing module, used for inputting the video to be encoded into the trained preprocessor to obtain a preprocessing result; The preprocessor is trained by the preprocessor training method according to any one of claims 1 to 5; The second encoding module is used to input the preprocessing result into a target encoder to obtain a target encoding result.
9. An electronic device, characterized in that: include: Processor and memory; The processor is used to execute the steps of the method according to any one of claims 1 to 6 by calling the program or instruction stored in the memory.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and the program or instruction enables a computer to execute the steps of the method according to any one of claims 1 to 6.