Image adjustment model training method and device, video coding method and device, electronic equipment, storage medium and program product

Abstract

The invention provides an image adjustment model training method and device, a video coding method and device, electronic equipment, a storage medium and a program product. The training method comprises the following steps: acquiring a first training picture set and a second training picture set; pre-training the initial image adjustment model by using a first loss function based on the first training picture set to obtain a pre-trained image adjustment model, the first loss function being composed of fidelity loss; based on the second training picture set, utilizing a second loss function to finely adjust the pre-training image adjustment model to obtain a final image adjustment model, the second loss function being composed of code rate loss and at least one preset loss, the code rate for the code rate loss is obtained by encoding the picture adjusted by the pre-training image adjustment model. According to the method, the training difficulty is reduced, and the output image quality of the model is guaranteed.

Description

Technical Field

[0001] The present disclosure relates to video coding and compression. More specifically, the present disclosure relates to a training method for an image adjustment model, a video coding method, an apparatus, an electronic device, a storage medium and a program product. Background Art

[0002] As the number of user experiences in the short video industry continues to increase, the bandwidth cost of videos has become one of the important expenses that video platforms cannot ignore. With the continuous iteration of codec algorithms, the compression rate of videos is slowly approaching its limit, and exploring "compression" algorithms based on image content has become another direction for cost saving.

[0003] There is a solution in the related art, which is to fine-tune the image content to make it easier for the encoder to compress it, and reduce the bit rate after encoding without losing image quality, thereby saving bandwidth costs. However, in this solution, easy image compression and no loss of image quality are two contradictory goals, which makes the training process difficult and difficult to achieve the global optimum. Summary of the invention

[0004] Embodiments of the present disclosure provide a method for training an image adjustment model, a video encoding method, an apparatus, an electronic device, a storage medium, and a program product, for solving at least one of the above problems.

[0005] According to one aspect of the present disclosure, a training method for an image adjustment model is provided, comprising: obtaining a first training picture set and a second training picture set; based on the first training picture set, using a first loss function, pre-training an initial image adjustment model to obtain a pre-trained image adjustment model, wherein the first loss function is composed of a fidelity loss; based on the second training picture set, using a second loss function, fine-tuning the pre-trained image adjustment model to obtain a final image adjustment model, wherein the second loss function is composed of a bit rate loss and at least one preset loss, and the bit rate loss targets a bit rate obtained by encoding a picture adjusted by the pre-trained image adjustment model.

[0006] Optionally, based on the first training picture set, the initial image adjustment model is pre-trained using a first loss function to obtain a pre-trained image adjustment model, including: using the initial image adjustment model to adjust the first training picture in the first training picture set to obtain a first adjusted picture; using an analog encoder to encode the first adjusted picture to obtain a first reconstructed picture; determining the value of the fidelity loss based on the first training picture, the first adjusted picture and the first reconstructed picture to obtain the value of the first loss function; based on the value of the first loss function, adjusting the parameters of the initial image adjustment model to obtain the pre-trained image adjustment model.

[0007] Optionally, based on the second training picture set, the pre-trained image adjustment model is fine-tuned using a second loss function to obtain a final image adjustment model, including: using the pre-trained image adjustment model to adjust the second training picture in the second training picture set to obtain a second adjusted picture; using an analog encoder to encode the second adjusted picture to obtain a second reconstructed picture and a bit rate; determining a value of the bit rate loss and a value of at least one preset loss based on the second training picture, the second adjusted picture, the second reconstructed picture and the bit rate to obtain a value of the second loss function; based on the value of the second loss function, adjusting the parameters of the pre-trained image adjustment model to obtain the image adjustment model.

[0008] Optionally, the at least one preset loss includes at least one of the following: perceptual loss, image quality loss, and the fidelity loss.

[0009] Optionally, the learning rate during the pre-training phase is at least an order of magnitude larger than the learning rate during the fine-tuning phase.

[0010] Optionally, based on the second training picture set, using the second loss function, fine-tuning the pre-trained image adjustment model to obtain the final image adjustment model, including: determining multiple sets of training parameters; for each set of training parameters, based on the second training picture set, using the second loss function, fine-tuning the pre-trained image adjustment model to obtain a candidate image adjustment model; according to a bit rate reduction metric, determining the final image adjustment model from multiple candidate image adjustment models corresponding to the multiple sets of training parameters, wherein the bit rate reduction metric is used to indicate the degree of reduction in the bit rate when the corresponding candidate image adjustment model is used compared to the bit rate when the image adjustment model is not used.

[0011] According to another aspect of the present disclosure, a video encoding method is provided, comprising: using an image adjustment model to adjust a current frame to obtain an adjusted frame; encoding the adjusted frame to obtain encoded data of the current frame; wherein the image adjustment model is an image adjustment model trained using the training method described above.

[0012] Optionally, the using of the image adjustment model to adjust the current frame to obtain the adjusted frame includes: performing a first sharpening process on the current frame to adjust the image texture to obtain a first sharpened frame; using the image adjustment model to adjust the first sharpened frame to obtain the adjusted frame; wherein, encoding the adjusted frame to obtain the encoded data of the current frame includes: performing a second sharpening process on the adjusted frame to adjust the image sharpness to obtain a second sharpened frame; encoding the second sharpened frame to obtain the encoded data of the current frame.

[0013] According to another aspect of the present disclosure, a training device for an image adjustment model is provided, comprising: an acquisition unit, configured to acquire a first training picture set and a second training picture set; a pre-training unit, configured to pre-train an initial image adjustment model based on the first training picture set using a first loss function to obtain a pre-trained image adjustment model, wherein the first loss function is composed of a fidelity loss; a fine-tuning unit, configured to fine-tune the pre-trained image adjustment model based on the second training picture set using a second loss function to obtain a final image adjustment model, wherein the second loss function is composed of a bit rate loss and at least one preset loss, and the bit rate loss targets a bit rate obtained by encoding a picture adjusted by the pre-trained image adjustment model.

[0014] Optionally, the pre-training unit is also configured to: use the initial image adjustment model to adjust the first training picture in the first training picture set to obtain a first adjusted picture; use an analog encoder to encode the first adjusted picture to obtain a first reconstructed picture; determine the value of the fidelity loss based on the first training picture, the first adjusted picture and the first reconstructed picture to obtain the value of the first loss function; based on the value of the first loss function, adjust the parameters of the initial image adjustment model to obtain the pre-trained image adjustment model.

[0015] Optionally, the fine-tuning unit is further configured to: use the pre-trained image adjustment model to adjust the second training picture in the second training picture set to obtain a second adjusted picture; use an analog encoder to encode the second adjusted picture to obtain a second reconstructed picture and a bit rate; determine the value of the bit rate loss and the value of at least one preset loss based on the second training picture, the second adjusted picture, the second reconstructed picture and the bit rate to obtain the value of the second loss function; based on the value of the second loss function, adjust the parameters of the pre-trained image adjustment model to obtain the image adjustment model.

[0016] Optionally, the at least one preset loss includes at least one of the following: perceptual loss, image quality loss, and the fidelity loss.

[0017] Optionally, the learning rate during the pre-training phase is at least an order of magnitude larger than the learning rate during the fine-tuning phase.

[0018] Optionally, the fine-tuning unit is further configured to: determine multiple groups of training parameters; for each group of training parameters, based on the second training picture set, use the second loss function to fine-tune the pre-trained image adjustment model to obtain a candidate image adjustment model; and determine a final image adjustment model from multiple candidate image adjustment models corresponding to the multiple groups of training parameters according to a bit rate reduction metric, wherein the bit rate reduction metric is used to indicate the degree of reduction in bit rate when using the corresponding candidate image adjustment model compared to the bit rate when not using the image adjustment model.

[0019] According to another aspect of the present disclosure, a video encoding device is provided, including: an adjustment unit, configured to use an image adjustment model to adjust a current frame to obtain an adjusted frame; an encoding unit, configured to encode the adjusted frame to obtain encoded data of the current frame; wherein the image adjustment model is an image adjustment model trained using the training method described above.

[0020] Optionally, the adjustment unit is further configured to: perform a first sharpening process on the current frame to adjust the image texture and obtain a first sharpened frame; use the image adjustment model to adjust the first sharpened frame to obtain the adjusted frame; the encoding unit is further configured to: perform a second sharpening process on the adjusted frame to adjust the image sharpness and obtain a second sharpened frame; encode the second sharpened frame to obtain the encoded data of the current frame.

[0021] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer executable instructions, wherein the computer executable instructions, when executed by the at least one processor, prompt the at least one processor to execute the image adjustment model training method or the video encoding method as described above.

[0022] According to another aspect of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the image adjustment model training method or video encoding method as described above.

[0023] According to another aspect of the present disclosure, a computer program product is provided, comprising computer instructions, which, when executed by at least one processor, implement the training method of the image adjustment model or the video encoding method as described above.

[0024] According to the training method, video encoding method, device, electronic device, storage medium and program product of the image adjustment model of the exemplary embodiment of the present disclosure, by splitting the model training process into a pre-training stage and a fine-tuning stage, and using the fidelity loss alone in the pre-training stage, a pre-trained image adjustment model with basically the same input and output can be trained first to ensure the output image quality of the model. Afterwards, the pre-trained image adjustment model is comprehensively fine-tuned using the bit rate loss and at least one preset loss. By using the bit rate loss, the compression bit rate of the picture output by the model can be reduced, making the picture output by the model easy to compress. By using at least one preset loss, the loss can be set on demand to meet the corresponding quality requirements. After such two-stage training, an image adjustment model that outputs easily compressible pictures as much as possible while keeping the picture quality close and meeting the specified requirements can be obtained, which not only reduces the difficulty of training, but also ensures the output picture quality of the model.

[0025] It is to be understood that both the foregoing general description and the following detailed description are examples only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate examples according to the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0027] Figure 1 is a block diagram illustrating an exemplary system for encoding and decoding video blocks according to some embodiments of the present disclosure.

[0028] Figure 2 is a block diagram illustrating an exemplary video encoder according to some embodiments of the present disclosure.

[0029] Figure 3is a block diagram illustrating an exemplary video decoder according to some embodiments of the present disclosure.

[0030] Figure 4 is a training framework diagram showing a training method of an image adjustment model of the related art.

[0031] Figure 5 is a flowchart illustrating a method for training an image adjustment model according to some embodiments of the present disclosure.

[0032] Figure 6 is a flowchart illustrating a video encoding method according to some embodiments of the present disclosure.

[0033] Figure 7 is a block diagram illustrating a training apparatus for an image adjustment model according to some embodiments of the present disclosure.

[0034] Figure 8 is a block diagram illustrating a video encoding apparatus according to some embodiments of the present disclosure.

[0035] Fig. 9 is a diagram illustrating a computing environment coupled with a user interface according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0036] Reference will now be made in detail to specific embodiments, examples of which are shown in the accompanying drawings. In the following detailed description, a number of non-limiting specific details are set forth to aid in understanding the subject matter presented herein. However, various alternatives may be used without departing from the scope of the claims, and the subject matter may be practiced without these specific details. For example, the subject matter presented herein may be implemented on many types of electronic devices having digital video capabilities.

[0037] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the drawings are used to distinguish objects, but not to describe any specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those shown in the drawings or described in the present disclosure.

[0038] Figure 1 is a block diagram illustrating an exemplary system 10 for encoding and decoding video blocks in parallel according to some embodiments of the present disclosure. Figure 1As shown in , system 10 includes a source device 12 that generates and encodes video data to be later decoded by a target device 14. Source device 12 and target device 14 may include any of a wide variety of electronic devices, including cloud servers, server computers, desktop or laptop computers, tablet computers, smart phones, set-top boxes, digital televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, etc. In some implementations, source device 12 and target device 14 are equipped with wireless communication capabilities.

[0039] In some implementations, target device 14 may receive the encoded video data to be decoded via link 16. Link 16 may include any type of communication medium or device capable of moving the encoded video data from source device 12 to target device 14.

[0040] In some other embodiments, the encoded video data may be sent from the output interface 22 to the storage device 32. The encoded video data in the storage device 32 may then be accessed by the target device 14 via the input interface 28.

[0041] like Figure 1 , source device 12 includes a video source 18, a video encoder 20, and an output interface 22. Video source 18 may include sources such as, or a combination of such sources: a video capture device (e.g., a camera), a video archive containing previously captured video, a video feed interface for receiving video from a video content provider, and / or a computer graphics system for generating computer graphics data as source video.

[0042] The captured, pre-captured, or computer-generated video may be encoded by video encoder 20. The encoded video data may be sent directly to target device 14 via output interface 22 of source device 12. The encoded video data may also (or alternatively) be stored on storage device 32 for later access by target device 14 or other devices for decoding and / or playback.

[0043] Target device 14 includes input interface 28, video decoder 30, and display device 34. Input interface 28 may include a receiver and / or a modem and receives encoded video data via link 16. The encoded video data transmitted via link 16 or provided on storage device 32 may include various syntax elements generated by video encoder 20 for use by video decoder 30 in decoding the video data. Such syntax elements may be included within the encoded video data sent over a communication medium, stored on a storage medium, or stored on a file server.

[0044] The video encoder 20 and the video decoder 30 can operate according to a proprietary standard or an industry standard (e.g., VVC, HEVC, Part 10 of MPEG-4, AVC) or an extension of such a standard. It should be understood that the present disclosure is not limited to a specific video encoding / decoding standard and can be applied to other video encoding / decoding standards. It is generally believed that the video encoder 20 of the source device 12 can be configured to encode the video data according to any of these current standards or future standards. Similarly, it is also generally believed that the video decoder 30 of the target device 14 can be configured to decode the video data according to any of these current standards or future standards.

[0045] The video encoder 20 and the video decoder 30 may be implemented as any of a variety of suitable encoder and / or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic devices, software, hardware, firmware, or any combination thereof. When partially implemented in software, the electronic device may store instructions for the software in a suitable non-transitory computer-readable medium, and use one or more processors to execute the instructions in the hardware to perform the video encoding / decoding operations disclosed in the present disclosure. Each of the video encoder 20 and the video decoder 30 may be included in one or more encoders or decoders, and either of the encoders or decoders may be integrated as part of a combined encoder / decoder (CODEC) in the corresponding device.

[0046] Figure 2 is a block diagram illustrating an exemplary video encoder 20 according to some embodiments described in the present disclosure. The video encoder 20 can perform intra-frame prediction coding and inter-frame prediction coding on video blocks within a video frame. Intra-frame prediction coding relies on spatial prediction to reduce or remove spatial redundancy in video data within a given video frame or picture. Inter-frame prediction coding relies on temporal prediction to reduce or remove temporal redundancy in video data within adjacent video frames or pictures of a video sequence. It should be noted that in the field of video coding and decoding, the term "frame" can be used as a synonym for the term "image" or "picture".

[0047] like Figure 2As shown in , the video encoder 20 includes a video data memory 40, a prediction processing unit 41, a decoded picture buffer (DPB) 64, an adder 50, a transform processing unit 52, a quantization unit 54, and an entropy coding unit 56. The prediction processing unit 41 further includes a motion estimation unit 42, a motion compensation unit 44, a segmentation unit 45, an intra-frame prediction processing unit 46, and an intra-frame block copy (BC) unit 48. In some embodiments, the video encoder 20 also includes an inverse quantization unit 58, an inverse transform processing unit 60, and an adder 62 for video block reconstruction. A loop filter 63 such as a deblocking filter can be located between the adder 62 and the DPB 64 to filter the block boundary to remove the block effect from the reconstructed video. In addition to the deblocking filter, another loop filter (e.g., a sample adaptive offset (SAO) filter, a cross-component sample adaptive offset (CCSAO) filter, and / or an adaptive loop filter (ALF)) can also be used to filter the output of the adder 62. In some examples, the loop filter may be omitted, and the decoded video blocks may be provided directly by adder 62 to DPB 64. Video encoder 20 may take the form of a fixed or programmable hardware unit, or may be distributed among one or more of the illustrated fixed or programmable hardware units.

[0048] Video data memory 40 may store video data to be encoded by components of video encoder 20. Figure 1 Video source 18 is shown obtaining video data in video data memory 40. DPB 64 is a buffer that stores reference video data (eg, reference frames or pictures) for use by video encoder 20 in encoding the video data (eg, in intra or inter prediction coding modes).

[0049] like Figure 2 As shown in , after receiving the video data, the segmentation unit 45 within the prediction processing unit 41 segments the video data into video blocks. This segmentation may also include segmenting the video frame into strips, tiles (e.g., a set of video blocks), or other larger coding units (CUs) according to a predefined splitting structure associated with the video data, such as a quadtree (QT) structure. It should be noted that the terms "block" or "video block" used herein may be a portion of a frame or picture, in particular a rectangular (square or non-square) portion. With reference to, for example, HEVC and VVC, a block or video block may be or correspond to a coding tree unit (CTU), a CU, a prediction unit (PU), or a transform unit (TU) and / or may be or correspond to a corresponding block (e.g., a coding tree block (CTB), a coding block (CB), a prediction block (PB), or a transform block (TB)) and / or a sub-block.

[0050] The prediction processing unit 41 may select one of a plurality of feasible prediction coding modes for the current video block based on the error results (e.g., coding rate and distortion level), such as one of one or more inter-frame prediction coding modes in a plurality of intra-frame prediction coding modes. The prediction processing unit 41 may provide the resulting intra-frame prediction coding block or inter-frame prediction coding block to the adder 50 to generate a residual block, and to the adder 62 to reconstruct the coding block for subsequent use as part of a reference frame. The prediction processing unit 41 also provides syntax elements (e.g., motion vectors, intra-frame mode indicators, partition information, and other such syntax information) to the entropy coding unit 56.

[0051] To select an appropriate intra-prediction coding mode for the current video block, intra-prediction processing unit 46 within prediction processing unit 41 may perform intra-prediction coding of the current video block in relation to one or more neighboring blocks in the same frame as the current block to be encoded to provide spatial prediction. Motion estimation unit 42 and motion compensation unit 44 within prediction processing unit 41 may perform inter-prediction coding of the current video block in relation to one or more prediction blocks in one or more reference frames to provide temporal prediction. Video encoder 20 may perform multiple encoding passes, for example, to select an appropriate coding mode for each block of video data.

[0052] In some embodiments, motion estimation unit 42 determines an inter-prediction mode for a current video frame by generating a motion vector according to a predetermined pattern within a sequence of video frames, the motion vector indicating the displacement of a video block within the current video frame relative to a prediction block within a reference video frame. Motion estimation performed by motion estimation unit 42 is the process of generating a motion vector that estimates the motion of a video block. For example, a motion vector may indicate the displacement of a video block within a current video frame or picture relative to a prediction block within a reference frame associated with a current block being encoded within the current frame. The predetermined pattern may designate a video frame in a sequence as a P frame or a B frame. Intra BC unit 48 may determine a vector (e.g., a block vector) for intra BC coding in a manner similar to the motion vector determined by motion estimation unit 42 for inter prediction, or may utilize motion estimation unit 42 to determine a block vector.

[0053] Regardless of whether the prediction block is from the same frame according to intra-frame prediction or from a different frame according to inter-frame prediction, the video encoder 20 can form a residual video block by subtracting the pixel values ​​of the prediction block from the pixel values ​​of the current video block being encoded. The pixel difference values ​​forming the residual video block may include both luma component differences and chroma component differences.

[0054] Intra-prediction processing unit 46 may encode the current block using various intra-prediction modes, for example, during separate encoding passes, and intra-prediction processing unit 46 (or, in some examples, mode selection unit) may select an appropriate intra-prediction mode to use from the tested intra-prediction modes. Intra-prediction processing unit 46 may provide information indicating the intra-prediction mode selected for the block to entropy encoding unit 56. Entropy encoding unit 56 may encode the information indicating the selected intra-prediction mode into the bitstream.

[0055] After prediction processing unit 41 determines a prediction block for the current video block via inter-prediction or intra-prediction, adder 50 forms a residual video block by subtracting the prediction block from the current video block. The residual video data in the residual block may be included in one or more TUs and provided to transform processing unit 52. Transform processing unit 52 transforms the residual video data into residual transform coefficients using a transform (e.g., a discrete cosine transform (DCT) or a conceptually similar transform).

[0056] The transform processing unit 52 may send the resulting transform coefficients to a quantization unit 54. The quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process may also reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, the quantization unit 54 may then perform a scan on the matrix including the quantized transform coefficients. Alternatively, the entropy coding unit 56 may perform the scan.

[0057] After quantization, entropy coding unit 56 entropy encodes the quantized transform coefficients into a video bitstream using, for example, context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), syntax-based context adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy coding method or technique. The encoded bitstream may then be sent to a video bitstream such as a Figure 1 The video decoder 30 shown, or archived as Figure 1 The video frame may be stored in storage device 32 as shown for later transmission to or retrieval by video decoder 30. Entropy encoding unit 56 may also entropy encode motion vectors and other syntax elements for the current video frame being encoded.

[0058] Inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual video block in the pixel domain for use in generating reference blocks for predicting other video blocks. As noted above, motion compensation unit 44 may generate a motion compensated prediction block from one or more reference blocks of a frame stored in DPB 64. Motion compensation unit 44 may also apply one or more interpolation filters to the prediction block to calculate sub-integer pixel values ​​for use in motion estimation.

[0059] Adder 62 adds the reconstructed residual block to the motion compensated prediction block generated by motion compensation unit 44 to generate a reference block for storage in DPB 64. The reference block may then be used as a prediction block by intra BC unit 48, motion estimation unit 42, and motion compensation unit 44 to inter-predict another video block in a subsequent video frame.

[0060] Figure 3 is a block diagram showing an exemplary video decoder 30 according to some embodiments of the present disclosure. The video decoder 30 includes a video data memory 79, an entropy decoding unit 80, a prediction processing unit 81, an inverse quantization unit 86, an inverse transform processing unit 88, an adder 90, and a DPB 92. The prediction processing unit 81 further includes a motion compensation unit 82, an intra-frame prediction unit 84, and an intra-frame BC unit 85. The video decoder 30 may perform the above-mentioned Figure 2 The encoding process is substantially the inverse of the decoding process described with respect to video encoder 20. For example, motion compensation unit 82 may generate prediction data based on motion vectors received from entropy decoding unit 80, and intra-prediction unit 84 may generate prediction data based on intra-prediction mode indicators received from entropy decoding unit 80.

[0061] In some examples, embodiments of the present disclosure may be dispersed in one or more of the units of the video decoder 30. For example, the intra BC unit 85 may perform embodiments of the present disclosure alone or in combination with other units of the video decoder 30, such as the motion compensation unit 82, the intra prediction unit 84, and the entropy decoding unit 80. In some examples, the video decoder 30 may not include the intra BC unit 85, and the functions of the intra BC unit 85 may be performed by other components of the prediction processing unit 81, such as the motion compensation unit 82.

[0062] Video data memory 79 may store video data, such as an encoded video bitstream, to be decoded by other components of video decoder 30. Video data stored in video data memory 79 may be obtained, for example, from storage device 32, from a local video source (e.g., a camera), via wired or wireless network communication of video data, or by accessing a physical data storage medium (e.g., a flash drive or hard disk).

[0063] During the decoding process, the video decoder 30 receives an encoded video bitstream representing video blocks of an encoded video frame and associated syntax elements. The entropy decoding unit 80 of the video decoder 30 entropy decodes the bitstream to generate quantization coefficients, motion vectors or intra-frame prediction mode indicators, and other syntax elements. The entropy decoding unit 80 then forwards the motion vectors or intra-frame prediction mode indicators, and other syntax elements to the prediction processing unit 81.

[0064] When a video frame is encoded as an intra-prediction coded (I) frame or for intra-coded prediction blocks in other types of frames, the intra-prediction unit 84 of the prediction processing unit 81 can generate prediction data for a video block of the current video frame based on the intra-prediction mode transmitted by the signal and reference data from a previously decoded block of the current frame.

[0065] When the video frame is encoded as an inter-frame prediction coding (i.e., B or P) frame, the motion compensation unit 82 of the prediction processing unit 81 generates one or more prediction blocks for the video block of the current video frame based on the motion vector and other syntax elements received from the entropy decoding unit 80. Each of the prediction blocks can be generated from a reference frame in one of the reference frame lists. The video decoder 30 can construct the reference frame lists, i.e., List 0 and List 1, based on the reference frames stored in the DPB 92 using a default construction technique.

[0066] In some examples, when a video block is encoded according to the intra BC mode described herein, intra BC unit 85 of prediction processing unit 81 generates a prediction block for the current video block based on the block vector and other syntax elements received from entropy decoding unit 80. The prediction block may be within a reconstructed region of the same picture as the current video block as defined by video encoder 20.

[0067] Motion compensation unit 82 and / or intra BC unit 85 determine prediction information for a video block of the current video frame by parsing motion vectors and other syntax elements, and then use the prediction information to generate a prediction block for the current video block being decoded.

[0068] Motion compensation unit 82 may also perform interpolation using interpolation filters as used by video encoder 20 during encoding of the video blocks to calculate interpolated values ​​for sub-integer pixels of reference blocks. In this case, motion compensation unit 82 may determine the interpolation filters used by video encoder 20 based on the received syntax elements and use these interpolation filters to produce the prediction blocks.

[0069] Inverse quantization unit 86 inverse quantizes the quantized transform coefficients provided in the bitstream and entropy decoded by entropy decoding unit 80, using the same quantization parameters calculated by video encoder 20 for each video block in the video frame to determine the degree of quantization. Inverse transform processing unit 88 applies an inverse transform (e.g., an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process) to the transform coefficients to reconstruct the residual block in the pixel domain.

[0070] After the motion compensation unit 82 or the intra BC unit 85 generates a prediction block for the current video block based on the vector and other syntax elements, the adder 90 reconstructs the decoded video block for the current video block by adding the residual block from the inverse transform processing unit 88 to the corresponding prediction block generated by the motion compensation unit 82 and the intra BC unit 85. A loop filter 91 (e.g., a deblocking filter, an SAO filter, a CCSAO filter, and / or an ALF) may be located between the adder 90 and the DPB 92 to further process the decoded video block. In some examples, the loop filter 91 may be omitted, and the decoded video block may be provided directly to the DPB 92 by the adder 90. The decoded video blocks in a given frame are then stored in the DPB 92, which stores reference frames for subsequent motion compensation of the next video block. The DPB 92 or a memory device separate from the DPB 92 may also store the decoded video for later presentation on a display device (e.g., Figure 1 on a display device 34).

[0071] Figure 4 is a training framework diagram showing a training method of an image adjustment model of the related art.

[0072] Reference Figure 4 , for the current frame using inter-frame prediction coding mode, record it as the original frame Record the reference frame of the current frame as the original reference frame First, the original frame and the original reference frame Perform motion estimation and obtain motion vectors for future use. and the original reference frame Degradation processing is performed respectively, such as but not limited to downsampling processing and blur processing, so as to reduce the quality of the original frame and obtain a degraded frame I t and degraded reference frame I t-1 Then the degraded frame I t and degraded reference frame I t-1 Input the image adjustment model (here specifically the enhancement model to enhance the quality of the degraded frame) respectively to obtain the processed frame I′ t and I′ t-1, and then simulate the pre-processing of the two to obtain the processing frame I″ t and I″ t-1 , and then input the two and the motion vector into the simulated encoder (using the model to simulate the encoder in video encoding), and the original frame can be output Reconstructed frame And the code rate R. Based on the original frame and reconstruct the frame By comparing the loss function, we can get the fidelity loss, perceptual loss, and image quality loss; based on the bit rate R, we can get the bit rate loss. The weighted sum of these loss items is used as the loss function for training to realize the training of the model.

[0073] For these losses, specifically, fidelity loss refers to the degree of information loss of the processed image (here it can be represented by a reconstructed frame or a processed frame output by an enhancement model) relative to the original image (here it refers to the original frame). For example, the L1 loss (Mean Absolute Error, MAE for short), L2 loss (Mean Squared Error, MSE for short) or Smooth L1 loss (a combination of L1 loss and L2 loss) between the reconstructed frame and the original frame can be calculated. The L1 loss, L2 loss or Smooth L1 loss between the processed frame output by the enhancement model and the original frame can also be calculated.

[0074] Perceptual loss refers to the degree of loss in the perceived quality of the processed image relative to the original image. It focuses on the perceived quality of the input data after processing, rather than just the accuracy of the prediction results. For example, you can refer to the fidelity loss, extract feature maps of the reconstructed or processed frame and the original frame using the VGG (Visual Geometry Group) model, and then calculate the L1 loss, L2 loss, or Smooth L1 loss between the feature maps of the two.

[0075] Image quality loss refers to the decrease in visual quality of the processed image relative to the original image. You can first use some IQA (Image Quality Assessment) algorithms to calculate the image score. For example, get the score a of the original frame and the score b of the reconstructed frame, and use MarginRankingLoss to constrain b to be greater than a. The specific IQA algorithm can be selected as needed without restriction.

[0076] Bit rate loss refers to a loss function related to bit rate. Since the goal is to reduce the bit rate, the bit rate output by the simulated encoder can be directly used as the bit rate loss.

[0077] In other words, this training method does not directly constrain the output of the model, but constrains the output of the simulated encoder. Two temporally adjacent frames are input into the model, and the model output is processed by the simulated pre-processing algorithm, and then sent to the simulated encoder together with the pre-calculated motion vector to obtain the final reconstruction result and predicted bit rate. During the training process, constraints are imposed on the fidelity and perceptual quality of the reconstructed frames, as well as the image quality and bit rate.

[0078] Figure 5 is a flowchart showing a method for training an image adjustment model according to some embodiments of the present disclosure. Figure 4 The training method was further improved on this basis.

[0079] Reference Figure 5 In step S501, a first training picture set and a second training picture set are obtained.

[0080] This step is used to obtain the training data set required for training. The first training picture set and the second training picture set are used for the pre-training stage and the fine-tuning stage of the model respectively, and can be different or the same, and the present disclosure does not limit this. Each training picture set may include a single video sequence or multiple video sequences, and each video sequence includes multiple continuous frames (i.e., pictures in the training picture set). For the current frame using the inter-frame prediction coding mode, its reference frame is also a picture in the video sequence to which the current frame belongs.

[0081] In step S502, based on the first training picture set, the initial image adjustment model is pre-trained using a first loss function to obtain a pre-trained image adjustment model, wherein the first loss function is composed of a fidelity loss.

[0082] This step is the pre-training stage. Its main feature is the use of the first loss function consisting of the fidelity loss. The specific meaning and setting method of the fidelity loss can be found in the previous article, and will not be repeated here. The other losses in the following text are similar and will not be described one by one.

[0083] In step S503, based on the second training picture set, the pre-trained image adjustment model is fine-tuned using a second loss function to obtain a final image adjustment model, wherein the second loss function is composed of a bit rate loss and at least one preset loss, and the bit rate targeted by the bit rate loss is obtained by encoding the picture adjusted by the pre-trained image adjustment model.

[0084] This step is the fine-tuning stage, and its main feature is the use of a second loss function consisting of a bit rate loss and at least one preset loss.

[0085] Optionally, the at least one preset loss includes at least one of the following: perceptual loss, image quality loss, and fidelity loss. By selecting at least one of the perceptual loss, image quality loss, and fidelity loss as the preset loss, the requirements for the image adjusted by the model in terms of perceptual quality, image quality, and information loss can be configured as needed, thereby achieving flexibility in training.

[0086] According to the training method of the image adjustment model of the exemplary embodiment of the present disclosure, by splitting the model training process into a pre-training stage and a fine-tuning stage, and using the fidelity loss alone in the pre-training stage, a pre-trained image adjustment model with basically the same input and output can be trained first to ensure the output image quality of the model. Afterwards, the pre-trained image adjustment model is comprehensively fine-tuned using the bit rate loss and at least one preset loss. By using the bit rate loss, the compression bit rate of the picture output by the model can be reduced, making the picture output by the model easy to compress. By using at least one preset loss, the loss can be set on demand to meet the corresponding quality requirements. After such two-stage training, an image adjustment model that outputs easily compressible pictures as much as possible while keeping the picture quality close and meeting the specified requirements can be obtained, which not only reduces the difficulty of training, but also ensures the output image quality of the model.

[0087] It should be understood that the overall framework of using the model to be trained to process the corresponding training image set, calculate the loss function, and adjust the model parameters during the training process can be referred to Figure 4 It can also be executed using other reasonable frameworks. For example, in the pre-training stage, considering that only the fidelity loss is involved, only the initial image adjustment model is used to adjust the first training image in the first training image set to obtain the adjusted first adjustment image, and then the value of the fidelity loss is determined based on the first adjustment image and the first training image to obtain the value of the first loss function. The parameters of the initial image adjustment model are adjusted based on the value of the first loss function to obtain the pre-trained image adjustment model. For example, in both the pre-training stage and the fine-tuning stage, the following is not executed. Figure 4 The simulation pre-processing shown. For another example, for the current frame using the intra-frame prediction coding mode, it is not necessary to use the reference frame, nor is it necessary to perform motion estimation to obtain the motion vector. The present disclosure does not limit the detailed data processing process and model parameter adjustment process during the training process.

[0088] Next, the training method of the image adjustment model according to the exemplary embodiment of the present disclosure is further introduced.

[0089] From the perspective of the training process, optionally, step S502 includes: using the initial image adjustment model to adjust the first training picture in the first training picture set to obtain a first adjusted picture; using the simulation encoder to encode the first adjusted picture to obtain a first reconstructed picture; determining the value of the fidelity loss based on the first training picture, the first adjusted picture and the first reconstructed picture to obtain the value of the first loss function; based on the value of the first loss function, adjusting the parameters of the initial image adjustment model to obtain a pre-trained image adjustment model; step S503 includes: using the pre-trained image adjustment model to adjust the second training picture in the second training picture set to obtain a second adjusted picture; using the simulation encoder to encode the second adjusted picture to obtain a second reconstructed picture and a bit rate; determining the value of the bit rate loss and the value of at least one preset loss based on the second training picture, the second adjusted picture, the second reconstructed picture and the bit rate to obtain the value of the second loss function; based on the value of the second loss function, adjusting the parameters of the pre-trained image adjustment model to obtain an image adjustment model. In the pre-training stage and fine-tuning stage, the overall process of using the model to adjust the image and encoding the adjusted image with a simulated encoder is adopted to process the data. The value of the loss function of the corresponding stage is determined according to the images before and after processing, and the model parameters are adjusted based on the loss. This process can not only restore the application process of the model as much as possible, but also realizes that the output results of the simulated encoder are constrained instead of directly constrained, which can ensure the final encoding effect.

[0090] As an example, in the pre-training stage, the fidelity loss may include the information loss of the first adjusted image relative to the first training image, and the information loss of the first reconstructed image relative to the first training image. For example, both of these information losses may be directly used as the fidelity loss, or a weighted sum of the two may be calculated as a comprehensive fidelity loss. Of course, other methods may be used to obtain the fidelity loss, and the present disclosure does not limit this. In the fine-tuning stage, if the preset loss includes the fidelity loss, the fidelity loss may be calculated in a manner consistent with the pre-training stage, or a different manner may be used to calculate the fidelity loss. For example, the information loss of the second adjusted image relative to the second training image, and the information loss of the second reconstructed image relative to the second training image are still calculated, but the two information losses are processed in a manner different from the pre-training stage to obtain the fidelity loss. For example, only one of the information losses may be calculated and directly used as the fidelity loss. The present disclosure does not limit this either. The use of other preset losses for images is similar to the fidelity loss, and will not be described one by one.

[0091] In addition, the difference between the two stages, in addition to the different loss functions used, is that the simulated encoder needs to output the bit rate in the fine-tuning stage in order to determine the bit rate loss. It should be understood that in the pre-training stage, the simulated encoder can also output the bit rate, so there is no need to adjust the simulated encoder, but since the bit rate loss does not need to be used when calculating the value of the first loss function, it is not emphasized. The present disclosure does not limit whether the simulated encoder outputs the bit rate in the pre-training stage. As an example, the simulated encoder includes but is not limited to a deep learning model and a traditional machine learning model.

[0092] It should be noted that regarding the image processing flow in the two model training stages and the model application stage, when using the model to adjust the image, the original image to be adjusted can be directly input into the model to obtain an almost identical adjusted image, or the original image to be adjusted can be first degraded and then the degraded image can be input into the model (the model at this time is an enhancement model, which is used to enhance the quality of the image) to obtain the adjusted image. Before encoding the adjusted image (whether it is encoded using an analog encoder in the model training stage or encoded using a traditional encoder in the model application stage), the adjusted image can be pre-processed first. Regarding specific pre-processing methods, in the model application stage, sharpening processing can be used, such as but not limited to USM (Unsharp Masing), or blurring processing can be used, such as but not limited to Gaussian Blur, BilateralFilter, Guided Filter, and sharpening or blurring processing guided by ROI (Region of Interest). In the model training stage, in order to improve processing efficiency, analog pre-processing can be used like using an analog encoder, such as using deep learning frameworks such as pytorch to simulate pre-processing, so that there is no need to perform detailed pre-processing calculations, but only to use the analog pre-processing algorithm to output the simulated pre-processing results.

[0093] From the perspective of model selection in training, optionally, step S503 includes: determining multiple sets of training parameters; for each set of training parameters, based on the second training picture set, using the second loss function, fine-tuning the pre-trained image adjustment model to obtain a candidate image adjustment model; according to the bit rate reduction metric, determining the final image adjustment model from the multiple candidate image adjustment models corresponding to the multiple sets of training parameters, wherein the bit rate reduction metric is used to indicate the degree of reduction in the bit rate when the corresponding candidate image adjustment model is used compared to the bit rate when the image adjustment model is not used. Since the fine-tuning stage may use multiple sets of different hyperparameters and learning rates to train the model separately, and there will be multiple epochs (a complete cycle in the model training process, which refers to the process in which the model performs forward propagation and back propagation on the entire training data set once), these different trainings can be regarded as multiple sets of training parameters. Fine-tuning training is performed according to a set of training parameters to obtain a corresponding optional model, that is, a candidate image adjustment model. The specific training process of a candidate image adjustment model can refer to the introduction in the previous text. By taking the optional model corresponding to each set of training parameters as a candidate image adjustment model and taking the above-mentioned bit rate reduction metric as a selection indicator for these candidate image adjustment models, a model that makes the final encoding bit rate relatively small can be selected, thereby optimizing the encoding result.

[0094] It should be understood that for a set of training parameters, the bit rate when the candidate image adjustment model is used refers to the final bit rate after the training of the set of training parameters is completed, which is equivalent to using the candidate image adjustment model corresponding to the set of training parameters to adjust the second training picture in the second training picture set to obtain the second adjusted picture, and then encoding the second adjusted picture (for example, encoding using an analog encoder) to obtain the bit rate, which can be recorded as the adjusted bit rate. When calculating the bit rate reduction metric, for the same second training picture, the bit rate obtained by directly encoding the second training picture (recorded as the initial bit rate) can be obtained, and then the degree of reduction of the adjusted bit rate relative to the initial bit rate can be calculated to obtain the bit rate reduction metric corresponding to the second training picture of the set of training parameters. Finally, still for the set of training parameters, the statistical value of the bit rate reduction metric (for example, including but not limited to the mean, mode, median, etc.) can be determined for all the second training pictures in the second training picture set as the bit rate reduction metric of the set of training parameters.

[0095] It should also be understood that for the pre-training stage, since the goal of this stage is to roughly select the model and the loss function used is the fidelity loss, the bit rate reduction metric selection indicator may not be used. Of course, this selection indicator may also be used, and the present disclosure does not limit this. For the embodiment using this selection indicator in the pre-training stage, in the pre-training stage, not only is it necessary to use the initial image adjustment model to adjust the first training picture in the first training picture set to obtain the first adjusted picture, but it is also necessary to encode the first adjusted picture to obtain the corresponding bit rate, thereby combining the bit rate obtained by encoding the first training picture to calculate the bit rate reduction metric.

[0096] Optionally, the learning rate in the pre-training phase is at least one order of magnitude larger than the learning rate in the fine-tuning phase. The learning rate determines the step size of the parameter update in each iteration, and a higher learning rate means a larger step size of the parameter update. In the field of machine learning and deep learning, the learning rate is a very important hyperparameter that controls the step size of the parameter update during the model training process. Specifically, the learning rate determines the magnitude of the model parameter update in each iteration (or each gradient descent step). In the gradient descent algorithm, the learning rate is used to scale the gradient (i.e., the derivative of the loss function with respect to the model parameters) to determine the step size of moving the parameters in the direction of the gradient of the loss function. This step size is the learning rate multiplied by the size of the gradient. The size of the learning rate directly affects the convergence speed and stability of the model training. A suitable learning rate can help the model converge to the optimal solution faster while maintaining the stability of the training process. By using a relatively larger (at least one order of magnitude larger) learning rate in the pre-training stage and a relatively smaller learning rate in the fine-tuning stage, a low-distortion pre-trained image adjustment model can be quickly obtained in the pre-training stage, and then a small range of adjustments can be made to it in the fine-tuning stage, so that the model gradually meets other requirements while maintaining the basic low-distortion capability, which can further reduce the difficulty of model training and help achieve global optimization. As an example, the order of magnitude of the learning rate in the pre-training stage is 1e-4, and the order of magnitude of the learning rate in the fine-tuning stage is 1e-6 to 1e-8. Of course, other orders of magnitude are selected according to the actual calculation situation, and the present disclosure does not limit this.

[0097] In summary of the above, in a specific embodiment, since the training goal is to obtain a model for fine-tuning an image, during the training process, firstly, only the fidelity loss is used in the pre-training stage to train a model with almost identical input and output, that is, to train a model that can reconstruct the input image with minimal distortion. After that, in the fine-tuning stage, a smaller learning rate (the learning rate in the pre-training stage is on the order of 1e-4, and the learning rate in the fine-tuning stage is on the order of 1e-6 to 1e-8) is used to fine-tune a model that can output an image that is as easy to compress as possible while maintaining close image quality, using perceptual loss, image quality loss, fidelity loss, and bitrate loss, to obtain the final image adjustment model.

[0098] Figure 6 is a flowchart illustrating an exemplary video encoding method according to some embodiments of the present disclosure.

[0099] Reference Figure 6 In step S601, the current frame is adjusted using the image adjustment model to obtain an adjusted frame.

[0100] For the specific adjustment method, please refer to the method of adjusting the training images in the pre-training stage and the fine-tuning stage mentioned above, which will not be repeated here.

[0101] The image adjustment model is an image adjustment model trained using the training method described above. Since the image adjustment model has a small number of parameters and a weak generation capability, the image adjustment model can be used as an adaptive blur operator in the process of performing video encoding to perform adaptive blur processing on the current frame to obtain an adjusted frame.

[0102] In step S602, the adjusted frame is encoded to obtain encoded data of the current frame.

[0103] This step can be implemented in the video encoder 20 introduced above.

[0104] According to the video encoding method of the exemplary embodiment of the present disclosure, since the image adjustment model trained by the training method as described above is used, it has all the beneficial technical effects of the training method and can reduce the encoding bit rate while maintaining similar image quality, which will not be repeated here.

[0105] Optionally, step S601 includes: performing a first sharpening process on the current frame to adjust the image texture to obtain a first sharpened frame; using an image adjustment model, adjusting the first sharpened frame to obtain an adjusted frame; step S602 includes: performing a second sharpening process on the adjusted frame to adjust the image sharpness to obtain a second sharpened frame; encoding the second sharpened frame to obtain encoded data of the current frame. Considering that the image quality is bound to be affected to a certain extent as the bit rate decreases when the encoding algorithm remains unchanged. By adding the first sharpening process and the second sharpening process before the image adjustment model processing and before encoding, respectively, it is possible to adjust the amount of texture retained and control the size of the sharpness, which helps to ensure the image quality as much as possible and improve the encoding effect. As an example, the first sharpening process and the second sharpening process can be the same or different. When the two are different, different sharpening strengths can be used for the same sharpening algorithm (such as the aforementioned USM algorithm), or different sharpening algorithms can be used. The present disclosure does not limit this. As an example, the complete process of encoding the current frame may include: first sharpening processing, image adjustment model processing, pre-processing, second sharpening processing, and encoding compression. The specific process can be referred to the previous introduction and will not be repeated here.

[0106] Figure 7 is a block diagram showing a training apparatus for an image adjustment model according to an exemplary embodiment of the present disclosure. Figure 7 The training device 700 of the image adjustment model includes an acquisition unit 701, a pre-training unit 702, and a fine-tuning unit 703.

[0107] The acquisition unit 701 may acquire a first training picture set and a second training picture set.

[0108] The pre-training unit 702 can pre-train the initial image adjustment model based on the first training picture set and using the first loss function to obtain a pre-trained image adjustment model, wherein the first loss function is composed of a fidelity loss.

[0109] The fine-tuning unit 703 can fine-tune the pre-trained image adjustment model based on the second training picture set and using the second loss function to obtain the final image adjustment model, wherein the second loss function is composed of a bit rate loss and at least one preset loss, and the bit rate targeted by the bit rate loss is obtained by encoding the picture adjusted by the pre-trained image adjustment model.

[0110] Optionally, the pre-training unit 702 may also: use the initial image adjustment model to adjust the first training picture in the first training picture set to obtain a first adjusted picture; use the analog encoder to encode the first adjusted picture to obtain a first reconstructed picture; determine the value of the fidelity loss based on the first training picture, the first adjusted picture and the first reconstructed picture to obtain the value of the first loss function; based on the value of the first loss function, adjust the parameters of the initial image adjustment model to obtain a pre-trained image adjustment model.

[0111] Optionally, the fine-tuning unit 703 can also: use a pre-trained image adjustment model to adjust the second training picture in the second training picture set to obtain a second adjusted picture; use an analog encoder to encode the second adjusted picture to obtain a second reconstructed picture and a bit rate; determine a bit rate loss value and at least one preset loss value based on the second training picture, the second adjusted picture, the second reconstructed picture and the bit rate to obtain a value of a second loss function; based on the value of the second loss function, adjust the parameters of the pre-trained image adjustment model to obtain an image adjustment model.

[0112] Optionally, the at least one preset loss includes at least one of the following: perception loss, image quality loss, and fidelity loss.

[0113] Optionally, the learning rate during the pre-training phase is at least an order of magnitude larger than the learning rate during the fine-tuning phase.

[0114] Optionally, the fine-tuning unit 703 may also: determine multiple groups of training parameters; for each group of training parameters, based on the second training picture set, use a second loss function to fine-tune the pre-trained image adjustment model to obtain a candidate image adjustment model; and determine a final image adjustment model from multiple candidate image adjustment models corresponding to the multiple groups of training parameters according to a bit rate reduction metric, wherein the bit rate reduction metric is used to indicate the degree of reduction in bit rate when the corresponding candidate image adjustment model is used compared to the bit rate when the image adjustment model is not used.

[0115] Figure 8 is a block diagram showing a video encoding apparatus according to an exemplary embodiment of the present disclosure. Figure 8 , the video encoding device 800 includes an adjustment unit 801 and an encoding unit 802.

[0116] The adjustment unit 801 may use an image adjustment model to adjust the current frame to obtain an adjusted frame, wherein the image adjustment model is an image adjustment model trained using the above training method.

[0117] The encoding unit 802 may encode the adjusted frame to obtain encoded data of the current frame.

[0118] Optionally, the adjustment unit 801 may also: perform a first sharpening process on the current frame to adjust the image texture to obtain a first sharpened frame; use an image adjustment model to adjust the first sharpened frame to obtain an adjusted frame; the encoding unit 802 may also: perform a second sharpening process on the adjusted frame to adjust the image sharpness to obtain a second sharpened frame; encode the second sharpened frame to obtain encoded data of the current frame.

[0119] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0120] Fig. 9 A computing environment 910 is shown coupled to a user interface 950. The computing environment 910 may be part of a data processing server. The computing environment 910 includes a processor 920, a memory 930, and an input / output (I / O) interface 940.

[0121] The processor 920 generally controls the overall operation of the computing environment 910, such as operations associated with display, data acquisition, data communication, and image processing. The processor 920 may include one or more processors for executing instructions to perform all or some steps in the above method. In addition, the processor 920 may include one or more modules that facilitate the interaction between the processor 920 and other components. The processor may be a central processing unit (CPU), a microprocessor, a single-chip microcomputer, a graphics processing unit (GPU), etc.

[0122] The memory 930 is configured to store various types of data to support the operation of the computing environment 910. The memory 930 may include predetermined software 932. Examples of such data include instructions for any application or method operating on the computing environment 910, video data sets, image data, etc. The memory 930 may be implemented by using any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0123] The I / O interface 940 provides an interface between the processor 920 and peripheral interface modules (e.g., keyboard, click wheel, buttons, etc.). The buttons may include, but are not limited to, a home button, a start scan button, and a stop scan button. The I / O interface 940 may be coupled to an encoder and a decoder.

[0124] In an embodiment, a non-transitory computer-readable storage medium including, for example, a plurality of programs in a memory 930 and / or storing a bit stream generated by the above encoding method or a bit stream to be decoded by the above decoding method is also provided, wherein the plurality of programs can be executed by a processor 920 in a computing environment 910 to perform the above method. In one example, the plurality of programs can be executed by a processor 920 in a computing environment 910 to (for example, from Figure 2 The video encoder 20 in the computing environment 910 receives a bit stream or data stream including encoded video information (e.g., video blocks representing encoded video frames, and / or one or more associated syntax elements, etc.), and can also be executed by the processor 920 in the computing environment 910 to perform the above-mentioned decoding method according to the received bit stream or data stream. In another example, the multiple programs can be executed by the processor 920 in the computing environment 910 to perform the above-mentioned encoding method to encode the video information (e.g., video blocks representing video frames, and / or one or more associated syntax elements, etc.) into a bit stream or data stream, and can also be executed by the processor 920 in the computing environment 910 to (e.g., to Figure 3 Alternatively, a non-transitory computer-readable storage medium may store the bit stream or data stream generated by the encoder (e.g., Figure 2 The video encoder 20 in FIG. 1 generates a video signal for a decoder (eg, Figure 3 A bitstream or data stream including encoded video information (e.g., video blocks representing encoded video frames, and / or associated one or more syntax elements, etc.) used by the video decoder 30 in the video decoder 30 when decoding video data. The non-transitory computer-readable storage medium can be, for example, a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0125] In an embodiment, a bit stream generated by the above encoding method or a bit stream to be decoded by the above decoding method is provided. In an embodiment, a bit stream including encoded video information generated by the above encoding method or encoded video information to be decoded by the above decoding method is provided.

[0126] In an embodiment, a computing device is also provided, which includes: one or more processors (e.g., processor 920); and a non-transitory computer-readable storage medium or memory 930 in which multiple programs that can be executed by the one or more processors are stored, wherein the one or more processors are configured to perform the above-mentioned method when executing the multiple programs.

[0127] In an embodiment, a computer program product having instructions for storing or transmitting a bitstream is also provided, wherein the bitstream includes encoded video information generated by the above encoding method or encoded video information to be decoded by the above decoding method. In an embodiment, a computer program product including, for example, a plurality of programs in a memory 930 is also provided, wherein the plurality of programs can be executed by a processor 920 in a computing environment 910 for performing the above method. For example, the computer program product may include a non-transitory computer-readable storage medium.

[0128] In an embodiment, the computing environment 910 may be implemented by one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), FPGAs, GPUs, controllers, microcontrollers, microprocessors, or other electronic components for performing the above methods.

[0129] In an embodiment, a method for storing a bit stream is also provided, comprising: storing the bit stream on a digital storage medium, wherein the bit stream comprises encoded video information generated by the above encoding method or encoded video information to be decoded by the above decoding method.

[0130] In an embodiment, a method for transmitting a bit stream generated by the above encoder is also provided. In an embodiment, a method for receiving a bit stream to be decoded by the above decoder is also provided.

[0131] The description of the present disclosure has been presented for purposes of illustration and is not intended to be exhaustive or limited to the present disclosure. Many modifications, variations, and alternative embodiments will be apparent to one of ordinary skill in the art having the benefit of the teachings presented in the foregoing description and the associated drawings.

[0132] Unless otherwise specifically stated, the order of steps of the method according to the present disclosure is intended to be illustrative only, and the steps of the method according to the present disclosure are not limited to the order specifically described above, but can be changed according to actual conditions. In addition, at least one of the steps of the method according to the present disclosure can be adjusted, combined or deleted according to actual needs.

[0133] The examples are chosen and described in order to explain the principles of the present disclosure and to enable other persons skilled in the art to understand the various embodiments of the present disclosure and to best utilize the basic principles and various embodiments with various modifications suitable for the specific use contemplated. Therefore, it will be understood that the scope of the present disclosure is not limited to the specific examples of the disclosed embodiments, and modifications and other embodiments are intended to be included within the scope of the present disclosure.

Claims

1. A training method for an image adjustment model, characterized in that: include: Obtain a first training picture set and a second training picture set; Based on the first training picture set, using a first loss function, pre-training the initial image adjustment model to obtain a pre-trained image adjustment model, wherein the first loss function is composed of a fidelity loss; Based on the second training picture set, the pre-trained image adjustment model is fine-tuned using a second loss function to obtain a final image adjustment model, wherein the second loss function is composed of a bit rate loss and at least one preset loss, and the bit rate targeted by the bit rate loss is obtained by encoding the picture adjusted by the pre-trained image adjustment model.

2. The training method according to claim 1, characterized in that: The method of pre-training the initial image adjustment model based on the first training picture set and using a first loss function to obtain a pre-trained image adjustment model includes: Using the initial image adjustment model, adjusting a first training picture in the first training picture set to obtain a first adjusted picture; Using an analog encoder, encoding the first adjusted picture to obtain a first reconstructed picture; Determine the value of the fidelity loss according to the first training picture, the first adjusted picture, and the first reconstructed picture, and obtain the value of the first loss function; Based on the value of the first loss function, the parameters of the initial image adjustment model are adjusted to obtain the pre-trained image adjustment model.

3. The training method according to claim 1, characterized in that: The method of fine-tuning the pre-trained image adjustment model based on the second training picture set and using a second loss function to obtain a final image adjustment model includes: Using the pre-trained image adjustment model, adjusting the second training picture in the second training picture set to obtain a second adjusted picture; Using an analog encoder, encoding the second adjusted picture to obtain a second reconstructed picture and a bit rate; Determine, according to the second training picture, the second adjusted picture, the second reconstructed picture and the bit rate, a value of the bit rate loss and a value of the at least one preset loss, and obtain a value of the second loss function; Based on the value of the second loss function, the parameters of the pre-trained image adjustment model are adjusted to obtain the image adjustment model.

4. The training method according to any one of claims 1 to 3, characterized in that: The at least one preset loss includes at least one of the following: perception loss, image quality loss, and the fidelity loss.

5. The training method according to any one of claims 1 to 3, characterized in that: The learning rate during pre-training is at least an order of magnitude larger than the learning rate during fine-tuning.

6. The training method according to any one of claims 1 to 3, characterized in that: The method of fine-tuning the pre-trained image adjustment model based on the second training picture set and using a second loss function to obtain a final image adjustment model includes: determining multiple sets of training parameters; For each set of training parameters, based on the second training picture set, using the second loss function, fine-tuning the pre-trained image adjustment model to obtain a candidate image adjustment model; According to a bit rate reduction metric, a final image adjustment model is determined from multiple candidate image adjustment models corresponding to the multiple groups of training parameters, wherein the bit rate reduction metric is used to indicate a degree of reduction in bit rate when the corresponding candidate image adjustment model is used compared to the bit rate when the image adjustment model is not used.

7. A video encoding method, characterized in that: include: Use the image adjustment model to adjust the current frame to obtain an adjusted frame; Encoding the adjusted frame to obtain encoded data of the current frame; The image adjustment model is an image adjustment model trained using the training method described in any one of claims 1 to 6.

8. The video encoding method according to claim 7, characterized in that: The method of using the image adjustment model to adjust the current frame to obtain an adjusted frame includes: Performing a first sharpening process on the current frame to adjust image texture and obtain a first sharpened frame; Using the image adjustment model, adjusting the first sharpened frame to obtain the adjusted frame; The step of encoding the adjusted frame to obtain the encoded data of the current frame includes: Performing a second sharpening process on the adjustment frame to adjust image sharpness to obtain a second sharpened frame; The second sharpened frame is encoded to obtain encoded data of the current frame.

9. A training device for an image adjustment model, characterized in that: include: An acquisition unit, configured to acquire a first training picture set and a second training picture set; A pre-training unit is configured to pre-train the initial image adjustment model based on the first training picture set using a first loss function to obtain a pre-trained image adjustment model, wherein the first loss function is composed of a fidelity loss; The fine-tuning unit is configured to fine-tune the pre-trained image adjustment model based on the second training picture set using a second loss function to obtain a final image adjustment model, wherein the second loss function is composed of a bit rate loss and at least one preset loss, and the bit rate targeted by the bit rate loss is obtained by encoding the picture adjusted by the pre-trained image adjustment model.

10. A video encoding device, characterized in that: include: An adjustment unit is configured to adjust the current frame using an image adjustment model to obtain an adjusted frame; an encoding unit, configured to encode the adjusted frame to obtain encoded data of the current frame; The image adjustment model is an image adjustment model trained using the training method described in any one of claims 1 to 6.

11. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, When the computer executable instructions are executed by at least one processor, they prompt the at least one processor to execute the image adjustment model training method as described in any one of claims 1 to 6 or the video encoding method as described in claim 7 or 8.

12. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the image adjustment model training method as described in any one of claims 1 to 6 or the video encoding method as described in claim 7 or 8.

13. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by at least one processor, the training method of the image adjustment model according to any one of claims 1 to 6 or the video encoding method according to claim 7 or 8 is implemented.

Images