Picture decoding method and related product

Through joint training and parameter adjustment technology, the problem of insufficient decoding capabilities of decoding models for code stream files in traditional codecs is solved, and multiple decoding models are realized to decoding the same codec stream files, improving the codecs iterative efficiency and decoding effect.

CN119011861AActive Publication Date: 2024-11-22XIAOHONGSHU TECH CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411078791.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-07
Publication Date
2024-11-22
Estimated Expiration
2044-08-07

AI Technical Summary

Technical Problem

In traditional optimization iterative codec technology, it is difficult for multiple different decoders to decode the code stream files encoded by the same encoder, resulting in the code stream files incompatible during the optimization iteration process and cannot be effectively decoded.

Method used

By jointly training the encoding model and the decoding model, the differences between the basic encoding model and the decoding model are used to adjust parameters, and a coding model and decoding model that are compatible with different decoding models can be generated, thereby realizing that multiple decoding models generate the same encoding model. Decoding of code stream files.

Benefits of technology

During the optimization iteration of the codec model, the decoding ability of different decoding models to the code stream files generated by the same codec model is realized, avoiding the problem of incompatibility of the codec file, and improving the iteration efficiency and decoding effect of the codec .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119011861A_ABST
    Figure CN119011861A_ABST
Patent Text Reader

Abstract

The invention relates to a picture decoding method and related products, and the method comprises the steps: obtaining a target code stream file obtained by encoding a to-be-compressed picture through a first encoding model, and enabling the first encoding model and a first decoding model according with a first target to be obtained through the encoding of a first training picture through a basic encoding model to obtain the first code stream file, decoding the first code stream file by a first basic decoding model to obtain a first reconstructed picture, and adjusting parameters of the basic coding model and the first basic decoding model based on the first reconstructed picture and the first training picture; decoding the target code stream file by using a second decoding model to obtain a target picture, wherein an acquisition mode of the second decoding model conforming to a second target is that the second code stream file is decoded by using a second basic decoding model to obtain a second reconstructed picture, and parameter adjustment is performed on the second basic decoding model based on the second reconstructed picture and a second training picture; the second code stream file is obtained by encoding a second training picture through the first encoding model; the first target is different from the second target.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image encoding and decoding, and in particular to a picture decoding method and related products. Background Art

[0002] As image coding and decoding technology is widely used in various information systems, deep learning technology has also made great progress in the field of image coding and decoding. As business scenarios continue to change, the demand for image coding and decoding capabilities is also changing. Using deep learning technology to train encoders and decoders, optimize and iterate codecs, and expand image coding and decoding capabilities has become a technical trend. However, in the traditional technology of optimizing and iterating codecs, there is a deficiency that it is difficult for multiple different decoders to decode the bitstream files encoded by the same encoder. In other words, in the optimization and iteration process of the codec model, it is difficult for multiple different decoding models to decode the bitstream files generated by the same encoding model. Summary of the invention

[0003] Based on this, it is necessary to provide a picture decoding method and related products in which multiple different decoding models can decode the bitstream file generated by the same coding model during the optimization and iteration process of the coding and decoding model in order to solve the above technical problems. The related products include a video frame display method, a decoding model training method, a picture decoding device, a video frame display device, a decoding model training device, a computer device, a computer-readable storage medium, and a computer program product.

[0004] In a first aspect, the present application provides a picture decoding method, the method comprising:

[0005] Obtaining a target bitstream file, wherein the target bitstream file is obtained by encoding a to-be-compressed picture using a first coding model, wherein the first coding model and the first decoding model are obtained by training in the following manner: using a first bitstream file output after encoding a first training picture using a basic coding model as an input of a first basic decoding model, decoding the first bitstream file using the first basic decoding model to obtain a first reconstructed picture, and adjusting parameters of the basic coding model and the first basic decoding model based on a difference between the first reconstructed picture and the first training picture to obtain a first coding model and the first decoding model that meets a first target;

[0006] Decoding the target bitstream file by using a second decoding model to obtain a target picture, wherein the second decoding model is trained in the following manner: using a second bitstream file output after encoding a second training picture by the first encoding model as an input of a second basic decoding model, decoding the second bitstream file by using the second basic decoding model to obtain a second reconstructed picture, and adjusting parameters of the second basic decoding model based on a difference between the second reconstructed picture and the second training picture to obtain the second decoding model that meets the second target;

[0007] Among them, the first goal of training the first basic decoding model is different from the second goal of training the second basic decoding model.

[0008] The picture decoding method provided by the first aspect is adopted, and the first coding model and the first decoding model are obtained by joint training. Specifically, the first code stream file output after the basic coding model encodes the first training picture is used as the input of the first basic decoding model, and the first code stream file is decoded using the first basic decoding model to obtain the first reconstructed picture. Based on the difference between the first reconstructed picture and the first training picture, the basic coding model and the first basic decoding model are adjusted to obtain the first coding model and the first decoding model that meets the first goal. The second decoding model is obtained by training based on the first coding model while the first coding model remains unchanged. Specifically, the second code stream file output after the first coding model encodes the second training picture is used as the input of the second basic decoding model, and the second code stream file is decoded using the second basic decoding model to obtain the second reconstructed picture. Based on the difference between the second reconstructed picture and the second training picture, the second basic decoding model is adjusted to obtain the second decoding model that meets the second goal.

[0009] Since the first goal of training the first basic decoding model is different from the second goal of training the second basic decoding model, that is, the first goal met by the first decoding model is different from the second goal met by the second decoding model, it shows that the optimization iteration of the decoding model is achieved between the second decoding model and the first decoding model. It can also be understood that the optimization iteration of the encoding and decoding model is achieved between the first coding model and the first decoding model, and between the first coding model and the second decoding model. For example, when the first decoding model and the first coding model are obtained by joint training, the first goal met by the first decoding model is to decode the code stream file output after the first coding model encodes the initial picture into a decoded picture similar to the initial picture; when the second decoding model is obtained based on the first coding model training, the second goal met by the second decoding model is to decode the code stream file output after the first coding model encodes the initial picture into a decoded picture similar to the initial picture and with the expected sharpness.

[0010] In addition, since the first decoding model is obtained by joint training with the first encoding model, the first decoding model can decode the bitstream file output by the first encoding model during the model reasoning application process. The second decoding model is obtained by training the second basic decoding model by using the second bitstream file output by the first encoding model after encoding the second training picture as the input of the second basic decoding model, so during the model reasoning application process, the second decoding model can also decode the bitstream file output by the first encoding model.

[0011] The first encoding model, the first decoding model, and the second decoding model obtained through the above training process not only reflect the optimization iteration of the encoding and decoding model, but also reflect that during the optimization iteration of the encoding and decoding model, multiple different decoding models can decode the bitstream file generated by the same encoding model. Based on this, the target bitstream file obtained by encoding the compressed image using the first encoding model is obtained, and the target bitstream file is decoded using the second decoding model to obtain the target image, thereby realizing that during the optimization iteration of the encoding and decoding model, multiple different decoding models can decode the bitstream file generated by the same encoding model.

[0012] In one embodiment, the second target of training the second basic decoding model matches the display effect of the decoded picture output by the second decoding model; before decoding the target bitstream file using the second decoding model to obtain the target picture, the method further includes:

[0013] Obtaining a display requirement for a display effect of the target picture, where the target picture refers to a picture obtained by encoding the target bitstream file;

[0014] When the display effect of the decoded picture output by the second decoding model meets the display requirement, it is determined that the second decoding model is used as the model for decoding the target code stream file.

[0015] In one embodiment, the second decoding model that meets the second target is obtained by adjusting the parameters of the second basic decoding model based on a loss, wherein the loss is calculated based on a difference between the second reconstructed picture and the second training picture using a target loss function;

[0016] Among them, at least one of the target training set including the second training picture and the target loss function matches the second target.

[0017] In one embodiment,

[0018] When the second target is that the first target attribute of the decoded picture generated by the second decoding model meets the expected requirements of the picture, and the first target attribute refers to any picture attribute that affects the visual effect of the picture, the target loss function includes information for judging the difference between the first target attribute of the second reconstructed picture generated by the second basic decoding model and the expected requirements of the picture.

[0019] In one embodiment,

[0020] When the second target is that the second target attribute of the area where the target object is located in the decoded picture generated by the second decoding model meets the local expected requirement, and the second target attribute refers to any picture attribute that affects the visual effect of the picture, the target loss function includes information for detecting the target object in the second reconstructed picture generated by the second basic decoding model, and information for judging the difference between the second target attribute of the area where the target object is located in the second reconstructed picture and the local expected requirement, the number of second training pictures with target content in the target training set is greater than or equal to the first value, and the target content refers to target objects with a preset number or more.

[0021] In one embodiment,

[0022] When the second target is that when the third target attribute of the initial picture is lower than the preset requirement, the second decoding model decodes the code stream file corresponding to the initial picture, and the similarity between the obtained decoded picture and the initial picture reaches the expected similarity value, and the third target attribute refers to any picture attribute that affects the visual effect of the picture, the number of second training pictures in the target training set whose third target attribute is lower than the preset requirement is greater than or equal to the second value.

[0023] In a second aspect, the present application further provides a video frame display method, the method comprising:

[0024] Obtaining a target bitstream file formed after a video frame is encoded, wherein the target bitstream file is obtained by encoding the video frame using a first encoding model, wherein the first encoding model and the first decoding model are trained in the following manner: using a first bitstream file output after a first training picture is encoded by a basic encoding model as an input of a first basic decoding model, decoding the first bitstream file using the first basic decoding model to obtain a first reconstructed picture, and adjusting parameters of the basic encoding model and the first basic decoding model based on a difference between the first reconstructed picture and the first training picture to obtain a first encoding model and the first decoding model that meets a first target;

[0025] Decoding the target bitstream file by using a second decoding model to obtain a target picture, wherein the second decoding model is trained in the following manner: using a second bitstream file output after encoding a second training picture by the first encoding model as an input of a second basic decoding model, decoding the second bitstream file by using the second basic decoding model to obtain a second reconstructed picture, and adjusting parameters of the second basic decoding model based on a difference between the second reconstructed picture and the second training picture to obtain the second decoding model that meets a second goal; wherein the first goal of training the first basic decoding model is different from the second goal of training the second basic decoding model;

[0026] The target image is displayed.

[0027] In a third aspect, the present application also provides a decoding model training method, the method comprising:

[0028] Get the target training set;

[0029] Using a first coding model to encode a second training picture in the target training set to obtain a second bitstream file, wherein the first coding model and the first decoding model are obtained by training in the following manner: using a first bitstream file output after encoding the first training picture by a basic coding model as an input of a first basic decoding model, decoding the first bitstream file by using the first basic decoding model to obtain a first reconstructed picture, and adjusting parameters of the basic coding model and the first basic decoding model based on a difference between the first reconstructed picture and the first training picture to obtain the first coding model and the first decoding model that meet the first target;

[0030] Inputting the second code stream file into a second basic decoding model to obtain a second reconstructed picture;

[0031] Based on the difference between the second reconstructed picture and the second training picture, adjusting the parameters of the second basic decoding model to obtain a second decoding model that meets the second objective;

[0032] Among them, the first goal of training the first basic decoding model is different from the second goal of training the second basic decoding model.

[0033] In one of the embodiments, when the second target is that a first target attribute of a decoded picture generated by the second decoding model meets the expected requirements of the picture, and the first target attribute refers to any picture attribute that affects the visual effect of the picture, adjusting the parameters of the second basic decoding model based on the difference between the second reconstructed picture and the second training picture to obtain a second decoding model that meets the second target includes:

[0034] Calculating a loss based on a difference between the second reconstructed image and the second training image, and a difference between the first target attribute of the second reconstructed image and the image expectation requirement;

[0035] The second basic decoding model is adjusted based on the loss to obtain the second decoding model that meets the second goal.

[0036] In one of the embodiments, when the second target is that the second target attribute of the area where the target object is located in the decoded picture generated by the second decoding model meets the local expected requirement, and the second target attribute refers to any picture attribute that affects the visual effect of the picture, the second basic decoding model is adjusted based on the difference between the second reconstructed picture and the second training picture to obtain a second decoding model that meets the second target, including:

[0037] Calculate the loss based on the difference between the second reconstructed picture and the second training picture, and the difference between the second target attribute of the region where the target object in the second reconstructed picture is located and the local expected requirement;

[0038] The second basic decoding model is adjusted based on the loss to obtain the second decoding model that meets the second goal.

[0039] In a fourth aspect, the present application further provides a picture decoding device, the device comprising:

[0040] a bitstream acquisition module, configured to acquire a target bitstream file, wherein the target bitstream file is obtained by encoding the to-be-compressed picture using a first encoding model, wherein the first encoding model and the first decoding model are obtained by training in the following manner: a first bitstream file output after encoding the first training picture using a basic encoding model is used as an input of a first basic decoding model, the first bitstream file is decoded using the first basic decoding model to obtain a first reconstructed picture, and based on a difference between the first reconstructed picture and the first training picture, the basic encoding model and the first basic decoding model are adjusted to obtain a first encoding model and the first decoding model that meets a first target;

[0041] a bitstream decoding module, configured to decode the target bitstream file using a second decoding model to obtain a target picture, wherein the second decoding model is trained in the following manner: a second bitstream file output after encoding the second training picture using the first encoding model is used as an input of a second basic decoding model, the second bitstream file is decoded using the second basic decoding model to obtain a second reconstructed picture, and based on a difference between the second reconstructed picture and the second training picture, the second basic decoding model is adjusted to obtain the second decoding model that meets the second target;

[0042] Among them, the first goal of training the first basic decoding model is different from the second goal of training the second basic decoding model.

[0043] In a fifth aspect, the present application further provides a video frame display device, the device comprising:

[0044] a target bitstream acquisition module, configured to acquire a target bitstream file formed after a video frame is encoded, wherein the target bitstream file is obtained by encoding the video frame using a first encoding model, wherein the first encoding model and the first decoding model are obtained by training in the following manner: a first bitstream file output after encoding a first training picture using a basic encoding model is used as an input of a first basic decoding model, the first bitstream file is decoded using the first basic decoding model to obtain a first reconstructed picture, and based on a difference between the first reconstructed picture and the first training picture, the basic encoding model and the first basic decoding model are adjusted to obtain a first encoding model and the first decoding model that meets a first target;

[0045] a target picture generating module, configured to decode the target bitstream file by using a second decoding model to obtain a target picture, wherein the second decoding model is obtained by training in the following manner: using the second bitstream file output after encoding the second training picture by the first encoding model as an input of the second basic decoding model, decoding the second bitstream file by using the second basic decoding model to obtain a second reconstructed picture, and adjusting the parameters of the second basic decoding model based on the difference between the second reconstructed picture and the second training picture to obtain the second decoding model that meets a second goal; wherein the first goal of training the first basic decoding model is different from the second goal of training the second basic decoding model;

[0046] The target image display module is used to display the target image.

[0047] In a sixth aspect, the present application also provides a decoding model training device, the device comprising:

[0048] A training set acquisition module is used to acquire a target training set;

[0049] a bitstream generation module, configured to encode a second training picture in the target training set by using a first encoding model to obtain a second bitstream file, wherein the first encoding model and the first decoding model are obtained by training in the following manner: using a first bitstream file output after encoding the first training picture by a basic encoding model as an input of a first basic decoding model, decoding the first bitstream file by using the first basic decoding model to obtain a first reconstructed picture, and adjusting parameters of the basic encoding model and the first basic decoding model based on a difference between the first reconstructed picture and the first training picture to obtain the first encoding model and the first decoding model that meets the first target;

[0050] A picture reconstruction module, used for inputting the second code stream file into a second basic decoding model to obtain a second reconstructed picture;

[0051] A model generation module, configured to adjust the parameters of the second basic decoding model based on the difference between the second reconstructed picture and the second training picture to obtain a second decoding model that meets the second objective;

[0052] Among them, the first goal of training the first basic decoding model is different from the second goal of training the second basic decoding model.

[0053] In the seventh aspect, the present application also provides a computer device, comprising: a memory and a processor, wherein the memory stores program instructions; when the program instructions are executed by the processor, the processor executes the method as shown in the first aspect or any embodiment of the first aspect, or executes the method as shown in the second aspect or any embodiment of the second aspect, or executes the method as shown in the third aspect or any embodiment of the third aspect.

[0054] In the eighth aspect, the present application also provides a computer-readable storage medium, which stores a computer program; when the computer program runs on one or more processors, it executes the method shown in the first aspect or any embodiment of the first aspect, or executes the method shown in the second aspect or any embodiment of the second aspect, or executes the method shown in the third aspect or any embodiment of the third aspect.

[0055] In the ninth aspect, the present application also provides a computer program product, comprising a computer program or instructions; when the computer program or instructions are run on a computer, the computer is caused to execute the method as shown in the first aspect or any embodiment of the first aspect, or execute the method as shown in the second aspect or any embodiment of the second aspect, or execute the method as shown in the third aspect or any embodiment of the third aspect.

[0056] It can be understood that the video frame display method provided in the second aspect, the decoding model training method provided in the third aspect, the picture decoding device provided in the fourth aspect, the video frame display device provided in the fifth aspect, the decoding model training device provided in the sixth aspect, the computer device provided in the seventh aspect, the computer readable storage medium provided in the eighth aspect, and the computer program product provided in the ninth aspect can be used to execute the picture decoding method shown in the first aspect of the present application or any embodiment of the first aspect, or is related to the picture decoding method shown in the first aspect of the present application or any embodiment of the first aspect. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the picture decoding method, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments are introduced as follows.

[0058] Figure 1 This is a flowchart of a method for decoding a picture provided in an embodiment of the present application;

[0059] Figure 2 A schematic diagram of the model structure of a basic encoding model and a first basic decoding model during joint training provided in an embodiment of the present application;

[0060] Figure 3 The second flowchart of the picture decoding method provided in the embodiment of the present application;

[0061] Figure 4 A schematic diagram of a flow chart of a video frame display method provided in an embodiment of the present application;

[0062] Figure 5 A flowchart of a decoding model training method provided in an embodiment of the present application;

[0063] Figure 6 One of the flow charts of adjusting the parameters of the second basic decoding model based on the difference between the second reconstructed picture and the second training picture to obtain the second decoding model that meets the second goal provided in the embodiment of the present application;

[0064] Figure 7A second flow chart of adjusting the parameters of the second basic decoding model based on the difference between the second reconstructed picture and the second training picture to obtain a second decoding model that meets the second goal provided in an embodiment of the present application;

[0065] Figure 8 A schematic diagram of the structure of a picture decoding device provided in an embodiment of the present application;

[0066] Fig. 9 A schematic diagram of the structure of a video frame display device provided in an embodiment of the present application;

[0067] Fig.10 A schematic diagram of the structure of a decoding model training device provided in an embodiment of the present application;

[0068] Fig.11 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to facilitate understanding of the embodiments of the present application, the embodiments of the present application will be described more fully below with reference to the relevant drawings. The preferred embodiments of the embodiments of the present application are given in the drawings. However, the embodiments of the present application can be implemented in many different forms and are not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the embodiments of the present application more thorough and comprehensive.

[0070] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the technical field of the embodiments of the present application. The terms used herein in the description of the embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the embodiments of the present application.

[0071] The terms "first", "second", etc. used in this article are used to distinguish different objects, rather than to describe a specific order. For the purpose of distinguishing different objects, the "first decoding model" can also be described as the "second decoding model", and the "second decoding model" can be described as the "first decoding model". When used, the singular expressions "one", "a kind", "said", "above", "the" and "this" are intended to also include plural expressions, unless the context clearly indicates otherwise. It should also be understood that the term "includes / comprising" specifies the presence of the stated features, wholes, steps, operations, parts or their fusions, but does not exclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, parts or their fusions.

[0072] In the traditional technology of optimizing iterative codecs, there is a shortcoming that it is difficult for multiple different decoders to decode the bitstream files encoded by the same encoder. Specifically, there are encoder E_a and decoder D_a, encoder E_b and decoder D_b. In the traditional technology of optimizing iterative codecs, the codec is obtained by synchronous joint training. That is, in the training stage, E_a and D_a are end-to-end jointly trained, and E_b and D_b are end-to-end jointly trained. The loss function used in the training process is usually the mean square error function. In the inference application stage, E_a and D_a can be inferred separately, and E_b and D_b can be inferred separately. Among them, the input of E_a is the initial picture, and the output is the encoded bitstream file S_a. The bitstream file S_a is used as the input of D_a, and D_a outputs the decoded picture after decoding; the input of E_b is the initial picture, and the output is the encoded bitstream file S_b. The bitstream file S_b is used as the input of D_a, and D_a outputs the decoded picture after decoding. Among them, the bitstream file S_a encoded by E_a can be decoded by the decoder D_a and reconstructed into a decoded picture, and the bitstream file S_b encoded by E_b can be decoded by the decoder D_b and reconstructed into a decoded picture; but D_b cannot decode the bitstream file S_a encoded by E_a, and similarly, D_a cannot decode the bitstream file S_b encoded by E_b. The codec needs to be continuously optimized and iterated, and with the continuous iteration of the codec version, there is a problem that the second version of the decoder is difficult to decode the bitstream file encoded by the first version of the encoder. In other words, only the second version of the decoder can decode the bitstream file encoded by the second version of the encoder, and only the first version of the decoder can decode the bitstream file encoded by the first version of the encoder. It is difficult for multiple different decoders to decode the bitstream file encoded by the same encoder. The codec is regarded as a codec model, the encoder is regarded as a coding model, and the decoder is regarded as a decoding model. The above technical problem can be understood as that in the optimization and iteration process of the codec model, the bitstream files between different codec models are not compatible, and multiple different decoding models are difficult to decode the bitstream files encoded by the same coding model.

[0073] In order to meet the new demands for image encoding and decoding capabilities caused by the ever-changing business scenarios, the codec needs to be continuously optimized and iterated. The above-mentioned deficiencies in the traditional technology of optimizing and iterating codecs will not only weaken the significance of optimizing and iterating codecs, but also lead to the risk that existing code stream files cannot be decoded. Based on this, an embodiment of the present application provides a picture decoding method, which can achieve compatibility between code stream files between different codec models during the optimization and iteration process of the codec, that is, during the optimization and iteration process of the codec model, different decoding models can decode the code stream files encoded by the same coding model, and the second version of the decoding model can also decode the code stream files encoded by the first version of the coding model.

[0074] The present application embodiment provides a method for decoding a picture. Figure 1 As shown, the picture decoding method includes the following steps S101 to S102.

[0075] S101, obtaining a target bitstream file. The target bitstream file is obtained by encoding a to-be-compressed picture using a first coding model, and the first coding model and the first decoding model are trained in the following manner: a first bitstream file output after encoding a first training picture using a basic coding model is used as an input of a first basic decoding model, the first bitstream file is decoded using the first basic decoding model to obtain a first reconstructed picture, and based on a difference between the first reconstructed picture and the first training picture, the basic coding model and the first basic decoding model are adjusted to obtain a first coding model and a first decoding model that meets a first target.

[0076] The first coding model and the first decoding model are obtained through joint training as a whole. Based on the difference between the first reconstructed picture and the first training picture, the basic coding model and the first basic decoding model are adjusted as a whole, that is, the parameters of the basic coding model and the parameters of the first basic decoding model affect each other. The first target refers to the decoding model training target that the first decoding model meets, and also refers to the target of training the first basic decoding model, which can also be understood as the encoding and decoding model training target that the first coding model and the first decoding model meet.

[0077] Optionally, the basic encoding model used to train the first encoding model may be Normalizer-Free ResNets (NFNet), Vision transformer (ViT), convolutional neural network (CNN), etc. The first basic decoding model used to train the first decoding model may also be a convolutional neural network (CNN), etc. Taking the example that both the basic encoding model and the first basic decoding model are CNN, the model structures of the basic encoding model and the first basic decoding model during joint training are as follows: Figure 2 shown. Figure 2 The meanings of the English symbols are as follows: Conv represents the convolution layer; TConv represents the transposed convolution layer; k5s2 represents the convolution kernel of the convolution layer is 5*5 and the step size is 2; k3s1 represents the convolution kernel of the convolution layer is 3*3 and the step size is 1; k3s2 represents the convolution kernel of the convolution layer is 3*3 and the step size is 2; k4s2 represents the convolution kernel of the convolution layer is 4*4 and the step size is 2; k4s1 represents the convolution kernel of the convolution layer is 4*4 and the step size is 1; k4s4 represents the convolution kernel of the convolution layer is 4*4 and the step size is 4; ReLU6 and ReLU represent activation functions respectively; x represents the first training image; y represents the first feature of the image; z represents the second feature of the image; represents the first reconstructed picture; and They respectively represent the data obtained after decoding the bit stream data; Q represents quantization; AE represents the arithmetic encoder; AD represents the arithmetic decoder; μ and σ represent the parameters of the arithmetic encoder and arithmetic decoder.

[0078] like Figure 2 As shown, first, the basic coding model includes a series of convolutional layers and activation functions, aiming to extract the first feature y of the image from the input first training image x. Specifically, the basic coding model can sequentially use four convolutional layers Conv and three activation functions ReLU6, where the four convolutional layers Conv and the three activation functions ReLU6 are as follows: Figure 2The intervals shown are arranged alternately. The convolution kernels of the four convolution layers Conv can all be 5*5, and the step size is 2, so as to achieve 16 times downsampling of the first training picture x. The activation function ReLU6 can effectively improve the performance of the deep learning model, so that the basic coding model can better capture and transmit the linear relationship between features. Through the design of the above-mentioned convolution layer Conv and the activation function ReLU6, it is possible to capture rich image information and extract the first feature y of the picture while reducing the image size. Since the code stream data compressed based on the first feature y of the picture may be too large, based on this, the basic coding model also includes another series of convolution layers and activation functions, which aims to extract the second feature z of the picture from the first feature y corresponding to the input first training picture x, and then compress the second feature z to obtain smaller code stream data. Specifically, the basic coding model can use three convolution layers Conv and three activation functions ReLU in sequence. Among them, a convolution layer with a convolution kernel of 3*3 and a step size of 1 is first used, and then two convolution layers with convolution kernels of 5*5 and a step size of 2 are used. The three convolution layers Conv and three activation functions ReLU are as follows Figure 2 The intervals shown are arranged alternately. Through the design of the above-mentioned convolution layer Conv and the activation function ReLU, it is possible to further downsample to reduce the image size, capture and transfer the linear relationship between the features, increase the depth and width of the network, and improve the richness and diversity of the second feature z obtained. Then, the second feature z of the image is quantized by Q and input into the arithmetic encoder AE to obtain the first part of the code stream data, such as 100111. It can be understood that the second feature z obtained by compressing and downsampling again, the first part of the code stream data obtained is relatively small. And, the first feature y of the image is quantized by Q and input into the arithmetic encoder AE to obtain the second part of the code stream data, such as 100111. Accordingly, the first part of the code stream data 100111 and the second part of the code stream data 100111 are spliced, and the first part of the code stream data is in front and the second part of the code stream data is in the back, then the first code stream file 100111100111 obtained by encoding the first training picture x by the basic coding model can be obtained.

[0079] It should be noted that if Figure 2 As shown, in the process of quantizing the first feature y of the image and inputting it into the arithmetic encoder AE to obtain the second part of the bitstream data, entropy coding needs to be used, that is, parameters μ and σ need to be obtained to more efficiently compress the first feature y, so that the second part of the bitstream data output by the arithmetic encoder AE is shorter and smaller. At this time, the first part of the bitstream data is first decoded by the arithmetic decoder AD to obtain data Then through Figure 2 The three transposed convolutional layers TConv and two activation functions ReLU are arranged alternately to transform the data Converted into parameters μ and σ. First, it passes through a convolution layer with a convolution kernel of 3*3 and a step size of 2, and then passes through two convolution layers with convolution kernels of 4*4 and step sizes of 2 and 1 respectively. Then, the parameters μ and σ are input into the basic coding model, so that the arithmetic encoder AE uses the parameters μ and σ to output a smaller second part of the code stream data.

[0080] like Figure 2 As shown, the first basic decoding model includes a series of transposed convolutional layers and activation functions, which is intended to reconstruct the first reconstructed picture based on the second part of the input code stream data. Specifically, the input second part of the code stream data is first decoded by the arithmetic decoder AD, and the output data Correspondingly, the arithmetic decoder AD also needs to obtain the parameters μ and σ in the process of decoding the second part of the bitstream data. Based on this, the first basic decoding model obtains the parameters μ and σ obtained by the above method, so that the arithmetic decoder AD uses the parameters μ and σ to decode the second part of the bitstream data. Further, the first basic decoding model can sequentially use three transposed convolutional layers TConv and two activation functions ReLU to transform the data Convert to the first reconstructed image We can first use two transposed convolution layers with a convolution kernel of 4*4 and a stride of 2, and then use a convolution layer with a convolution kernel of 4*4 and a stride of 4 to achieve 16 times upsampling. Three transposed convolution layers TConv and two activation functions ReLU are as follows: Figure 2 It can be understood that, in order to achieve upsampling 16 times corresponding to downsampling 16 times, the first basic decoding model can also include four transposed convolution layers with a step size of 2. Figure 2 The model structure including three transposed convolutional layers TConv shown in FIG can speed up the decoding and reconstruction. Through the design of the transposed convolutional layer TConv and the activation function ReLU, a first reconstructed image similar to the first training image x and with higher quality can be reconstructed.

[0081] go through Figure 2 The model structure of the basic encoding model and the first basic decoding model during joint training shown can encode and decode the first training picture to obtain the first reconstructed picture. It should be understood that Figure 2The model structure shown is only an optional example. On the premise that the first training picture can be encoded and decoded to obtain the first reconstructed picture, the embodiment of the present application does not specifically limit the model structure of the basic coding model and the first basic decoding model. Furthermore, based on the difference between the first reconstructed picture and the first training picture, the basic coding model and the first basic decoding model can be adjusted to obtain the first coding model and the first decoding model that meets the first goal. Specifically, the parameter adjustment can be to adjust the parameters of each convolutional layer contained in the basic coding model, the parameters of each loss function, and the parameters of each transposed convolutional layer contained in the first basic decoding model, and the parameters of each loss function.

[0082] S102, using the second decoding model to decode the target bitstream file to obtain the target picture. The second decoding model is obtained by training in the following manner: using the second bitstream file output after the first encoding model encodes the second training picture as the input of the second basic decoding model, decoding the second bitstream file using the second basic decoding model to obtain a second reconstructed picture, and adjusting the parameters of the second basic decoding model based on the difference between the second reconstructed picture and the second training picture to obtain a second decoding model that meets the second goal; wherein the first goal of training the first basic decoding model is different from the second goal of training the second basic decoding model.

[0083] The second decoding model is obtained by training based on the first coding model while the first coding model remains unchanged. Based on the difference between the second reconstructed picture and the second training picture, the parameters of the second basic decoding model are adjusted. At this time, the first coding model is in a fixed state, and the adjustment of the parameters of the second basic decoding model has no effect on the parameters of the first coding model. The second target refers to the decoding model training target that the second decoding model meets, and also refers to the target of training the second basic decoding model. It can also be understood as the coding and decoding model training target that the first coding model and the second decoding model meet. Optionally, the second basic decoding model can also be CNN.

[0084] Since the first goal of training the first basic decoding model is different from the second goal of training the second basic decoding model, that is, the first goal met by the first decoding model is different from the second goal met by the second decoding model, it shows that the optimization iteration of the decoding model is achieved between the second decoding model and the first decoding model. It can also be understood that the optimization iteration of the encoding and decoding model is achieved between the first coding model and the first decoding model, and between the first coding model and the second decoding model. For example, when the first decoding model and the first coding model are obtained by joint training, the first goal met by the first decoding model is to decode the code stream file output after the first coding model encodes the initial picture into a decoded picture similar to the initial picture; when the second decoding model is obtained based on the first coding model training, the second goal met by the second decoding model is to decode the code stream file output after the first coding model encodes the initial picture into a decoded picture similar to the initial picture and with the expected sharpness.

[0085] In addition, since the first decoding model is obtained by joint training with the first encoding model, the first decoding model can decode the bitstream file output by the first encoding model during the model reasoning application process. The second decoding model is obtained by training the second basic decoding model by using the second bitstream file output by the first encoding model after encoding the second training picture as the input of the second basic decoding model, so during the model reasoning application process, the second decoding model can also decode the bitstream file output by the first encoding model.

[0086] Taking the first encoding model as E_a and the first decoding model as D_a as an example, E_a and D_a are obtained by jointly training the basic encoding model and the first basic decoding model, and D_a meets the first goal of decoding the code stream file output after E_a encodes the initial picture into a decoded picture similar to the initial picture.

[0087] Optionally, on the basis of maintaining the first encoding model E_a unchanged, the optimization iteration of the encoding and decoding model is performed, that is, the optimization iteration of the decoding model is performed. Based on different decoding model training objectives, D_a1, D_a2, D_a3... are trained respectively. The second decoding model is any one of D_a1, D_a2, D_a3..., and the second decoding model is obtained by training the second basic decoding model. Taking the second decoding model as D_a1 as an example, D_a1 meets the second goal of decoding the bitstream file output after E_a encodes the initial picture into a decoded picture similar to the initial picture and with higher quality than the initial picture. Higher quality such as clearer details, richer details, higher pixels, etc. In the optimization iteration process of the encoding and decoding model, D_a1 and D_a can both decode the bitstream file encoded by E_a. Among them, the first basic decoding model and the second basic decoding model can be the same or different; the second basic decoding model when the second decoding model is D_a1 and the second basic decoding model when the second decoding model is D_a2 can be the same or different.

[0088] Optionally, on the basis of the first coding model E_a remaining unchanged, the optimization iteration of the coding and decoding model is performed, that is, the optimization iteration of the decoding model is performed, and D_aN is obtained by training based on the decoding model training target. Further, on the basis of the first coding model E_a remaining unchanged, the optimization iteration of the coding and decoding model is performed again, that is, the optimization iteration of the decoding model is performed again. Based on different decoding model training targets, D_aN1, D_aN2, D_aN3... are obtained by training D_aN respectively. The second decoding model is any one of D_a1, D_a2, D_a3..., and the second decoding model is obtained by training the second basic decoding model. Taking the second decoding model as D_aN1 as an example, D_aN1 meets the second target of decoding the bitstream file output after E_a encodes the initial picture into a decoded picture that is similar to the initial picture and has higher quality than the initial picture. Higher quality means clearer details, richer details, and higher pixels. It should be understood that in this optional example, based on the first decoding model D_a, through the optimization iteration of the decoding model, D_aN is trained, and then the process of training the second decoding model through the optimization iteration of the decoding model can still be regarded as the optimization iteration of the decoding model between the second decoding model and the first decoding model, and the optimization iteration of the encoding and decoding model is realized between the first encoding model and the first decoding model, and between the first encoding model and the second decoding model. In the optimization iteration process of the encoding and decoding model, D_aN1 and D_a can both decode the bitstream file encoded by E_a. Different from the aforementioned optional example in which the second decoding model is obtained by only one optimization iteration of the decoding model, the second basic decoding model trained to obtain the second decoding model in this optional example is D_aN, and D_aN is obtained by optimizing and iterating compared to the first decoding model D_a, so the second basic decoding model D_aN is different from the first basic decoding model trained to obtain the first decoding model D_a. Furthermore, the second basic decoding model when the second decoding model is D_aN1 is the same as the second basic decoding model when the second decoding model is D_aN2, both of which are D_aN.

[0089] The first encoding model, the first decoding model, and the second decoding model obtained through the above training process not only reflect the optimization iteration of the encoding and decoding model, but also reflect that during the optimization iteration of the encoding and decoding model, multiple different decoding models can decode the code stream file generated by the same encoding model. Based on this, the picture decoding method in the embodiment of the present application is adopted to obtain the target code stream file obtained by encoding the compressed picture using the first encoding model, and the target code stream file is decoded using the second decoding model to obtain the target picture. This realizes that during the optimization iteration of the encoding and decoding model, multiple different decoding models can decode the code stream file generated by the same encoding model, and the second decoding model of the second version can decode the target code stream file generated by the first encoding model of the first version. This picture decoding method is conducive to promoting the development of deep learning technology in the field of image encoding and decoding, and is also conducive to improving the efficiency of image encoding and decoding technology based on deep learning technology in actual business.

[0090] In one embodiment, the second goal of training the second basic decoding model matches the display effect of the decoded picture output by the second decoding model. Figure 3 As shown, the picture decoding method further includes the following steps S301 to S304. Among them, steps S301 and S304 correspond to steps S101 and S102 in the aforementioned embodiment. The specific discussion of steps S301 and S304 in this embodiment can refer to the specific discussion of steps S101 and S102 in the aforementioned embodiment, and will not be repeated in this embodiment.

[0091] S301, obtaining a target bitstream file. The target bitstream file is obtained by encoding a to-be-compressed picture using a first coding model, and the first coding model and the first decoding model are trained in the following manner: a first bitstream file output after encoding a first training picture using a basic coding model is used as an input of a first basic decoding model, the first bitstream file is decoded using the first basic decoding model to obtain a first reconstructed picture, and based on the difference between the first reconstructed picture and the first training picture, the basic coding model and the first basic decoding model are adjusted to obtain a first coding model and a first decoding model that meets the first target.

[0092] S302, obtaining display requirements for display effects of a target picture. The target picture refers to a picture obtained by encoding a target bitstream file.

[0093] Among them, the display effect of the target image can be understood as the display effect of the target image in terms of quality. For example, in terms of quality, in addition to being similar to the image to be compressed, the details of the target objects such as portraits or animals in the target image are clearer and the saturation is higher than that of the image to be compressed. The display requirements correspond to the display effect. For example, in terms of quality, in addition to being similar to the image to be compressed, the details of the target objects such as portraits or animals in the target image are clearer and the saturation is higher than that of the image to be compressed.

[0094] The display requirements of the display effect of the target image can be set manually, or the basic image attributes such as brightness, saturation, and sharpness of the image to be compressed can be identified by modules, systems, devices, etc., and the display requirements of the display effect of the target image obtained after encoding and decoding the image to be compressed can be determined based on the identified basic image attributes. Alternatively, the display requirements of the display effect of the target image can be determined based on the purpose of the target image obtained after encoding and decoding the image to be compressed by modules, systems, devices, etc. The embodiment of the present application does not limit the determination subject and determination method of the display requirements of the display effect of the target image.

[0095] S303: When the display effect of the decoded picture output by the second decoding model meets the display requirement, determine to use the second decoding model as a model for decoding the target bitstream file.

[0096] Since both the second decoding model and the first decoding model can decode the target code stream file encoded by the first encoding model. Moreover, based on different training objectives, the decoding model is optimized and iterated to obtain multiple different optimized decoding models, and the second decoding model can be any one of the multiple different optimized decoding models. Therefore, when there are multiple decoding models that can be used to decode the target code stream file, the training objectives met by each decoding model are known, and the display requirements of the display effect of the target picture are obtained, the target decoding model can be adaptively determined based on the display requirements of the display effect of the target picture to decode the target code stream file, and the display effect of the decoded picture output by this target decoding model meets the display requirements.

[0097] Based on this, when the display effect of the decoded picture output by the second decoding model meets the display requirements, it is determined that the second decoding model is used as the model for decoding the target code stream file so that the target picture has the required display effect.

[0098] Optionally, if the target image has multiple display requirements, such as requiring the details of the target objects such as portraits or animals to be clearer than those of the image to be compressed, and requiring the saturation to be higher than that of the image to be compressed. Then obtain the importance of each of the multiple display requirements, and determine the first important display requirement, the second most important display requirement... The importance of each of the multiple display requirements can be set manually or determined by preset rules. When the importance of each of the multiple display requirements is determined, the decoding model of the first target is adaptively determined to decode the target code stream file, and the display effect of the decoded picture output by the decoding model of the first target meets the first important display requirement. Furthermore, the decoding model of the second target can be adaptively determined to decode the code stream file corresponding to the target picture decoded by the decoding model of the first target, and the display effect of the decoded picture output by the decoding model of the second target meets the second most important display requirement. And so on.

[0099] S304, using the second decoding model to decode the target bitstream file to obtain the target picture. The second decoding model is obtained by training in the following manner: using the second bitstream file output after the first encoding model encodes the second training picture as the input of the second basic decoding model, decoding the second bitstream file using the second basic decoding model to obtain a second reconstructed picture, and adjusting the parameters of the second basic decoding model based on the difference between the second reconstructed picture and the second training picture to obtain a second decoding model that meets the second goal; wherein the first goal of training the first basic decoding model is different from the second goal of training the second basic decoding model.

[0100] In this embodiment, when there are multiple decoding models that can be used to decode the target code stream file, the training objectives met by each decoding model are known, and the display requirements of the display effect of the target image are obtained, based on the display requirements of the display effect of the target image, the target decoding model can be adaptively determined to decode the target code stream file, and the display effect of the decoded image output by this target decoding model meets the display requirements.

[0101] In one embodiment, the second decoding model that meets the second goal is obtained by adjusting the parameters of the second basic decoding model based on the loss, and the loss is calculated based on the difference between the second reconstructed picture and the second training picture using the target loss function; wherein at least one of the target training set and the target loss function including the second training picture matches the second goal.

[0102] In the picture decoding method of this embodiment, at least one of a specific training set and a specific loss function is set to meet the training target required for the trained decoding model. The second basic decoding model is trained with the target training set and the target loss function so that the trained second decoding model meets the specific training target, that is, meets the second target. Among them, at least one of the target training set and the target loss function matches the second target. It should be noted that matching can be understood as a specific target training set and / or a specific target loss function set to meet the second target so that the second target met by the second decoding model is different from the training target met by other decoding models. The target training set does not match the second target, which does not mean that the second decoding model that meets the second target cannot be trained by the target training set, but refers to the target training set used to train the second decoding model that meets the second target. It may be the same as the training set for training other decoding models, and this target training set is not a specific training set set to meet the second target.

[0103] Specifically, on the basis that the first encoding model encodes the second training picture in the target training set to obtain the target bitstream file, and the second basic decoding model decodes the target bitstream file to obtain the second reconstructed picture, the target loss function is used to calculate the loss based on the difference between the second reconstructed picture and the second training picture. Then, based on the loss, the second basic decoding model is adjusted to obtain a second decoding model that meets the second target. The second decoding model trained by the target training set and the target loss function meets the second target.

[0104] In this implementation, at least one of a specific training set and a specific loss function can be set to meet the training target requirements of the trained decoding model. Furthermore, different decoding models are obtained by training with different training sets and / or different loss functions, so that different decoding models meet different training targets, which is conducive to achieving optimization iteration between decoding models.

[0105] In one embodiment,

[0106] When the second target is that the first target attribute of the decoded picture generated by the second decoding model meets the expected requirements of the picture, and the first target attribute refers to any picture attribute that affects the visual effect of the picture, the target loss function includes information for judging the difference between the first target attribute of the second reconstructed picture generated by the second basic decoding model and the expected requirements of the picture.

[0107] The image attributes include but are not limited to pixels, resolution, size, color, bit depth, hue, saturation, brightness, sharpness, contrast, color channels, and image levels. The first target attribute is any image attribute that can affect the visual effect of the image, such as saturation, brightness, sharpness, resolution, etc. The image expectation requirement refers to the expected value of the first target attribute of the preset decoded image.

[0108] In order to determine which of the following needs to be met by setting a specific training set, setting a specific loss function, setting a specific training set and a specific loss function to meet the training target required by the trained decoding model, it is necessary to first determine the type of training target that the decoding model needs to meet. The first target type is to further optimize the first target attribute of the decoded picture generated by the decoding model. For example, the first target attribute of the decoded picture generated by the first decoding model before the decoding model optimization iteration is the initial value, and then it is expected that the first target attribute of the decoded picture generated by the decoding model will meet the expected requirements of the picture through the decoding model optimization iteration, that is, the preset expected value. Among them, the expected value is not equal to the initial value. For this type of target, setting a specific training set has little effect on the display effect of the reconstructed picture obtained by decoding. However, setting a specific loss function can make it possible to pay more attention to the loss of the reconstructed picture obtained by decoding about the first target attribute when calculating the loss using the loss function, which has a greater impact on the display effect of the reconstructed picture. Based on this, when the second target is that the first target attribute of the decoded picture generated by the second decoding model meets the expected requirements of the picture, a specific target loss function can be set to meet the needs of the second decoding model to meet the second target. Among them, the target training set used to train the second decoding model may be the same as or different from the training sets used to train other decoding models. This embodiment does not impose special restrictions on the target training set.

[0109] Specifically, the target loss function is L=R+a*D+b_1*D_1. Wherein, R represents the bitrate; D is used to judge the degree of difference between the second reconstructed picture and the second training picture, which can be expressed as a mean-square error (MSE) function, a multi-scale structural similarity (MS-SSIM) function, etc.; a is a coefficient greater than 0 and less than 1. Since the training target that the decoding model meets is usually based on the decoded picture obtained by decoding being similar to the initial picture before encoding and decoding, that is, the training target that the decoding model meets must at least meet the similarity between the decoded picture and the initial picture. Based on this, "R+a*D" in the target loss function is still used to calculate the first loss based on the difference between the second reconstructed picture and the second training picture. In particular, D_1 is used to judge the degree of difference between the first target attribute of the second reconstructed picture and the expected requirement of the picture; b_1 is a coefficient matching D_1. That is, "b_1*D_1" in the target loss function is used to calculate the second loss based on the difference between the first target attribute of the second reconstructed picture and the expected requirement of the picture. Exemplarily, if the first target attribute is sharpness, D_1 in the target loss function is used to determine the degree of difference between the sharpness of the second reconstructed image and the expected value of sharpness, and "b_1*D_1" in the target loss function is used to calculate the second loss based on the difference between the sharpness of the second reconstructed image and the expected value of sharpness. Using the target loss function, the first loss is calculated based on the difference between the second reconstructed image and the second training image, and the second loss is calculated based on the difference between the first target attribute of the second reconstructed image and the expected requirement of the image, to obtain the final comprehensive loss. Then, the second basic decoding model is adjusted using the final comprehensive loss to obtain the second decoding model. Since the second loss of the second reconstructed image with respect to the first target attribute can be focused on when calculating the comprehensive loss using the target loss function, the parameters of the second basic decoding model are adjusted based on the first loss and the second loss, so that the first loss and the second loss converge. The second decoding model finally trained can meet the training goal of the first target attribute of the decoded image to meet the expected requirements of the image.

[0110] In this embodiment, when the second target is that the first target attribute of the decoded picture generated by the second decoding model meets the expected requirements of the picture, a specific target loss function is set to meet the requirement that the second decoding model meets the second target. Specifically, in addition to including information for judging the difference between the second reconstructed picture and the second training picture, the target loss function also includes information for judging the difference between the first target attribute of the second reconstructed picture and the expected requirements of the picture. Through this specific target loss function, the second decoding model that meets the second target can be trained to achieve optimization iteration of the decoding model.

[0111] In one embodiment,

[0112] When the second target is that the second target attribute of the area where the target object is located in the decoded picture generated by the second decoding model meets the local expected requirement, and the second target attribute refers to any image attribute that affects the visual effect of the picture, the target loss function includes information for detecting the target object in the second reconstructed picture generated by the second basic decoding model, and information for judging the difference between the second target attribute of the area where the target object is located in the second reconstructed picture and the local expected requirement, the number of second training pictures with target content in the target training set is greater than or equal to the first value, and having target content refers to having a preset number or more of target objects.

[0113] The area where the target object is located can be understood as the area where target objects such as portraits and animals are located. The picture attributes can refer to the explanation in the above embodiment. The second target attribute is any picture attribute that can affect the visual effect of the picture, such as saturation, brightness, sharpness, resolution, etc. The local expected requirement refers to the expected value of the second target attribute of the area where the target object is located in the preset decoded picture.

[0114] In order to determine which of the following needs to be met by setting a specific training set, setting a specific loss function, setting a specific training set and a specific loss function to meet the training target required by the trained decoding model, it is necessary to first determine the type of training target that the decoding model needs to meet. The second target type is to further optimize the second target attribute of the area where the target object is located in the decoded picture generated by the decoding model, improve the attention to the details of the area where the target object is located, and make the details of the area where the target object is located clearer. For example, the second target attribute of the area where the target object is located in the decoded picture generated by the first decoding model before the decoding model optimization iteration is the initial value, and then it is expected that the second target attribute of the area where the target object is located in the decoded picture generated by the decoding model will meet the local expected requirements, that is, reach the preset expected value through the decoding model optimization iteration. Among them, the expected value is not equal to the initial value. For this type of target, a specific loss function can be set so that when the loss is calculated using the loss function, more attention can be paid to the loss of the second target attribute of the area where the target object is located in the reconstructed picture obtained by decoding. The specific loss function has a great influence on the display effect of the reconstructed picture. In addition, in order to strengthen the role of a specific loss function, so that in the process of training the decoding model, the loss function can better focus on the loss of the second target attribute of the reconstructed image in the area where the target object is located, the reconstructed image needs to contain the target object, and the more target objects, the better. Based on this, it is also necessary to set a specific training set so that the training images in the training set contain more target objects, so as to obtain a reconstructed image containing more target objects after encoding and decoding. Based on this, when the second target is that the second target attribute of the area where the target object is located in the decoded image generated by the second decoding model meets the local expected requirements, a specific target training set and a specific target loss function can be set to meet the requirements of the second decoding model meeting the second target.

[0115] Specifically, let the target loss function be L=R+a*D+b_2*D_2. Referring to the specific discussion in the aforementioned embodiment, it can be known that "R+a*D" in the target loss function is still used to calculate the first loss based on the difference between the second reconstructed image and the second training image. In particular, D_2 is used to detect the target object in the second reconstructed image, and when the target object is detected, determine the degree of difference between the second target attribute of the area where the target object in the second reconstructed image is located and the local expected requirement; b_2 is a coefficient matching D_2. That is, "b_2*D_2" in the target loss function is used to calculate the second loss based on the difference between the second target attribute of the area where the target object in the second reconstructed image is located and the local expected requirement. Exemplarily, if the target object is a portrait and the second target attribute is resolution, D_2 in the target loss function is used to detect the portrait in the second reconstructed image, and when the portrait is detected, the difference between the resolution of the region where the portrait is located in the second reconstructed image and the expected resolution value is determined, and "b_2*D_2" in the target loss function is used to calculate the second loss based on the difference between the resolution of the region where the portrait is located in the second reconstructed image and the expected resolution value. Using the target loss function, the first loss is calculated based on the difference between the second reconstructed image and the second training image, and the second loss is calculated based on the difference between the second target attribute of the region where the target object is located in the second reconstructed image and the local expected requirement, to obtain the final comprehensive loss. Then, the second basic decoding model is adjusted using the final comprehensive loss to obtain the second decoding model. Since the second loss of the second target attribute of the region where the target object is located in the second reconstructed image can be focused on when calculating the comprehensive loss using the target loss function, the compression loss of the region where the target object is located can be reduced. Then, the parameters of the second basic decoding model are adjusted based on the first loss and the second loss, so that the first loss and the second loss converge. The second decoding model finally obtained through training can meet the training goal of ensuring that the second target attribute of the area where the target object in the decoded image is located meets the local expected requirements.

[0116] Accordingly, the number of second training pictures with target content in the target training set is made greater than or equal to the first value, and having target content means having a preset number or more of target objects. In order to make the second reconstructed picture obtained after encoding and decoding the second training picture contain the target object, and contain more target objects, thereby strengthening the effect of the target loss function, the larger the first value and the preset number, the more advantageous it is.

[0117] In this embodiment, when the second target is that the second target attribute of the area where the target object in the decoded picture generated by the second decoding model is located meets the local expected requirements, a specific target loss function and a specific target training set are set to meet the requirement that the second decoding model meets the second target. Specifically, in addition to including information for judging the difference between the second reconstructed picture and the second training picture, the target loss function also includes information for detecting the target object in the second reconstructed picture, and information for judging the difference between the second target attribute of the area where the target object in the second reconstructed picture is located and the local expected requirements. The number of second training pictures with target content in the target training set is greater than or equal to the first value, and having target content means having a preset number or more of target objects. Through this specific target loss function and specific target loss function, a second decoding model that meets the second target can be trained to achieve optimization iteration of the decoding model.

[0118] In one embodiment,

[0119] When the second target is that when the third target attribute of the initial picture is lower than the preset requirement, the second decoding model decodes the bitstream file corresponding to the initial picture, and the similarity between the obtained decoded picture and the initial picture reaches the expected similarity value, and the third target attribute refers to any picture attribute that affects the visual effect of the picture, the number of second training pictures in the target training set whose third target attribute is lower than the preset requirement is greater than or equal to the second value.

[0120] The image attributes can refer to the explanation in the above embodiment, and the third target attribute is any image attribute that can affect the visual effect of the image, such as saturation, brightness, sharpness, resolution, etc. The preset requirement refers to the preset basic value of the third target attribute of the initial image.

[0121] In order to determine which of the following needs to be met by setting a specific training set, setting a specific loss function, setting a specific training set and a specific loss function to meet the training target required by the trained decoding model, it is necessary to first determine the type of training target that the decoding model needs to meet. The third target type is that when the third target attribute of the initial picture is lower than the preset requirement, the second decoding model decodes the bitstream file corresponding to the initial picture, and the similarity between the obtained decoded picture and the initial picture can still reach the expected similarity value. For example, when the third target attribute of the initial picture is lower than the preset requirement, the decoded picture generated by the first decoding model before the decoding model optimization iteration fails to reach the expected similarity value with the initial picture, that is, the decoding effect deviation. It is further expected that through the decoding model optimization iteration, when the third target attribute of the initial picture is lower than the preset requirement, the similarity between the decoded picture generated by the decoding model and the initial picture can still reach the expected similarity value, that is, it can still have a good decoding effect. For this type of target, setting a specific loss function has little effect on the display effect of the reconstructed picture obtained by decoding. However, by setting a specific training set so that the training set contains more training pictures whose third target attribute is lower than the preset requirement, the basic decoding model can be trained to decode special code stream files. Special code stream files refer to code stream files corresponding to training pictures whose third target attribute is lower than the preset requirement. Furthermore, a decoding model can be trained to better decode the code stream file corresponding to the initial picture whose third target attribute is lower than the preset requirement and generate a decoded picture similar to the initial picture. Among them, similarity with the initial picture means that the similarity with the initial picture reaches the expected similarity value. Based on this, when the second goal is that when the third target attribute of the initial picture is lower than the preset requirement, the second decoding model decodes the code stream file corresponding to the initial picture, and the similarity between the obtained decoded picture and the initial picture reaches the expected similarity value, a specific target training set can be set to meet the second decoding model to meet the second goal. Among them, the target loss function used to train the second decoding model may be the same as the loss function of training other decoding models, or it may be different. This embodiment does not impose special restrictions on the target loss function.

[0122] Specifically, the number of second training pictures whose third target attribute in the target training set is lower than the preset requirement is greater than or equal to the second value. In order to better train the second basic decoding model's decoding ability for the code stream files corresponding to the second training pictures whose third target attribute is lower than the preset requirement, the larger the second value, the better. Exemplarily, if the third target attribute is brightness, the number of second training pictures whose brightness in the target training set is lower than the preset requirement is greater than or equal to the second value, so as to train the second basic decoding model's decoding ability for the code stream files corresponding to the second training pictures with low brightness. Based on this, the trained second decoding model can better decode the code stream files corresponding to the initial pictures whose third target attribute is lower than the preset requirement, and generate decoded pictures similar to the initial pictures. That is, the second decoding model can meet the second target.

[0123] In this embodiment, when the second target is that the third target attribute of the initial picture is lower than the preset requirement, the second decoding model decodes the code stream file corresponding to the initial picture, and when the similarity between the obtained decoded picture and the initial picture reaches the expected similarity value, a specific target training set is set to meet the requirement that the second decoding model meets the second target. Specifically, the number of second training pictures whose third target attribute in the target training set is lower than the preset requirement is greater than or equal to the second value. Through this specific target training set, a second decoding model that meets the second target can be trained to achieve optimization iteration of the decoding model.

[0124] Taking the initial encoding model as E_a and the initial decoding model as D_a as an example, E_a and D_a are obtained through joint training, and E_a and D_a meet the initial goal. When there is no need to solve the technical problem of incompatibility of the bitstream file encoded by E_a, that is, when there is no need to enable multiple different decoding models to decode the bitstream file encoded by E_a, the encoding and decoding model can be optimized and iterated based on more and more complex training goals. Specifically, in order to enable the encoding and decoding model to meet the iteration goals, such as further improving the speed of encoding and decoding the initial picture to obtain the decoded picture, further improving the detection accuracy of each object in the decoded picture, etc., the encoding and decoding model can be optimized and iterated to obtain the iterative encoding model E_c and the iterative decoding model D_c. E_c and D_c are obtained through joint training. Although D_c cannot decode the bitstream file encoded by E_a, E_c and D_c meet the iteration goals. Furthermore, taking E_c as the first encoding model and D_c as the first decoding model, on the basis of the optimization iteration of the above encoding and decoding model, the decoding model can still be optimized and iterated based on D_c while E_c remains unchanged, and D_c1, D_c2, D_c3, etc. can be obtained by training with different training objectives. Among them, the second decoding model is any one of D_c1, D_c2, D_c3, etc. E_c and D_c are obtained by optimizing and iterating E_a and D_a, and then E_c and D_c1, E_c and D_c2, E_c and D_c3, etc. obtained by optimizing E_c and D_c may meet both the iteration objective and the second objective, and D_c1, D_c2, D_c3, etc. can also decode the bitstream file encoded by E_c. In addition, compared with E_a before the optimization iteration, the bitstream file generated by E_c encoding has higher performance. Based on this, when the second decoding model after optimization iteration decodes the bitstream file generated by E_c encoding, it is beneficial to improve the performance of the decoded image obtained, and to enable the decoded image to better achieve the image display effect corresponding to the second goal.

[0125] In the process of optimizing and iterating to obtain E_c and D_c that meet the iteration target based on E_a and D_a, at least one of the training set, loss function and basic codec model used matches the iteration target. The training picture in the training set is input into the basic codec model to obtain a reconstructed picture by encoding and decoding, and then the loss function is used to calculate the loss based on the difference between the reconstructed picture and the training picture to adjust the parameters of the basic codec model to obtain E_c and D_c that meet the iteration target. For example, when the iteration target type is that the codec model further improves the detection accuracy of each object in the decoded picture, a specific loss function is set to calculate the loss based on the detection accuracy of each object in the reconstructed picture obtained by encoding and decoding; when the iteration target is that the codec model further improves the speed of encoding and decoding the initial picture to obtain the decoded picture, the number of parameters of the basic codec model is reduced to speed up the training speed, thereby speeding up the picture reconstruction speed.

[0126] The embodiment of the present application also provides a video frame display method. The video frame display method is based on the picture decoding method described in the aforementioned embodiment, and uses the video frame as the picture to be compressed, decodes the target code stream file formed after the video frame is encoded, obtains the target picture, and then displays the target picture. Based on this, for the explanation of the nouns in the video frame display method provided in the embodiment of the present application, the specific discussion of each step, the derivation of the technical effect, and the example of a feasible embodiment, please refer to the description of the corresponding content in the aforementioned picture decoding method embodiment. It will not be repeated in the embodiment of the present application.

[0127] The video frame display method can be applied to any application scenario that requires displaying video frames, such as live video, video on demand, etc. The embodiment of the present application does not limit the application scenario of the video frame display method. Figure 4 As shown, the video frame display method includes the following steps S401 to S403.

[0128] S401, obtaining a target bitstream file formed after the video frame is encoded. The target bitstream file is obtained by encoding the video frame using a first encoding model, and the first encoding model and the first decoding model are trained in the following manner: a first bitstream file output after the basic encoding model encodes the first training picture is used as an input of the first basic decoding model, the first bitstream file is decoded using the first basic decoding model to obtain a first reconstructed picture, and based on the difference between the first reconstructed picture and the first training picture, the basic encoding model and the first basic decoding model are adjusted to obtain the first encoding model and the first decoding model that meets the first target.

[0129] S402, using the second decoding model to decode the target bitstream file to obtain the target picture. The second decoding model is obtained by training in the following manner: using the second bitstream file output after the first encoding model encodes the second training picture as the input of the second basic decoding model, decoding the second bitstream file using the second basic decoding model to obtain a second reconstructed picture, and adjusting the parameters of the second basic decoding model based on the difference between the second reconstructed picture and the second training picture to obtain a second decoding model that meets the second goal; wherein the first goal of training the first basic decoding model is different from the second goal of training the second basic decoding model.

[0130] S403, display the target image.

[0131] In this embodiment, a target bitstream file obtained by encoding a video frame using a first coding model is obtained, and the target bitstream file is decoded using a second decoding model to obtain a target picture, so that in the optimization and iteration process of the coding and decoding model, multiple different decoding models can decode the bitstream file generated by the same coding model, and the second decoding model of the second version can decode the target bitstream file generated by the first coding model of the first version. Furthermore, it avoids the situation where the video frame and its corresponding target bitstream file are invalid when the bitstream files are incompatible, the second decoding model has difficulty decoding the target bitstream file generated by the first coding model, and the first decoding model cannot be used. The display of the video frame is achieved by decoding to obtain the target picture and displaying the target picture.

[0132] The embodiment of the present application also provides a decoding model training method. The second decoding model that meets the second goal obtained by the present decoding model training method is inferred and applied in the picture decoding method described in the aforementioned embodiment. That is, the present decoding model training method discusses the training process of the second decoding model, and the picture decoding method described in the aforementioned embodiment discusses the reasoning and application process of the second decoding model, and when discussing the reasoning and application process of the second decoding model, the training process of the second decoding model is also introduced. Based on this, for the explanation of the nouns in the decoding model training method provided in the embodiment of the present application, the specific discussion of each step, the derivation of the technical effects, and the examples of feasible embodiments, please refer to the description of the corresponding content in the aforementioned picture decoding method embodiment. It will not be repeated in the embodiment of the present application.

[0133] like Figure 5 As shown, the decoding model training method includes the following steps S501 to S504.

[0134] S501, obtaining a target training set.

[0135] S502, using the first coding model to encode the second training picture in the target training set to obtain a second bitstream file. The first coding model and the first decoding model are obtained by training in the following manner: using the first bitstream file output after the basic coding model encodes the first training picture as the input of the first basic decoding model, decoding the first bitstream file using the first basic decoding model to obtain a first reconstructed picture, and adjusting the basic coding model and the first basic decoding model based on the difference between the first reconstructed picture and the first training picture to obtain the first coding model and the first decoding model that meets the first target.

[0136] S503: Input the second code stream file into the second basic decoding model to obtain a second reconstructed picture.

[0137] S504, adjusting the parameters of the second basic decoding model based on the difference between the second reconstructed picture and the second training picture to obtain a second decoding model that meets the second goal. The first goal of training the first basic decoding model is different from the second goal of training the second basic decoding model.

[0138] In this embodiment, the first coding model and the first decoding model are obtained through joint training, and then, when the first coding model remains unchanged, the second decoding model is obtained based on the first coding model training. Among them, the first goal met by the first decoding model is different from the second goal met by the second decoding model. The decoding model training method in this implementation not only realizes the optimization iteration of the coding and decoding model, but also realizes that during the optimization and iteration process of the coding and decoding model, multiple different decoding models can decode the code stream file generated by the same coding model, and the second decoding model of the second version can decode the target code stream file generated by the first coding model of the first version.

[0139] In one embodiment, when the second target is that the first target attribute of the decoded picture generated by the second decoding model meets the expected requirements of the picture, and the first target attribute refers to any picture attribute that affects the visual effect of the picture, such as Figure 6 As shown, the above step S504, based on the difference between the second reconstructed picture and the second training picture, adjusts the parameters of the second basic decoding model to obtain a second decoding model that meets the second goal, including the following steps S601 to S602.

[0140] S601, calculating the loss based on the difference between the second reconstructed image and the second training image, and the difference between the first target attribute of the second reconstructed image and the expected requirement of the image.

[0141] Refer to the specific discussion of the embodiment of the aforementioned picture decoding method regarding "when the second target is that the first target attribute of the decoded picture generated by the second decoding model meets the expected requirements of the picture, and the first target attribute refers to any picture attribute that affects the visual effect of the picture, the target loss function contains information for judging the difference between the first target attribute of the second reconstructed picture generated by the second basic decoding model and the expected requirements of the picture." It can be seen that when the second target is that the first target attribute of the decoded picture generated by the second decoding model meets the expected requirements of the picture, a specific target loss function is set to meet the requirement that the second decoding model meets the second target. Using the target loss function, the loss is calculated based on the difference between the second reconstructed picture and the second training picture and the difference between the first target attribute of the second reconstructed picture and the expected requirements of the picture.

[0142] S602: Adjust parameters of the second basic decoding model based on the loss to obtain a second decoding model that meets the second objective.

[0143] When the second goal is that the first target attribute of the decoded picture generated by the second decoding model meets the expected requirements of the picture, the loss is calculated based on the above two differences, which not only considers the similarity between the second reconstructed picture and the second training picture, but also pays attention to the loss of the second reconstructed picture with respect to the first target attribute. Then, the second basic decoding model is adjusted using the loss calculated based on the above two differences to converge the loss, and the second decoding model finally trained can meet the second goal that the first target attribute of the decoded picture meets the expected requirements of the picture.

[0144] In one embodiment, when the second target is that the second target attribute of the area where the target object is located in the decoded picture generated by the second decoding model meets the local expected requirement, and the second target attribute refers to any picture attribute that affects the visual effect of the picture, such as Figure 7 As shown, the above step S504, based on the difference between the second reconstructed picture and the second training picture, adjusts the parameters of the second basic decoding model to obtain a second decoding model that meets the second goal, including the following steps S701 to S702.

[0145] S701, calculating the loss based on the difference between the second reconstructed image and the second training image, and the difference between the second target attribute of the region where the target object in the second reconstructed image is located and the local expected requirement.

[0146] Refer to the specific discussion of the embodiment of the aforementioned picture decoding method regarding "when the second target is that the second target attribute of the area where the target object in the decoded picture generated by the second decoding model is located meets the local expected requirement, the second target attribute refers to any picture attribute that affects the visual effect of the picture, the target loss function includes information for detecting the target object in the second reconstructed picture generated by the second basic decoding model, and information for judging the difference between the second target attribute of the area where the target object in the second reconstructed picture is located and the local expected requirement, the number of second training pictures with target content in the target training set is greater than or equal to the first value, and having target content refers to having a preset number or more of target objects". It can be seen from this that in order to meet the requirement that the second decoding model meets the second target, a specific target loss function needs to be set. Using the target loss function, the loss is calculated based on the difference between the second reconstructed picture and the second training picture, and the difference between the second target attribute of the area where the target object in the second reconstructed picture is located and the local expected requirement.

[0147] S702: Adjust parameters of the second basic decoding model based on the loss to obtain a second decoding model that meets the second objective.

[0148] When the second target is that the second target attribute of the area where the target object is located in the decoded picture generated by the second decoding model meets the local expected requirements, the loss is calculated based on the above two differences, which not only considers the similarity between the second reconstructed picture and the second training picture, but also pays attention to the loss of the second target attribute of the area where the target object is located in the second reconstructed picture. Then, the second basic decoding model is adjusted using the loss calculated based on the above two differences to make the loss converge, and the second decoding model finally trained can meet the second target that the second target attribute of the area where the target object is located in the decoded picture meets the local expected requirements.

[0149] like Figure 8As shown, the embodiment of the present application further provides a picture decoding device 800. The picture decoding device 800 includes a code stream acquisition module 801 and a code stream decoding module 802. The code stream acquisition module 801 is used to acquire a target code stream file, which is obtained by encoding a to-be-compressed picture using a first coding model, and the first coding model and the first decoding model are trained in the following manner: the first code stream file output after the basic coding model encodes the first training picture is used as the input of the first basic decoding model, the first code stream file is decoded using the first basic decoding model to obtain a first reconstructed picture, and based on the difference between the first reconstructed picture and the first training picture, the basic coding model and the first basic decoding model are adjusted to obtain the first coding model and the first decoding model that meets the first target. The bitstream decoding module 802 is used to decode the target bitstream file by using the second decoding model to obtain the target picture, where the second decoding model is trained in the following manner: the second bitstream file output after the first encoding model encodes the second training picture is used as the input of the second basic decoding model, the second bitstream file is decoded by using the second basic decoding model to obtain a second reconstructed picture, and based on the difference between the second reconstructed picture and the second training picture, the second basic decoding model is adjusted to obtain a second decoding model that meets the second goal; wherein the first goal of training the first basic decoding model is different from the second goal of training the second basic decoding model.

[0150] In one embodiment, the second goal of training the second basic decoding model matches the display effect of the decoded picture output by the second decoding model. The above-mentioned picture decoding device 800 also includes a demand acquisition module and a model determination module. The demand acquisition module is used to obtain the display requirements for the display effect of the target picture, and the target picture refers to the picture obtained by encoding the target code stream file. The model determination module is used to determine that the second decoding model is used as the model for decoding the target code stream file when the display effect of the decoded picture output by the second decoding model meets the display requirements.

[0151] In one embodiment, the second decoding model that meets the second goal is obtained by adjusting the parameters of the second basic decoding model based on the loss, and the loss is calculated based on the difference between the second reconstructed picture and the second training picture using the target loss function; wherein at least one of the target training set and the target loss function including the second training picture matches the second goal.

[0152] In one embodiment, when the second target is that the first target attribute of the decoded picture generated by the second decoding model meets the expected requirements of the picture, and the first target attribute refers to any picture attribute that affects the visual effect of the picture, the target loss function includes information for judging the difference between the first target attribute of the second reconstructed picture generated by the second basic decoding model and the expected requirements of the picture.

[0153] In one of the embodiments, when the second target is that the second target attribute of the area where the target object is located in the decoded picture generated by the second decoding model meets the local expected requirement, and the second target attribute refers to any picture attribute that affects the visual effect of the picture, the target loss function includes information for detecting the target object in the second reconstructed picture generated by the second basic decoding model, and information for judging the difference between the second target attribute of the area where the target object is located in the second reconstructed picture and the local expected requirement, the number of second training pictures with target content in the target training set is greater than or equal to the first value, and having target content means having a preset number or more of target objects.

[0154] In one of the embodiments, when the second target is that the third target attribute of the initial image is lower than the preset requirement, the second decoding model decodes the bitstream file corresponding to the initial image, and the similarity between the obtained decoded image and the initial image reaches the expected similarity value, and the third target attribute refers to any image attribute that affects the visual effect of the image, the number of second training images in the target training set whose third target attribute is lower than the preset requirement is greater than or equal to the second value.

[0155] For the explanation of each term, please refer to the relevant description in the aforementioned picture decoding method embodiment, which will not be elaborated here.

[0156] It should be noted that the specific execution process of the above-mentioned picture decoding device 800 can be found in Figures 1 to 3 The specific description of the illustrated embodiment will not be repeated here.

[0157] Each module in the above-mentioned picture decoding device 800 can be implemented in whole or in part by software, hardware or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0158] like Fig. 9As shown, the embodiment of the present application also provides a video frame display device 900. The video frame display device 900 includes a target code stream acquisition module 901, a target image generation module 902 and a target image display module 903. The target code stream acquisition module 901 is used to obtain a target code stream file formed after the video frame is encoded. The target code stream file is obtained by encoding the video frame using a first encoding model. The first encoding model and the first decoding model are obtained by training in the following manner: the first code stream file output after the basic encoding model encodes the first training image is used as the input of the first basic decoding model, the first code stream file is decoded using the first basic decoding model to obtain a first reconstructed image, and based on the difference between the first reconstructed image and the first training image, the basic encoding model and the first basic decoding model are adjusted to obtain the first encoding model and the first decoding model that meets the first target. The target picture generation module 902 is used to decode the target code stream file using the second decoding model to obtain the target picture. The second decoding model is obtained by training in the following manner: the second code stream file output after the first coding model encodes the second training picture is used as the input of the second basic decoding model, the second code stream file is decoded using the second basic decoding model to obtain a second reconstructed picture, and based on the difference between the second reconstructed picture and the second training picture, the second basic decoding model is adjusted to obtain a second decoding model that meets the second goal; wherein the first goal of training the first basic decoding model is different from the second goal of training the second basic decoding model. The target picture display module 903 is used to display the target picture.

[0159] For the explanation of each term, please refer to the relevant description in the aforementioned video frame display method embodiment, which will not be elaborated here.

[0160] It should be noted that the specific execution process of the video frame display device 900 can be found in Figure 4 The specific description of the illustrated embodiment will not be repeated here.

[0161] Each module in the video frame display device 900 can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0162] like Fig.10As shown, the embodiment of the present application also provides a decoding model training device 1000. The decoding model training device 1000 includes a training set acquisition module 1001, a code stream generation module 1002, a picture reconstruction module 1003 and a model generation module 1004. The training set acquisition module 1001 is used to obtain a target training set. The code stream generation module 1002 is used to encode the second training picture in the target training set using the first coding model to obtain a second code stream file. The first coding model and the first decoding model are obtained by training in the following manner: the first code stream file output after the basic coding model encodes the first training picture is used as the input of the first basic decoding model, the first code stream file is decoded using the first basic decoding model to obtain a first reconstructed picture, and based on the difference between the first reconstructed picture and the first training picture, the basic coding model and the first basic decoding model are adjusted to obtain the first coding model and the first decoding model that meets the first target. The picture reconstruction module 1003 is used to input the second code stream file into the second basic decoding model to obtain a second reconstructed picture. The model generation module 1004 is used to adjust the parameters of the second basic decoding model based on the difference between the second reconstructed image and the second training image to obtain a second decoding model that meets the second goal; wherein the first goal of training the first basic decoding model is different from the second goal of training the second basic decoding model.

[0163] In one of the embodiments, when the second target is that the first target attribute of the decoded picture generated by the second decoding model meets the expected requirements of the picture, and the first target attribute refers to any picture attribute that affects the visual effect of the picture, the above-mentioned model generation module 1004 is also used to calculate the loss based on the difference between the second reconstructed picture and the second training picture, and the difference between the first target attribute of the second reconstructed picture and the expected requirements of the picture; adjust the parameters of the second basic decoding model based on the loss to obtain a second decoding model that meets the second target.

[0164] In one of the embodiments, when the second target is that the second target attribute of the area where the target object is located in the decoded picture generated by the second decoding model meets the local expected requirement, and the second target attribute refers to any image attribute that affects the visual effect of the picture, the above-mentioned model generation module 1004 is also used to calculate the loss based on the difference between the second reconstructed picture and the second training picture, and the difference between the second target attribute of the area where the target object is located in the second reconstructed picture and the local expected requirement; adjust the parameters of the second basic decoding model based on the loss to obtain a second decoding model that meets the second target.

[0165] For the explanation of each term, please refer to the relevant description in the aforementioned decoding model training method embodiment, which will not be elaborated here.

[0166] It should be noted that the specific execution process of the above decoding model training device 1000 can be found in Figures 5 to 7 The specific description of the illustrated embodiment will not be repeated here.

[0167] Each module in the above decoding model training device 1000 can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module above.

[0168] like Fig.11 As shown, the embodiment of the present application further provides a computer device 1100. Exemplarily, the computer device 1100 may include a processor 1101, a communication interface 1102, a communication bus 1103, and a memory 1104. Specifically, the computer device 1100 may include:

[0169] At least one processor 1101, such as a CPU, at least one communication interface 1102, a memory 1104, and at least one communication bus 1103. The communication bus 1103 is used to realize the connection and communication between these components. The communication interface 1102 may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface or a Bluetooth interface, etc.). The memory 1104 may be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk memory. The memory 1104 may optionally be at least one storage device located away from the aforementioned processor 1101. Fig.11 As shown, the memory 1104 as a computer storage medium may include an operating system and program instructions.

[0170] Exemplarily, the processor 1101 may be used to implement the above Figure 8 The steps or methods performed by the code stream acquisition module 801 and the code stream decoding module 802 in FIG.

[0171] It is understandable that the above method is only an example, and the processor 1101 and other modules in the above computer device 1100 may also cooperate to perform the above method. Figure 8 The steps or methods performed by the bit stream acquisition module 801 and the bit stream decoding module 802 in the embodiment are not limited in this document.

[0172] exist Fig.11 In the computer device 1100 shown, the processor 1101 may be used to load program instructions stored in the memory 1104 and specifically perform the following operations:

[0173] Obtaining a target bitstream file, where the target bitstream file is obtained by encoding the to-be-compressed picture using a first coding model, and the first coding model and the first decoding model are trained in the following manner: using a first bitstream file output after encoding a first training picture using a basic coding model as an input of a first basic decoding model, decoding the first bitstream file using the first basic decoding model to obtain a first reconstructed picture, and adjusting parameters of the basic coding model and the first basic decoding model based on a difference between the first reconstructed picture and the first training picture to obtain a first coding model and a first decoding model that meets a first target;

[0174] The target bitstream file is decoded by using the second decoding model to obtain a target picture, wherein the second decoding model is trained in the following manner: a second bitstream file output after the second training picture is encoded by the first encoding model is used as an input of the second basic decoding model, the second bitstream file is decoded by using the second basic decoding model to obtain a second reconstructed picture, and based on the difference between the second reconstructed picture and the second training picture, the second basic decoding model is adjusted to obtain a second decoding model that meets the second target;

[0175] Among them, the first goal of training the first basic decoding model is different from the second goal of training the second basic decoding model.

[0176] For the explanation of each term, please refer to the relevant description in the aforementioned picture decoding method embodiment, which will not be elaborated here.

[0177] It should be noted that the specific implementation process can be found in Figures 1 to 3 The specific description of the illustrated embodiment will not be repeated here.

[0178] Exemplarily, the processor 1101 may be used to implement the above Fig. 9 The steps or methods performed by the target code stream acquisition module 901, the target image generation module 902 and the target image display module 903 in the embodiment.

[0179] It is understandable that the above method is only an example, and the processor 1101 and other modules in the above computer device 1100 may also cooperate to perform the above method. Fig. 9 The steps or methods executed by the target code stream acquisition module 901, the target image generation module 902 and the target image display module 903 are not limited in this document.

[0180] exist Fig.11 In the computer device 1100 shown, the processor 1101 may be used to load program instructions stored in the memory 1104 and specifically perform the following operations:

[0181] A target bitstream file formed after the video frame is encoded is obtained, where the target bitstream file is obtained by encoding the video frame using a first encoding model, and the first encoding model and the first decoding model are trained in the following manner: a first bitstream file output after the first training picture is encoded by the basic encoding model is used as an input of the first basic decoding model, the first bitstream file is decoded using the first basic decoding model to obtain a first reconstructed picture, and based on the difference between the first reconstructed picture and the first training picture, the basic encoding model and the first basic decoding model are adjusted to obtain the first encoding model and the first decoding model that meets the first target;

[0182] The target bitstream file is decoded by using the second decoding model to obtain a target picture, wherein the second decoding model is trained in the following manner: a second bitstream file output after the second training picture is encoded by the first encoding model is used as an input of the second basic decoding model, the second bitstream file is decoded by using the second basic decoding model to obtain a second reconstructed picture, and based on the difference between the second reconstructed picture and the second training picture, the second basic decoding model is adjusted to obtain a second decoding model that meets the second goal; wherein the first goal of training the first basic decoding model is different from the second goal of training the second basic decoding model;

[0183] Display the target image.

[0184] For the explanation of each term, please refer to the relevant description in the aforementioned video frame display method embodiment, which will not be elaborated here.

[0185] It should be noted that the specific implementation process can be found in Figure 4 The specific description of the illustrated embodiment will not be repeated here.

[0186] Exemplarily, the processor 1101 may be used to implement the above Fig.10 The steps or methods performed by the training set acquisition module 1001, the code stream generation module 1002, the image reconstruction module 1003 and the model generation module 1004.

[0187] It is understandable that the above method is only an example, and the processor 1101 and other modules in the above computer device 1100 may also cooperate to perform the above method. Fig.10 The steps or methods performed by the training set acquisition module 1001, the code stream generation module 1002, the image reconstruction module 1003 and the model generation module 1004 are not limited in this document.

[0188] exist Fig.11 In the computer device 1100 shown, the processor 1101 may be used to load program instructions stored in the memory 1104 and specifically perform the following operations:

[0189] Get the target training set;

[0190] The second training picture in the target training set is encoded by using the first encoding model to obtain a second bitstream file, and the first encoding model and the first decoding model are obtained by training in the following manner: a first bitstream file output after the first training picture is encoded by the basic encoding model is used as an input of the first basic decoding model, the first bitstream file is decoded by using the first basic decoding model to obtain a first reconstructed picture, and based on the difference between the first reconstructed picture and the first training picture, the basic encoding model and the first basic decoding model are adjusted to obtain the first encoding model and the first decoding model that meets the first target;

[0191] Inputting the second code stream file into the second basic decoding model to obtain a second reconstructed picture;

[0192] Based on the difference between the second reconstructed picture and the second training picture, adjusting the parameters of the second basic decoding model to obtain a second decoding model that meets the second goal;

[0193] Among them, the first goal of training the first basic decoding model is different from the second goal of training the second basic decoding model.

[0194] For the explanation of each term, please refer to the relevant description in the aforementioned decoding model training method embodiment, which will not be elaborated here.

[0195] It should be noted that the specific implementation process can be found in Figures 5 to 7 The specific description of the illustrated embodiment will not be repeated here.

[0196] The present application also provides a computer-readable storage medium, which can store a plurality of instructions, and the instructions are suitable for being loaded and executed by a processor as described above. Figures 1 to 7 The method steps of the embodiment shown in the figure can be found in the specific execution process. Figures 1 to 7 The specific description of the illustrated embodiment will not be repeated here.

[0197] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.

[0198] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any fusion thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0199] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

[0200] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0201] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A picture decoding method, characterized in that: The method comprises: Obtaining a target bitstream file, wherein the target bitstream file is obtained by encoding a to-be-compressed picture using a first coding model, wherein the first coding model and the first decoding model are obtained by training in the following manner: using a first bitstream file output after encoding a first training picture using a basic coding model as an input of a first basic decoding model, decoding the first bitstream file using the first basic decoding model to obtain a first reconstructed picture, and adjusting parameters of the basic coding model and the first basic decoding model based on a difference between the first reconstructed picture and the first training picture to obtain a first coding model and the first decoding model that meets a first target; Decoding the target bitstream file by using a second decoding model to obtain a target picture, wherein the second decoding model is trained in the following manner: using a second bitstream file output after encoding a second training picture by the first encoding model as an input of a second basic decoding model, decoding the second bitstream file by using the second basic decoding model to obtain a second reconstructed picture, and adjusting parameters of the second basic decoding model based on a difference between the second reconstructed picture and the second training picture to obtain the second decoding model that meets the second target; Among them, the first goal of training the first basic decoding model is different from the second goal of training the second basic decoding model.

2. The method according to claim 1, characterized in that The second objective of training the second basic decoding model matches the display effect of the decoded picture output by the second decoding model; Before decoding the target code stream file using the second decoding model to obtain the target picture, the method further includes: Obtaining a display requirement for a display effect of the target picture, where the target picture refers to a picture obtained by encoding the target bitstream file; When the display effect of the decoded picture output by the second decoding model meets the display requirement, it is determined that the second decoding model is used as the model for decoding the target code stream file.

3. The method according to claim 1 or 2, characterized in that: The second decoding model that meets the second goal is obtained by adjusting the parameters of the second basic decoding model based on the loss, where the loss is calculated based on the difference between the second reconstructed picture and the second training picture using a target loss function; Among them, at least one of the target training set including the second training picture and the target loss function matches the second target.

4. The method according to claim 3, characterized in that When the second target is that the first target attribute of the decoded picture generated by the second decoding model meets the expected requirements of the picture, and the first target attribute refers to any picture attribute that affects the visual effect of the picture, the target loss function includes information for judging the difference between the first target attribute of the second reconstructed picture generated by the second basic decoding model and the expected requirements of the picture.

5. The method according to claim 3, characterized in that: When the second target is that the second target attribute of the area where the target object is located in the decoded picture generated by the second decoding model meets the local expected requirement, and the second target attribute refers to any picture attribute that affects the visual effect of the picture, the target loss function includes information for detecting the target object in the second reconstructed picture generated by the second basic decoding model, and information for judging the difference between the second target attribute of the area where the target object is located in the second reconstructed picture and the local expected requirement, the number of second training pictures with target content in the target training set is greater than or equal to the first value, and the target content refers to target objects with a preset number or more.

6. The method according to claim 3, characterized in that When the second target is that when the third target attribute of the initial picture is lower than the preset requirement, the second decoding model decodes the code stream file corresponding to the initial picture, and the similarity between the obtained decoded picture and the initial picture reaches the expected similarity value, and the third target attribute refers to any picture attribute that affects the visual effect of the picture, the number of second training pictures in the target training set whose third target attribute is lower than the preset requirement is greater than or equal to the second value.

7. A video frame display method, characterized in that: The method comprises: Obtaining a target bitstream file formed after a video frame is encoded, wherein the target bitstream file is obtained by encoding the video frame using a first encoding model, wherein the first encoding model and the first decoding model are trained in the following manner: using a first bitstream file output after a first training picture is encoded by a basic encoding model as an input of a first basic decoding model, decoding the first bitstream file using the first basic decoding model to obtain a first reconstructed picture, and adjusting parameters of the basic encoding model and the first basic decoding model based on a difference between the first reconstructed picture and the first training picture to obtain a first encoding model and the first decoding model that meets a first target; Decoding the target bitstream file by using a second decoding model to obtain a target picture, wherein the second decoding model is trained in the following manner: using a second bitstream file output after encoding a second training picture by the first encoding model as an input of a second basic decoding model, decoding the second bitstream file by using the second basic decoding model to obtain a second reconstructed picture, and adjusting parameters of the second basic decoding model based on a difference between the second reconstructed picture and the second training picture to obtain the second decoding model that meets a second goal; wherein the first goal of training the first basic decoding model is different from the second goal of training the second basic decoding model; The target image is displayed.

8. A decoding model training method, characterized in that: The method comprises: Get the target training set; Using a first coding model to encode a second training picture in the target training set to obtain a second bitstream file, wherein the first coding model and the first decoding model are obtained by training in the following manner: using a first bitstream file output after encoding the first training picture by a basic coding model as an input of a first basic decoding model, decoding the first bitstream file by using the first basic decoding model to obtain a first reconstructed picture, and adjusting parameters of the basic coding model and the first basic decoding model based on a difference between the first reconstructed picture and the first training picture to obtain the first coding model and the first decoding model that meet the first target; Inputting the second code stream file into a second basic decoding model to obtain a second reconstructed picture; Based on the difference between the second reconstructed picture and the second training picture, adjusting the parameters of the second basic decoding model to obtain a second decoding model that meets the second objective; Among them, the first goal of training the first basic decoding model is different from the second goal of training the second basic decoding model.

9. The method according to claim 8, characterized in that When the second target is that a first target attribute of a decoded picture generated by the second decoding model meets an expected requirement of the picture, and the first target attribute refers to any picture attribute that affects a visual effect of the picture, adjusting the parameters of the second basic decoding model based on a difference between the second reconstructed picture and the second training picture to obtain a second decoding model that meets the second target includes: Calculating a loss based on a difference between the second reconstructed image and the second training image, and a difference between the first target attribute of the second reconstructed image and the image expectation requirement; The second basic decoding model is adjusted based on the loss to obtain the second decoding model that meets the second goal.

10. The method according to claim 8, characterized in that When the second target is that the second target attribute of the area where the target object is located in the decoded picture generated by the second decoding model meets the local expected requirement, and the second target attributes all refer to any picture attributes that affect the visual effect of the picture, adjusting the parameters of the second basic decoding model based on the difference between the second reconstructed picture and the second training picture to obtain the second decoding model that meets the second target includes: Calculate the loss based on the difference between the second reconstructed picture and the second training picture, and the difference between the second target attribute of the region where the target object in the second reconstructed picture is located and the local expected requirement; The second basic decoding model is adjusted based on the loss to obtain the second decoding model that meets the second goal.

11. A picture decoding device, characterized in that: The device comprises: a bitstream acquisition module, configured to acquire a target bitstream file, wherein the target bitstream file is obtained by encoding the to-be-compressed picture using a first encoding model, wherein the first encoding model and the first decoding model are obtained by training in the following manner: a first bitstream file output after encoding the first training picture using a basic encoding model is used as an input of a first basic decoding model, the first bitstream file is decoded using the first basic decoding model to obtain a first reconstructed picture, and based on a difference between the first reconstructed picture and the first training picture, the basic encoding model and the first basic decoding model are adjusted to obtain a first encoding model and the first decoding model that meets a first target; a bitstream decoding module, configured to decode the target bitstream file using a second decoding model to obtain a target picture, wherein the second decoding model is trained in the following manner: a second bitstream file output after encoding the second training picture using the first encoding model is used as an input of a second basic decoding model, the second bitstream file is decoded using the second basic decoding model to obtain a second reconstructed picture, and based on a difference between the second reconstructed picture and the second training picture, the second basic decoding model is adjusted to obtain the second decoding model that meets the second target; Among them, the first goal of training the first basic decoding model is different from the second goal of training the second basic decoding model.

12. A video frame display device, characterized in that: The device comprises: a target bitstream acquisition module, configured to acquire a target bitstream file formed after a video frame is encoded, wherein the target bitstream file is obtained by encoding the video frame using a first encoding model, wherein the first encoding model and the first decoding model are obtained by training in the following manner: a first bitstream file output after encoding a first training picture using a basic encoding model is used as an input of a first basic decoding model, the first bitstream file is decoded using the first basic decoding model to obtain a first reconstructed picture, and based on a difference between the first reconstructed picture and the first training picture, the basic encoding model and the first basic decoding model are adjusted to obtain a first encoding model and the first decoding model that meets a first target; a target picture generating module, configured to decode the target bitstream file by using a second decoding model to obtain a target picture, wherein the second decoding model is obtained by training in the following manner: using the second bitstream file output after encoding the second training picture by the first encoding model as an input of the second basic decoding model, decoding the second bitstream file by using the second basic decoding model to obtain a second reconstructed picture, and adjusting the parameters of the second basic decoding model based on the difference between the second reconstructed picture and the second training picture to obtain the second decoding model that meets a second goal; wherein the first goal of training the first basic decoding model is different from the second goal of training the second basic decoding model; The target image display module is used to display the target image.

13. A decoding model training device, characterized in that: The device comprises: A training set acquisition module is used to acquire a target training set; a bitstream generation module, configured to encode a second training picture in the target training set by using a first encoding model to obtain a second bitstream file, wherein the first encoding model and the first decoding model are obtained by training in the following manner: using a first bitstream file output after encoding the first training picture by a basic encoding model as an input of a first basic decoding model, decoding the first bitstream file by using the first basic decoding model to obtain a first reconstructed picture, and adjusting parameters of the basic encoding model and the first basic decoding model based on a difference between the first reconstructed picture and the first training picture to obtain the first encoding model and the first decoding model that meets the first target; A picture reconstruction module, used for inputting the second code stream file into a second basic decoding model to obtain a second reconstructed picture; A model generation module, configured to adjust the parameters of the second basic decoding model based on the difference between the second reconstructed picture and the second training picture to obtain a second decoding model that meets the second objective; Among them, the first goal of training the first basic decoding model is different from the second goal of training the second basic decoding model.

14. A computer device, characterized in that: include: A memory and a processor, wherein the memory stores program instructions; when the program instructions are executed by the processor, the processor executes the method as claimed in any one of claims 1 to 10.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program; when the computer program is run on one or more processors, the method according to any one of claims 1 to 10 is executed.

16. A computer program product, characterized in that The computer program product comprises a computer program or instructions; when the computer program or instructions are run on a computer, the computer is caused to perform the method according to any one of claims 1 to 10.

Citation Information

Patent Citations

  • Channel information acquisition method and device and related equipment

    CN114726413A

  • Systems, methods, and bitstream structures for hybrid feature video bitstreams and decoders

    CN117356092A

  • Model training method and device, facial expression migration method and device, equipment and medium

    CN117540789A

  • Processing method and device and storage medium

    CN117546159A

  • Method and system for performing domain adaptation of end-to-end automatic speech recognition model

    US11315548B1