Parameter updating method of video coding and decoding network and related equipment

By iteratively updating the model parameters of the encoder and decoder using reconstructed frames in the video codec network, the problem of poor adaptability of the training set in the prior art is solved, which improves the video codec efficiency and reduces the code rate consumption .

CN118945365APending Publication Date: 2024-11-12HUAWEI TECH CO LTD +1
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202310539205.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-12
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The training set of existing video codec networks is difficult to adapt to the image video data in actual scenes, resulting in limited matching ability of the trained parameters to the content of the encoded video image frame, limiting the encoding efficiency of the video image frame.

Method used

By using reconstructed frames as training data in the encoding device and the decoder device, the model parameters of the encoder and the decoder are iteratively updated, and the code rate and distortion measurement are optimized using time domain information, thereby improving the matching of model parameters to the content of the encoded frame to be encoded. to avoid the additional bit rate consumption caused by transmitting the decoder model parameters.

Benefits of technology

The matching degree of the model parameters of the video codec to the encoded frame content is improved, the encoding and decoding efficiency is improved, and the code rate consumption of the encoding terminal to the decoding terminal is reduced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118945365A_ABST
    Figure CN118945365A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses a parameter updating method of a video coding and decoding network and related equipment, and the method comprises the steps: employing a first encoder in the video coding network to carry out the coding of a reconstructed frame, and obtaining a first coding result; the reconstructed frame is obtained by decoding a code stream output by a first encoder through a first decoder in the video encoding network; the code stream is obtained by encoding the encoded frame through the first encoder; obtaining a first distortion measure and a first code rate of the reconstructed frame according to the first decoder and the first coding result; determining a first loss according to the first distortion measurement and the first code rate; and updating parameters of the first encoder and the first decoder according to the first loss to obtain a second encoder and a second decoder. According to the embodiment of the invention, the matching degree of the model parameters of the video codec to the content of the to-be-coded frame can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of video coding and decoding technology, and in particular to a parameter updating method and related equipment of a video coding and decoding network. Background Art

[0002] At present, most of the research related to artificial intelligence will talk about neural networks. The training of neural networks is usually based on large-scale data sets, and the trained fixed parameters can be used for reasoning in actual scenes. The distribution of image and video data has the characteristics of spatial and temporal variability, that is, image and video data is a non-stationary source. Although there are many open source image and video databases that can be used as training sets for image and video codec networks, these training sets are difficult to adapt to the image and video data to be encoded in actual scenes in terms of image resolution, image content and image quality, resulting in the limited matching ability of the trained parameters to the content of the video image frames to be encoded, which limits the encoding efficiency of the video image frames. Summary of the invention

[0003] The embodiments of the present application provide a parameter updating method and related devices for a video codec network, which can improve the matching degree of the model parameters of the video codec to the content of the frame to be encoded, thereby facilitating improving the encoding efficiency of the frame to be encoded.

[0004] In a first aspect, an embodiment of the present application provides a method for updating parameters of a video coding network, which is applied to a coding device, and the method includes:

[0005] The reconstructed frame is encoded by a first encoder in the video encoding network to obtain a first encoding result; the reconstructed frame is obtained by decoding a bit stream output by the first encoder by a first decoder in the video encoding network; the bit stream is obtained by encoding the encoded frame by the first encoder;

[0006] Obtaining a first distortion metric and a first bit rate of a reconstructed frame according to the first decoder and the first encoding result;

[0007] Determining a first loss based on the first distortion metric and the first bit rate;

[0008] The parameters of the first encoder and the first decoder are updated according to the first loss to obtain the second encoder and the second decoder.

[0009] It can be seen that in the embodiment of the present application, before encoding the frame to be encoded, the encoding device uses the reconstructed frame before the frame to be encoded as training data, obtains the model parameters of the reconstructed frame as the initial model parameters of the encoder and decoder, and uses the optimization of the bit rate and distortion metric of the reconstructed frame as the objective function, and iteratively updates the parameters of the encoder and decoder through the first loss. In this way, by using time domain information to update the model parameters of the encoder and decoder of the video coding network, it is possible to improve the matching degree of the model parameters to the content of the frame to be encoded, which is beneficial to improving the encoding efficiency of the frame to be encoded. In addition, the encoding device and the decoding device synchronously and independently use the reconstructed frame to update the model parameters of the encoder and decoder. The encoding end does not need to transmit the model parameters of the decoder to the decoding end, which can avoid the additional bit rate consumption caused by the encoding end transmitting the decoder model parameters to the decoding end.

[0010] In a possible implementation, the method further includes:

[0011] Obtaining a second loss for encoding and decoding a frame to be encoded using an initial encoder and an initial decoder;

[0012] Obtaining a third loss of encoding and decoding the frame to be encoded using the second encoder and the second decoder;

[0013] According to the second loss and the third loss, it is determined whether to reset the parameters of the second encoder and the second decoder to the parameters of the initial encoder and the parameters of the initial decoder, respectively.

[0014] In this implementation, an initial encoder and an initial decoder are used to encode and decode the frame to be encoded, and a corresponding second loss is obtained, and a second encoder and a second decoder are used to encode and decode the frame to be encoded, and a corresponding third loss is obtained. The second loss and the third loss can determine the matching degree and encoding and decoding efficiency of the pre-trained model parameters and the updated model parameters for the frame to be encoded, so that a model parameter reset decision based on encoding and decoding efficiency can be executed, and then parameters that are more adapted to the frame to be encoded and have higher encoding and decoding efficiency are selected for the second encoder and the second decoder.

[0015] In a possible implementation, determining whether to reset parameters of the second encoder and the second decoder to parameters of the initial encoder and the initial decoder, respectively, according to the second loss and the third loss, includes:

[0016] If the second loss is less than the third loss, resetting the parameters of the second encoder and the second decoder to the parameters of the initial encoder and the parameters of the initial decoder respectively;

[0017] If the second loss is greater than or equal to the third loss, the parameters of the second encoder and the second decoder are not reset.

[0018] In this implementation, the second loss is less than the third loss, indicating that the initial encoder and the initial decoder can encode and decode the to-be-encoded frame with a lower distortion metric and bit rate, and therefore, the parameters of the second encoder and the second decoder can be reset to the parameters of the initial encoder and the parameters of the initial decoder, respectively. The second loss is greater than or equal to the third loss, indicating that the second encoder and the second decoder can encode and decode the to-be-encoded frame with a lower distortion metric and bit rate, and therefore, the parameters of the second encoder and the second decoder are not reset.

[0019] In a possible implementation, the method further includes:

[0020] Obtaining a fourth loss of encoding and decoding the to-be-encoded frame by using the reset second encoder and the second decoder;

[0021] updating the parameters of the reset second encoder according to the fourth loss;

[0022] or,

[0023] Obtain a fifth loss for encoding and decoding the frame to be encoded using the second encoder and the second decoder;

[0024] Parameters of the second encoder are updated according to the fifth loss.

[0025] In this implementation, the model parameters of the encoder are updated at the encoding end using the spatial domain information of the frame to be encoded, which can further improve the matching degree of the encoder to the content of the frame to be encoded.

[0026] In a possible implementation, the method further includes:

[0027] Sending the flag bit to the decoding device so that the decoding device selects parameters of the video decoding network according to the flag bit;

[0028] Among them, when the flag bit is a first value, the flag bit is used to indicate that the parameters of the second encoder and the second decoder are reset to the parameters of the initial encoder and the parameters of the initial decoder respectively; when the flag bit is a second value, the flag bit is used to indicate that the parameters of the second encoder and the second decoder are not reset.

[0029] In this implementation, a flag can be used to represent the result of the model parameter reset decision based on coding efficiency on the encoding device side. The encoding device sends the flag to the decoding device so that after the decoding device completes the model parameter update based on time domain information, it can choose to use the updated parameters or the pre-trained model parameters according to the decision result on the encoding device side.

[0030] In a possible implementation, the first encoding result includes an encoded bitstream of a reconstructed frame; and obtaining a first distortion metric and a first bitrate of the reconstructed frame according to the first decoder and the first encoding result includes:

[0031] Decoding the coded bitstream using a first decoder to obtain a decoded frame of the reconstructed frame;

[0032] determining a first distortion metric based on the reconstructed frame and a decoded frame of the reconstructed frame;

[0033] A first bit rate is determined according to the encoded bit stream.

[0034] In this implementation, the encoded code stream of the reconstructed frame is decoded to obtain a corresponding decoded frame; based on the decoded frame and the reconstructed frame, a first distortion metric for encoding and decoding the reconstructed frame using the first encoder and the first decoder can be determined; based on the first distortion metric and the first bit rate of the encoded code stream, a first loss for encoding and decoding the reconstructed frame using the first encoder and the first decoder can be obtained, and the encoding device can update the parameters of the first encoder and the first decoder according to the first loss.

[0035] In a possible implementation, the second encoder and the second decoder are obtained by iteratively updating the parameters of the first encoder and the first decoder N times according to the first loss; N is an integer greater than 0;

[0036] The method further includes:

[0037] The number of iterations N is sent to the decoding device, so that the decoding device iteratively updates the parameters of the video decoding network according to the number of iterations N.

[0038] In this implementation, the encoding device can send a preset number of iterations N to the decoding device, so that the decoding device also iterates the reconstructed frame N times, thereby updating the model parameters of the network on the decoding device side to improve the degree of matching between the model parameters on the decoding device side and the content of the frame to be encoded.

[0039] In a second aspect, an embodiment of the present application provides a parameter updating method of a video decoding network, which is applied to a decoding device; the method includes:

[0040] The reconstructed frame is encoded by a third encoder in the video decoding network to obtain a second encoding result; the reconstructed frame is obtained by decoding a bit stream output by a third encoder by a third decoder in the video encoding network; the bit stream is obtained by encoding the encoded frame by the third encoder;

[0041] Obtaining a second distortion metric and a second bit rate of the reconstructed frame according to the third decoder and the second encoding result;

[0042] determining a sixth loss based on the second distortion metric and the second bit rate;

[0043] The parameters of the third encoder and the third decoder are updated according to the sixth loss to obtain a fourth encoder and a fourth decoder.

[0044] It can be seen that in the embodiment of the present application, before encoding the frame to be encoded, the decoding device uses the reconstructed frame before the frame to be encoded as training data, obtains the model parameters of the reconstructed frame as the initial model parameters of the encoder and decoder, and optimizes the bit rate and distortion metric of the reconstructed frame as the objective function, and iteratively updates the parameters of the encoder and decoder through the sixth loss. In this way, the model parameters of the encoder and decoder of the video decoding network are updated using time domain information, which can improve the matching degree of the model parameters to the content of the frame to be encoded, which is conducive to improving the decoding efficiency of the frame to be encoded. In addition, the decoding device and the encoding device synchronously and independently use the reconstructed frame to update the model parameters of the encoder and decoder. The decoding end does not need to receive the model parameters of the decoder transmitted by the encoding end, which can avoid the additional bit rate consumption caused by the encoder end transmitting the decoder model parameters to the decoding end.

[0045] In a possible implementation, the method further includes:

[0046] Receive the flag bit sent by the encoding device;

[0047] If the flag bit is the first value, the parameters of the fourth encoder and the fourth decoder are reset to the parameters of the initial encoder and the parameters of the initial decoder respectively;

[0048] If the flag bit is the second value, the parameters of the fourth encoder and the fourth decoder are not reset.

[0049] In this implementation, the decoding device selects parameters for the fourth encoder and the fourth decoder according to the flag bit sent by the encoding device, and can obtain parameters that are more adapted to the frame to be encoded and have higher encoding and decoding efficiency.

[0050] In a possible implementation, the second encoding result includes an encoded code stream of the reconstructed frame; and obtaining a second distortion metric and a second code rate of the reconstructed frame according to the third decoder and the second encoding result includes:

[0051] Decoding the encoded bitstream using a third decoder to obtain a decoded frame of the reconstructed frame;

[0052] determining a second distortion metric based on the reconstructed frame and a decoded frame of the reconstructed frame;

[0053] A second bit rate is determined according to the encoded bit stream.

[0054] In this implementation, the encoded code stream of the reconstructed frame is decoded to obtain a corresponding decoded frame; based on the decoded frame and the reconstructed frame, a second distortion metric for encoding and decoding the reconstructed frame using a third encoder and a third decoder can be determined; based on the second distortion metric and the second bit rate of the encoded code stream, a sixth loss for encoding and decoding the reconstructed frame using the third encoder and the third decoder can be obtained, and the decoding device can update the parameters of the third encoder and the third decoder according to the sixth loss.

[0055] In a possible implementation manner, updating parameters of the third encoder and the third decoder according to the sixth loss to obtain a fourth encoder and a fourth decoder includes:

[0056] Iteratively updating the parameters of the third encoder and the third decoder according to the sixth loss until the number of iterations reaches M times, fixing the parameters updated at the Mth iteration, and obtaining a fourth encoder and a fourth decoder; M is an integer greater than 0;

[0057] or,

[0058] Receiving the number of iterations N sent by the encoding device;

[0059] Iteratively update the parameters of the third encoder and the third decoder according to the sixth loss until the number of iterations reaches N times, fix the parameters updated in the Nth iteration, and obtain the fourth encoder and the fourth decoder.

[0060] In this implementation, the decoding device may iteratively update the parameters of the third encoder and the third decoder according to the number of iterations M preset on its own side, or iteratively update the parameters of the third encoder and the third decoder according to the number of iterations N preset on the encoding device side.

[0061] In a third aspect, an embodiment of the present application provides a parameter updating device for a video coding network, which is applied to a coding device; the device includes a first coding unit, a first decoding unit and a first updating unit;

[0062] A first encoding unit is used to encode the reconstructed frame by using a first encoder in the video encoding network to obtain a first encoding result; the reconstructed frame is obtained by decoding a code stream output by the first encoder by a first decoder in the video encoding network; the code stream is obtained by encoding the encoded frame by the first encoder;

[0063] A first decoding unit, configured to obtain a first distortion metric and a first bit rate of a reconstructed frame according to the first decoder and the first encoding result;

[0064] The first updating unit is used to determine a first loss according to a first distortion metric and a first bit rate; and update parameters of a first encoder and a first decoder according to the first loss to obtain a second encoder and a second decoder.

[0065] In a possible implementation manner, the first updating unit is further configured to:

[0066] Obtaining a second loss for encoding and decoding a frame to be encoded using an initial encoder and an initial decoder;

[0067] Obtaining a third loss of encoding and decoding the frame to be encoded using the second encoder and the second decoder;

[0068] According to the second loss and the third loss, it is determined whether to reset the parameters of the second encoder and the second decoder to the parameters of the initial encoder and the parameters of the initial decoder, respectively.

[0069] In a possible implementation manner, in determining whether to reset the parameters of the second encoder and the second decoder to the parameters of the initial encoder and the parameters of the initial decoder respectively according to the second loss and the third loss, the first updating unit is specifically used to:

[0070] If the second loss is less than the third loss, resetting the parameters of the second encoder and the second decoder to the parameters of the initial encoder and the parameters of the initial decoder respectively;

[0071] If the second loss is greater than or equal to the third loss, the parameters of the second encoder and the second decoder are not reset.

[0072] In a possible implementation manner, the first updating unit is further configured to:

[0073] Obtaining a fourth loss of encoding and decoding the to-be-encoded frame by using the reset second encoder and the second decoder;

[0074] updating the parameters of the reset second encoder according to the fourth loss;

[0075] or,

[0076] Obtaining a fifth loss for encoding and decoding the frame to be encoded using the second encoder and the second decoder;

[0077] Parameters of the second encoder are updated according to the fifth loss.

[0078] In a possible implementation manner, the first updating unit is further configured to:

[0079] Sending the flag bit to the decoding device so that the decoding device selects parameters of the video decoding network according to the flag bit;

[0080] Among them, when the flag bit is a first value, the flag bit is used to indicate that the parameters of the second encoder and the second decoder are reset to the parameters of the initial encoder and the parameters of the initial decoder respectively; when the flag bit is a second value, the flag bit is used to indicate that the parameters of the second encoder and the second decoder are not reset.

[0081] In a possible implementation, the first encoding result includes an encoded code stream of a reconstructed frame; in terms of obtaining a first distortion metric and a first code rate of the reconstructed frame according to the first decoder and the first encoding result, the first decoding unit is specifically configured to:

[0082] Decoding the coded bitstream using a first decoder to obtain a decoded frame of the reconstructed frame;

[0083] determining a first distortion metric based on the reconstructed frame and a decoded frame of the reconstructed frame;

[0084] A first bit rate is determined according to the encoded bit stream.

[0085] In a possible implementation, the second encoder and the second decoder are obtained by iteratively updating parameters of the first encoder and the first decoder N times according to the first loss; N is an integer greater than 0; and the first updating unit is further used for:

[0086] The number of iterations N is sent to the decoding device, so that the decoding device iteratively updates the parameters of the video decoding network according to the number of iterations N.

[0087] It can be understood that since the method embodiment and the device embodiment are different presentation forms of the same technical concept, the content of the first aspect of the embodiment of the present application should be synchronously adapted to the third aspect of the embodiment of the present application, and can achieve the same or similar beneficial effects, which will not be repeated here.

[0088] In a fourth aspect, an embodiment of the present application provides a parameter updating device for a video decoding network, which is applied to a decoding device; the device includes a second encoding unit, a second decoding unit, and a second updating unit;

[0089] The second encoding unit is used to encode the reconstructed frame by using the third encoder in the video decoding network to obtain a second encoding result; the reconstructed frame is obtained by decoding the code stream output by the third encoder by the third decoder in the video encoding network; the code stream is obtained by encoding the encoded frame by the third encoder;

[0090] A second decoding unit, configured to obtain a second distortion metric and a second bit rate of the reconstructed frame according to the third decoder and the second encoding result;

[0091] The second updating unit is used to determine a sixth loss according to the second distortion metric and the second bit rate; and update parameters of the third encoder and the third decoder according to the sixth loss to obtain a fourth encoder and a fourth decoder.

[0092] In a possible implementation manner, the second updating unit is further configured to:

[0093] Receive the flag bit sent by the encoding device;

[0094] If the flag bit is the first value, the parameters of the fourth encoder and the fourth decoder are reset to the parameters of the initial encoder and the parameters of the initial decoder respectively;

[0095] If the flag bit is the second value, the parameters of the fourth encoder and the fourth decoder are not reset.

[0096] In a possible implementation, the second encoding result includes an encoded code stream of the reconstructed frame; in terms of obtaining the second distortion metric and the second code rate of the reconstructed frame according to the third decoder and the second encoding result, the second decoding unit is specifically configured to:

[0097] Decoding the encoded bitstream using a third decoder to obtain a decoded frame of the reconstructed frame;

[0098] determining a second distortion metric based on the reconstructed frame and a decoded frame of the reconstructed frame;

[0099] A second bit rate is determined according to the encoded bit stream.

[0100] In a possible implementation manner, in terms of updating the parameters of the third encoder and the third decoder according to the sixth loss to obtain the fourth encoder and the fourth decoder, the second updating unit is specifically used to:

[0101] Iteratively updating the parameters of the third encoder and the third decoder according to the sixth loss until the number of iterations reaches M times, fixing the parameters updated at the Mth iteration, and obtaining a fourth encoder and a fourth decoder; M is an integer greater than 0;

[0102] or,

[0103] Receiving the number of iterations N sent by the encoding device;

[0104] Iteratively update the parameters of the third encoder and the third decoder according to the sixth loss until the number of iterations reaches N times, fix the parameters updated in the Nth iteration, and obtain the fourth encoder and the fourth decoder.

[0105] It can be understood that the content of the second aspect of the embodiment of the present application should be synchronously adapted to the fourth aspect of the embodiment of the present application, and can achieve the same or similar beneficial effects, which will not be repeated here.

[0106] In a fifth aspect, an embodiment of the present application provides an encoding device, comprising a processor and a memory, wherein the memory is used to store one or more programs, the program comprising instructions for executing the method in the first aspect as described above.

[0107] In a sixth aspect, an embodiment of the present application provides a decoding device, comprising a processor and a memory, wherein the memory is used to store one or more programs, the program comprising instructions for executing the method in the second aspect as described above.

[0108] In a seventh aspect, an embodiment of the present application provides a chip, comprising: a processor, configured to call and run a computer program from a memory, so that a device equipped with the chip executes a method as described in any one of the first or second aspects above.

[0109] In an eighth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed, it implements the method of any one of the first aspect or the second aspect mentioned above.

[0110] In a ninth aspect, an embodiment of the present application provides a computer program product, the computer program product comprising: a computer program code, when the computer program code is executed, it implements the method as described in any one of the first aspect or the second aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0111] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0112] Figure 1 A schematic diagram of a parameter updating method provided in an embodiment of the present application;

[0113] Figure 2 A schematic diagram of another parameter updating method provided in an embodiment of the present application;

[0114] Figure 3 A schematic diagram of another parameter updating method provided in an embodiment of the present application;

[0115] Figure 4 A system architecture diagram of a video encoding end and a decoding end provided in an embodiment of the present application;

[0116] Figure 5 A schematic diagram of a flow chart of a method for updating parameters of a video coding network provided in an embodiment of the present application;

[0117] Figure 6 A schematic diagram of a flow chart of a method for updating parameters of a video decoding network provided in an embodiment of the present application;

[0118] Figure 7 A schematic diagram of a flow chart of another method for updating parameters of a video coding network provided in an embodiment of the present application;

[0119] Figure 8 A schematic diagram of a flow chart of another method for updating parameters of a video decoding network provided in an embodiment of the present application;

[0120] Fig. 9 An overall schematic diagram of a parameter updating method for a video encoding and decoding network provided in an embodiment of the present application;

[0121] Fig.10 A schematic diagram of the structure of a parameter updating device for a video coding network provided in an embodiment of the present application;

[0122] Fig.11 A schematic diagram of the structure of a parameter updating device for a video decoding network provided in an embodiment of the present application;

[0123] Fig.12 A schematic diagram of the structure of an encoding device provided in an embodiment of the present application;

[0124] Fig.13 A schematic diagram of the structure of a decoding device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0125] The terms "first", "second", "third" and "fourth" etc. in the specification and claims of the present application and the drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0126] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0127] The terms "component", "module", "system", etc. used in this specification are used to represent computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process, a processor, an object, an executable file, an execution thread, a program and / or a computer running on a processor. By way of illustration, both applications running on a terminal device and a terminal device can be components. One or more components can reside in a process and / or an execution thread, and a component can be located on a computer and / or distributed between two or more computers. In addition, these components can be executed from various computer-readable media having various data structures stored thereon. Components can, for example, communicate through local and / or remote processes according to signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system and / or a network, such as the Internet interacting with other systems through signals).

[0128] The following is a brief introduction to the relevant technical background of this application to facilitate a better understanding of this application.

[0129] One of the related technologies provides a partial network parameter update scheme, which takes the video frame to be encoded as input, obtains the initial value of the latent variable through the pre-trained video encoding neural network, and then updates the latent variable without adjusting any other parameters to obtain higher video encoding efficiency as the optimization goal. The latent variable update operation is mainly performed on the encoder side, and it does not update the decoder parameters, such as Figure 1 As shown, only some network model parameters are updated, which may result in limited matching ability of the model parameters to the picture content of the encoded video frame, thereby limiting the encoding efficiency of the video frame.

[0130] Among them, X t Indicates the frame to be encoded, E m 、E h represents the encoder, represents the quantized representation of the encoded sub-features, represents the quantized representation of the encoded image features, and To optimize the data required by the function, D m , D h represents the corresponding decoder, P represents the probability distribution estimation module, that is, using Characteristics of the coded frames Estimating probability distribution Represents the decoded reconstructed frame of the frame to be encoded. Figure 1 The dotted box is the part where the network model parameters are updated.

[0131] The second related technology provides an encoder model parameter update scheme, which does not update the latent variables, such as Figure 2 As shown, only the encoder model parameters are updated at the encoding end, and then the updated encoder model parameters are used to encode the frame to be encoded. It also does not update the model parameters of the decoder. Although it does not introduce additional transmission costs, it may still result in limited matching ability of the model parameters to the content of the video frame to be encoded. Figure 2 The dotted box is the part where the network model parameters are updated.

[0132] The third related technology provides an encoder and decoder model parameter update scheme, such as Figure 3 As shown in the figure, the scheme updates the model parameters of the encoder and decoder at the encoding end, and then uses the updated encoder model parameters to encode the frame to be encoded into a bitstream, and transmits the bitstream and the updated decoder model parameters to the decoding end. The decoding end first decodes and reconstructs the updated decoder model parameters, and then uses the decoder model parameters to decode and reconstruct the current frame. The advantage of this scheme is that all model parameters are updated, but it takes the video frame sequence as input, and all model parameters of the video are updated frame by frame. At the same time, the decoder model parameters updated frame by frame need to be transmitted to the decoding end, which consumes a lot of bit rate and reduces the overall coding efficiency. One solution is to use a long video sequence to update all model parameters once, so that each frame in the sequence shares the same model parameters. However, this may lead to higher transmission delays, and the matching degree of the model parameters to the content of each frame will also decrease to a certain extent.

[0133] Among them, X0, X1, …, n represents a sequence of video frames, Represents the decoded and reconstructed frame sequence corresponding to the video frame sequence, represents the quantized representation of the encoded sub-features of each frame, Represents the quantized representation of the encoded image features of each frame, and To optimize the data required by the function, Represents the probability distribution estimate of the features of each frame. Among them, Figure 3 The dotted box is the part where the network model parameters are updated.

[0134] It can be seen that in neural network video encoding and decoding, how to improve the matching degree between network model parameters and video frame content, and avoid the additional bit rate consumption caused by transmitting decoder model parameters, is an urgent problem that needs to be solved.

[0135] The following is a brief description of the architecture to which this application can be applied.

[0136] See also Figure 4 , Figure 4The system architecture diagram of a video encoding end and a decoding end provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the encoding end includes a model parameter updating module based on time domain information, a model parameter reset decision module based on coding efficiency, and a model parameter updating module based on spatial domain information. The decoding end adopts a symmetrical architecture with the encoding end, including a model parameter updating module based on time domain information and a decoding mode switching module.

[0137] The encoder uses the reconstructed frame before the frame to be encoded as training data, and the model parameter update module based on time domain information is configured to use the model parameters of the reconstructed frame as the initial model parameters of the encoder and decoder. For example, the reconstructed frame is obtained by reconstructing the bitstream of the encoded frame, then the initial model parameters of the encoder are the model parameters for encoding the encoded frame, and the initial model parameters of the decoder are the model parameters for reconstructing the bitstream of the encoded frame. The initial model parameters of the encoder and decoder are iteratively updated with the reconstructed frame as input, and after a preset number of iterations, the updated model parameters of the encoder and decoder are obtained.

[0138] The model parameter reset decision module based on coding efficiency is configured to use the updated model parameters to encode and decode the frames to be coded, and to use the pre-trained model parameters of the encoder and decoder to encode and decode the frames to be coded, and then compare the coding and decoding efficiency between the updated model parameters and the pre-trained model parameters, so as to decide whether to reset the model parameters of the encoder and decoder to the pre-trained model parameters according to the coding and decoding efficiency, and the coding end device transmits the decision result to the decoding end.

[0139] The model parameter update module based on spatial domain information is configured to use the updated model parameters to encode and decode the frame to be encoded if the model parameters of the updated encoder and decoder are not reset, obtain the value of the optimization function based on the encoding and decoding information, and continue to update the model parameters of the encoder according to the value of the optimization function. If the model parameters of the updated encoder and decoder are reset to pre-trained model parameters, the reset pre-trained model parameters are used to encode and decode the frame to be encoded, obtain the value of the optimization function based on the encoding and decoding information, and continue to update the model parameters of the encoder according to the value of the optimization function. In this way, the model parameters of the encoder are updated using the spatial domain information of the frame to be encoded itself at the encoding end. After the model parameters based on spatial domain information are updated, the updated encoder model parameters are used to encode the frame to be encoded.

[0140] The decoding end also uses the reconstructed frame before the frame to be encoded as training data, and the model parameter updating module based on time domain information is configured to use the model parameters of the reconstructed frame as the initial model parameters of the encoder and decoder, and iteratively update the initial model parameters of the encoder and decoder with the reconstructed frame as input, and obtain the updated model parameters of the encoder and decoder after a preset number of iterations. In this way, the encoding end and the decoding end can synchronously and independently use the reconstructed frame to update the model parameters of the encoder and decoder, avoiding the additional bit rate consumption caused by the encoder transmitting the decoder model parameters to the decoding end.

[0141] The decoding mode switching module is configured to receive the bitstream sent by the encoder and decode the decision result therefrom. If the decision result is reset, the updated model parameters of the decoder are also reset to the pre-trained model parameters; if the decision result is not reset, the model parameters of the decoder are the model parameters updated based on the time domain information. If the updated model parameters are reset, the pre-trained model parameters of the decoder are used to decode and reconstruct the bitstream output by the encoder; if the updated model parameters are not reset, the updated model parameters of the decoder are used to decode and reconstruct the bitstream output by the encoder.

[0142] The decoding end can store the reconstructed frame of the frame to be encoded for updating the encoding and decoding network parameters of the next frame of video.

[0143] The technical solution provided by this application is introduced in detail below in conjunction with specific implementation methods.

[0144] See also Figure 5 , Figure 5 A flow chart of a method for updating parameters of a video coding network provided in an embodiment of the present application. The method can be based on Figure 4 The architecture shown in FIG. 1 is implemented and applied to a coding device (i.e., a coding end device). Figure 5 As shown, the method may include:

[0145] S501: Use the first encoder in the video coding network to encode the reconstructed frame to obtain a first coding result; the reconstructed frame is obtained by decoding the code stream output by the first encoder by the first decoder in the video coding network; the code stream is obtained by encoding the encoded frame by the first encoder.

[0146] In an embodiment of the present application, the video coding network is a pre-trained neural network deployed on the coding device side or a neural network whose parameters are updated by using the scheme provided in the embodiment of the present application based on the pre-trained neural network, and the video coding network includes an encoder and a decoder. The parameters of the first encoder are the parameters used to encode the encoded frame, and the parameters of the first decoder are the parameters used to decode the code stream output by the first encoder. The reconstructed frame is a frame reconstructed before the current frame to be encoded, including a reconstructed frame of a time domain adjacent frame to the frame to be encoded or a reconstructed frame of a non-time domain adjacent frame. For example: if the frame to be encoded is the third frame in the video sequence, the encoded frame is the first frame or the second frame, and the reconstructed frame can be a reconstructed frame of the second frame or a reconstructed frame of the first frame. When the reconstructed frame is a reconstructed frame of the second frame, the first encoder is an encoder that encodes the second frame, and the first decoder is a decoder that decodes the code stream of the second frame. When the reconstructed frame is the reconstructed frame of the first frame, the first encoder is an initial encoder for encoding the first frame, and its parameters are pre-trained model parameters. The first decoder is an initial decoder for decoding the bitstream of the first frame, and its parameters are pre-trained model parameters.

[0147] The reconstructed frame is input into the first encoder as training data, and the image feature is extracted by the first encoder, and then compression encoding and quantization are performed to obtain a first encoding result, wherein the first encoding result includes an encoding code stream of the reconstructed frame.

[0148] S502: Obtain a first distortion metric and a first bit rate of a reconstructed frame according to the first decoder and the first encoding result.

[0149] In the embodiment of the present application, the coded code stream of the reconstructed frame is decoded by the first decoder to obtain a decoded frame of the reconstructed frame. It should be understood that if the reconstructed frame is X1 t , then its encoded code stream can be The first decoder pair Decode and reconstruct to obtain the decoded frame of the reconstructed frame Encoding stream The size of is known, then according to the encoding stream The size of the first bit rate for transmitting the reconstructed frame can be obtained According to the reconstructed frame X1 t and decoded frames of reconstructed frames The first distortion metric of the reconstructed frame codec can be obtained

[0150] S503: Determine a first loss according to the first distortion metric and the first bit rate.

[0151] In the embodiment of the present application, the loss function takes the distortion metric D(·) and the bit rate R(·) as small as possible as the optimization goal, and the loss function can be:

[0152]

[0153] The first loss can be obtained through this loss function. In which, λ represents the weight, and reducing λ can provide a higher compression rate, but at the cost of greater distortion of the reconstructed image.

[0154] It should be noted that the structure of the video coding network that can be used in the embodiments of the present application is not limited, such as whether the decoder includes a probability distribution estimation module, whether the encoder includes an auxiliary feature encoder, whether the decoder includes an auxiliary feature decoder, etc. In a possible example, for a network structure including an auxiliary feature code stream and a probability distribution estimation module, the loss function can be:

[0155]

[0156] in, Represents an auxiliary feature code stream, such as motion information, residual information, etc., which is not limited in the embodiments of the present application.

[0157] S504: Update the parameters of the first encoder and the first decoder according to the first loss to obtain a second encoder and a second decoder.

[0158] In the embodiment of the present application, the second encoder and the second decoder are obtained by iteratively updating the parameters of the first encoder and the first decoder N times according to the first loss. N is the preset number of iterations on the encoding device side, and N is an integer greater than 0. When the first loss does not converge, the parameters of the first encoder and the first decoder are updated by the back propagation algorithm, and the reconstructed frame is input into the updated first encoder for further iteration until the number of iterations reaches the preset number of iterations N, and the model parameters updated at the Nth iteration are fixed to obtain the second encoder and the second decoder.

[0159] Exemplarily, the method further includes: sending the number of iterations N to the decoding device, so that the decoding device iteratively updates the parameters of the video decoding network according to the number of iterations N.

[0160] In this implementation, the encoding device can send a preset number of iterations N to the decoding device, so that the decoding device also iterates the reconstructed frame N times, thereby updating the model parameters of the network on the decoding device side to improve the degree of matching between the model parameters on the decoding device side and the content of the frame to be encoded.

[0161] Exemplarily, the method further includes:

[0162] Obtaining a second loss for encoding and decoding a frame to be encoded using an initial encoder and an initial decoder;

[0163] Obtaining a third loss of encoding and decoding the frame to be encoded using the second encoder and the second decoder;

[0164] According to the second loss and the third loss, it is determined whether to reset the parameters of the second encoder and the second decoder to the parameters of the initial encoder and the parameters of the initial decoder, respectively.

[0165] In the embodiment of the present application, the initial encoder refers to an encoder that uses pre-trained model parameters, that is, an encoder in a video coding network that has not been updated after pre-training. The initial decoder refers to a decoder that uses pre-trained model parameters, that is, a decoder in a video coding network that has not been updated after pre-training, and the encoding end stores the pre-trained model parameters. The initial encoder is used to encode frame X2 t Encode and get the corresponding code stream Use the initial decoder to decode the bitstream Decode and get the corresponding reconstructed frame Based on the reconstructed frame and frame to be encoded X2 t Determining the distortion metric According to the code stream Determine the bit rate According to the distortion metric And the code rate Through the loss function L2(E m ,D m ) to determine the second loss. Where:

[0166]

[0167] Similarly, a second encoder is used to encode the frame to be encoded to obtain a corresponding code stream; a second decoder is used to decode the code stream to obtain a corresponding reconstructed frame; a distortion metric is determined based on the reconstructed frame and the frame to be encoded, and a bit rate is determined based on the code stream; a third loss is determined through a loss function based on the distortion metric and the bit rate.

[0168] If the second loss is less than the third loss, it means that the initial encoder and the initial decoder can encode and decode the frame to be encoded with a lower distortion metric and bit rate, that is, the encoding and decoding efficiency of the pre-trained model parameters is higher. Therefore, the parameters of the second encoder and the second decoder can be reset to the parameters of the initial encoder and the initial decoder, that is, reset to the pre-trained model parameters; if the second loss is greater than or equal to the third loss, it means that the second encoder and the second decoder can encode and decode the frame to be encoded with a lower distortion metric and bit rate, that is, the encoding and decoding efficiency of the updated parameters is higher, therefore, the parameters of the second encoder and the second decoder are not reset. It should be understood that the temporally adjacent frames in the video sequence may have large differences in picture content. The parameters of the second encoder and the second decoder are obtained by training the reconstructed frames before the frame to be encoded. They may be overfitted to the reconstructed frames. When the encoding and decoding efficiency of the frame to be encoded is low, it means that the frame to be encoded is greatly different from the previous frame in the scene, and the updated parameters are not well adapted to the content of the frame to be encoded. Therefore, it is necessary to reset the parameters of the second encoder and the second decoder to the pre-trained model parameters with stronger generalization ability.

[0169] In this implementation, an initial encoder and an initial decoder are used to encode and decode the frame to be encoded, and a corresponding second loss is obtained, and a second encoder and a second decoder are used to encode and decode the frame to be encoded, and a corresponding third loss is obtained. The second loss and the third loss can determine the matching degree and encoding and decoding efficiency of the pre-trained model parameters and the updated model parameters for the frame to be encoded, so that a model parameter reset decision based on encoding and decoding efficiency can be executed, and then parameters that are more adapted to the frame to be encoded and have higher encoding and decoding efficiency are selected for the second encoder and the second decoder.

[0170] Exemplarily, the method further includes:

[0171] A flag is generated according to the second loss and the third loss; for example, a first value is generated when the second loss is less than the third loss, and a second value is generated when the second loss is greater than or equal to the third loss. When the flag is a first value (for example, 1), the flag is used to indicate that the parameters of the second encoder and the second decoder are reset to the parameters of the initial encoder and the initial decoder, respectively; when the flag is a second value (for example, 0), the flag is used to indicate that the parameters of the second encoder and the second decoder are not reset;

[0172] The flag bit is sent to the decoding device so that the decoding device selects parameters of the video decoding network according to the flag bit.

[0173] In an embodiment of the present application, a flag bit can be used to represent the result of the model parameter reset decision based on the coding efficiency on the encoding device side, and the flag bit is sent to the decoding device in the form of a code stream. The decoding device decodes and reconstructs the code stream to obtain the flag bit, thereby facilitating the decoding device to choose whether to use the updated parameters or the pre-trained model parameters after completing the model parameter update based on the time domain information.

[0174] Exemplarily, the method further includes:

[0175] Obtaining a fourth loss of encoding and decoding the to-be-encoded frame by using the reset second encoder and the second decoder;

[0176] updating the parameters of the reset second encoder according to the fourth loss;

[0177] or,

[0178] Obtain a fifth loss for encoding and decoding the frame to be encoded using the second encoder and the second decoder;

[0179] Parameters of the second encoder are updated according to the fifth loss.

[0180] In the embodiment of the present application, the parameters of the reset second encoder and the second decoder are pre-trained model parameters. If the encoding device resets the parameters of the second encoder and the second decoder, the pre-trained model parameters are used to encode and decode the frames to be encoded, and the encoding and decoding code rate and distortion metric are obtained, thereby calculating the fourth loss. The parameters of the reset second encoder are iteratively updated according to the fourth loss until the number of iterations reaches the preset number of iterations K, and the model parameters updated at the Kth iteration are fixed to complete the parameter update of the reset second encoder. Wherein, K is an integer greater than 0. If the encoding device does not reset the parameters of the second encoder and the second decoder, the parameters of the second encoder and the second decoder are used to encode and decode the frames to be encoded, and the encoding and decoding code rate and distortion metric are obtained, thereby calculating the fifth loss. The parameters of the second encoder are iteratively updated according to the fifth loss until the number of iterations reaches the preset number of iterations K, and the model parameters updated at the Kth iteration are fixed to complete the parameter update of the second encoder.

[0181] In this implementation, the model parameters of the encoder are updated at the encoding end using the spatial domain information of the frame to be encoded, which can further improve the matching degree of the encoder to the content of the frame to be encoded.

[0182] Exemplarily, if the parameters of the second encoder and the second decoder are not reset, the embodiment of the present application can use the second encoder to encode the frame to be encoded. If the parameters of the second encoder and the second decoder are reset, the embodiment of the present application can use the reset second encoder (that is, use the pre-trained model parameters) to encode the frame to be encoded. Of course, the embodiment of the present application can also use an encoder updated based on spatial domain information to encode the frame to be encoded.

[0183] It can be seen that in the embodiment of the present application, before encoding the frame to be encoded, the encoding device uses the reconstructed frame before the frame to be encoded as training data, obtains the model parameters of the reconstructed frame as the initial model parameters of the encoder and decoder, and uses the optimization of the bit rate and distortion metric of the reconstructed frame as the objective function, and iteratively updates the parameters of the encoder and decoder through the first loss. In this way, by using time domain information to update the model parameters of the encoder and decoder of the video coding network, it is possible to improve the matching degree of the model parameters to the content of the frame to be encoded, which is beneficial to improving the encoding efficiency of the frame to be encoded. In addition, the encoding device and the decoding device synchronously and independently use the reconstructed frame to update the model parameters of the encoder and decoder. The encoding end does not need to transmit the model parameters of the decoder to the decoding end, which can avoid the additional bit rate consumption caused by the encoding end transmitting the decoder model parameters to the decoding end.

[0184] See also Figure 6 , Figure 6 A flow chart of a method for updating parameters of a video decoding network provided in an embodiment of the present application. The method can be based on Figure 4 The architecture shown in FIG. 1 is implemented and applied to a decoding device (i.e., a decoding end device). Figure 6 As shown, the method may include:

[0185] S601: Use the third encoder in the video decoding network to encode the reconstructed frame to obtain a second encoding result; the reconstructed frame is obtained by decoding the code stream output by the third encoder through the third decoder in the video encoding network; the code stream is obtained by encoding the encoded frame through the third encoder.

[0186] In the embodiment of the present application, the video decoding network is a pre-trained neural network deployed on the decoding device side or a neural network whose parameters are updated using the scheme provided in the embodiment of the present application based on the pre-trained neural network. The video decoding network includes an encoder and a decoder. The network structure of the video decoding network is the same as that of the video encoding network. The reconstructed frame can be Figure 5The same reconstructed frame in the illustrated embodiment may also be different reconstructed frames. In order to improve the matching degree between the parameters and the frame to be encoded, the same reconstructed frame may be preferably used. The parameters of the third encoder are the parameters used to encode the encoded frame, and the parameters of the third decoder are the parameters used to decode the bitstream output by the third encoder. Exemplarily, when the reconstructed frame is Figure 5 When the reconstructed frames in the illustrated embodiment are the same reconstructed frames, the parameters used by the third encoder and the first encoder are the same, and the parameters used by the third decoder and the first decoder are the same.

[0187] The reconstructed frame is input into the third encoder as training data, and the image feature is extracted by the third encoder, and then compression encoding and quantization are performed to obtain a second encoding result, wherein the second encoding result includes an encoding code stream of the reconstructed frame.

[0188] S602: Obtain a second distortion metric and a second bit rate of a reconstructed frame according to the third decoder and the second encoding result.

[0189] In the embodiment of the present application, the coded code stream of the reconstructed frame is decoded by the third decoder to obtain a decoded frame of the reconstructed frame. It should be understood that if the reconstructed frame is X3 t , then its encoded code stream can be The third decoder pair Decode and reconstruct to obtain the decoded frame of the reconstructed frame Encoding stream The size of is known, then according to the encoding stream The size of the reconstructed frame can be obtained by transmitting the second code rate Based on the reconstructed frame X3 t and decoded frames of reconstructed frames The second distortion metric of the reconstructed frame codec can be obtained

[0190] S603: Determine a sixth loss according to the second distortion metric and the second bit rate.

[0191] In the embodiment of the present application, the loss function takes the distortion metric D(·) and the bit rate R(·) as small as possible as the optimization goal, and the loss function can be:

[0192]

[0193] The sixth loss can be obtained by using this loss function. It should be noted that the structure of the video decoding network that can be used in the embodiment of the present application is not limited, such as: whether the decoder includes a probability distribution estimation module, whether the encoder includes an auxiliary feature encoder, whether the decoder includes an auxiliary feature decoder, and so on.

[0194] S604: Update parameters of the third encoder and the third decoder according to the sixth loss to obtain a fourth encoder and a fourth decoder.

[0195] Exemplarily, in a possible implementation, updating parameters of the third encoder and the third decoder according to the sixth loss to obtain a fourth encoder and a fourth decoder includes:

[0196] Iteratively update the parameters of the third encoder and the third decoder according to the sixth loss until the number of iterations reaches M times, fix the parameters updated in the Mth iteration, and obtain the fourth encoder and the fourth decoder; M is an integer greater than 0.

[0197] In an embodiment of the present application, M is a preset number of iterations on the decoding device side. When the sixth loss has not converged, the parameters of the third encoder and the third decoder are updated by the back propagation algorithm, and the reconstructed frame is input into the updated third encoder for further iteration until the number of iterations reaches the preset number of iterations M. The model parameters updated in the Mth iteration are fixed to obtain the fourth encoder and the fourth decoder.

[0198] Exemplarily, in another possible implementation, updating parameters of the third encoder and the third decoder according to the sixth loss to obtain a fourth encoder and a fourth decoder includes:

[0199] Receiving the number of iterations N sent by the encoding device;

[0200] Iteratively update the parameters of the third encoder and the third decoder according to the sixth loss until the number of iterations reaches N times, fix the parameters updated in the Nth iteration, and obtain the fourth encoder and the fourth decoder.

[0201] In an embodiment of the present application, when the sixth loss has not converged, the parameters of the third encoder and the third decoder are updated through the back propagation algorithm, and the reconstructed frame is input into the updated third encoder for further iteration until the number of iterations reaches a preset number of iterations N, and the model parameters updated at the Nth iteration are fixed to obtain the fourth encoder and the fourth decoder.

[0202] In this implementation, the decoding device may iteratively update the parameters of the third encoder and the third decoder according to the number of iterations M preset on its own side, or iteratively update the parameters of the third encoder and the third decoder according to the number of iterations N preset on the encoding device side.

[0203] Exemplarily, the method further includes:

[0204] Receive the flag bit sent by the encoding device;

[0205] If the flag bit is the first value, the parameters of the fourth encoder and the fourth decoder are reset to the parameters of the initial encoder and the parameters of the initial decoder respectively;

[0206] If the flag bit is the second value, the parameters of the fourth encoder and the fourth decoder are not reset.

[0207] In an embodiment of the present application, the encoding end stores pre-trained model parameters. The decoding device can receive a code stream sent by the encoding device, which may include a code stream and a flag of a frame to be encoded. The decoding device decodes the flag from the code stream. If the flag is a first value (for example: 1), the parameters of the fourth encoder and the fourth decoder are reset to the parameters of the initial encoder and the initial decoder, respectively, that is, reset to the pre-trained model parameters; if the flag is a second value (for example: 0), the parameters of the fourth encoder and the fourth decoder are not reset. It should be understood that if the decision result of the encoding device is reset, it means that there is a large difference in the scene between the frame to be encoded and the previous frame, and the updated parameters are not well adapted to the content of the frame to be encoded. Therefore, the decoding device also needs to reset the parameters. If the decision result of the encoding device is not reset, it means that there is no large difference in the scene between the frame to be encoded and the previous frame, and the updated parameters are adapted to the content of the frame to be encoded.

[0208] In this implementation, the decoding device selects parameters for the fourth encoder and the fourth decoder according to the flag bit sent by the encoding device, and can obtain parameters that are more adapted to the frame to be encoded and have higher encoding and decoding efficiency.

[0209] Exemplarily, if the decoding device resets the parameters of the fourth decoder, the pre-trained model parameters after the fourth decoder is reset are used to decode and reconstruct the code stream of the frame to be encoded. If the decoding device does not reset the parameters of the fourth decoder, the parameters of the fourth decoder are used to decode and reconstruct the code stream of the frame to be encoded. The decoding device stores the reconstructed frame of the frame to be encoded for updating the model parameters of the next frame.

[0210] It can be seen that in the embodiment of the present application, before encoding the frame to be encoded, the decoding device uses the reconstructed frame before the frame to be encoded as training data, obtains the model parameters of the reconstructed frame as the initial model parameters of the encoder and decoder, and optimizes the bit rate and distortion metric of the reconstructed frame as the objective function, and iteratively updates the parameters of the encoder and decoder through the sixth loss. In this way, the model parameters of the encoder and decoder of the video decoding network are updated using time domain information, which can improve the matching degree of the model parameters to the content of the frame to be encoded, which is conducive to improving the decoding efficiency of the frame to be encoded. In addition, the decoding device and the encoding device synchronously and independently use the reconstructed frame to update the model parameters of the encoder and decoder. The decoding end does not need to receive the model parameters of the decoder transmitted by the encoding end, which can avoid the additional bit rate consumption caused by the encoder end transmitting the decoder model parameters to the decoding end.

[0211] See also Figure 7 , Figure 7 A flow chart of another method for updating parameters of a video coding network provided in an embodiment of the present application. The method is applied to a coding device, such as Figure 7 As shown, the method may include:

[0212] S701: Encode the reconstructed frame using a first encoder in a video encoding network to obtain a first encoding result; the reconstructed frame is obtained by decoding a bitstream output by a first encoder using a first decoder in the video encoding network; the bitstream is obtained by encoding the encoded frame using the first encoder;

[0213] S702: Obtain a first distortion metric and a first bit rate of a reconstructed frame according to the first decoder and the first encoding result;

[0214] S703: Determine a first loss according to the first distortion metric and the first bit rate;

[0215] S704: Update parameters of the first encoder and the first decoder according to the first loss to obtain a second encoder and a second decoder;

[0216] S705: Make a parameter reset decision for the second encoder and the second decoder, and execute the decision result.

[0217] In the embodiment of the present application, the specific implementation of S701-S704 can refer to Figure 5 Related description of the illustrated embodiment.

[0218] Exemplarily, making a parameter reset decision for the second encoder and the second decoder includes:

[0219] Obtaining a second loss for encoding and decoding a frame to be encoded using an initial encoder and an initial decoder;

[0220] Obtaining a third loss of encoding and decoding the frame to be encoded using the second encoder and the second decoder;

[0221] According to the second loss and the third loss, it is determined whether to reset the parameters of the second encoder and the second decoder to the parameters of the initial encoder and the parameters of the initial decoder, respectively.

[0222] That is, the frame to be encoded is encoded by the initial encoder to obtain the corresponding code stream; the initial decoder is used to decode the code stream to obtain the corresponding reconstructed frame; the distortion metric is determined according to the reconstructed frame and the frame to be encoded, and the code rate is determined according to the code stream; the second loss is determined by the loss function according to the distortion metric and the code rate. The frame to be encoded is then encoded by the second encoder to obtain the corresponding code stream; the second decoder is used to decode the code stream to obtain the corresponding reconstructed frame; the distortion metric is determined according to the reconstructed frame and the frame to be encoded, and the code rate is determined according to the code stream; the third loss is determined by the loss function according to the distortion metric and the code rate. The encoding device determines the decision result according to the second loss and the third loss. If the second loss is less than the third loss, the decision result can be represented by a flag bit 1 to indicate that the parameters of the second encoder and the second decoder are reset to the parameters of the initial encoder and the initial decoder, that is, reset to the pre-trained model parameters; if the second loss is greater than or equal to the third loss, the decision result can be represented by a flag bit 0 to indicate that the parameters of the second encoder and the second decoder are not reset.

[0223] The encoding device sends the decision result to the decoding device in the form of a code stream, so that the decoding device can select parameters of the video decoding network according to the decision result.

[0224] Exemplarily, if the parameters of the second encoder and the second decoder are not reset, the encoding device uses the second encoder to encode the frame to be encoded; if the parameters of the second encoder and the second decoder are reset, the encoding device uses the reset second encoder to encode the frame to be encoded.

[0225] It can be seen that in the embodiment of the present application, after making a model parameter reset decision based on coding efficiency, the encoding device can use the second encoder to encode the frame to be encoded, or use the reset second encoder to encode the frame to be encoded.

[0226] See also Figure 8 , Figure 8 A flow chart of another method for updating parameters of a video decoding network provided in an embodiment of the present application. The method is applied to a decoding device, such as Figure 8 As shown, the method may include:

[0227] S801: Encode the reconstructed frame using a third encoder in the video decoding network to obtain a second encoding result; the reconstructed frame is obtained by decoding a bitstream output by a third encoder using a third decoder in the video encoding network; the bitstream is obtained by encoding the encoded frame using the third encoder;

[0228] S802: Obtain a second distortion metric and a second bit rate of a reconstructed frame according to the third decoder and the second encoding result;

[0229] S803: Determine a sixth loss according to the second distortion metric and the second bit rate;

[0230] S804: updating parameters of the third encoder and the third decoder according to the sixth loss to obtain a fourth encoder and a fourth decoder;

[0231] S805: According to the decision result sent by the encoding device, the parameters of the fourth encoder and the fourth decoder are reset to the parameters of the initial encoder and the initial decoder respectively, or the parameters of the fourth encoder and the fourth decoder are not reset.

[0232] In the embodiment of the present application, the decision result sent by the encoding device can be represented by a flag bit, and the decoding device decodes and reconstructs the code stream on the encoding device side to obtain the flag bit. If the flag bit is 1, the parameters of the fourth encoder and the fourth decoder are reset to the parameters of the initial encoder and the parameters of the initial decoder respectively; if the flag bit is 0, the parameters of the fourth encoder and the fourth decoder are not reset.

[0233] It can be seen that in the embodiment of the present application, based on the decision result sent by the encoding device, the decoding device side can be synchronized with the encoding device side in the selection of parameters.

[0234] To better understand this solution, the following uses the reconstructed frame of the first frame in the video sequence as training data and the second frame as the frame to be encoded as an example for a brief description. Fig. 9 , Fig. 9 The overall schematic diagram of a parameter updating method of a video coding network provided in an embodiment of the present application. Fig. 9 As shown, the encoder I_Enc and the decoder I_Dnc are deployed on both the encoding device side and the decoding device side. For the first frame, the parameters of the encoder I_Enc and the decoder I_Dnc are pre-trained model parameters. The encoding device encodes the first frame through the encoder I_Enc and sends the code stream to the decoding device, which decodes it to obtain the reconstructed frame of the first frame. For the second frame, the reconstructed frame of the first frame is the reconstructed frame. The encoding device inputs the reconstructed frame into the model parameter update module based on time domain information and executes Figure 5The operations of S501-S504 in the illustrated embodiment obtain an updated encoder I_Enc' and an updated decoder I_Dnc'. The encoding device uses the encoder I_Enc to encode the second frame, and uses the decoder I_Dnc to decode the corresponding code stream, and obtains the loss. The encoding device uses the updated encoder I_Enc' to encode the second frame, and uses the updated decoder I_Dnc' to decode the corresponding code stream, and obtains the loss. According to the obtained loss, the rate-distortion optimization (RDO) encoding method is selected to reset the parameters of the updated encoder I_Enc' and the updated decoder I_Dnc' to the parameters of the encoder I_Enc and the decoder I_Dnc, respectively, or the parameters of the updated encoder I_Enc' and the updated decoder I_Dnc' are not reset. Based on the selection of the RDO encoding method, the encoding device performs a model parameter update based on spatial domain information to further update the parameters of the encoder at the encoding end. The encoding device encodes the second frame using the encoder updated based on the spatial domain information, compresses the flag bit corresponding to the decision result into the encoded code stream, and sends it to the decoding device.

[0235] The decoding device inputs the reconstructed frame into the model parameter update module based on time domain information and executes Figure 6 The operations of S601-S604 in the illustrated embodiment obtain an updated encoder I_Enc' and an updated decoder I_Dnc'. The decoding device decodes the code stream sent by the encoding device to obtain a flag bit, and switches the decoding mode according to the flag bit, for example: resetting the parameters of the updated encoder I_Enc' and the updated decoder I_Dnc' to the parameters of the encoder I_Enc and the decoder I_Dnc, respectively, or not resetting the parameters of the updated encoder I_Enc' and the updated decoder I_Dnc'. The decoding device decodes and reconstructs the code stream of the second frame based on the decoder after the decoding mode is switched to obtain a reconstructed frame of the second frame.

[0236] In order to verify the effect of the parameter updating method of the video coding network provided in the embodiment of the present application, relevant personnel used a publicly available image coding network as a single-frame intra-coding network and conducted experiments on different data sets. The coding efficiency gain was used as an evaluation indicator in the experiment, where the coding efficiency gain was calculated using BD-rate ( rate, which is used to evaluate the rate-distortion performance of different video encoders. BD-rate refers to the average bit rate savings of each comparison technology compared to the scheme for obtaining pre-trained model parameters under the same peak signal-to-noise ratio (PSNR). A negative BD-rate value indicates that the comparison technology saves bit rate. The larger the absolute value of the value, the greater the bit rate savings. Table 1 shows the coding efficiency gains of the schemes provided by Related Technology 1, Related Technology 2, and the embodiments of the present application on multiple data sets:

[0237] Table 1

[0238] Dataset One of the related technologies Related Technology 2 This program Xiph-5-2fps -1.92% -5.30% -13.15% MCL-JCV-2fps -3.03% -6.07% -10.04% HEVC Class B -2.45% -6.12% -6.86% HEVC Class C -6.40% -13.71% -16.91% HEVC Class D -10.21% -19.07% -19.30% HEVC Class E -5.16% -9.85% -25.69% average -4.86% -10.02% -15.33%

[0239] It is not difficult to see from Table 1 that the coding efficiency gain of the scheme provided in the embodiment of the present application on multiple data sets is basically higher than that of the first and second related technologies. On the E category (i.e., HEVC Class E) sequence of High Efficiency Video Coding (HEVC), a coding efficiency gain of up to 25.69% was obtained. The reason for the improvement in coding efficiency is that the scheme provided in the embodiment of the present application uses spatiotemporal domain information to update the model parameters, which improves the matching degree between the model parameters and the frames to be encoded, while avoiding the additional bit rate consumption caused by the transmission of parameters from the encoding end to the decoding end.

[0240] The method of the embodiment of the present application is described above, and the device of the embodiment of the present application is provided below.

[0241] See also Fig.10 , Fig.10 The present invention provides a schematic diagram of a parameter updating device 1000 for a video coding network according to an embodiment of the present invention. The parameter updating device 1000 for the video coding network is applied to a coding device or a chip in a coding device, and the parameter updating device 1000 for the video coding network includes a first coding unit 1001, a first decoding unit 1002, and a first updating unit 1003. Among them:

[0242] The first encoding unit 1001 is used to encode the reconstructed frame by using the first encoder in the video encoding network to obtain a first encoding result; the reconstructed frame is obtained by decoding the bit stream output by the first encoder by the first decoder in the video encoding network; the bit stream is obtained by encoding the encoded frame by the first encoder;

[0243] A first decoding unit 1002, configured to obtain a first distortion metric and a first bit rate of a reconstructed frame according to the first decoder and the first encoding result;

[0244] The first updating unit 1003 is configured to determine a first loss according to the first distortion metric and the first bit rate; and update parameters of the first encoder and the first decoder according to the first loss to obtain a second encoder and a second decoder.

[0245] It can be seen that in Fig.10 In the device shown, before encoding the frame to be encoded, the encoding device uses the reconstructed frame before the frame to be encoded as training data, obtains the model parameters of the reconstructed frame as the initial model parameters of the encoder and decoder, and optimizes the bit rate and distortion metric of the reconstructed frame as the objective function, and iteratively updates the parameters of the encoder and decoder through the first loss. In this way, by using time domain information to update the model parameters of the encoder and decoder of the video coding network, the matching degree of the model parameters to the content of the frame to be encoded can be improved, which is conducive to improving the encoding efficiency of the frame to be encoded. In addition, the encoding device and the decoding device synchronously and independently use the reconstructed frame to update the model parameters of the encoder and decoder. The encoding end does not need to transmit the model parameters of the decoder to the decoding end, which can avoid the additional bit rate consumption caused by the transmission of the decoder model parameters by the encoding end to the decoding end.

[0246] In a possible implementation, the first updating unit 1003 is further configured to:

[0247] Obtaining a second loss for encoding and decoding a frame to be encoded using an initial encoder and an initial decoder;

[0248] Obtaining a third loss of encoding and decoding the frame to be encoded using the second encoder and the second decoder;

[0249] According to the second loss and the third loss, it is determined whether to reset the parameters of the second encoder and the second decoder to the parameters of the initial encoder and the parameters of the initial decoder, respectively.

[0250] In a possible implementation manner, in terms of determining whether to reset the parameters of the second encoder and the second decoder to the parameters of the initial encoder and the parameters of the initial decoder, respectively, according to the second loss and the third loss, the first updating unit 1003 is specifically used to:

[0251] If the second loss is less than the third loss, resetting the parameters of the second encoder and the second decoder to the parameters of the initial encoder and the parameters of the initial decoder respectively;

[0252] If the second loss is greater than or equal to the third loss, the parameters of the second encoder and the second decoder are not reset.

[0253] In a possible implementation, the first updating unit 1003 is further configured to:

[0254] Obtaining a fourth loss of encoding and decoding the to-be-encoded frame by using the reset second encoder and the second decoder;

[0255] updating the parameters of the reset second encoder according to the fourth loss;

[0256] or,

[0257] Obtaining a fifth loss for encoding and decoding the frame to be encoded using the second encoder and the second decoder;

[0258] Parameters of the second encoder are updated according to the fifth loss.

[0259] In a possible implementation, the first updating unit 1003 is further configured to:

[0260] Sending the flag bit to the decoding device so that the decoding device selects parameters of the video decoding network according to the flag bit;

[0261] Among them, when the flag bit is a first value, the flag bit is used to indicate that the parameters of the second encoder and the second decoder are reset to the parameters of the initial encoder and the parameters of the initial decoder respectively; when the flag bit is a second value, the flag bit is used to indicate that the parameters of the second encoder and the second decoder are not reset.

[0262] In a possible implementation, the first encoding result includes an encoded code stream of a reconstructed frame; in terms of obtaining a first distortion metric and a first code rate of the reconstructed frame according to the first decoder and the first encoding result, the first decoding unit 1002 is specifically configured to:

[0263] Decoding the coded bitstream using a first decoder to obtain a decoded frame of the reconstructed frame;

[0264] determining a first distortion metric based on the reconstructed frame and a decoded frame of the reconstructed frame;

[0265] A first bit rate is determined according to the encoded bit stream.

[0266] In a possible implementation, the second encoder and the second decoder are obtained by iteratively updating the parameters of the first encoder and the first decoder N times according to the first loss; N is an integer greater than 0; and the first updating unit 1003 is further used to:

[0267] The number of iterations N is sent to the decoding device, so that the decoding device iteratively updates the parameters of the video decoding network according to the number of iterations N.

[0268] See also Fig.11 , Fig.11The present invention provides a schematic diagram of a parameter updating device 1100 for a video decoding network according to an embodiment of the present invention. The parameter updating device 1100 for the video decoding network is applied to a decoding device or a chip in a decoding device, and the parameter updating device 1100 for the video decoding network includes a second encoding unit 1101, a second decoding unit 1102, and a second updating unit 1103. Among them:

[0269] The second encoding unit 1101 is used to encode the reconstructed frame by using the third encoder in the video decoding network to obtain a second encoding result; the reconstructed frame is obtained by decoding the bit stream output by the third encoder by the third decoder in the video encoding network; the bit stream is obtained by encoding the encoded frame by the third encoder;

[0270] A second decoding unit 1102, configured to obtain a second distortion metric and a second bit rate of the reconstructed frame according to the third decoder and the second encoding result;

[0271] The second updating unit 1103 is configured to determine a sixth loss according to the second distortion metric and the second bit rate; and update parameters of the third encoder and the third decoder according to the sixth loss to obtain a fourth encoder and a fourth decoder.

[0272] It can be seen that in Fig.11 In the device shown, before encoding the frame to be encoded, the decoding device uses the reconstructed frame before the frame to be encoded as training data, obtains the model parameters of the reconstructed frame as the initial model parameters of the encoder and decoder, optimizes the bit rate and distortion metric of the reconstructed frame as the objective function, and iteratively updates the parameters of the encoder and decoder through the sixth loss. In this way, the model parameters of the encoder and decoder of the video decoding network are updated using time domain information, which can improve the matching degree of the model parameters to the content of the frame to be encoded, which is conducive to improving the decoding efficiency of the frame to be encoded. In addition, the decoding device and the encoding device synchronously and independently use the reconstructed frame to update the model parameters of the encoder and decoder. The decoding end does not need to receive the model parameters of the decoder transmitted by the encoding end, which can avoid the additional bit rate consumption caused by the encoder transmitting the decoder model parameters to the decoding end.

[0273] In a possible implementation, the second updating unit 1103 is further configured to:

[0274] Receive the flag bit sent by the encoding device;

[0275] If the flag bit is the first value, the parameters of the fourth encoder and the fourth decoder are reset to the parameters of the initial encoder and the parameters of the initial decoder respectively;

[0276] If the flag bit is the second value, the parameters of the fourth encoder and the fourth decoder are not reset.

[0277] In a possible implementation, the second encoding result includes an encoded code stream of a reconstructed frame; in terms of obtaining a second distortion metric and a second code rate of the reconstructed frame according to the third decoder and the second encoding result, the second decoding unit 1102 is specifically configured to:

[0278] Decoding the encoded bitstream using a third decoder to obtain a decoded frame of the reconstructed frame;

[0279] determining a second distortion metric based on the reconstructed frame and a decoded frame of the reconstructed frame;

[0280] A second bit rate is determined according to the encoded bit stream.

[0281] In a possible implementation, the parameters of the third encoder and the third decoder are updated according to the sixth loss to obtain the fourth encoder and the fourth decoder. The second updating unit 1103 is specifically used to:

[0282] Iteratively updating the parameters of the third encoder and the third decoder according to the sixth loss until the number of iterations reaches M times, fixing the parameters updated at the Mth iteration, and obtaining a fourth encoder and a fourth decoder; M is an integer greater than 0;

[0283] or,

[0284] Receiving the number of iterations N sent by the encoding device;

[0285] Iteratively update the parameters of the third encoder and the third decoder according to the sixth loss until the number of iterations reaches N times, fix the parameters updated in the Nth iteration, and obtain the fourth encoder and the fourth decoder.

[0286] See also Fig.12 , Fig.12 The schematic diagram of the structure of a coding device 1200 provided in an embodiment of the present application is as follows. The coding device 1200 includes a processor 1201 , a memory 1202 , and a communication interface 1203 , and the processor 1201 , the memory 1202 , and the communication interface 1203 are interconnected via a bus 1204 .

[0287] The memory 1202 includes, but is not limited to, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or a portable read-only memory (CD-ROM), and the memory 1202 is used for related computer programs and data. The communication interface 1203 is used to receive and send data.

[0288] The processor 1201 may be one or more central processing units (CPUs). When the processor 1201 is a CPU, the CPU may be a single-core CPU or a multi-core CPU.

[0289] The processor 1201 in the encoding device 1200 is used to read the computer program code stored in the memory 1202 and perform the following operations:

[0290] The reconstructed frame is encoded by a first encoder in the video encoding network to obtain a first encoding result; the reconstructed frame is obtained by decoding a bit stream output by the first encoder by a first decoder in the video encoding network; the bit stream is obtained by encoding the encoded frame by the first encoder;

[0291] Obtaining a first distortion metric and a first bit rate of a reconstructed frame according to the first decoder and the first encoding result;

[0292] Determining a first loss based on the first distortion metric and the first bit rate;

[0293] The parameters of the first encoder and the first decoder are updated according to the first loss to obtain the second encoder and the second decoder.

[0294] It should be noted that the implementation of each operation can also refer to Figure 5 The corresponding description of the method embodiment shown.

[0295] It can be seen that in Fig.12 In the encoding device 1200 shown, before encoding the frame to be encoded, the encoding device 1200 uses the reconstructed frame before the frame to be encoded as training data, obtains the model parameters of the reconstructed frame as the initial model parameters of the encoder and decoder, takes the optimization of the bit rate and distortion metric of the reconstructed frame as the objective function, and iteratively updates the parameters of the encoder and decoder through the first loss. In this way, by using time domain information to update the model parameters of the encoder and decoder of the video coding network, it is possible to improve the matching degree of the model parameters to the content of the frame to be encoded, which is conducive to improving the encoding efficiency of the frame to be encoded. In addition, the encoding device 1200 and the decoding device synchronously and independently use the reconstructed frame to update the model parameters of the encoder and decoder. The encoding end does not need to transmit the model parameters of the decoder to the decoding end, which can avoid the additional bit rate consumption caused by the transmission of the decoder model parameters by the encoding end to the decoding end.

[0296] See also Fig.13 , Fig.13The schematic diagram of the structure of a decoding device 1300 provided in an embodiment of the present application is as follows. The decoding device 1300 includes a processor 1301 , a memory 1302 , and a communication interface 1303 , and the processor 1301 , the memory 1302 , and the communication interface 1303 are interconnected via a bus 1304 .

[0297] The memory 1302 includes but is not limited to RAM, ROM, EPROM, and CD-ROM, and is used for related computer programs and data. The communication interface 1303 is used to receive and send data.

[0298] The processor 1301 may be one or more CPUs. When the processor 1301 is a CPU, the CPU may be a single-core CPU or a multi-core CPU.

[0299] The processor 1301 in the decoding device 1300 is used to read the computer program code stored in the memory 1302 and perform the following operations:

[0300] The reconstructed frame is encoded by a third encoder in the video decoding network to obtain a second encoding result; the reconstructed frame is obtained by decoding a bit stream output by a third encoder by a third decoder in the video encoding network; the bit stream is obtained by encoding the encoded frame by the third encoder;

[0301] Obtaining a second distortion metric and a second bit rate of the reconstructed frame according to the third decoder and the second encoding result;

[0302] determining a sixth loss based on the second distortion metric and the second bit rate;

[0303] The parameters of the third encoder and the third decoder are updated according to the sixth loss to obtain a fourth encoder and a fourth decoder.

[0304] It should be noted that the implementation of each operation can also refer to Figure 6 The corresponding description of the method embodiment shown.

[0305] It can be seen that in Fig.13In the decoding device 1300 shown, before encoding the frame to be encoded, the decoding device 1300 uses the reconstructed frame before the frame to be encoded as training data, obtains the model parameters of the reconstructed frame as the initial model parameters of the encoder and decoder, optimizes the bit rate and distortion metric of the reconstructed frame as the objective function, and iteratively updates the parameters of the encoder and decoder through the sixth loss. In this way, the model parameters of the encoder and decoder of the video decoding network are updated using time domain information, which can improve the matching degree of the model parameters to the content of the frame to be encoded, which is conducive to improving the decoding efficiency of the frame to be encoded. In addition, the decoding device 1300 and the encoding device 1200 synchronously and independently use the reconstructed frame to update the model parameters of the encoder and decoder. The decoding end does not need to receive the model parameters of the decoder transmitted by the encoding end, which can avoid the additional bit rate consumption caused by the encoder transmitting the decoder model parameters to the decoding end.

[0306] The present application also provides a chip, including: a processor, configured to call and run a computer program from a memory, so that a device equipped with the chip executes the above-mentioned Figure 5 or Figure 6 The chip may be a chip in the encoding device 1200 or the decoding device 1300.

[0307] The present application also provides a computer-readable storage medium (Memory), which stores a computer program. When the computer program is executed, the above-mentioned Figure 5 or Figure 6 The method described in any one of the embodiments. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the device and the extended storage medium supported by the device. The computer-readable storage medium provides a storage space, which stores the operating system of the device. In addition, one or more computer programs suitable for being loaded and executed by the processor of the device are also stored in the storage space. It should be noted that the computer-readable storage medium here can be a high-speed RAM or a non-volatile memory, such as at least one disk storage; optionally, it can also be at least one computer-readable storage medium located away from the aforementioned processor.

[0308] The present application also provides a computer program product, the computer program product comprising: a computer program code, when the computer program code is executed, Figure 5 or Figure 6 The method process described in any one of the embodiments is implemented.

[0309] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0310] It should be understood that the processor mentioned in the embodiments of the present application may be a CPU, or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0311] It should also be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a ROM, a programmable read-only memory (Programmable ROM, PROM), an EPROM, an electrically erasable programmable read-only memory (ElectricallyEPROM, EEPROM) or a flash memory. The volatile memory may be a RAM, which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (Static RAM, SRAM), dynamic random access memory (Dynamic RAM, DRAM), synchronous dynamic random access memory (Synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (Double Data Rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (Enhanced SDRAM, ESDRAM), synchronous connection dynamic random access memory (Synchlink DRAM, SLDRAM) and direct memory bus random access memory (Direct Rambus RAM, DRRAM).

[0312] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, the memory (storage module) is integrated in the processor.

[0313] It should be noted that the memory described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0314] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0315] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely exemplary. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0316] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0317] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. If the above integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium.

[0318] In this application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. In the text description of this application, the character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0319] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs.

[0320] The modules in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.

[0321] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for updating parameters of a video coding network, characterized in that: Applied to an encoding device; the method comprises: The reconstructed frame is encoded by a first encoder in the video encoding network to obtain a first encoding result; the reconstructed frame is obtained by decoding a bit stream output by the first encoder by a first decoder in the video encoding network; the bit stream is obtained by encoding the encoded frame by the first encoder; Obtaining a first distortion metric and a first bit rate of the reconstructed frame according to the first decoder and the first encoding result; determining a first loss based on the first distortion metric and the first bit rate; The parameters of the first encoder and the first decoder are updated according to the first loss to obtain a second encoder and a second decoder.

2. The method according to claim 1, characterized in that The method further comprises: Obtaining a second loss for encoding and decoding a frame to be encoded using an initial encoder and an initial decoder; Obtaining a third loss of encoding and decoding the to-be-encoded frame by using the second encoder and the second decoder; According to the second loss and the third loss, it is determined whether to reset the parameters of the second encoder and the second decoder to the parameters of the initial encoder and the parameters of the initial decoder, respectively.

3. The method according to claim 2, characterized in that The determining, according to the second loss and the third loss, whether to reset the parameters of the second encoder and the second decoder to the parameters of the initial encoder and the parameters of the initial decoder, respectively, comprises: If the second loss is less than the third loss, resetting the parameters of the second encoder and the second decoder to the parameters of the initial encoder and the parameters of the initial decoder respectively; If the second loss is greater than or equal to the third loss, the parameters of the second encoder and the second decoder are not reset.

4. The method according to claim 3, characterized in that The method further comprises: Obtaining a fourth loss of encoding and decoding the to-be-encoded frame by using the reset second encoder and the second decoder; updating the parameters of the reset second encoder according to the fourth loss; or, Obtaining a fifth loss of encoding and decoding the to-be-encoded frame by using the second encoder and the second decoder; Parameters of the second encoder are updated according to the fifth loss.

5. The method according to claim 3, characterized in that: The method further comprises: Sending the flag bit to a decoding device so that the decoding device selects parameters of a video decoding network according to the flag bit; When the flag bit is a first value, the flag bit is used to indicate that the parameters of the second encoder and the second decoder are reset to the parameters of the initial encoder and the parameters of the initial decoder respectively; when the flag bit is a second value, the flag bit is used to indicate that the parameters of the second encoder and the second decoder are not reset.

6. The method according to any one of claims 1 to 5, characterized in that: The first encoding result includes an encoded code stream of the reconstructed frame; and obtaining a first distortion metric and a first code rate of the reconstructed frame according to the first decoder and the first encoding result includes: Decoding the encoded code stream using the first decoder to obtain a decoded frame of the reconstructed frame; determining the first distortion metric based on the reconstructed frame and a decoded frame of the reconstructed frame; The first bit rate is determined according to the encoded bit stream.

7. The method according to any one of claims 1 to 6, characterized in that: The second encoder and the second decoder are obtained by iteratively updating parameters of the first encoder and the first decoder N times according to the first loss; N is an integer greater than 0; The method further comprises: The number of iterations N is sent to a decoding device, so that the decoding device iteratively updates the parameters of the video decoding network according to the number of iterations N.

8. A method for updating parameters of a video decoding network, characterized in that: Applied to a decoding device; the method comprises: The reconstructed frame is encoded by a third encoder in the video decoding network to obtain a second encoding result; the reconstructed frame is obtained by decoding a code stream output by the third encoder by a third decoder in the video encoding network; the code stream is obtained by encoding the encoded frame by the third encoder; Obtaining a second distortion metric and a second bit rate of the reconstructed frame according to the third decoder and the second encoding result; determining a sixth loss based on the second distortion metric and the second bit rate; The parameters of the third encoder and the third decoder are updated according to the sixth loss to obtain a fourth encoder and a fourth decoder.

9. The method according to claim 8, characterized in that The method further comprises: Receive the flag bit sent by the encoding device; If the flag bit is the first value, resetting the parameters of the fourth encoder and the fourth decoder to the parameters of the initial encoder and the parameters of the initial decoder respectively; If the flag bit is the second value, the parameters of the fourth encoder and the fourth decoder are not reset.

10. The method according to claim 8 or 9, characterized in that: The second encoding result includes an encoded code stream of the reconstructed frame; and obtaining a second distortion metric and a second code rate of the reconstructed frame according to the third decoder and the second encoding result includes: Decoding the encoded bitstream using the third decoder to obtain a decoded frame of the reconstructed frame; determining the second distortion metric based on the reconstructed frame and a decoded frame of the reconstructed frame; The second bit rate is determined according to the encoded bit stream.

11. The method according to any one of claims 8 to 10, characterized in that: The updating of the parameters of the third encoder and the third decoder according to the sixth loss to obtain a fourth encoder and a fourth decoder comprises: Iteratively updating the parameters of the third encoder and the third decoder according to the sixth loss until the number of iterations reaches M times, fixing the parameters updated in the Mth iteration, and obtaining the fourth encoder and the fourth decoder; M is an integer greater than 0; or, Receiving the number of iterations N sent by the encoding device; Iteratively update the parameters of the third encoder and the third decoder according to the sixth loss until the number of iterations reaches N times, fix the parameters updated by the Nth iteration, and obtain the fourth encoder and the fourth decoder.

12. A parameter updating device for a video coding network, characterized in that: Applied to a coding device; the device comprises a first coding unit, a first decoding unit and a first updating unit; The first encoding unit is used to encode the reconstructed frame using the first encoder in the video encoding network to obtain a first encoding result; the reconstructed frame is obtained by decoding the code stream output by the first encoder by the first decoder in the video encoding network; the code stream is obtained by encoding the encoded frame by the first encoder; The first decoding unit is used to obtain a first distortion metric and a first bit rate of the reconstructed frame according to the first decoder and the first encoding result; The first updating unit is used to determine a first loss according to the first distortion metric and the first bit rate; and update parameters of the first encoder and the first decoder according to the first loss to obtain a second encoder and a second decoder.

13. The device according to claim 12, characterized in that The first updating unit is further configured to: Obtaining a second loss for encoding and decoding a frame to be encoded using an initial encoder and an initial decoder; Obtaining a third loss of encoding and decoding the to-be-encoded frame by using the second encoder and the second decoder; According to the second loss and the third loss, it is determined whether to reset the parameters of the second encoder and the second decoder to the parameters of the initial encoder and the parameters of the initial decoder, respectively.

14. The device according to claim 13, characterized in that In terms of determining whether to reset the parameters of the second encoder and the second decoder to the parameters of the initial encoder and the parameters of the initial decoder respectively according to the second loss and the third loss, the first updating unit is specifically configured to: If the second loss is less than the third loss, resetting the parameters of the second encoder and the second decoder to the parameters of the initial encoder and the parameters of the initial decoder respectively; If the second loss is greater than or equal to the third loss, the parameters of the second encoder and the second decoder are not reset.

15. The device according to claim 14, characterized in that The first updating unit is further configured to: Obtaining a fourth loss of encoding and decoding the to-be-encoded frame by using the reset second encoder and the second decoder; updating the parameters of the reset second encoder according to the fourth loss; or, Obtaining a fifth loss of encoding and decoding the to-be-encoded frame by using the second encoder and the second decoder; Parameters of the second encoder are updated according to the fifth loss.

16. The device according to claim 14, characterized in that The first updating unit is further configured to: Sending the flag bit to a decoding device so that the decoding device selects parameters of a video decoding network according to the flag bit; When the flag bit is a first value, the flag bit is used to indicate that the parameters of the second encoder and the second decoder are reset to the parameters of the initial encoder and the parameters of the initial decoder respectively; when the flag bit is a second value, the flag bit is used to indicate that the parameters of the second encoder and the second decoder are not reset.

17. The device according to any one of claims 12 to 16, characterized in that: The first encoding result includes an encoded code stream of the reconstructed frame; in terms of obtaining a first distortion metric and a first code rate of the reconstructed frame according to the first decoder and the first encoding result, the first decoding unit is specifically configured to: Decoding the encoded code stream using the first decoder to obtain a decoded frame of the reconstructed frame; determining the first distortion metric based on the reconstructed frame and a decoded frame of the reconstructed frame; The first bit rate is determined according to the encoded bit stream.

18. The device according to any one of claims 12 to 17, characterized in that: The second encoder and the second decoder are obtained by iteratively updating the parameters of the first encoder and the first decoder N times according to the first loss; N is an integer greater than 0; the first updating unit is further used for: The number of iterations N is sent to a decoding device, so that the decoding device iteratively updates the parameters of the video decoding network according to the number of iterations N.

19. A parameter updating device for a video decoding network, characterized in that: Applied to a decoding device; the device comprises a second encoding unit, a second decoding unit and a second updating unit; The second encoding unit is used to encode the reconstructed frame by using the third encoder in the video decoding network to obtain a second encoding result; the reconstructed frame is obtained by decoding the code stream output by the third encoder by the third decoder in the video encoding network; the code stream is obtained by encoding the encoded frame by the third encoder; The second decoding unit is configured to obtain a second distortion metric and a second bit rate of the reconstructed frame according to the third decoder and the second encoding result; The second updating unit is used to determine a sixth loss according to the second distortion metric and the second bit rate; and update parameters of the third encoder and the third decoder according to the sixth loss to obtain a fourth encoder and a fourth decoder.

20. The device according to claim 19, characterized in that The second updating unit is further used for: Receive the flag bit sent by the encoding device; If the flag bit is the first value, resetting the parameters of the fourth encoder and the fourth decoder to the parameters of the initial encoder and the parameters of the initial decoder respectively; If the flag bit is the second value, the parameters of the fourth encoder and the fourth decoder are not reset.

21. The device according to claim 19 or 20, characterized in that The second encoding result includes an encoded code stream of the reconstructed frame; in terms of obtaining a second distortion metric and a second code rate of the reconstructed frame according to the third decoder and the second encoding result, the second decoding unit is specifically configured to: Decoding the encoded bitstream using the third decoder to obtain a decoded frame of the reconstructed frame; determining the second distortion metric based on the reconstructed frame and a decoded frame of the reconstructed frame; The second bit rate is determined according to the encoded bit stream.

22. The device according to any one of claims 19 to 21, characterized in that In terms of updating the parameters of the third encoder and the third decoder according to the sixth loss to obtain a fourth encoder and a fourth decoder, the second updating unit is specifically used for: Iteratively updating the parameters of the third encoder and the third decoder according to the sixth loss until the number of iterations reaches M times, fixing the parameters updated in the Mth iteration, and obtaining the fourth encoder and the fourth decoder; M is an integer greater than 0; or, Receiving the number of iterations N sent by the encoding device; Iteratively update the parameters of the third encoder and the third decoder according to the sixth loss until the number of iterations reaches N times, fix the parameters updated by the Nth iteration, and obtain the fourth encoder and the fourth decoder.

23. A coding device, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store one or more programs, and the programs include instructions for executing the method according to any one of claims 1 to 7.

24. A decoding device, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store one or more programs, and the programs include instructions for executing the method according to any one of claims 8 to 11.

25. A chip, characterized in that: include: A processor, configured to call and run a computer program from a memory, so that a device equipped with the chip executes a method as described in any one of claims 1-7 or 8-11.

26. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 7 or 8 to 11 is implemented.

27. A computer program product, characterized in that The computer program product comprises: a computer program code, and when the computer program code is executed, the method according to any one of claims 1-7 or 8-11 is implemented.

Citation Information

Cited By

  • Video processing method and device and electronic equipment

    CN119520858A