Network training method and electronic equipment

Abstract

The embodiment of the invention provides a network training method and electronic equipment. The method comprises the following steps: acquiring video data; encoding the video data based on the first video processing network, and decoding the encoding result of the video data based on the first video processing network to obtain reconstructed video data; determining a first loss of each frame of reconstructed image in the plurality of frames of reconstructed images, and determining a first loss weight of each frame of reconstructed image; wherein if the first loss weight of the ith frame of reconstructed image is greater than the first loss weight of the kth frame of reconstructed image, the accumulative error of the ith frame of reconstructed image is greater than the accumulative error of the kth frame of reconstructed image, and the ith frame of reconstructed image and the kth frame of reconstructed image belong to the same image group; and training the first video processing network based on the target loss of the multiple frames of reconstructed images. And subsequently, the trained first video processing network is used for video coding and decoding, so that the accumulative error of the reconstructed video data can be reduced, and the compression performance of the video data is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of coding and decoding, and in particular to a network training method and an electronic device. Background Art

[0002] As videos evolve from high-definition to ultra-high-definition, people have higher and higher requirements for video quality. At the same time, high-definition videos have higher and higher requirements for bandwidth and storage. Accordingly, due to cost control considerations for bandwidth, transmission delay, and storage, the demand for video encoding, i.e., video processing, has become more and more urgent.

[0003] Among them, the artificial intelligence (AI) video compression (or codec) algorithm is based on deep learning and has better compression effect than traditional video compression technologies (such as H265 and H266).

[0004] Since there is temporal continuity between video frames, the amount of change between adjacent frames is relatively small, which means that there is temporal redundancy between video frames; thus, inter-frame coding can be used to reduce the bitrate overhead. In the inter-frame coding process of AI video coding, many steps such as prediction and entropy coding of residual information will introduce errors; thus, in the process of encoding each group of pictures (Group of Pictures, GOP), as the timing increases, the cumulative error will continue to increase, and as the cumulative error increases, the compression performance of the video will also decrease. Summary of the invention

[0005] In order to solve the above technical problems, the present application provides a network training method and an electronic device. The first video processing network trained by the network training method is used for video encoding and decoding, which can reduce the cumulative error of reconstructed video data and improve compression performance.

[0006] In a first aspect, an embodiment of the present application provides a network training method, the method comprising: first, obtaining video data, the video data comprising multiple frames of original images; then, encoding the video data based on a first video processing network and then decoding the encoded result of the video data based on the first video processing network to obtain reconstructed video data; wherein the reconstructed video data comprises multiple frames of reconstructed images; thereafter, determining a first loss of each reconstructed image in the multiple frames of reconstructed images, and determining a first loss weight of each reconstructed image in the multiple frames of reconstructed images; wherein, if the first loss weight of the i-th frame of reconstructed image is greater than the first loss weight of the k-th frame of reconstructed image, then the cumulative error of the i-th frame of reconstructed image is greater than the cumulative error of the k-th frame of reconstructed image; the i-th frame of reconstructed image and the k-th frame of reconstructed image belong to the same image group, i and k are positive integers; then, based on the target loss of the multiple frames of reconstructed images, the first video processing network is trained; wherein the target loss of the j-th frame of reconstructed image in the multiple frames of reconstructed images is determined according to the product of the first loss of the j-th frame of reconstructed image and the first loss weight of the j-th frame of reconstructed image, and j is a positive integer.

[0007] Among them, during the training process, the network parameters of the first video processing network are adjusted according to the target loss of each frame of the reconstructed image; for the i-th frame of the reconstructed image with a large cumulative error, the first loss weight of the i-th frame of the reconstructed image is determined to be large, and correspondingly, the target loss of the i-th frame of the reconstructed image will also be increased; and during the training process, the first video processing network is usually trained with the goal of minimizing the loss (i.e., minimizing the target loss of each frame of the reconstructed image); therefore, after the first video processing network is trained with the goal of minimizing the target loss of the i-th frame of the reconstructed image, the cumulative error of the i-th frame of the reconstructed image obtained by encoding and decoding using the trained first video processing network will be reduced, and then the compression performance of the trained first video processing network for the i-th frame of the original image will also be improved, so that the compression performance of each frame can be balanced. Subsequently, using the trained first video processing network for video encoding and decoding can reduce the cumulative error of the reconstructed video data and improve the compression performance of the video data.

[0008] Exemplarily, the first loss may refer to a loss used to measure the image quality of a reconstructed image, such as distortion loss, etc., which is not limited in the present application.

[0009] Exemplarily, the cumulative error of the reconstructed image of the i-th frame may refer to the error introduced in the process of encoding the i-th frame original image by the error between the reconstructed image of the i-th frame corresponding to the reference frame and the original image of the i-th frame corresponding to the reference frame. For example, in a GOP, the second frame original image is encoded with reference to the first frame reconstructed image. Since some steps in the encoding process may introduce errors, there is an error 1 between the second frame reconstructed image and the second frame original image. The third frame original image is encoded with reference to the second frame reconstructed image. There is an error 2 between the third frame reconstructed image and the third frame original image; wherein, error 2 includes error 3 introduced by error 1 in the process of encoding the third frame original image and error 4 introduced by some steps in encoding the third frame original image; error 3 may be referred to as the cumulative error.

[0010] Exemplarily, multiple video data can be obtained as training data for training the first video processing network. Among them, a video data can include at least one image group (image group, i.e., GOP, refers to a group of images between I frames and I frames, and I frame, i.e., Intra frame, is an intra-frame coded frame), and each image group can include N (N is an integer greater than 1) frames of original images. Exemplarily, the N frames of original images included in an image group can be used as a batch to train the first video processing network. Specifically, the target losses of the multiple frames of reconstructed images contained in the image group can be added to obtain the target loss of the image group; then, based on the target loss of the image group, the first video processing network is trained.

[0011] Exemplarily, there may be multiple reconstructed video data, each of which may include at least one image group, and each image group may include N frames of reconstructed images.

[0012] Exemplarily, i and k are integers between 1 and N (the values ​​of i and k can be 1 or N).

[0013] It should be noted that the present application only performs inter-frame encoding and inter-frame decoding on the video data based on the first video processing network, that is, the present application only encodes and decodes the original images of the 2nd to Nth frames in each image group of the video data based on the first video processing network; for the original image of the 1st frame of each image group of the video data, the original image of the 1st frame in each image group of the video data can be encoded based on the AI ​​image encoding framework, and the encoding result of the original image of the 1st frame in each image group of the video data can be decoded based on the AI ​​image decoding framework. Correspondingly, during the training process, the first video processing network can be trained based on the first loss and the first loss weight of the reconstructed image of the 2nd to Nth frames of each image group.

[0014] According to the first aspect, determining the first loss weight of each reconstructed image frame includes: determining the first loss weight of each reconstructed image frame based on the cumulative error of each reconstructed image frame. In this way, the size of the first loss weight of each reconstructed image frame can be determined according to the size of the cumulative error of each reconstructed image frame.

[0015] That is to say, when the first loss weight of the reconstructed image of the i-th frame is greater than the first loss weight of the reconstructed image of the k-th frame, the cumulative error of the reconstructed image of the i-th frame is greater than the cumulative error of the reconstructed image of the k-th frame. There are two understandings here: only when the cumulative error of the i-th reconstructed image is greater than the cumulative error of the reconstructed image of the k-th frame, the first loss weight of the reconstructed image of the i-th frame is greater than the first loss weight of the reconstructed image of the k-th frame; or, when the cumulative error of the i-th reconstructed image is much larger than the cumulative error of the reconstructed image of the k-th frame (for example, the difference between the cumulative error of the i-th reconstructed image and the cumulative error of the reconstructed image of the k-th frame is greater than 2δ, δ is a preset value), the first loss weight of the reconstructed image of the i-th frame is greater than the first loss weight of the reconstructed image of the k-th frame.

[0016] According to the first aspect, or any implementation of the first aspect above, based on the cumulative error of each reconstructed image frame, determining the first loss weight of each reconstructed image frame includes: for the j-th reconstructed image frame in the multiple reconstructed images: determining the difference between the first loss of the j-th reconstructed image frame and the first loss of the reference reconstructed image corresponding to the j-th reconstructed image frame, and obtaining the loss difference of the j-th reconstructed image frame; wherein the loss difference of the j-th reconstructed image frame is used to characterize the cumulative error of the j-th reconstructed image frame; based on the loss difference of the j-th reconstructed image frame and a preset range, determining the first loss weight of the j-th reconstructed image frame. That is, the size of the first loss weight of each reconstructed image frame is determined according to the size of the difference between the first loss of each reconstructed image frame and the first loss of the reference reconstructed image corresponding to each reconstructed image frame.

[0017] Exemplarily, the cumulative error of the reference reconstructed image corresponding to each frame of the reconstructed image is smaller than the cumulative error of each frame of the reconstructed image.

[0018] Exemplarily, the loss difference of the j-th frame reconstructed image may be a positive number, a negative number, or zero.

[0019] It should be noted that, when the cumulative error of the reconstructed image is larger, if the loss difference of the reconstructed image is the first loss of the reconstructed image minus the first loss of the reconstructed image corresponding to the reference reconstructed image, the loss difference of the reconstructed image is larger; if the loss difference of the reconstructed image is the first loss of the reconstructed image corresponding to the reference reconstructed image minus the first loss of the reconstructed image, the loss difference of the reconstructed image is smaller.

[0020] For example, j is an integer between 1 and N (the value of j can be 1 or N).

[0021] According to the first aspect, or any implementation of the first aspect above, based on the loss difference of the j-th frame reconstructed image and the preset range, determining the first loss weight of the j-th frame reconstructed image includes: determining the weight coefficient of the j-th frame reconstructed image based on the loss difference of the j-th frame reconstructed image and the preset range; taking the product value of the weight coefficient of the j-th frame reconstructed image and the first loss of the j-th frame reconstructed image in the previous iteration cycle as the first loss weight of the j-th frame reconstructed image in the current iteration cycle. That is, according to the size of the loss difference of the reconstructed image, the first loss weight of the reconstructed image determined in the previous iteration cycle is increased or decreased, and then the first loss weight of the reconstructed image in the current cycle is obtained; thereby setting the first loss weight of the reconstructed images with different cumulative error sizes to different values.

[0022] According to the first aspect, or any implementation method of the first aspect above, the preset range includes a first boundary value and a second boundary value, and the first boundary value is less than the second boundary value; based on the loss difference of the j-th frame reconstructed image and the preset range, the weight coefficient of the j-th frame reconstructed image is determined, including: when the loss difference of the j-th frame reconstructed image belongs to the preset range, the first preset coefficient is used as the weight coefficient of the j-th frame reconstructed image; when the loss difference of the j-th frame reconstructed image is greater than the second boundary value, the second preset coefficient is used as the weight coefficient of the j-th frame reconstructed image, and the second preset coefficient is greater than 1; when the loss difference of the j-th frame reconstructed image is less than the first boundary value, the third preset coefficient is used as the weight coefficient of the j-th frame reconstructed image, and the third preset coefficient is less than 1; wherein the first preset coefficient is greater than the third preset coefficient and less than the second preset coefficient, and j is a positive integer.

[0023] The loss difference of the reconstructed image is the first loss of the reconstructed image minus the first loss of the reference reconstructed image corresponding to the reconstructed image.

[0024] That is to say, the greater the difference between the first loss of the reconstructed image and the first loss of the reference reconstructed image corresponding to the reconstructed image, the greater the weight coefficient; in this way, the weight coefficient of the reconstructed image with a larger loss difference can be made larger, thereby improving the compression performance of the first video processing network for the original image with a large loss difference corresponding to the reconstructed image.

[0025] Exemplarily, if the loss difference of the reconstructed image is the first loss of the reconstructed image corresponding to the reference reconstructed image minus the first loss of the reconstructed image, then when the loss difference of the reconstructed image of the j-th frame belongs to the preset range, the first preset coefficient is used as the weight coefficient of the reconstructed image of the j-th frame; when the loss difference of the reconstructed image of the j-th frame is greater than the second boundary value, the third preset coefficient is used as the weight coefficient of the reconstructed image of the j-th frame, and the third preset coefficient is greater than 1; when the loss difference of the reconstructed image of the j-th frame is less than the first boundary value, the second preset coefficient is used as the weight coefficient of the reconstructed image of the j-th frame, and the second preset coefficient is less than 1; wherein the first preset coefficient is greater than the third preset coefficient and less than the second preset coefficient, and j is a positive integer.

[0026] For example, the second preset coefficient is 0.9, the third preset coefficient is 1.1, and the first preset coefficient may be 1.

[0027] It should be understood that more preset ranges may be pre-set, so that the weight coefficients of reconstructed images with loss differences belonging to different preset ranges are different, and the present application does not impose any limitation on this.

[0028] According to the first aspect, or any implementation of the first aspect above, the reference reconstructed image corresponding to the j-th frame reconstructed image includes at least one of the following: a reference reconstructed image obtained by encoding the j-th frame original image based on the second video processing network and then decoding the encoded result of the j-th frame original image based on the second video processing network; wherein the cumulative error of the reference reconstructed image corresponding to the j-th frame reconstructed image is less than the cumulative error of the j-th frame reconstructed image; or, the first m frames reconstructed image of the j-th frame reconstructed image in the image group; wherein the first loss of the reference reconstructed image corresponding to the j-th frame reconstructed image is the average of the first losses of the m frames reconstructed image before the j-th frame reconstructed image, and m is a positive integer.

[0029] Exemplarily, the second video processing network can be a video processing network composed of an AI condition I video encoding network and an AI condition I video decoding network (the AI ​​condition I video encoding network encodes the feature information of the original image, and the calculated first optical flow is not used to calculate the residual information, but only to calculate the probability distribution of the feature information of the original image. Therefore, the reconstructed images output by the AI ​​condition I video encoding network and the AI ​​condition I video decoding network do not have cumulative errors).

[0030] According to the first aspect, or any implementation method of the first aspect above, based on the cumulative error of each frame of reconstructed image, the first loss weight of each frame of reconstructed image is determined, including: determining the timing information corresponding to each frame of reconstructed image, the timing information of each frame of reconstructed image is used to characterize the cumulative error of each frame of reconstructed image; based on the timing information corresponding to each frame of reconstructed image, determining the first loss weight of each frame of reconstructed image; wherein i is greater than k.

[0031] Since the current frame original image is encoded based on the reconstructed images of other frames before it during the inter-frame coding process, the cumulative error of the reconstructed image that is later in time in an image group is greater; furthermore, the present application can determine the timing information corresponding to each reconstructed image frame, and determine the first loss weight of each reconstructed image frame based on the timing information of each reconstructed image frame.

[0032] According to the first aspect, or any implementation of the first aspect above, the method further includes: obtaining the second loss of each reconstructed image frame in the multiple reconstructed images and the second loss weight of each reconstructed image frame; wherein the first loss is the distortion loss, and the second loss is the bit rate loss; wherein the target loss of the j-th reconstructed image frame is determined by weighted calculation of the first loss of the j-th reconstructed image frame and the second loss of the j-th reconstructed image frame according to the first loss weight of the j-th reconstructed image frame and the second loss weight of the j-th reconstructed image frame. In this way, the distortion of the reconstructed image output by the first video processing network and the bit rate of the bit stream obtained by encoding the original image can be balanced.

[0033] Exemplarily, the distortion loss may refer to the difference (eg, error) between the original image and the reconstructed image.

[0034] Exemplarily, the bit rate loss may refer to a bit rate obtained by estimating the bit rate during the encoding process.

[0035] According to the first aspect, or any implementation of the first aspect above, the reconstructed image includes multiple components; the first loss of each frame of the reconstructed image includes the first losses of the multiple components of each frame of the reconstructed image, and the first loss weight of each frame of the reconstructed image includes the first loss weights of the multiple components of each frame of the reconstructed image. Since the proportions of different components of the reconstructed image are different, determining the corresponding first loss weights for different components can effectively reduce the accumulated errors of different components.

[0036] Exemplarily, the reconstructed image includes multiple components, which can also be referred to as the reconstructed image including multiple channels. For example, if the reconstructed image is a YUV image, the reconstructed image includes a Y component, a U component, and a V component, or in other words, the reconstructed image includes a Y channel, a U channel, and a V channel.

[0037] According to the first aspect, or any implementation of the first aspect above, the first video processing network includes a video encoding network and a video decoding network; encoding the video data based on the first video processing network and then decoding the encoded result of the video data based on the first video processing network to obtain reconstructed video data, including: encoding the video data based on the video encoding network and then decoding the encoded result of the video data based on the video decoding network to obtain reconstructed video data.

[0038] In one possible approach, the AI ​​video encoding framework can be implemented only by a video encoding network, and the AI ​​video decoding framework can be implemented only by a video decoding network; in this case, the video data can be input into the video encoding network to obtain a bit stream; the bit stream can be input into the video decoding network to obtain reconstructed video data.

[0039] In one possible approach, the AI ​​video encoding framework can be implemented by a video encoding network and a module implemented by an algorithm, and the AI ​​video decoding framework can be implemented by a video decoding network and a module implemented by an algorithm; at this time, the video data can be encoded based on the video encoding network and the module implemented by the algorithm, and then the encoded result of the video data can be decoded based on the video decoding network and the module implemented by the algorithm to obtain reconstructed video data.

[0040] In a second aspect, an embodiment of the present application provides an encoding method, the method comprising: acquiring video data; encoding the video data based on a video encoding network to obtain a code stream; wherein the video encoding network is trained based on a target loss of multiple frames of reconstructed images, and the target loss of the j-th frame reconstructed image in the multiple frames of reconstructed images is determined based on the product of the first loss of the j-th frame reconstructed image and the first loss weight of the j-th frame reconstructed image, where j is a positive integer; if the first loss weight of the i-th frame reconstructed image is greater than the first loss weight of the k-th frame reconstructed image, then the cumulative error of the i-th frame reconstructed image is greater than the cumulative error of the k-th frame reconstructed image; the i-th frame reconstructed image and the k-th frame reconstructed image belong to the same image group, and i and k are positive integers.

[0041] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0042] In a third aspect, an embodiment of the present application also provides a decoding method, the method comprising: receiving a code stream; decoding the code stream based on a video decoding network to obtain reconstructed video data; wherein the video decoding network is trained based on a target loss of multiple frames of reconstructed images, and the target loss of the j-th frame reconstructed image in the multiple frames of reconstructed images is determined based on the product of the first loss of the j-th frame reconstructed image and the first loss weight of the j-th frame reconstructed image, where j is a positive integer; if the first loss weight of the i-th frame reconstructed image is greater than the first loss weight of the k-th frame reconstructed image, then the cumulative error of the i-th frame reconstructed image is greater than the cumulative error of the k-th frame reconstructed image; the i-th frame reconstructed image and the k-th frame reconstructed image belong to the same image group, and i and k are positive integers.

[0043] The third aspect and any implementation of the third aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the third aspect and any implementation of the third aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0044] In a fourth aspect, an embodiment of the present application further provides a network training device, the device comprising:

[0045] An acquisition module is used to acquire video data, where the video data includes multiple frames of original images;

[0046] A compression module, configured to encode the video data based on the first video processing network and then decode the encoded video data based on the first video processing network to obtain reconstructed video data; wherein the reconstructed video data includes a plurality of frames of reconstructed images;

[0047] A loss and weight determination module, used to determine a first loss of each reconstructed image in a plurality of reconstructed image frames, and to determine a first loss weight of each reconstructed image frame; wherein, if the first loss weight of the i-th reconstructed image frame is greater than the first loss weight of the k-th reconstructed image frame, then the cumulative error of the i-th reconstructed image frame is greater than the cumulative error of the k-th reconstructed image frame, the i-th reconstructed image frame and the k-th reconstructed image frame belong to the same image group, and i and k are positive integers;

[0048] A training module is used to train the first video processing network based on the target loss of multiple frames of reconstructed images; wherein the target loss of the j-th frame reconstructed image in the multiple frames of reconstructed images is determined according to the product of the first loss of the j-th frame reconstructed image and the first loss weight of the j-th frame reconstructed image, and j is a positive integer.

[0049] It should be understood that the network training device of the fourth aspect can be used to execute the above-mentioned first aspect or any implementation method of the first aspect, which will not be described in detail here.

[0050] The fourth aspect and any implementation of the fourth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fourth aspect and any implementation of the fourth aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0051] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, wherein the memory is coupled to the processor; the memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the network training method in the first aspect or any possible implementation of the first aspect.

[0052] The fifth aspect and any implementation of the fifth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fifth aspect and any implementation of the fifth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0053] In the sixth aspect, an embodiment of the present application provides a chip, comprising one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the method in the first aspect or any implementation of the first aspect is executed.

[0054] The sixth aspect and any implementation of the sixth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the sixth aspect and any implementation of the sixth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0055] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, the computer or the processor executes the network training method in the first aspect or any possible implementation of the first aspect.

[0056] The seventh aspect and any implementation of the seventh aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the seventh aspect and any implementation of the seventh aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0057] In an eighth aspect, an embodiment of the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer or the processor executes the network training method in the first aspect or any possible implementation of the first aspect.

[0058] The eighth aspect and any implementation of the eighth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the eighth aspect and any implementation of the eighth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0059] In the ninth aspect, an embodiment of the present application provides a coding device, comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, and when the program instructions are executed by the processor, the coding device executes the coding method in the second aspect or any possible implementation of the second aspect.

[0060] The ninth aspect and any implementation of the ninth aspect correspond to the second aspect and any implementation of the second aspect respectively. The technical effects corresponding to the ninth aspect and any implementation of the ninth aspect can refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect.

[0061] In the tenth aspect, an embodiment of the present application provides a decoding device, comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, and when the program instructions are executed by the processor, the decoding device executes the decoding method in the third aspect or any possible implementation of the third aspect.

[0062] The tenth aspect and any implementation of the tenth aspect correspond to the third aspect and any implementation of the third aspect, respectively. The technical effects corresponding to the tenth aspect and any implementation of the tenth aspect can refer to the technical effects corresponding to the third aspect and any implementation of the third aspect, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 The figure is a schematic diagram of an artificial intelligence main framework shown as an example;

[0064] Figure 2A is a schematic diagram of an application framework shown as an example;

[0065] Figure 2B A schematic diagram of a storage framework is shown as an example;

[0066] Figure 2C is a schematic diagram of an exemplary transmission framework;

[0067] Figure 3A is a schematic diagram of an exemplary training process;

[0068] Figure 3B is a schematic diagram of the structure of a video coding network shown as an example;

[0069] Figure 3C is a schematic diagram of the structure of a video decoding network shown as an example;

[0070] Figure 4 is a schematic diagram of an exemplary training process;

[0071] Figure 5A is a schematic diagram of an exemplary training process;

[0072] Figure 5B is a schematic diagram of an exemplary training process;

[0073] Fig. 6A is a schematic diagram of an exemplary training process;

[0074] Figure 6B is a schematic diagram of a first loss weight of a multi-frame reconstructed image shown as an example;

[0075] Fig. 7A is a schematic diagram of an exemplary training process;

[0076] Figure 7B is a schematic diagram of an exemplary training process;

[0077] Figure 7C is a schematic diagram showing an exemplary compression performance comparison;

[0078] Figure 8 is a schematic diagram of an exemplary network training device;

[0079] Fig. 9 Schematic diagram of the structure of the device shown as an example. DETAILED DESCRIPTION

[0080] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0081] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0082] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects rather than to describe a specific order of objects. For example, a first target object and a second target object are used to distinguish different target objects rather than to describe a specific order of target objects.

[0083] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0084] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.

[0085] Figure 1 This is a schematic diagram of an artificial intelligence main framework, which describes the overall workflow of the artificial intelligence system and is applicable to general artificial intelligence field requirements.

[0086] The following is an explanation of the above artificial intelligence theme framework from two dimensions: "intelligent information chain" (horizontal axis) and "IT (Information Technology) value chain" (vertical axis).

[0087] The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, data undergoes a condensed process of "data-information-knowledge-wisdom".

[0088] The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of artificial intelligence, information (providing and processing technology implementation) to the system's industrial ecological process.

[0089] (1) Infrastructure:

[0090] The infrastructure provides computing power support for the artificial intelligence system, enables communication with the outside world, and supports it through the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips (central processing unit (CPU), embedded neural network processor (NPU), graphics processing unit (GPU), application specific integrated circuit (ASIC), field programmable gate array (FPGA) and other hardware acceleration chips); the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

[0091] (2) Data

[0092] The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature, and humidity.

[0093] (3) Data processing

[0094] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

[0095] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0096] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0097] Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.

[0098] (4) General ability

[0099] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, material map generation, etc.

[0100] (5) Smart products and industry applications

[0101] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical applications. Its application areas mainly include: smart manufacturing, smart transportation, smart home, smart medical care, smart security, autonomous driving, smart terminals, etc.

[0102] The first video processing network involved in this application can be used to implement machine learning, deep learning, search, reasoning, decision-making, etc. The first video processing network involved in this application can include various types of neural networks, such as deep neural networks (DNN), convolutional neural networks (CNN), recurrent neural networks (RNN), residual networks, neural networks using transformer models or other neural networks, etc., and this application does not limit this.

[0103] The work of each layer in the neural network can be expressed mathematically as To describe: From a physical level, the work of each layer in a neural network can be understood as completing the transformation from input space to output space (i.e., from row space to column space of a matrix) through five operations on the input space (a set of input vectors). These five operations include: 1. Dimension increase / reduction; 2. Enlargement / reduction; 3. Rotation; 4. Translation; 5. "Bending". Operations 1, 2, and 3 are represented by Completed, operation 4 is completed by +b, and operation 5 is implemented by a(). The word "space" is used here because the object being classified is not a single thing, but a class of things, and space refers to the collection of all individuals of this class of things. Among them, W is a weight vector, and each value in the vector represents the weight value of a neuron in this layer of neural network. The vector W determines the spatial transformation from the input space to the output space above, that is, the weight W of each layer controls how to transform the space.

[0104] The purpose of training a neural network is to eventually obtain the weight matrix of all layers of the trained neural network (the weight matrix formed by many layers of vectors W). Therefore, the training process of a neural network is essentially learning how to control spatial transformations, and more specifically learning the weight matrix.

[0105] The first video processing network involved in this application may include a video encoding network and a video decoding network, wherein the video encoding network can be used to encode video data, and the video decoding network can be used to decode based on the encoding result of the video data. The first video processing network involved in this application can be applied to all services of video transmission and video storage, such as terminal recording, video album, Huawei Cloud, video surveillance, vehicle-mounted camera, live broadcast, etc., and this application does not limit this.

[0106] It should be noted that the AI ​​video encoding framework can be implemented only by a video encoding network, or the AI ​​video encoding framework can be implemented by a video encoding network and a module implemented by an algorithm; the AI ​​video decoding framework can be implemented only by a video decoding network, or the AI ​​video decoding framework can be implemented by a video decoding network and a module implemented by an algorithm.

[0107] This application uses the AI ​​video encoding framework implemented only by the video encoding network, and the AI ​​video decoding framework implemented only by the video decoding network as an example for explanation. It should be understood that when the AI ​​video encoding framework is implemented by the video encoding network and the module implemented by the algorithm, and the AI ​​video decoding framework is implemented by the video decoding network and the module implemented by the algorithm, in the forward calculation process, the video encoding network can cooperate with the module implemented by the algorithm to encode the video data, and the video decoding network can cooperate with the module implemented by the algorithm to decode based on the encoding result of the video data; in the back propagation process, only the video encoding network and the video decoding network can be trained.

[0108] Figure 2A The following is a schematic diagram of an application framework shown as an example.

[0109] Reference Figure 2A For example, a camera can capture video and obtain video data; then, a video encoding network can perform AI video encoding on the video data to obtain a bitstream. After that, the bitstream can be stored locally or transmitted to a remote device.

[0110] Continue to refer to Figure 2A In one possible approach, after the bitstream is stored locally, when the video needs to be played or edited, the video decoding network can perform AI video decoding on the bitstream to obtain reconstructed video data; then, the reconstructed video data can be played or edited.

[0111] Continue to refer to Figure 2A In one possible approach, after the remote device receives the bitstream, when it needs to play or edit the video, the video decoding network can perform AI video decoding on the bitstream to obtain reconstructed video data; then, the reconstructed video data can be played or edited.

[0112] Figure 2B A schematic diagram of a storage framework is shown as an example.

[0113] Reference Figure 2B , exemplarily, the video encoding network may include an AI encoding network and an entropy encoding network, and the video decoding network may include an AI decoding network and an entropy decoding network; it should be understood that, Figure 2B This is just an example of the present application. The video encoding network and the video decoding network may include Figure 2B More networks are shown, but this application is not limited to them.

[0114] Exemplarily, the AI ​​encoding network and the AI ​​decoding network can be set in the NPU or GPU. Exemplarily, the entropy encoding network and the entropy decoding network can be set in the CPU. Exemplarily, the file saving module and the file loading module can be set in the CPU.

[0115] Reference Figure 2B , exemplarily, after the Camera captures the video data, it can be input into the AI ​​coding network. Then, the AI ​​coding network can perform the following processing for each frame of the video data: perform prediction (including intra-frame prediction and inter-frame prediction, this application takes inter-frame prediction as an example), determine the prediction information; and determine the residual information based on the prediction information and the original image, and determine the probability distribution of the residual information; then, input the residual information and the probability distribution of the residual information into the entropy coding network. Afterwards, the entropy coding network can entropy encode the residual information according to the probability distribution of the residual information to obtain a code stream; and input the code stream into the file saving module for saving to obtain a file.

[0116] Continue to refer to Figure 2B For example, when a video needs to be played or edited, the file loading module can load the file and then input the bitstream in the file into the entropy decoding network. Then, the entropy decoding network can obtain the probability distribution of the residual information from the AI ​​decoding network, and according to the probability distribution of the residual information, entropy decode the residual information from the bitstream and output the residual information to the AI ​​decoding network. After that, the AI ​​decoding network can perform inter-frame prediction to obtain prediction information; then, reconstruct according to the prediction information and the residual information to obtain a reconstructed image; and then, reconstructed video data can be obtained.

[0117] Figure 2CFIG. 1 is a schematic diagram of an exemplary transmission framework.

[0118] Reference Figure 2C , exemplarily, the encoding process of the encoding end may be as follows:

[0119] Exemplarily, after the camera captures the video data, it can be input into the AI ​​coding network. Then, the AI ​​coding network can perform the following processing for each frame of the video data: perform prediction (including intra-frame prediction and inter-frame prediction, this application takes inter-frame prediction as an example), determine the prediction information; and determine the residual information based on the prediction information and the original image, and determine the probability distribution of the residual information; then, input the residual information and the probability distribution of the residual information into the entropy coding network. Afterwards, the entropy coding network can entropy encode the residual information according to the probability distribution of the residual information to obtain a bitstream; and send the bitstream to a server in the cloud.

[0120] Afterwards, the server can send the code stream to the decoding end. The server can be a single server or a server cluster, which is not limited in this application.

[0121] Continue to refer to Figure 2C , for example, after the decoding end receives the code stream sent by the server, the decoding process can be as follows:

[0122] Exemplarily, the file loading module can load the file and then send the code stream in the file to the entropy decoding network. Then, the AI ​​decoding network can perform inter-frame prediction to obtain prediction information; the entropy decoding network can obtain the probability distribution of residual information from the AI ​​decoding network, and according to the probability distribution of residual information, entropy decode the residual information from the code stream and output the residual information to the AI ​​decoding network. Subsequently, reconstruction is performed based on the prediction information and residual information to obtain a reconstructed image; and then, reconstructed video data can be obtained.

[0123] It should be understood that the encoding end can also directly send the code stream to the decoding end without forwarding through the server, and this application does not impose any restrictions on this.

[0124] Figure 3A FIG. 1 is a schematic diagram of an exemplary training process.

[0125] Reference Figure 3A Exemplarily, the video data can be input into a video encoding network, encoded by the video encoding network, and output a code stream to a video decoding network; the video decoding network can decode the code stream and output reconstructed video data; then, a first loss can be determined based on the video data and the reconstructed video data; the first loss is used for back propagation to train the video encoding network and the video decoding network.

[0126] Figure 3B FIG. 1 is a schematic diagram showing the structure of an exemplary video coding network. Figure 3B In the embodiment, the inter-frame coding process is shown.

[0127] Reference Figure 3B , exemplarily, the AI ​​coding network may include a motion estimation network, an optical flow coding network, a first entropy estimation network, an optical flow decoding network, a prediction network, a feature extraction network, a residual coding network, a second entropy estimation network, a residual decoding network, and a reconstruction network; it should be understood that the AI ​​coding network may also include Figure 3B More or fewer networks may be shown, and the present application is not limited thereto.

[0128] Reference Figure 3B , exemplarily, the entropy coding network may include a first entropy coding network and a second entropy coding network.

[0129] Reference Figure 3B Exemplarily, the video encoding network may further include an entropy decoding network, and the entropy decoding network may include a first entropy decoding network and a second entropy decoding network.

[0130] It should be noted that Figure 3B The feature extraction network in the dashed box is an optional network.

[0131] Continue to refer to Figure 3B , taking inter-frame coding of an image frame in video data as an example for explanation; wherein the AI ​​video coding process can be as follows:

[0132] Exemplarily, the original image of the current frame and the reconstructed image of the reference frame corresponding to the current frame can be input into the motion estimation network. Then, the motion estimation network can perform motion estimation based on the original image of the current frame and the reconstructed image of the reference frame to determine the optical flow of the current frame (hereinafter referred to as the first optical flow); and output the first optical flow of the current frame to the optical flow encoding network. The optical flow encoding network (also referred to as the optical flow feature extraction network) can encode the first optical flow of the current frame (it should be noted that the encoding performed by the optical flow encoding network is essentially feature extraction), obtain the feature information of the first optical flow of the current frame (also referred to as the compressed feature of the first optical flow of the current frame) and output it to the first entropy estimation network.

[0133] Exemplarily, the first entropy estimation network can perform probability estimation on the feature information of the first optical flow of the current frame, and determine the probability distribution of the feature information of the first optical flow of the current frame (hereinafter referred to as the optical flow probability distribution of the current frame); then, the feature information of the first optical flow of the current frame and the optical flow probability distribution of the current frame are output to the first entropy coding network.

[0134] Exemplarily, the first entropy coding network may perform entropy coding on feature information of the first optical flow of the current frame according to the optical flow probability distribution of the current frame to obtain an optical flow code stream.

[0135] Exemplarily, on the one hand, the optical flow code stream can be sent / saved (for example, the optical flow code stream is sent to a server or a decoding end). On the other hand, the optical flow code stream can be output to the first entropy decoding network; in addition, the first entropy estimation network can also output the optical flow probability distribution of the current frame to the first entropy decoding network.

[0136] Exemplarily, the first entropy decoding network can entropy decode the optical flow code stream according to the optical flow probability distribution of the current frame to obtain the feature information of the first optical flow after decoding the current frame (which can also be called the feature information of the second optical flow of the current frame); and output the feature information of the second optical flow of the current frame to the optical flow decoding network.

[0137] Exemplarily, the optical flow decoding network can decode the feature information of the second optical flow of the current frame (it should be noted that the decoding performed by the optical flow decoding network is essentially feature recovery), obtain the second optical flow of the current frame and output the second optical flow of the current frame to the prediction network.

[0138] In one possible way, when the AI ​​coding network includes a feature extraction network, the reconstructed image of the reference frame can be input into the feature extraction network, and the feature extraction network extracts the features of the reconstructed image of the reference frame to obtain the feature information of the reconstructed image of the reference frame; and the feature information of the reconstructed image of the reference frame is output to the prediction network. The prediction network can make a prediction based on the second optical flow of the current frame and the feature information of the reconstructed image of the reference frame to obtain the prediction information of the current frame; and the prediction information of the current frame is output to the residual coding network. Correspondingly, the original image of the current frame can be input into the feature extraction network, and the feature extraction network extracts the features of the original image of the current frame to obtain the feature information of the original image of the current frame, and the feature information of the original image of the current frame is output to the residual coding network. The residual coding network outputs the residual information of the current frame (hereinafter referred to as the first residual information of the current frame) to the second entropy estimation network based on the feature information of the original image of the current frame and the prediction information of the current frame. Among them, the first residual information of the current frame can be the first residual of the current frame (wherein, the first residual can refer to the residual between the feature information of the original image of the current frame and the prediction information of the current frame, or it can be the feature information of the first residual of the current frame (wherein, the feature information of the first residual can also be called the compressed feature of the first residual, which can be encoded by the first residual of the residual coding network (it should be noted that the encoding performed by the residual coding network is essentially feature extraction)), and the present application does not impose any restrictions on this. The present application takes the first residual information of the current frame as the feature information of the first residual of the current frame as an example for explanation.

[0139] In one possible way, when the AI ​​coding network does not include a feature extraction network, the reconstructed image of the reference frame can be directly input into the prediction network, and the prediction network predicts the current frame based on the second optical flow of the current frame and the reconstructed image of the reference frame, and obtains the prediction information of the current frame; and outputs the prediction information of the current frame to the residual coding network. Correspondingly, the original image of the current frame can be directly input into the residual coding network; the residual coding network can output the first residual information of the current frame to the second entropy estimation network based on the original image of the current frame and the prediction information of the current frame.

[0140] Exemplarily, the second entropy estimation network can perform probability estimation on the first residual information of the current frame to determine the probability distribution of the first residual information of the current frame (hereinafter referred to as the residual probability distribution of the current frame); then, the first residual information of the current frame and the residual probability distribution of the current frame are output to the second entropy coding network.

[0141] Exemplarily, the second entropy coding network may perform entropy coding on the first residual information of the current frame according to the residual probability distribution of the current frame to obtain a residual code stream.

[0142] Exemplarily, on the one hand, the residual code stream can be sent / saved (for example, the residual code stream is sent to a server or a decoding end). On the other hand, the residual code stream can be output to the second entropy decoding network; in addition, the second entropy estimation network can also output the residual probability distribution of the current frame to the second entropy decoding network.

[0143] Exemplarily, the second entropy decoding network can entropy decode the residual code stream according to the residual probability distribution of the current frame to obtain the first residual information after entropy decoding of the current frame. When the first residual information of the current frame is the feature information of the first residual of the current frame, the first residual information after entropy decoding of the current frame can be output to the residual decoding network.

[0144] Exemplarily, the residual decoding network can decode the first residual information after entropy decoding of the current frame (it should be noted that the decoding performed by the residual decoding network is essentially feature recovery), obtain the second residual information of the current frame and output the second residual information of the current frame to the reconstruction network. It should be noted that when the first residual information of the current frame is the first residual of the current frame, the AI ​​video encoding framework may not include a residual decoding network; at this time, the second residual information of the current frame is the decoded first residual information obtained by entropy decoding of the current frame. In addition, it can also be said that the second residual information of the current frame is obtained by decoding the encoding result of the first residual information of the current frame (i.e., the residual code stream); when the first residual information of the current frame is the feature information of the first residual of the current frame, the decoding in this step includes entropy decoding and feature recovery; when the first residual information of the current frame is the first residual of the current frame, the decoding in this step is entropy decoding.

[0145] Exemplarily, the prediction network can also output the prediction information of the current frame to the reconstruction network. When the AI ​​coding network includes a feature extraction network, the reconstruction network can first reconstruct according to the second residual information of the current frame and the prediction information of the current frame to obtain the feature information of the reconstructed image of the current frame; then, the reconstruction network can perform feature transformation on the feature information of the reconstructed image of the current frame to obtain the reconstructed image of the current frame. When the AI ​​coding network does not include a feature extraction network, the reconstruction network reconstructs according to the second residual information of the current frame and the prediction information of the current frame to obtain the reconstructed image of the current frame.

[0146] Optionally, the present application may also encode information for determining the optical flow probability distribution of the current frame, and send it to the decoding end; so that the decoding end determines the optical flow probability distribution to obtain the second optical flow of the current frame; and encode information for determining the residual probability distribution of the current frame, and send it to the decoding end; so that the decoding end determines the residual probability distribution to obtain the second residual information of the current frame.

[0147] Figure 3C FIG. 4 is a schematic diagram of the structure of a video decoding network shown as an example.

[0148] Reference Figure 3C , exemplarily, the AI ​​decoding network may include: a first entropy estimation network, a second entropy estimation network, an optical flow decoding network, a prediction network, a feature extraction network, a residual decoding network, and a reconstruction network; it should be understood that the AI ​​encoding network may also include Figure 3C More or fewer networks may be shown, and the present application is not limited thereto.

[0149] Reference Figure 3C Exemplarily, the entropy decoding network may include a first entropy decoding network and a second entropy encoding network.

[0150] Continue to refer to Figure 3C , taking inter-frame decoding of an image frame in video data as an example for explanation; wherein the AI ​​video decoding process can be as follows:

[0151] Exemplarily, the decoding end may receive a code stream, which may include an optical flow code stream and a residual code stream. When the code stream received by the decoding end does not include a code stream for determining the optical flow probability distribution of the current frame, the first entropy estimation network may perform probability estimation based on the reconstructed image of the reference frame or the prediction information of the reference frame or the feature information of the reconstructed image of the reference frame, etc., determine the optical flow probability distribution of the current frame and output it to the first entropy decoding network. When the received code stream includes a code stream for determining the optical flow probability distribution of the current frame, the first entropy decoding network may entropy decode the information for determining the optical flow probability distribution of the current frame from the code stream and output it to the first entropy estimation network; then, the first entropy estimation network may perform probability estimation based on the information for determining the optical flow probability distribution of the current frame, determine the optical flow probability distribution of the current frame and output it to the first entropy decoding network.

[0152] Exemplarily, the first entropy decoding network can perform entropy decoding on the optical flow code stream according to the optical flow probability distribution of the current frame, obtain feature information of the second optical flow of the current frame, and output it to the optical flow decoding network.

[0153] Exemplarily, the optical flow decoding network can perform feature recovery on feature information of the second optical flow of the current frame, obtain the second optical flow of the current frame, and output the second optical flow of the current frame to the prediction network.

[0154] Exemplarily, when the received bitstream does not include a bitstream containing information for determining the residual probability distribution of the current frame, the second entropy estimation network may perform probability estimation based on the reconstructed image of the reference frame or the prediction information of the reference frame or the feature information of the reconstructed image of the reference frame, etc., determine the residual probability distribution of the current frame and output it to the second entropy decoding network. When the received bitstream includes a bitstream containing information for determining the residual probability distribution of the current frame, the second entropy decoding network may entropy decode the information for determining the residual probability distribution of the current frame from the bitstream; then, the second entropy estimation network performs probability estimation based on the information for determining the residual probability distribution of the current frame, determines the residual probability distribution of the current frame and outputs it to the second entropy decoding network.

[0155] Exemplarily, the second entropy decoding network can entropy decode the bitstream of information used to determine the residual probability distribution, obtain information used to determine the residual probability distribution of the current frame, and output the information used to determine the residual probability distribution of the current frame to the second entropy estimation network; then, the second entropy estimation network performs probability estimation based on the information for determining the residual probability distribution of the current frame, determines the residual probability distribution of the current frame and outputs it to the second entropy decoding network.

[0156] Exemplarily, the second entropy decoding network may perform entropy decoding on the residual code stream according to the residual probability distribution of the current frame, obtain the first residual information after entropy decoding of the current frame, and output it to the residual decoding network.

[0157] Exemplarily, the residual decoding network can perform feature recovery on the first residual information after entropy decoding of the current frame, obtain the second residual information of the current frame, and output the second residual information of the current frame to the reconstruction network.

[0158] In one possible way, when the AI ​​decoding network includes a feature extraction network, the reconstructed image of the reference frame can be input into the feature extraction network, and the feature extraction network extracts the features of the reconstructed image of the reference frame to obtain the feature information of the reconstructed image of the reference frame; and the feature information of the reconstructed image of the reference frame is output to the prediction network. The prediction network can make a prediction based on the second optical flow of the current frame and the feature information of the reconstructed image of the reference frame to obtain the prediction information of the current frame; and the prediction information of the current frame is output to the reconstruction network. Then, the reconstruction network can first reconstruct based on the second residual information of the current frame and the prediction information of the current frame to obtain the feature information of the reconstructed image of the current frame; thereafter, the reconstruction network can perform feature transformation (such as feature recovery) on the feature information of the reconstructed image of the current frame to obtain the reconstructed image of the current frame.

[0159] In one possible approach, when the AI ​​decoding network does not include a feature extraction network, the reconstructed image of the reference frame can be directly input into the prediction network, which predicts the current frame based on the second optical flow of the current frame and the reconstructed image of the reference frame, and outputs the prediction information of the current frame to the reconstruction network. Then, the reconstruction network reconstructs based on the second residual information of the current frame and the prediction information of the current frame, and obtains the reconstructed image of the current frame.

[0160] It should be understood that the present invention is Figure 3B The AI ​​video encoding framework shown in the figure contains the names of the various networks, as well as Figure 3C There is no restriction on the names of the various networks included in the AI ​​video decoding framework shown in .

[0161] The following is based on the above Figure 3A to Figure 3C , the training process of the first video processing network is explained.

[0162] Figure 4 FIG. 1 is a schematic diagram of an exemplary training process.

[0163] S401, acquiring video data, where the video data includes multiple frames of original images.

[0164] Exemplarily, multiple pieces of video data may be acquired as training data for training the first video processing network.

[0165] Exemplarily, a piece of video data may include at least one group of images (a group of images, i.e., GOP, refers to a group of images between I frames, where I frame is Intra frame, i.e., intra-coded frame), and each group of images may include N (N is an integer greater than 1) frame images (hereinafter referred to as original images).

[0166] Next, the N frames of original images included in one image group may be used as a batch to train the first video processing network. The following S402 to S405 are described by taking the use of one image group to train the first video processing network as an example.

[0167] S402, encoding the video data based on the first video processing network and then decoding the encoded video data based on the first video processing network to obtain reconstructed video data; wherein the reconstructed video data includes multiple frames of reconstructed images.

[0168] It should be noted that the first video processing network involved in the present application is a network for inter-frame encoding and inter-frame decoding. Therefore, for the first frame of the original image of the image group, it can be encoded based on the AI ​​image encoding framework (including the AI ​​image encoding network), and the encoding result of the first frame of the original image can be decoded based on the AI ​​image decoding framework (including the AI ​​image decoding network) to obtain the first frame of the reconstructed image. For the second to Nth frames of the original image in the image group, they can be input into the first video processing network to obtain the second to Nth frames of the reconstructed image.

[0169] For example, for an original image in the last N-1 original images in the image group, the original image can be input into the first video processing network, and the video encoding network included in the first video processing network encodes the original image. Then, the video decoding network included in the first video processing network decodes the original image based on the encoding result of the original image, and outputs a reconstructed image corresponding to the original image. The specific encoding and decoding process can refer to the above Figure 3A and Figure 3B In this way, N frames of reconstructed images output by the first video processing network can be obtained, and the N frames of reconstructed images belong to one image group.

[0170] S403, determining a first loss of each reconstructed image frame in the multiple reconstructed image frames.

[0171] Exemplarily, the first loss may refer to a loss used to measure the image quality of a reconstructed image, such as a distortion loss (the distortion loss may refer to the difference (such as an error) between the original image and the reconstructed image), etc. This application does not impose any limitation on this.

[0172] Exemplarily, the first loss of the multiple reconstructed images can be determined based on the multiple original images and the multiple reconstructed images. Specifically, for each reconstructed image, the reconstructed image can be compared with a corresponding original image to determine the first loss of the reconstructed image.

[0173] S404, determining a first loss weight of each reconstructed image in a plurality of reconstructed image frames; wherein, if the first loss weight of the i-th reconstructed image frame is greater than the first loss weight of the k-th reconstructed image frame, then the cumulative error of the i-th reconstructed image frame is greater than the cumulative error of the k-th reconstructed image frame; the i-th reconstructed image frame and the k-th reconstructed image frame belong to the same image group, and i and k are positive integers.

[0174] Exemplarily, the first loss weight of each reconstructed image frame can be determined according to the cumulative error of each reconstructed image frame. If the first loss weight of the reconstructed image of the i-th frame is greater than the first loss weight of the reconstructed image of the k-th frame, the cumulative error of the reconstructed image of the i-th frame is greater than the cumulative error of the reconstructed image of the k-th frame. There are two understandings here: only when the cumulative error of the i-th reconstructed image is greater than the cumulative error of the reconstructed image of the k-th frame, the first loss weight of the reconstructed image of the i-th frame is greater than the first loss weight of the reconstructed image of the k-th frame; or, when the cumulative error of the i-th reconstructed image is much larger than the cumulative error of the reconstructed image of the k-th frame (for example, the difference between the cumulative error of the i-th reconstructed image and the cumulative error of the reconstructed image of the k-th frame is greater than 2δ, δ is a preset value), the first loss weight of the reconstructed image of the i-th frame is greater than the first loss weight of the reconstructed image of the k-th frame.

[0175] Wherein, i and k are integers between 1 and N (the values ​​of i and k may be 1 or N). That is, a reconstructed image with a large cumulative error has a large first loss weight.

[0176] S405: Training the first video processing network according to the target loss of the multi-frame reconstructed images.

[0177] The target loss of the j-th reconstructed image in the multiple reconstructed images is determined according to the product of the first loss of the j-th reconstructed image and the first loss weight of the j-th reconstructed image, and j is a positive integer.

[0178] Exemplarily, the target loss of the image group may be determined by referring to the following formula (1); and then the target loss of the image group is used to train the first video processing network:

[0179]

[0180] In formula (1), Loss is the target loss of the image group, Loss1 j is the first loss of the image reconstructed from the jth (an integer between 1 and N, where j can be N or 1) frame in the image group, βj The first loss weight for reconstructing the image for the jth frame.

[0181] Referring to formula (1), that is, the first loss of each frame of the reconstructed image is multiplied by the first loss weight to obtain the target loss of each frame of the reconstructed image; then the target losses of the N frames of the reconstructed image in the image group are added to obtain the target loss of the image group; then, the target loss of the image group is used to train the first video processing network.

[0182] It should be noted that when j is equal to 1, β 1 *Loss1 1 Train the AI ​​image encoding network and AI image decoding network. When j is 2 to N, β 2 *Loss1 2 +β 3 *Loss1 3 +......+β N *Loss1 N The first video processing network is trained.

[0183] It should be noted that the use of β 2 *Loss1 2 +β 3 *Loss1 3 +......+β N *Loss1 N During the training of the first video processing network, the network parameters of the first video processing network are adjusted according to the target loss of each frame of the reconstructed image.

[0184] In this way, for the reconstructed image with large cumulative error (such as the reconstructed image of the i-th frame), the first loss weight of the reconstructed image of the i-th frame is determined to be large, and correspondingly, the target loss of the reconstructed image of the i-th frame will also be increased; and in the training process, the first video processing network is usually trained with the goal of minimizing the loss (that is, minimizing the target loss of each frame of the reconstructed image); therefore, after the first video processing network is trained with the goal of minimizing the target loss of the reconstructed image of the i-th frame, the cumulative error of the reconstructed image of the i-th frame obtained by encoding and decoding using the trained first video processing network will be reduced, and then the compression performance of the trained first video processing network frame for the i-th frame original image will also be improved, so that the compression performance of each frame can be balanced. Subsequently, using the trained first video processing network for video encoding and decoding can reduce the cumulative error of the reconstructed video data and improve the compression performance of the video data.

[0185] Figure 5A FIG. 1 is a schematic diagram of an exemplary training process.

[0186] Reference Figure 5A , the present application can also obtain the bit rate loss of each reconstructed image in the multiple reconstructed images (the bit rate loss may refer to the bit rate estimated during the encoding process, and the bit rate loss is hereinafter referred to as the second loss) and the corresponding loss weight (hereinafter referred to as the second loss weight); then, the first loss and the first loss weight of each reconstructed image, as well as the second loss and the second loss weight of each reconstructed image, can be combined to train the first video processing network. The first loss may be a distortion loss.

[0187] Figure 5B FIG. 1 is a schematic diagram of an exemplary training process. Figure 5B For Figure 5A The training process is shown based on .

[0188] S501, acquiring video data, where the video data includes multiple frames of original images.

[0189] S502, encoding the video data based on the first video processing network and then decoding the encoded video data based on the first video processing network to obtain reconstructed video data; wherein the reconstructed video data includes multiple frames of reconstructed images.

[0190] S503, determining a first loss of each reconstructed image frame in the multiple reconstructed image frames.

[0191] S504, determining a first loss weight of each reconstructed image in a plurality of reconstructed image frames; wherein the first loss weight of the i-th reconstructed image frame is greater than the first loss weight of the k-th reconstructed image frame, the cumulative error of the i-th reconstructed image frame is greater than the cumulative error of the k-th reconstructed image frame, and the i-th reconstructed image frame and the k-th reconstructed image frame belong to the same image group.

[0192] Exemplarily, S501 to S504 may refer to the description of S401 to S404, which will not be repeated here.

[0193] S505 , obtaining a second loss of each reconstructed image frame in the multiple reconstructed image frames and a second loss weight of each reconstructed image frame.

[0194] Exemplarily, the second loss may include a bit rate loss corresponding to the optical stream bit stream and a bit rate loss corresponding to the residual bit stream; refer to the following formula (2):

[0195] Loss2 j =Loss j v +Loss j r (2)

[0196] In formula (2), Loss2 j The second loss for reconstructing the image of the jth frame, Lossj v Loss is the bit rate loss corresponding to the optical flow code stream of the j-th frame reconstructed image. j v The bit rate loss corresponding to the residual bit stream of the reconstructed image of the jth frame.

[0197] Among them, the bit rate loss corresponding to the optical flow code stream can be determined and output by the first entropy estimation network during the entropy estimation process; and the bit rate loss corresponding to the residual code stream can be determined and output by the second entropy estimation network during the entropy estimation process.

[0198] Exemplarily, the second loss weight may be a fixed value, such as 1, which is not limited in the present application. In this case, the second loss weight of each frame of the reconstructed image is the same.

[0199] S506 , based on the first loss weight of the j-th frame reconstructed image and the second loss weight of the j-th frame reconstructed image, perform weighted calculation on the first loss of the j-th frame reconstructed image and the second loss of the j-th frame reconstructed image.

[0200] S507: Training the first video processing network based on the weighted calculation result of the multiple-frame reconstructed images.

[0201] Among them, the result of the weighted calculation of the j-th frame reconstructed image is the target loss of the j-th frame reconstructed image.

[0202] Exemplarily, the first loss of the j-th frame reconstructed image and the second loss of the j-th frame reconstructed image may be weightedly calculated by referring to the content in brackets in the following formula (3):

[0203]

[0204] In formula (3), Loss is the target loss of the image group, Loss1 j is the first loss for reconstructing the image of the jth frame in the image group, β j The first loss weight for reconstructing the image for the jth frame; Loss2 j is the second loss for reconstructing the image of the jth frame in the image group, α j The second loss weight for reconstructing the image for the jth frame.

[0205] Referring to formula (3), that is, for the j-th frame reconstructed image, according to the first loss weight of the j-th frame reconstructed image and the second loss weight of the j-th frame reconstruction, the first loss of the j-th frame reconstructed image and the second loss of the j-th frame reconstruction are weightedly calculated to obtain the target loss of the j-th frame reconstructed image; then, the target losses of the N-frame reconstructed images in the image group are added to obtain the target loss of the image group; then, the first video processing network is trained using the target loss of the image group. In this way, the distortion of the reconstructed image output by the first video processing network and the bit rate of the bitstream obtained by encoding the original image can be balanced.

[0206] The following Figure 5A Examples and Figure 5B Based on this, the process of determining the first loss weight of each frame of reconstructed image is explained.

[0207] In a possible manner, the timing information of each reconstructed image frame may be used to characterize the cumulative error of each reconstructed image frame; thus, the first loss weight of each reconstructed image frame may be determined according to the timing information of each reconstructed image frame.

[0208] Fig. 6A FIG. 1 is a schematic diagram of an exemplary training process.

[0209] S601, acquiring video data, where the video data includes multiple frames of original images.

[0210] S602, encoding the video data based on the first video processing network and then decoding the encoded video data based on the first video processing network to obtain reconstructed video data; wherein the reconstructed video data includes multiple frames of reconstructed images.

[0211] Illustratively, S601 to S602 may refer to the description of S401 to S402 above, which will not be repeated here.

[0212] S603, determining a first loss of each reconstructed image frame in the multiple reconstructed image frames.

[0213] This application is described by taking the first loss as distortion loss as an example.

[0214] In one possible approach, a point-to-point loss (i.e., point-based loss) between each frame of the original image and the corresponding reconstructed image may be calculated as the distortion loss. The point-to-point loss may include Ln distance (such as L1 distance (Manhattan distance), L2 distance (Euclidean distance), L-Inf distance (Chebyshev distance), etc.), and this application does not limit this.

[0215] In one possible way, the feature-based loss (i.e., feature-based loss) between each frame of the original image and the corresponding reconstructed image can be calculated as a rate loss weight. Among them, the feature-based loss may include structural similarity (SSIM), multi-scale structural similarity (MSSSIM), learned perceptual image patch similarity (LPIPS), etc., and this application does not limit this. In addition, each frame of the original image and the corresponding reconstructed image can be input into a neural network (such as a convolutional network, VGG (Visual Geometry Group Network) network, etc.), and the neural network outputs the first feature of each frame of the original image and the second feature of the corresponding reconstructed image; then, the distance between the first feature and the second feature is calculated to obtain the feature-based loss.

[0216] It should be understood that the present application does not limit the manner of determining the distortion loss.

[0217] Exemplarily, the process of determining the first loss weight of each reconstructed image frame in the multiple reconstructed image frames may refer to the following S604 to S605:

[0218] S604, determining the timing information corresponding to each reconstructed image frame in the multiple reconstructed image frames, where the timing information of each reconstructed image frame is used to characterize the accumulated error of each reconstructed image frame.

[0219] S605 , determining a first loss weight of each reconstructed image frame in the multiple reconstructed image frames based on the timing information corresponding to each reconstructed image frame in the multiple reconstructed image frames.

[0220] Exemplarily, since during the inter-frame coding process, the original image of the current frame is encoded based on the reconstructed images of other frames before it, the cumulative error of the reconstructed image that is later in time in an image group is greater; furthermore, the present application can determine the timing information corresponding to each reconstructed image frame, and determine the first loss weight of each reconstructed image frame based on the timing information of each reconstructed image frame.

[0221] Exemplarily, the relationship between the timing information and the first loss weight can be set in advance; further, during the training process, the first loss weight of each frame of the reconstructed image can be determined based on the timing information of each frame of the reconstructed image, and the relationship between the timing information and the first loss weight.

[0222] Exemplarily, the timing information may be a frame number or time information, which is not limited in the present application.

[0223] For example, the relationship between the timing information and the first loss weight can be expressed as follows:

[0224]

[0225] In formula (4), β j is the first loss weight of the j-th (j is an integer between 1 and N, and the value of j can be 1 or N) frame reconstructed image, β is the preset weight, η is a hyperparameter set according to experience, and j is the timing information (i.e., frame number) of the j-th frame reconstructed image.

[0226] For example, the relationship between the timing information and the first loss weight can be expressed as follows:

[0227] β j =β*(1+log A j) (5)

[0228] In formula (5), β j is the first loss weight of the j-th frame reconstructed image, β is the preset weight, A is a hyperparameter set according to experience (A>1), and j is the timing information (ie, frame number) of the j-th frame reconstructed image.

[0229] It should be noted that β in formula (4) and formula (5) can also be the first loss weight of the reconstructed image of the jth frame in the previous iteration cycle. In this case, β j The first loss weight of the reconstructed image of the jth frame in the current frame iteration cycle.

[0230] It should be understood that the present application does not limit the specific expression of the relationship between the timing information and the first loss weight, and it is sufficient to indicate that the larger the timing is, the larger the first loss weight is.

[0231] Exemplarily, when the reconstructed image includes multiple components, the first loss of each frame of the reconstructed image includes the first losses of the multiple components of each frame of the reconstructed image, and the first loss weight of each frame of the reconstructed image includes the first loss weights of the multiple components of each frame of the reconstructed image.

[0232] In this way, assuming that the reconstructed image is a YUV image, for formula (4), the first loss weights of the three YUV components of each frame of the reconstructed image are respectively as shown in the following formulas (6), (7) and (8):

[0233]

[0234]

[0235]

[0236] Among them, in formula (6) to formula (8), β j Y , β j U and β j V are the first loss weight of the Y component of the j-th frame reconstructed image, the first loss weight of the U component of the j-th frame reconstructed image, and the first loss weight of the V component of the j-th frame reconstructed image; β Y , β Y and β Y are the preset weights of the Y component, the U component and the V component, respectively, Y , η U and η U are the hyperparameters corresponding to the Y component, the hyperparameters corresponding to the U component, and the hyperparameters corresponding to the V component set according to experience.

[0237] Similarly, assuming that the reconstructed image is a YUV image, for formula (5), the first loss weights of the three YUV components of each frame of the reconstructed image are respectively as shown in the following formulas (9), (10) and (11):

[0238] β j Y =β Y *(1+log a j) (9)

[0239] β j U =β U *(1+log b j) (10)

[0240] β j V =β V *(1+log c j) (11)

[0241] Among them, in formula (9) to formula (11), β j Y , β j U and β j V are the first loss weight of the Y component of the j-th frame reconstructed image, the first loss weight of the U component of the j-th frame reconstructed image, and the first loss weight of the V component of the j-th frame reconstructed image; β Y , β Y and β Yare the preset weights of the Y component, the U component, and the V component, respectively. a, b, and c are the hyperparameters corresponding to the Y component, the U component, and the V component, respectively, which are set according to experience. Among them, a, b, and c are all greater than 1.

[0242] In this embodiment, i is greater than k, that is, the i-th reconstructed image is located after the k-th reconstructed image.

[0243] Figure 6B FIG. 1 is a schematic diagram of a first loss weight of a multi-frame reconstructed image, where the reconstructed image is a YUV image having a Y component, a U component and a V component.

[0244] Reference Figure 6B , the first loss weights of the first 10 reconstructed images in an image group are shown; wherein the upper curve is the first loss weight of the UV component of each of the 10 reconstructed images, and the lower curve is the first loss weight of the Y component of each of the 10 reconstructed images. It should be understood that Figure 6B This is only an example of the present application. The first loss weight of the U component and the first loss weight of the V component of each frame of the reconstructed image may be different, and the present application does not impose any limitation on this.

[0245] It should be noted that Figure 6B The reconstructed image is a YUV420 image, with Y component: U component: V component = 8:1:1; therefore, the cumulative error between the U component and V component of the reconstructed image and the U component and V component of the corresponding original image is greater. Correspondingly, the ratio of the first loss weight of the Y component, the first loss weight of the U component, and the first loss weight of the V component of the reconstructed image is 8:1:1, and the first loss weight of the U component and the first loss weight of the V component of the reconstructed image are greater than the first loss weight of the V component.

[0246] S606, obtaining a second loss of each reconstructed image frame in the multiple reconstructed image frames and a second loss weight of each reconstructed image frame.

[0247] S607 , based on the first loss weight of the j-th frame reconstructed image and the second loss weight of the j-th frame reconstructed image, perform weighted calculation on the first loss of the j-th frame reconstructed image and the second loss of the j-th frame reconstructed image.

[0248] S608: Training the first video processing network based on the weighted calculation result of the multiple-frame reconstructed images.

[0249] Exemplarily, S606 to S608 may refer to the description of S505 to S507 above, which will not be repeated here.

[0250] The following test set is used to test the first video processing network, the traditional encoding and decoding method of the prior art (such as H266) and the AI ​​encoding and decoding method of the prior art (that is, the video processing network of the prior art) to compare the compression performance of the first video processing network of the present application and the video processing network of the prior art.

[0251] For example, the test set includes 5: two 4k test sets (wherein, a 4K (A1) test set has a complex motion type, and a 4K test set (A2) has a simple motion type), a 1080p test set, an 832*480 test set (target object motion) and a 720p test set (such as conference video); the peak signal-to-noise ratio (PSNR) of the first video processing network of the present application relative to H266, and the PSNR of the video processing network of the prior art relative to H266 can be shown in Table 1 below:

[0252] Table 1

[0253]

[0254] Referring to Table 1, Y-PSNR represents the PSNR of the Y component, U-PSNR represents the PSNR of the U component, and V-PSNR represents the PSNR of the V component.

[0255] Compared with the video processing network using the prior art, the first video processing network of the present application is used for encoding and decoding to improve the compression effect; especially in the UV component, the compression effect is more significant.

[0256] In one possible approach, the error of each reconstructed image frame can be characterized by the difference between the first loss of each reconstructed image frame and the first loss of a reference reconstructed image corresponding to each reconstructed image frame; in this way, the first loss weight of the reconstructed image of each frame can be determined based on the difference between the first loss of each reconstructed image frame and the first loss of the reference reconstructed image corresponding to each reconstructed image frame.

[0257] Fig. 7A FIG. 1 is a schematic diagram of an exemplary training process.

[0258] S701, acquiring video data, where the video data includes multiple frames of original images.

[0259] S702, encoding the video data based on the first video processing network and then decoding the encoded video data based on the first video processing network to obtain reconstructed video data; wherein the reconstructed video data includes multiple frames of reconstructed images.

[0260] S703, determining a first loss of each reconstructed image frame in the multiple reconstructed image frames.

[0261] Illustratively, S701 to S703 may refer to the description of S501 to S503 above, which will not be repeated here.

[0262] Exemplarily, the process of determining the first loss weight of each reconstructed image frame may refer to the following S704 to S706; in S704 to S706, the first loss weight of the j-th reconstructed image is determined as an example for explanation:

[0263] S704, determining the difference between the first loss of the j-th frame reconstructed image and the first loss of the reference reconstructed image corresponding to the j-th frame reconstructed image, and obtaining the loss difference of the j-th frame reconstructed image, wherein the loss difference of the j-th frame reconstructed image is used to characterize the cumulative error of the reconstructed image.

[0264] In one possible manner, the reference reconstructed image corresponding to the j-th frame reconstructed image is a reference reconstructed image obtained by encoding the j-th frame original image based on the second video processing network and then decoding the encoded result of the j-th frame original image based on the second video processing network; wherein the cumulative error of the reference reconstructed image corresponding to the j-th frame reconstructed image is less than the cumulative error of the j-th frame reconstructed image.

[0265] Exemplarily, a video processing network whose output reconstructed image has a cumulative error that is smaller than the cumulative error of the reconstructed image output by the first video processing network can be pre-selected as the second video processing network; for example, a video processing network composed of an AI condition I video encoding network and an AI condition I video decoding network (the AI ​​condition I video encoding network encodes the feature information of the original image, and the calculated first optical flow is not used to calculate the residual information, but only to calculate the probability distribution of the feature information of the original image, so there is no cumulative error in the reconstructed image output by the AI ​​condition I video encoding network and the AI ​​condition I video decoding network) is selected as the second video processing network.

[0266] Figure 7B FIG. 1 is a schematic diagram of an exemplary training process.

[0267] Reference Figure 7B During the training process, while the video data is input into the first video processing network, the video data can also be input into the second video processing network, which encodes each frame of the original image and then decodes it based on the encoding result of each frame of the original image, and outputs multiple frames of reference reconstructed images. In this way, each frame of the original image has a corresponding reconstructed image and a reference reconstructed image.

[0268] exist Figure 7BIn the figure, X_2, X_3, X_4, ..., X_N are the original images of the last N frames in an image group, Y1_2, Y1_3, Y1_4, ..., Y1_N are the original images of the last N frames in an image group X_2, X_3, X_4, ..., X_N, which are respectively input to the first video processing network and the reconstructed images output by the first video processing network; Y2_2, Y2_3, Y2_4, ..., Y2_N are the original images of the last N frames in an image group X_2, X_3, X_4, ..., X_N, which are respectively input to the second video processing network and the reconstructed images output by the second video processing network. Among them, the reference reconstructed image corresponding to the second reconstructed image Y1_2 is Y2_2, the reference reconstructed image corresponding to the third reconstructed image Y1_3 is Y2_3, the reference reconstructed image corresponding to the fourth reconstructed image Y1_4 is Y2_4, ..., and the reference reconstructed image corresponding to the Nth reconstructed image Y1_N is Y2_N.

[0269] Exemplarily, the first loss of the reference reconstructed image corresponding to the j-th frame reconstructed image can be determined based on the j-th frame original image and the j-th frame reference reconstructed image output by the second video processing network (that is, the reference reconstructed image corresponding to the j-th frame reconstructed image); the specific method is similar to the method for determining the first loss of the j-th frame reconstructed image, and will not be repeated here.

[0270] In one possible manner, the reference reconstructed image corresponding to the j-th reconstructed image is the m-frame reconstructed image before the j-th reconstructed image in the image group. Wherein, m is a positive integer. In this case, the first loss of the reference reconstructed image corresponding to the j-th reconstructed image may be the average of the first losses of the m-frame reconstructed images before the j-th reconstructed image, which may be shown in the following formula (12):

[0271]

[0272] In formula (12), Loss1 j The first loss of the j-th frame reconstructed image, Loss1 j ref It is the first loss of the reference reconstructed image corresponding to the j-th frame reconstructed image.

[0273] Subsequently, the difference between the first loss of the j-th frame reconstructed image and the first loss of the reference reconstructed image corresponding to the j-th frame reconstructed image is determined to obtain the loss difference of the j-th frame reconstructed image; the loss difference of the j-th frame reconstructed image is used to characterize the cumulative error of the reconstructed image. When the cumulative error of the reconstructed image is larger, if the loss difference of the reconstructed image is the first loss of the reconstructed image minus the first loss of the reference reconstructed image corresponding to the reconstructed image, then the loss difference of the reconstructed image is larger; if the loss difference of the reconstructed image is the first loss of the reference reconstructed image corresponding to the reconstructed image minus the first loss of the reconstructed image, then the loss difference of the reconstructed image is smaller.

[0274] Afterwards, the weight coefficient of the j-th frame reconstructed image can be determined based on the loss difference of the j-th frame reconstructed image and the preset range. Specifically, the loss difference of the j-th frame reconstructed image can be compared with the preset range to determine the weight coefficient of the j-th frame reconstructed image; refer to the following S705 to S706:

[0275] S705 , determining a weight coefficient of the j-th frame reconstructed image based on the loss difference of the j-th frame reconstructed image and a preset range.

[0276] In one possible approach, the loss difference of the j-th frame reconstructed image is the first loss of the j-th frame reconstructed image minus the first loss of the reference reconstructed image corresponding to the j-th frame reconstructed image; this can be shown in the following formula (13):

[0277] ΔLoss j =Loss1 j -Loss1 j ref (13)

[0278] In formula (13), ΔLoss j Loss1 is the loss difference of the reconstructed image of the jth frame j The first loss of the j-th frame reconstructed image, Loss1 j ref It is the first loss of the reference reconstructed image corresponding to the j-th frame reconstructed image.

[0279] In this case, when the loss difference of the reconstructed image of the j-th frame belongs to the preset range, the first preset coefficient is used as the weight coefficient of the reconstructed image of the j-th frame; when the loss difference of the reconstructed image of the j-th frame is greater than the second boundary value, the second preset coefficient is used as the weight coefficient of the reconstructed image of the j-th frame; when the loss difference of the reconstructed image of the j-th frame is less than the first boundary value, the third preset coefficient is used as the weight coefficient of the reconstructed image of the j-th frame.

[0280] Assume that the preset range is set to [-δ, +δ], where δ is a positive integer; -δ is the first boundary value, and +δ is the second boundary value. That is, when ΔLoss1j ∈[-δ,+δ], the weight coefficient of the j-th frame reconstructed image is the first preset coefficient; when ΔLoss1 j >+δ, the weight coefficient of the j-th frame reconstructed image is the second preset coefficient; when ΔLoss1 j <-δ, the weight coefficient of the j-th frame reconstructed image is the third preset coefficient.

[0281] The second preset coefficient is greater than 1, for example, the second preset coefficient is 1.1; the third preset coefficient is less than 1, for example, the third preset coefficient is 0.9; the first preset coefficient is greater than the third preset coefficient and less than the second preset coefficient, for example, the first preset coefficient may be 1. In this way, the weight coefficient of the reconstructed image with a larger difference between the first loss and the first loss of the corresponding reference reconstructed image is larger, and the weight coefficient of the reconstructed image with a smaller difference between the first loss and the first loss of the corresponding reference reconstructed image is smaller.

[0282] In one possible approach, the loss difference of the j-th frame reconstructed image is the first loss of the reference reconstructed image corresponding to the j-th frame reconstructed image minus the first loss of the j-th frame reconstructed image; this can be shown in the following formula (14):

[0283] ΔLoss j =Loss1 j ref -Loss1 j (14)

[0284] In formula (14), ΔLoss j Loss1 is the loss difference of the reconstructed image of the jth frame j The first loss of the j-th frame reconstructed image, Loss1 j ref It is the first loss of the reference reconstructed image corresponding to the j-th frame reconstructed image.

[0285] In this case, when the loss difference of the reconstructed image of the j-th frame belongs to the preset range, the first preset coefficient is used as the weight coefficient of the reconstructed image of the j-th frame; when the loss difference of the reconstructed image of the j-th frame is greater than the second boundary value, the third preset coefficient is used as the weight coefficient of the reconstructed image of the j-th frame; when the loss difference of the reconstructed image of the j-th frame is less than the first boundary value, the second preset coefficient is used as the weight coefficient of the reconstructed image of the j-th frame.

[0286] Assume that the preset range is set to [-δ, +δ], where δ is a positive integer; -δ is the first boundary value, and +δ is the second boundary value. That is, when ΔLoss1 j ∈[-δ,+δ], the weight coefficient of the j-th frame reconstructed image is the first preset coefficient; when ΔLoss1 j>+δ, the weight coefficient of the j-th frame reconstructed image is the third preset coefficient; when ΔLoss1 j <-δ, the weight coefficient of the j-th frame reconstructed image is the second preset coefficient.

[0287] S706: The product of the weight coefficient of the j-th frame reconstructed image and the first loss weight of the j-th frame reconstructed image in the previous iteration period is used as the first loss weight of the j-th frame reconstructed image in the current iteration period.

[0288] Afterwards, the product value of the weight coefficient of the j-th frame reconstructed image and the first loss weight of the j-th frame reconstructed image in the previous iteration cycle is used as the first loss weight of the j-th frame reconstructed image in the current iteration cycle. It should be understood that using one image group to train the first video processing network corresponds to one iteration cycle.

[0289] For example, assuming that the loss difference of the j-th frame reconstructed image is calculated according to formula (13), when ΔLoss1 j ∈[-δ,+δ], the first loss weight β of the reconstructed image of the jth frame in the current iteration cycle j H =β j H-1 *x1, where β j H-1 is the first loss weight of the j-th frame reconstructed image in the previous iteration cycle, and x1 is the first preset coefficient. j > +δ, the first loss weight β of the reconstructed image of the jth frame in the current iteration cycle j H =β j H-1 *x2, where β j H-1 is the first loss weight of the j-th frame reconstructed image in the previous iteration cycle, and x2 is the second preset coefficient. j <-δ, the first loss weight β of the reconstructed image of the jth frame in the current iteration cycle j H =β j H-1 *x1, where β j H-1 is the first loss weight of the j-th frame reconstructed image in the previous iteration cycle, and x3 is the third preset coefficient. Similarly, when the loss difference of the j-th frame reconstructed image is calculated according to formula (13), the method for determining the first loss weight of the j-th frame reconstructed image is not repeated.

[0290] Where H is a positive integer. When H = 1, β j 0 The weights can be preset.

[0291] In this way, the larger the difference between the first loss and the first loss of the corresponding reference reconstructed image, the larger the first loss weight of the reconstructed image; the smaller the difference between the first loss and the first loss of the corresponding reference reconstructed image, the smaller the first loss weight of the reconstructed image.

[0292] For example, when the reconstructed image includes multiple components, a corresponding preset range may be set for each component in advance. Assuming that the reconstructed image is a YUV image, the preset range corresponding to the Y component may be expressed as: [-δ Y ,+δ Y ], the preset range corresponding to the U component can be expressed as: [-δ U ,+δ U ], the preset range corresponding to the V component can be expressed as: [-δ V ,+δ V ].

[0293] Assuming that the loss difference of the j-th frame reconstructed image is calculated according to formula (13), then

[0294] When the loss difference of the Y component of the j-th frame reconstructed image is ΔLoss1 j Y ∈[-δ Y ,+δ Y ], the first loss weight β of the Y component of the reconstructed image of the jth frame in the current iteration cycle j HY =β j (H-1)Y *x1 Y , where β j (H-1)Y The first loss weight of the Y component of the reconstructed image of the jth frame in the previous iteration cycle, x1 Y is the first preset coefficient of the Y component. j Y >+δ Y When the first loss weight β of the Y component of the reconstructed image of the jth frame in the current iteration period is j HY =β j (H-1)Y *x2 Y , where β j (H-1)Y The first loss weight for the Y component of the image reconstructed for the jth frame in the previous iteration cycle, x2 Y is the second preset coefficient of the Y component. j Y <-δ Y When the first loss weight β of the reconstructed image of the jth frame in the current iteration period is jHY =β j (H-1)Y *x3 Y , where β j (H-1)Y The first loss weight for the Y component of the reconstructed image of the jth frame in the previous iteration cycle, x3 Y is the third preset coefficient of the Y component.

[0295] When the loss difference of the U component of the reconstructed image of the jth frame is ΔLoss1 j U ∈[-δ U ,+δ U ], the first loss weight β of the U component of the j-th frame reconstructed image in the current iteration cycle j HU =β j (H-1)U *x1 U , where β j (H-1)U The first loss weight of the U component of the reconstructed image of the jth frame in the previous iteration cycle, x1 U is the first preset coefficient of the U component. j U >+δ U When the first loss weight β of the U component of the j-th frame reconstructed image in the current iteration period is j HU =β j (H-1)U *x2 U , where β j (H-1)U The first loss weight of the U component of the j-th frame reconstructed image in the previous iteration cycle, x2 U is the second preset coefficient of the U component. j U <-δ U When the first loss weight β of the reconstructed image of the jth frame in the current iteration period is j HU =β j (H-1)U *x3 U , where β j (H-1)U The first loss weight of the U component of the j-th frame reconstructed image in the previous iteration cycle, x3 U is the third preset coefficient of the U component.

[0296] When the loss difference ΔLoss1 of the V component of the j-th frame reconstructed image j V ∈[-δ V ,+δ V], the first loss weight β of the V component of the j-th frame reconstructed image in the current iteration cycle j HV =β j (H-1)V *x1 V , where β j (H-1)V The first loss weight of the V component of the reconstructed image of the jth frame in the previous iteration cycle, x1 V is the first preset coefficient of the V component. j V >+δ V When the first loss weight β of the V component of the reconstructed image of the jth frame in the current iteration period is j HV =β j (H-1)V *x2 V , where β j (H-1)V The first loss weight of the V component of the reconstructed image of the jth frame in the previous iteration cycle, x2 V is the second preset coefficient of the V component. j V <-δ V When the first loss weight β of the reconstructed image of the jth frame in the current iteration period is j HV =β j (H-1)V *x3 V , where β j (H-1)V The first loss weight of the V component of the j-th frame reconstructed image in the previous iteration cycle, x3 V is the third preset coefficient of the V component.

[0297] By analogy, when the loss difference of the j-th frame reconstructed image is calculated according to formula (14), the method for determining the first loss weights of each component of the j-th frame reconstructed image is no longer repeated.

[0298] S707, obtaining a second loss of each reconstructed image frame in the multiple reconstructed image frames and a second loss weight of each reconstructed image frame.

[0299] S708 , performing weighted calculation on the first loss of the j-th frame reconstructed image and the second loss of the j-th frame reconstructed image based on the first loss weight of the j-th frame reconstructed image and the second loss weight of the j-th frame reconstructed image.

[0300] S709: Training the first video processing network based on the weighted calculation result of the multiple-frame reconstructed images.

[0301] Exemplarily, S707 to S709 may refer to the description of S505 to S507 above, which will not be repeated here.

[0302] Figure 7C FIG. 1 is a schematic diagram showing an exemplary compression performance comparison. Figure 7C The unit of the horizontal axis is frame. Figure 7C Two image groups with a total of 100 reconstructed images are shown in FIG.

[0303] Reference Figure 7C (1) Figure 7C In (1), the solid curve is the loss curve of the video processing network of the prior art, and the dotted curve is the loss curve of the first video processing network involved in the present application. By comparing the two curves, it can be seen that after the 20th frame in an image group, the target loss of the reconstructed image output by the first video processing network involved in the present application is less than the target loss of the reconstructed image output by the video processing network involved in the prior art. It can be seen that the target loss of the reconstructed image output by the video processing network involved in the present application is less than the target loss of the reconstructed image output by the video processing network involved in the prior art. Fig. 7A The training method of the embodiment can improve the compression performance of the first video processing network for image frames in an image group that use inter-frame coding.

[0304] Reference Figure 7C (2) Figure 7C In (2), the solid curve is the PSNR curve of the video processing network of the prior art, and the dashed curve is the PSNR curve of the first video processing network involved in this application; by comparing the two curves, it can be seen that under the same bit rate, the PSNR of the reconstructed image output by the first video processing network involved in this application is better than the PSNR of the reconstructed image output by the video processing network of the prior art. Fig. 7A The training method of the embodiment can improve the image quality of the image frames in the image group using inter-frame coding by the first video processing network.

[0305] Exemplarily, after the training of the first video processing network is completed, the video encoding network in the trained first video processing network can be used for encoding; specifically, the video data to be encoded can be obtained, and then, based on the video encoding network in the trained first video processing network, the video data is encoded to obtain a code stream. And the video decoding network in the trained first video processing network can be used for decoding; specifically, the code stream can be received; the code stream is decoded based on the video decoding network to obtain reconstructed video data.

[0306] The embodiment of the present application also provides a network training device, which can be used to execute the training method of the aforementioned embodiment. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0307] Figure 8Schematic diagram of a network training device shown as an example. The network training device may include:

[0308] An acquisition module 801 is used to acquire video data, where the video data includes multiple frames of original images;

[0309] A compression module 802 is used to encode the video data based on the first video processing network and then decode the encoded video data based on the first video processing network to obtain reconstructed video data; wherein the reconstructed video data includes multiple frames of reconstructed images;

[0310] The loss and weight determination module 803 is used to determine the first loss of each reconstructed image in the multiple reconstructed image frames, and determine the first loss weight of each reconstructed image frame; wherein, if the first loss weight of the i-th reconstructed image frame is greater than the first loss weight of the k-th reconstructed image frame, then the cumulative error of the i-th reconstructed image frame is greater than the cumulative error of the k-th reconstructed image frame; the i-th reconstructed image frame and the k-th reconstructed image frame belong to the same image group, and i and k are positive integers;

[0311] The training module 804 is used to train the first video processing network based on the target loss of the multi-frame reconstructed image; the target loss of the j-th frame reconstructed image in the multi-frame reconstructed image is determined according to the product of the first loss of the j-th frame reconstructed image and the first loss weight of the j-th frame reconstructed image, where j is a positive integer.

[0312] Exemplarily, the loss and weight determination module 803 is specifically configured to determine a first loss weight of each reconstructed image frame based on a cumulative error of each reconstructed image frame.

[0313] Exemplarily, the loss and weight determination module 803 is specifically used to reconstruct an image for the jth frame in the multi-frame reconstructed image:

[0314] Determine the difference between the first loss of the j-th frame reconstructed image and the first loss of the reference reconstructed image corresponding to the j-th frame reconstructed image, and obtain the loss difference of the j-th frame reconstructed image; wherein the loss difference of the j-th frame reconstructed image is used to characterize the cumulative error of the j-th frame reconstructed image; based on the loss difference of the j-th frame reconstructed image and a preset range, determine the first loss weight of the j-th frame reconstructed image.

[0315] Exemplarily, the loss and weight determination module 803 is specifically used to determine the weight coefficient of the j-th frame reconstructed image based on the loss difference of the j-th frame reconstructed image and a preset range;

[0316] The product value of the weight coefficient of the j-th frame reconstructed image and the first loss weight of the j-th frame reconstructed image in the previous iteration cycle is used as the first loss weight of the j-th frame reconstructed image in the current iteration cycle.

[0317] Exemplarily, the preset range includes a first boundary value and a second boundary value, and the first boundary value is less than the second boundary value; the loss and weight determination module 803 is specifically configured to use the first preset coefficient as the weight coefficient of the j-th frame reconstructed image when the loss difference of the j-th frame reconstructed image belongs to the preset range;

[0318] When the loss difference of the j-th frame reconstructed image is greater than the second boundary value, the second preset coefficient is used as the weight coefficient of the j-th frame reconstructed image, and the second preset coefficient is greater than 1;

[0319] When the loss difference of the j-th frame reconstructed image is less than the first boundary value, the third preset coefficient is used as the weight coefficient of the j-th frame reconstructed image, and the third preset coefficient is less than 1;

[0320] The first preset coefficient is greater than the third preset coefficient and less than the second preset coefficient, and j is a positive integer.

[0321] Exemplarily, the reference reconstructed image corresponding to the j-th frame reconstructed image includes at least one of the following: a reference reconstructed image obtained by encoding the j-th frame original image based on the second video processing network and then decoding the encoded result of the j-th frame original image based on the second video processing network; wherein the cumulative error of the reference reconstructed image corresponding to the j-th frame reconstructed image is less than the cumulative error of the j-th frame reconstructed image; or, the m-frame reconstructed images before the j-th frame reconstructed image in the image group; wherein the first loss of the reference reconstructed image corresponding to the j-th frame reconstructed image is the average of the first losses of the m-frame reconstructed images before the j-th frame reconstructed image, and m is a positive integer.

[0322] Exemplarily, the loss and weight determination module 803 is specifically used to determine the timing information corresponding to each frame of the reconstructed image, and the timing information of each frame of the reconstructed image is used to characterize the cumulative error of each frame of the reconstructed image;

[0323] Based on the timing information corresponding to each frame of the reconstructed image, a first loss weight of each frame of the reconstructed image is determined; wherein i is greater than k.

[0324] Exemplarily, the loss and weight determination module 803 is also used to obtain the second loss of each reconstructed image frame and the second loss weight of each reconstructed image frame in multiple reconstructed images; wherein the first loss is the distortion loss and the second loss is the bit rate loss; wherein the target loss of the j-th frame reconstructed image is determined by weighted calculation of the first loss of the j-th frame reconstructed image and the second loss of the j-th frame reconstructed image based on the first loss weight of the j-th frame reconstructed image and the second loss weight of the j-th frame reconstructed image.

[0325] Exemplarily, the reconstructed image includes multiple components; the first loss of each frame of the reconstructed image includes the first losses of the multiple components of each frame of the reconstructed image, and the first loss weight of each frame of the reconstructed image includes the first loss weights of the multiple components of each frame of the reconstructed image.

[0326] Exemplarily, the first video processing network includes a video encoding network and a video decoding network, the video encoding network is used to encode the video data, and the video decoding network is used to decode based on the encoding result of the video data.

[0327] In one example, Fig. 9 A schematic block diagram of a device 900 according to an embodiment of the present application is shown. The device 900 may include: a processor 901 and a transceiver / transceiver pin 902 , and optionally, a memory 903 .

[0328] The components of the device 900 are coupled together via a bus 904, wherein the bus 904 includes a power bus, a control bus, and a status signal bus in addition to a data bus. However, for the sake of clarity, all buses are referred to as bus 904 in the figure.

[0329] Optionally, the memory 903 may be used to store instructions in the aforementioned method embodiment. The processor 901 may be used to execute instructions in the memory 903, and control the receiving pin to receive a signal, and control the sending pin to send a signal.

[0330] The apparatus 900 may be the electronic device or a chip of the electronic device in the above method embodiment.

[0331] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.

[0332] The embodiment of the present application also provides a chip, including one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the above-mentioned related method steps implement the steps of the method in the above-mentioned embodiment. The interface circuit is a transceiver / transceiver pin 902.

[0333] This embodiment further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the method in the above-mentioned embodiment.

[0334] This embodiment further provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer executes the above-mentioned related steps to implement the method in the above-mentioned embodiment.

[0335] In addition, an embodiment of the present application also provides a device, which may specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor may execute the computer-executable instructions stored in the memory so that the chip executes the methods in the above-mentioned method embodiments.

[0336] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here.

[0337] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0338] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0339] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0340] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0341] Any content of each embodiment of the present application, as well as any content of the same embodiment, can be freely combined. Any combination of the above content is within the scope of the present application.

[0342] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0343] The steps of the method or algorithm described in conjunction with the disclosure of the embodiments of the present application can be implemented in a hardware manner, or can be implemented by a processor executing a software instruction. The software instruction can be composed of corresponding software modules, and the software module can be stored in a random access memory (Random Access Memory, RAM), a flash memory, a read-only memory (ReadOnly Memory, ROM), an erasable programmable read-only memory (Erasable Programmable ROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), a register, a hard disk, a mobile hard disk, a read-only compact disk (CD-ROM) or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0344] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer-readable storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0345] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A network training method, characterized in that: The method comprises: Acquire video data, wherein the video data includes multiple frames of original images; After encoding the video data based on a first video processing network, decoding the encoding result of the video data based on the first video processing network to obtain reconstructed video data; wherein the reconstructed video data includes multiple frames of reconstructed images; Determine a first loss of each reconstructed image in the multiple reconstructed image frames, and determine a first loss weight of each reconstructed image frame; wherein, if the first loss weight of the i-th reconstructed image frame is greater than the first loss weight of the k-th reconstructed image frame, then the cumulative error of the i-th reconstructed image frame is greater than the cumulative error of the k-th reconstructed image frame; the i-th reconstructed image frame and the k-th reconstructed image frame belong to the same image group, and i and k are positive integers; The first video processing network is trained based on the target loss of the multiple-frame reconstructed images; wherein the target loss of the j-th frame reconstructed image in the multiple-frame reconstructed images is determined according to the product of the first loss of the j-th frame reconstructed image and the first loss weight of the j-th frame reconstructed image, and j is a positive integer.

2. The method according to claim 1, characterized in that The determining of the first loss weight of the reconstructed image per frame comprises: Based on the accumulated error of the reconstructed image of each frame, a first loss weight of the reconstructed image of each frame is determined.

3. The method according to claim 2, characterized in that The determining, based on the accumulated error of the reconstructed image of each frame, a first loss weight of the reconstructed image of each frame comprises: Reconstruct an image for the jth frame in the multiple reconstructed images: Determine the difference between the first loss of the j-th frame reconstructed image and the first loss of the reference reconstructed image corresponding to the j-th frame reconstructed image, and obtain the loss difference of the j-th frame reconstructed image; wherein the loss difference of the j-th frame reconstructed image is used to characterize the cumulative error of the j-th frame reconstructed image; Based on the loss difference of the j-th frame reconstructed image and a preset range, a first loss weight of the j-th frame reconstructed image is determined.

4. The method according to claim 3, characterized in that The determining, based on the loss difference of the j-th frame reconstructed image and a preset range, a first loss weight of the j-th frame reconstructed image comprises: Determining a weight coefficient of the j-th frame reconstructed image based on a loss difference of the j-th frame reconstructed image and a preset range; The product value of the weight coefficient of the j-th frame reconstructed image and the first loss weight of the j-th frame reconstructed image in the previous iteration cycle is used as the first loss weight of the j-th frame reconstructed image in the current iteration cycle.

5. The method according to claim 4, characterized in that The preset range includes a first boundary value and a second boundary value, and the first boundary value is smaller than the second boundary value; The step of determining a weight coefficient of the j-th frame reconstructed image based on the loss difference of the j-th frame reconstructed image and a preset range includes: When the loss difference of the j-th frame reconstructed image falls within a preset range, using the first preset coefficient as the weight coefficient of the j-th frame reconstructed image; When the loss difference of the j-th frame reconstructed image is greater than a second boundary value, using a second preset coefficient as a weight coefficient of the j-th frame reconstructed image, and the second preset coefficient is greater than 1; When the loss difference of the j-th frame reconstructed image is less than the first boundary value, using a third preset coefficient as a weight coefficient of the j-th frame reconstructed image, wherein the third preset coefficient is less than 1; The first preset coefficient is greater than the third preset coefficient and less than the second preset coefficient, and j is a positive integer.

6. The method according to any one of claims 3 to 5, characterized in that: The reference reconstructed image corresponding to the j-th reconstructed image includes at least one of the following: A reference reconstructed image obtained by encoding the j-th frame original image based on a second video processing network and then decoding the encoding result of the j-th frame original image based on the second video processing network; wherein the cumulative error of the reference reconstructed image corresponding to the j-th frame reconstructed image is less than the cumulative error of the j-th frame reconstructed image; or, The m reconstructed images before the j-th reconstructed image in the image group; wherein the first loss of the reference reconstructed image corresponding to the j-th reconstructed image is the average of the first losses of the m reconstructed images before the j-th reconstructed image, and m is a positive integer.

7. The method according to claim 2, characterized in that The determining, based on the accumulated error of the reconstructed image of each frame, a first loss weight of the reconstructed image of each frame comprises: Determine the timing information corresponding to each frame of the reconstructed image, wherein the timing information of each frame of the reconstructed image is used to characterize the accumulated error of each frame of the reconstructed image; Based on the timing information corresponding to each frame of the reconstructed image, a first loss weight of each frame of the reconstructed image is determined; wherein i is greater than k.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: Obtaining a second loss of each reconstructed image in the multiple reconstructed image frames and a second loss weight of each reconstructed image frame; wherein the first loss is a distortion loss, and the second loss is a bit rate loss; Among them, the target loss of the j-th frame reconstructed image is determined by weighted calculation of the first loss of the j-th frame reconstructed image and the second loss of the j-th frame reconstructed image according to the first loss weight of the j-th frame reconstructed image and the second loss weight of the j-th frame reconstructed image.

9. The method according to any one of claims 1 to 8, characterized in that: The reconstructed image includes multiple components; the first loss of each frame of the reconstructed image includes the first losses of the multiple components of the reconstructed image of each frame, and the first loss weight of each frame of the reconstructed image includes the first loss weights of the multiple components of the reconstructed image of each frame.

10. A coding method, characterized in that: The method comprises: Get video data; The video data is encoded based on a video coding network to obtain a code stream; wherein the video coding network is trained based on a target loss of multiple reconstructed images, and the target loss of the j-th reconstructed image in the multiple reconstructed images is determined according to the product of the first loss of the j-th reconstructed image and the first loss weight of the j-th reconstructed image, where j is a positive integer; if the first loss weight of the i-th reconstructed image is greater than the first loss weight of the k-th reconstructed image, then the cumulative error of the i-th reconstructed image is greater than the cumulative error of the k-th reconstructed image; the i-th reconstructed image and the k-th reconstructed image belong to the same image group, and i and k are positive integers.

11. A decoding method, characterized in that: The method comprises: Receive code stream; The code stream is decoded based on a video decoding network to obtain reconstructed video data; wherein the video decoding network is trained based on a target loss of multiple frames of reconstructed images, and the target loss of the j-th frame of the multiple frames of reconstructed images is determined according to the product of the first loss of the j-th frame of reconstructed image and the first loss weight of the j-th frame of reconstructed image, where j is a positive integer; if the first loss weight of the i-th frame of reconstructed image is greater than the first loss weight of the k-th frame of reconstructed image, then the cumulative error of the i-th frame of reconstructed image is greater than the cumulative error of the k-th frame of reconstructed image; the i-th frame of reconstructed image and the k-th frame of reconstructed image belong to the same image group, and i and k are positive integers.

12. A network training device, characterized in that: The device comprises: An acquisition module, used for acquiring video data, wherein the video data includes multiple frames of original images; A compression module, configured to encode the video data based on a first video processing network and then decode the encoding result of the video data based on the first video processing network to obtain reconstructed video data; wherein the reconstructed video data includes a plurality of frames of reconstructed images; a loss and weight determination module, used to determine a first loss of each reconstructed image frame in the plurality of reconstructed image frames, and to determine a first loss weight of each reconstructed image frame; wherein, if the first loss weight of the i-th reconstructed image frame is greater than the first loss weight of the k-th reconstructed image frame, then the cumulative error of the i-th reconstructed image frame is greater than the cumulative error of the k-th reconstructed image frame; the i-th reconstructed image frame and the k-th reconstructed image frame belong to the same image group, and i and k are positive integers; A training module is used to train the first video processing network based on the target loss of the multiple frames of reconstructed images; wherein the target loss of the j-th frame reconstructed image in the multiple frames of reconstructed images is determined according to the product of the first loss of the j-th frame reconstructed image and the first loss weight of the j-th frame reconstructed image, and j is a positive integer.

13. An electronic device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 9.

14. A coding device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the encoding device executes the method according to claim 10 .

15. A decoding device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the decoding device executes the method according to claim 11.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program runs on a computer or a processor, the computer or the processor executes the method according to any one of claims 1 to 11.

17. A computer program product, characterized in that The computer program product comprises computer instructions, and when the computer instructions are executed by a computer or a processor, the steps of the method according to any one of claims 1 to 11 are performed.

Images