Network training method and electronic device

Abstract

Embodiments of the present application provide a network training method and an electronic device. The method comprises: acquiring video data; coding the video data on the basis of a first video processing network, and then decoding a coding result of the video data on the basis of the first video processing network to obtain reconstructed video data; determining a first loss of each of a plurality of frames of reconstructed pictures, and determining a first loss weight of each frame of reconstructed picture, wherein if a first loss weight of an i-th frame of reconstructed picture is greater than a first loss weight of a k-th frame of reconstructed picture, the accumulated error of the i-th frame of reconstructed picture is greater than the accumulated error of the k-th frame of reconstructed picture, and the i-th frame of reconstructed picture and the k-th frame of reconstructed picture belong to a same group of pictures; and training the first video processing network on the basis of a target loss of the plurality of frames of reconstructed pictures. The trained first video processing network is subsequently used for video coding and decoding, so that the accumulated error of the reconstructed video data can be reduced, thereby improving the compression performance of the video data.

Description

Network training method and electronic equipment

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on August 15, 2023, with application number 202311034757.8 and application name “Network Training Method and Electronic Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of coding and decoding, and in particular to a network training method and electronic device. Background Art

[0003] As video evolves from HD to UHD, people's expectations for video quality are growing. At the same time, HD video places increasing demands on bandwidth and storage. Consequently, the need for video encoding, or video processing, is becoming increasingly urgent due to cost controls on bandwidth, transmission latency, and storage.

[0004] Among them, the artificial intelligence (AI) video compression (or codec) algorithm is based on deep learning and has better compression effect than traditional video compression technologies (such as H265 and H266).

[0005] Since there is temporal continuity between video frames, the amount of change between adjacent frames is relatively small, which means that there is temporal redundancy between video frames. This allows inter-frame coding to reduce bitrate overhead. In the inter-frame coding process of AI video coding, many steps, such as prediction and entropy coding of residual information, introduce errors. Thus, in the process of encoding each group of pictures (GOP), the cumulative error increases with the increase in timing, and as the cumulative error increases, the compression performance of the video also decreases.

[0006] Summary of the Invention

[0007] To solve the above technical problems, the present application provides a network training method and electronic device. Using the first video processing network trained by the network training method for video encoding and decoding can reduce the cumulative error of reconstructed video data and improve compression performance.

[0008] In a first aspect, an embodiment of the present application provides a network training method, which includes: first, obtaining video data, where the video data includes multiple frames of original images; then, encoding the video data based on a first video processing network and then decoding the encoded result of the video data based on the first video processing network to obtain reconstructed video data; wherein the reconstructed video data includes multiple frames of reconstructed images; then, determining a first loss of each reconstructed image in the multiple frames of reconstructed images, and determining a first loss weight for each reconstructed image; wherein, if the first loss weight of the i-th reconstructed image is greater than the first loss weight of the k-th reconstructed image, then the cumulative error of the i-th reconstructed image is greater than the cumulative error of the k-th reconstructed image; the i-th reconstructed image and the k-th reconstructed image belong to the same image group, where i and k are positive integers; then, training the first video processing network based on the target loss of the multiple frames of reconstructed images; wherein the target loss of the j-th reconstructed image in the multiple frames of reconstructed images is determined according to the product of the first loss of the j-th reconstructed image and the first loss weight of the j-th reconstructed image, where j is a positive integer.

[0009] Among them, during the training process, the network parameters of the first video processing network are adjusted according to the target loss of each frame of the reconstructed image; for the i-th frame reconstructed image with a large cumulative error, the first loss weight of the i-th frame reconstructed image is determined to be large, and correspondingly, the target loss of the i-th frame reconstructed image will also be increased; and in the training process, the first video processing network is usually trained with the goal of minimizing the loss (that is, minimizing the target loss of each frame of the reconstructed image); therefore, after the first video processing network is trained with the goal of minimizing the target loss of the i-th frame reconstructed image, the cumulative error of the i-th frame reconstructed image obtained by encoding and decoding using the trained first video processing network will be reduced, and then the compression performance of the trained first video processing network for the i-th frame original image will also be improved. In this way, the compression performance of each frame can be balanced. Subsequently, using the trained first video processing network for video encoding and decoding can reduce the cumulative error of the reconstructed video data and improve the compression performance of the video data.

[0010] Exemplarily, the first loss may refer to a loss used to measure the image quality of a reconstructed image, such as distortion loss, etc., which is not limited in this application.

[0011] Exemplarily, the cumulative error of the reconstructed image of the i-th frame may refer to the error introduced during the encoding of the i-th frame original image by the error between the reconstructed image of the i-th frame corresponding to the reference frame and the original image of the i-th frame corresponding to the reference frame. For example, in a GOP, the second frame original image is encoded with reference to the first frame reconstructed image. Since some steps in the encoding process may introduce errors, there is an error 1 between the second frame reconstructed image and the second frame original image. The third frame original image is encoded with reference to the second frame reconstructed image. There is an error 2 between the third frame reconstructed image and the third frame original image; wherein, error 2 includes error 3 introduced by error 1 during the encoding of the third frame original image and error 4 introduced by some steps in the encoding of the third frame original image; error 3 may be referred to as a cumulative error.

[0012] Exemplarily, multiple video data can be obtained as training data for training the first video processing network. Among them, a video data can include at least one image group (image group, i.e., GOP, refers to a group of images between I frames and I frames, I frame, i.e., Intra frame, i.e., intra-frame coded frame), and each image group can include N (N is an integer greater than 1) frames of original images. Exemplarily, the N frames of original images included in an image group can be used as a batch to train the first video processing network. Specifically, the target losses of the multiple frames of reconstructed images contained in the image group can be added together to obtain the target loss of the image group; then, based on the target loss of the image group, the first video processing network can be trained.

[0013] Exemplarily, there may be multiple pieces of reconstructed video data, each piece of reconstructed video data may include at least one image group, and each image group may include N frames of reconstructed images.

[0014] Exemplarily, i and k are integers between 1 and N (the values ​​of i and k can be 1 or N).

[0015] It should be noted that the present application only performs inter-frame encoding and inter-frame decoding on the video data based on the first video processing network. That is, the present application only encodes and decodes the original images of the second to Nth frames in each image group of the video data based on the first video processing network. For the original image of the first frame of each image group of the video data, the original image of the first frame in each image group of the video data can be encoded based on the AI ​​image encoding framework, and the encoding result of the original image of the first frame in each image group of the video data can be decoded based on the AI ​​image decoding framework. Correspondingly, during the training process, the first video processing network can be trained based on the first loss and first loss weight of the reconstructed images of the second to Nth frames of each image group.

[0016] According to the first aspect, determining a first loss weight for each reconstructed image frame includes determining the first loss weight for each reconstructed image frame based on a cumulative error of each reconstructed image frame. Thus, the magnitude of the first loss weight for each reconstructed image frame can be determined based on the magnitude of the cumulative error of each reconstructed image frame.

[0017] That is to say, when the first loss weight of the reconstructed image of the i-th frame is greater than the first loss weight of the reconstructed image of the k-th frame, the cumulative error of the reconstructed image of the i-th frame is greater than the cumulative error of the reconstructed image of the k-th frame. There are two ways of understanding this: only when the cumulative error of the i-th reconstructed image is greater than the cumulative error of the reconstructed image of the k-th frame, the first loss weight of the reconstructed image of the i-th frame is greater than the first loss weight of the reconstructed image of the k-th frame; or, when the cumulative error of the i-th reconstructed image is much greater than the cumulative error of the reconstructed image of the k-th frame (for example, the difference between the cumulative error of the i-th reconstructed image and the cumulative error of the reconstructed image of the k-th frame is greater than 2δ, where δ is a preset value), the first loss weight of the reconstructed image of the i-th frame is greater than the first loss weight of the reconstructed image of the k-th frame.

[0018] According to the first aspect, or any implementation of the first aspect above, determining a first loss weight for each reconstructed image frame based on the cumulative error of each reconstructed image frame includes: for the j-th reconstructed image frame among the multiple reconstructed image frames: determining the difference between the first loss of the j-th reconstructed image frame and the first loss of a reference reconstructed image corresponding to the j-th reconstructed image frame, to obtain a loss difference value for the j-th reconstructed image frame; wherein the loss difference value for the j-th reconstructed image frame is used to characterize the cumulative error of the j-th reconstructed image frame; and determining the first loss weight for the j-th reconstructed image frame based on the loss difference value for the j-th reconstructed image frame and a preset range. In other words, the first loss weight for each reconstructed image frame is determined based on the difference between the first loss of each reconstructed image frame and the first loss of the reference reconstructed image frame corresponding to each reconstructed image frame.

[0019] Exemplarily, the cumulative error of the reference reconstructed image corresponding to each frame of the reconstructed image is smaller than the cumulative error of each frame of the reconstructed image.

[0020] Exemplarily, the loss difference of the j-th frame reconstructed image can be a positive number, a negative number, or 0.

[0021] It should be noted that when the cumulative error of the reconstructed image is larger, if the loss difference of the reconstructed image is the first loss of the reconstructed image minus the first loss of the reconstructed image corresponding to the reference reconstructed image, the loss difference of the reconstructed image is larger; if the loss difference of the reconstructed image is the first loss of the reconstructed image corresponding to the reference reconstructed image minus the first loss of the reconstructed image, the loss difference of the reconstructed image is smaller.

[0022] Exemplarily, j is an integer between 1 and N (the value of j can be 1 or N).

[0023] According to the first aspect, or any implementation of the first aspect above, determining the first loss weight of the j-th reconstructed image based on the loss difference of the j-th reconstructed image and a preset range includes: determining the weight coefficient of the j-th reconstructed image based on the loss difference of the j-th reconstructed image and the preset range; and multiplying the weight coefficient of the j-th reconstructed image by the first loss of the j-th reconstructed image in the previous iteration cycle as the first loss weight of the j-th reconstructed image in the current iteration cycle. That is, based on the size of the loss difference of the reconstructed image, the first loss weight of the reconstructed image determined in the previous iteration cycle is increased or decreased to obtain the first loss weight of the reconstructed image in the current cycle; thereby setting the first loss weights of reconstructed images with different cumulative error sizes to different values.

[0024] According to the first aspect, or any implementation method of the first aspect above, the preset range includes a first boundary value and a second boundary value, and the first boundary value is less than the second boundary value; based on the loss difference of the reconstructed image of the j-th frame and the preset range, the weight coefficient of the reconstructed image of the j-th frame is determined, including: when the loss difference of the reconstructed image of the j-th frame belongs to the preset range, the first preset coefficient is used as the weight coefficient of the reconstructed image of the j-th frame; when the loss difference of the reconstructed image of the j-th frame is greater than the second boundary value, the second preset coefficient is used as the weight coefficient of the reconstructed image of the j-th frame, and the second preset coefficient is greater than 1; when the loss difference of the reconstructed image of the j-th frame is less than the first boundary value, the third preset coefficient is used as the weight coefficient of the reconstructed image of the j-th frame, and the third preset coefficient is less than 1; wherein, the first preset coefficient is greater than the third preset coefficient and less than the second preset coefficient, and j is a positive integer.

[0025] The loss difference of the reconstructed image is the first loss of the reconstructed image minus the first loss of the reference reconstructed image corresponding to the reconstructed image.

[0026] That is to say, the greater the difference between the first loss of the reconstructed image and the first loss of the reference reconstructed image corresponding to the reconstructed image, the greater the weight coefficient; in this way, the weight coefficient of the reconstructed image with a larger loss difference can be made larger, thereby improving the compression performance of the first video processing network for the original image with a larger loss difference corresponding to the reconstructed image.

[0027] Exemplarily, if the loss difference of the reconstructed image is the first loss of the reconstructed image corresponding to the reference reconstructed image minus the first loss of the reconstructed image, then when the loss difference of the reconstructed image of the j-th frame belongs to the preset range, the first preset coefficient is used as the weight coefficient of the reconstructed image of the j-th frame; when the loss difference of the reconstructed image of the j-th frame is greater than the second boundary value, the third preset coefficient is used as the weight coefficient of the reconstructed image of the j-th frame, and the third preset coefficient is greater than 1; when the loss difference of the reconstructed image of the j-th frame is less than the first boundary value, the second preset coefficient is used as the weight coefficient of the reconstructed image of the j-th frame, and the second preset coefficient is less than 1; wherein the first preset coefficient is greater than the third preset coefficient and less than the second preset coefficient, and j is a positive integer.

[0028] For example, the second preset coefficient is 0.9, the third preset coefficient is 1.1, and the first preset coefficient may be 1.

[0029] It should be understood that more preset ranges can be pre-set, so that the weight coefficients of reconstructed images with loss differences belonging to different preset ranges are different, and this application does not impose any restrictions on this.

[0030] According to the first aspect, or any implementation of the first aspect above, the reference reconstructed image corresponding to the j-th frame reconstructed image includes at least one of the following: a reference reconstructed image obtained by encoding the j-th frame original image based on the second video processing network and then decoding the encoded result of the j-th frame original image based on the second video processing network; wherein the cumulative error of the reference reconstructed image corresponding to the j-th frame reconstructed image is less than the cumulative error of the j-th frame reconstructed image; or, the m-frame reconstructed images preceding the j-th frame reconstructed image in the image group; wherein the first loss of the reference reconstructed image corresponding to the j-th frame reconstructed image is the average of the first losses of the m-frame reconstructed images preceding the j-th frame reconstructed image, and m is a positive integer.

[0031] Exemplarily, the second video processing network can be a video processing network composed of an AI condition I video encoding network and an AI condition I video decoding network (the AI ​​condition I video encoding network encodes the feature information of the original image, and the calculated first optical flow is not used to calculate the residual information, but only to calculate the probability distribution of the feature information of the original image. Therefore, the reconstructed image output by the AI ​​condition I video encoding network and the AI ​​condition I video decoding network does not have cumulative errors).

[0032] According to the first aspect, or any implementation method of the first aspect above, based on the cumulative error of each frame of reconstructed image, the first loss weight of each frame of reconstructed image is determined, including: determining the timing information corresponding to each frame of reconstructed image, the timing information of each frame of reconstructed image is used to characterize the cumulative error of each frame of reconstructed image; based on the timing information corresponding to each frame of reconstructed image, determining the first loss weight of each frame of reconstructed image; wherein i is greater than k.

[0033] Since the original image of the current frame is encoded based on the reconstructed images of other frames before it during the inter-frame coding process, the cumulative error of the reconstructed image that is later in time in an image group is greater; furthermore, the present application can determine the timing information corresponding to each frame of the reconstructed image, and determine the first loss weight of each frame of the reconstructed image based on the timing information of each frame of the reconstructed image.

[0034] According to the first aspect, or any implementation of the first aspect above, the method further includes: obtaining a second loss for each reconstructed image frame in the multiple reconstructed image frames and a second loss weight for each reconstructed image frame; wherein the first loss is a distortion loss and the second loss is a bit rate loss; wherein the target loss of the j-th reconstructed image frame is determined by weighted calculation of the first loss of the j-th reconstructed image frame and the second loss weight of the j-th reconstructed image frame. In this way, the distortion of the reconstructed image output by the first video processing network and the bit rate of the bitstream obtained by encoding the original image can be balanced.

[0035] For example, the distortion loss may refer to the difference (eg, error) between the original image and the reconstructed image.

[0036] Exemplarily, the bit rate loss may refer to a bit rate obtained by bit rate estimation during the encoding process.

[0037] According to the first aspect, or any implementation of the first aspect above, the reconstructed image includes multiple components; the first loss for each reconstructed image frame includes the first losses of the multiple components for each reconstructed image frame; and the first loss weight for each reconstructed image frame includes the first loss weights of the multiple components for each reconstructed image frame. Because different components of the reconstructed image have different proportions, determining corresponding first loss weights for different components can effectively reduce the cumulative error of the different components.

[0038] Exemplarily, the reconstructed image includes multiple components, which can also be referred to as the reconstructed image including multiple channels. For example, if the reconstructed image is a YUV image, the reconstructed image includes a Y component, a U component, and a V component, or in other words, the reconstructed image includes a Y channel, a U channel, and a V channel.

[0039] According to the first aspect, or any implementation of the first aspect above, the first video processing network includes a video encoding network and a video decoding network; encoding the video data based on the first video processing network and then decoding the encoded result of the video data based on the first video processing network to obtain reconstructed video data includes: encoding the video data based on the video encoding network and then decoding the encoded result of the video data based on the video decoding network to obtain reconstructed video data.

[0040] In one possible approach, the AI ​​video encoding framework can be implemented only by the video encoding network, and the AI ​​video decoding framework can be implemented only by the video decoding network; in this case, the video data can be input into the video encoding network to obtain a bit stream; the bit stream can be input into the video decoding network to obtain reconstructed video data.

[0041] In one possible approach, the AI ​​video encoding framework can be implemented by a video encoding network and a module implemented by an algorithm, and the AI ​​video decoding framework can be implemented by a video decoding network and a module implemented by an algorithm; at this time, the video data can be encoded based on the video encoding network and the module implemented by the algorithm, and then the encoded result of the video data can be decoded based on the video decoding network and the module implemented by the algorithm to obtain reconstructed video data.

[0042] In a second aspect, an embodiment of the present application provides an encoding method, the method comprising: obtaining video data; encoding the video data based on a video encoding network to obtain a code stream; wherein the video encoding network is trained based on a target loss of a multi-frame reconstructed image, and the target loss of the j-th frame reconstructed image in the multi-frame reconstructed image is determined based on the product of the first loss of the j-th frame reconstructed image and the first loss weight of the j-th frame reconstructed image, where j is a positive integer; if the first loss weight of the i-th frame reconstructed image is greater than the first loss weight of the k-th frame reconstructed image, then the cumulative error of the i-th frame reconstructed image is greater than the cumulative error of the k-th frame reconstructed image; the i-th frame reconstructed image and the k-th frame reconstructed image belong to the same image group, and i and k are positive integers.

[0043] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0044] In a third aspect, an embodiment of the present application also provides a decoding method, which includes: receiving a code stream; decoding the code stream based on a video decoding network to obtain reconstructed video data; wherein the video decoding network is trained based on the target loss of multiple frames of reconstructed images, and the target loss of the j-th frame reconstructed image in the multiple frames of reconstructed images is determined based on the product of the first loss of the j-th frame reconstructed image and the first loss weight of the j-th frame reconstructed image, where j is a positive integer; if the first loss weight of the i-th frame reconstructed image is greater than the first loss weight of the k-th frame reconstructed image, then the cumulative error of the i-th frame reconstructed image is greater than the cumulative error of the k-th frame reconstructed image; the i-th frame reconstructed image and the k-th frame reconstructed image belong to the same image group, and i and k are positive integers.

[0045] The third aspect and any implementation of the third aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the third aspect and any implementation of the third aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0046] In a fourth aspect, an embodiment of the present application further provides a network training device, comprising:

[0047] An acquisition module is used to acquire video data, where the video data includes multiple frames of original images;

[0048] A compression module, configured to encode the video data based on the first video processing network and then decode the encoded video data based on the first video processing network to obtain reconstructed video data; wherein the reconstructed video data includes multiple frames of reconstructed images;

[0049] a loss and weight determination module, configured to determine a first loss for each reconstructed image frame in the plurality of reconstructed image frames, and to determine a first loss weight for each reconstructed image frame; wherein if the first loss weight of the i-th reconstructed image frame is greater than the first loss weight of the k-th reconstructed image frame, then the cumulative error of the i-th reconstructed image frame is greater than the cumulative error of the k-th reconstructed image frame; the i-th reconstructed image frame and the k-th reconstructed image frame belong to the same image group, where i and k are positive integers;

[0050] A training module is used to train the first video processing network based on the target loss of multiple frames of reconstructed images; wherein the target loss of the j-th frame reconstructed image in the multiple frames of reconstructed images is determined according to the product of the first loss of the j-th frame reconstructed image and the first loss weight of the j-th frame reconstructed image, and j is a positive integer.

[0051] It should be understood that the network training device of the fourth aspect can be used to execute the above-mentioned first aspect or any implementation method of the first aspect, and will not be described in detail here.

[0052] The fourth aspect and any implementation of the fourth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fourth aspect and any implementation of the fourth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0053] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, which, when executed by the processor, enables the electronic device to execute the network training method of the first aspect or any possible implementation of the first aspect.

[0054] The fifth aspect and any implementation of the fifth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fifth aspect and any implementation of the fifth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0055] In a sixth aspect, an embodiment of the present application provides a chip comprising one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the method in the first aspect or any implementation of the first aspect is executed.

[0056] The sixth aspect and any implementation of the sixth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the sixth aspect and any implementation of the sixth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0057] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, the computer or the processor executes the network training method in the first aspect or any possible implementation of the first aspect.

[0058] The seventh aspect and any implementation of the seventh aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the seventh aspect and any implementation of the seventh aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0059] In an eighth aspect, an embodiment of the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer or the processor executes the network training method in the first aspect or any possible implementation of the first aspect.

[0060] The eighth aspect and any implementation of the eighth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the eighth aspect and any implementation of the eighth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0061] In the ninth aspect, an embodiment of the present application provides a coding device, comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, which, when executed by the processor, enables the coding device to execute the coding method in the second aspect or any possible implementation of the second aspect.

[0062] The ninth aspect and any implementation of the ninth aspect correspond to the second aspect and any implementation of the second aspect, respectively. The technical effects corresponding to the ninth aspect and any implementation of the ninth aspect can refer to the technical effects corresponding to the second aspect and any implementation of the second aspect.

[0063] In the tenth aspect, an embodiment of the present application provides a decoding device, comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, which, when executed by the processor, enables the decoding device to execute the decoding method in the third aspect or any possible implementation of the third aspect.

[0064] The tenth aspect and any implementation of the tenth aspect correspond to the third aspect and any implementation of the third aspect, respectively. The technical effects corresponding to the tenth aspect and any implementation of the tenth aspect can be referred to the technical effects corresponding to the third aspect and any implementation of the third aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] FIG1 is a schematic diagram of an exemplary artificial intelligence main body framework;

[0066] FIG2A is a schematic diagram of an exemplary application framework;

[0067] FIG2B is a schematic diagram of an exemplary storage framework;

[0068] FIG2C is a schematic diagram illustrating an exemplary transmission framework;

[0069] FIG3A is a schematic diagram illustrating an exemplary training process;

[0070] FIG3B is a schematic diagram illustrating the structure of an exemplary video coding network;

[0071] FIG3C is a schematic diagram illustrating the structure of an exemplary video decoding network;

[0072] FIG4 is a schematic diagram illustrating an exemplary training process;

[0073] FIG5A is a schematic diagram illustrating an exemplary training process;

[0074] FIG5B is a schematic diagram illustrating an exemplary training process;

[0075] FIG6A is a schematic diagram illustrating an exemplary training process;

[0076] FIG6B is a schematic diagram showing an exemplary first loss weight of a multi-frame reconstructed image;

[0077] FIG7A is a schematic diagram illustrating an exemplary training process;

[0078] FIG7B is a schematic diagram illustrating an exemplary training process;

[0079] FIG7C is a schematic diagram showing an exemplary compression performance comparison;

[0080] FIG8 is a schematic diagram of an exemplary network training device;

[0081] FIG9 is a schematic structural diagram of an exemplary device. DETAILED DESCRIPTION

[0082] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0083] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0084] In the description and claims of the embodiments of this application, the terms "first" and "second" are used to distinguish different objects, rather than to describe a specific order of objects. For example, the terms "first target object" and "second target object" are used to distinguish different objects, rather than to describe a specific order of objects.

[0085] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0086] In the description of the embodiments of this application, unless otherwise specified, "multiple" means two or more. For example, "multiple processing units" means two or more processing units; "multiple systems" means two or more systems.

[0087] Figure 1 is a schematic diagram of an illustrative artificial intelligence main framework, which describes the overall workflow of the artificial intelligence system and is applicable to general artificial intelligence field needs.

[0088] The following is an explanation of the above artificial intelligence theme framework from the two dimensions of "intelligent information chain" (horizontal axis) and "IT (Information Technology) value chain" (vertical axis).

[0089] The "intelligent information chain" reflects the entire process from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensed process of "data-information-knowledge-wisdom."

[0090] The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of artificial intelligence, information (providing and processing technology implementation) to the system's industrial ecological process.

[0091] (1) Infrastructure

[0092] Infrastructure provides computing power for AI systems, enabling communication with the outside world and supporting this through a foundational platform. Communication with the outside world occurs through sensors; computing power is provided by intelligent chips (such as central processing units (CPUs), embedded neural network processing units (NPUs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), and field programmable gate arrays (FPGAs)). The foundational platform includes distributed computing frameworks and networks, along with other related platform guarantees and support. This can include cloud storage and computing, and interconnected networks. For example, sensors communicate with the outside world to acquire data, which is then fed into the intelligent chips within the distributed computing system provided by the foundational platform for computation.

[0093] (2) Data

[0094] Data above the infrastructure layer represents data sources for AI. This data includes graphics, images, voice, and text, as well as IoT data from traditional devices. This includes business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0095] (3) Data processing

[0096] Data processing generally includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.

[0097] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0098] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0099] Decision-making refers to the process of making decisions after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

[0100] (4) General ability

[0101] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, material map generation, etc.

[0102] (5) Smart products and industry applications

[0103] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart manufacturing, smart transportation, smart home, smart medical care, smart security, autonomous driving, smart terminals, etc.

[0104] The first video processing network involved in this application can be used to implement machine learning, deep learning, search, reasoning, decision-making, etc. The first video processing network involved in this application can include various types of neural networks, such as deep neural networks (DNN), convolutional neural networks (CNN), recurrent neural networks (RNN), residual networks, neural networks using transformer models or other neural networks, etc., and this application does not limit this.

[0105] The work of each layer in the neural network can be expressed mathematically as To describe: From a physical perspective, the work of each layer in a neural network can be understood as completing the transformation from input space to output space (i.e., from the row space to the column space of a matrix) through five operations on the input space (a set of input vectors). These five operations include: 1. Dimensionality increase / decrease; 2. Zoom in / out; 3. Rotation; 4. Translation; 5. "Bending". Operations 1, 2, and 3 are represented by Completed, operation 4 is completed by +b, and operation 5 is performed by a(). The word "space" is used here because the object being classified is not a single thing, but a category of things, and space refers to the collection of all individuals in this category. W is the weight vector, and each value in this vector represents the weight of a neuron in that layer of the neural network. This vector W determines the spatial transformation from the input space to the output space mentioned above. In other words, the weight W of each layer controls how the space is transformed.

[0106] The purpose of training a neural network is to ultimately obtain the weight matrices of all layers of the trained neural network (a weight matrix formed by many layers of vectors W). Therefore, the training process of a neural network is essentially learning how to control spatial transformations, and more specifically, learning the weight matrices.

[0107] The first video processing network involved in this application may include a video encoding network and a video decoding network, wherein the video encoding network can be used to encode video data, and the video decoding network can be used to decode based on the encoding result of the video data. The first video processing network involved in this application can be applied to all services of video transmission and video storage, such as terminal recording, video album, Huawei Cloud, video surveillance, vehicle-mounted camera, live broadcast, etc., and this application does not limit this.

[0108] It should be noted that the AI ​​video encoding framework can be implemented only by the video encoding network, or the AI ​​video encoding framework can be implemented by the video encoding network and a module implemented by an algorithm; the AI ​​video decoding framework can be implemented only by the video decoding network, or the AI ​​video decoding framework can be implemented by the video decoding network and a module implemented by an algorithm.

[0109] This application uses the AI ​​video encoding framework implemented only by the video encoding network, and the AI ​​video decoding framework implemented only by the video decoding network as an example for explanation. It should be understood that when the AI ​​video encoding framework is implemented by the video encoding network and the module implemented by the algorithm, and the AI ​​video decoding framework is implemented by the video decoding network and the module implemented by the algorithm, in the forward calculation process, the video encoding network can cooperate with the module implemented by the algorithm to encode the video data, and the video decoding network can cooperate with the module implemented by the algorithm to decode based on the encoding result of the video data; in the back propagation process, only the video encoding network and the video decoding network can be trained.

[0110] FIG2A is a schematic diagram illustrating an exemplary application framework.

[0111] Referring to Figure 2A , for example, a camera can capture video and obtain video data. A video encoding network can then perform AI video encoding on the video data to generate a bitstream. This bitstream can then be stored locally or transmitted to a remote device.

[0112] Continuing with reference to FIG2A , in one possible approach, after the code stream is stored locally, when a video needs to be played or edited, the video decoding network can perform AI video decoding on the code stream to obtain reconstructed video data; subsequently, the reconstructed video data can be played or edited.

[0113] Continuing with reference to FIG2A , in one possible approach, after the remote device receives the code stream, when it needs to play or edit a video, the video decoding network can perform AI video decoding on the code stream to obtain reconstructed video data; subsequently, the reconstructed video data can be played or edited.

[0114] FIG2B is a schematic diagram of an exemplary storage framework.

[0115] 2B , exemplarily, the video encoding network may include an AI encoding network and an entropy encoding network, and the video decoding network may include an AI decoding network and an entropy decoding network; it should be understood that FIG2B is only an example of the present application, and the video encoding network and the video decoding network may include more networks than shown in FIG2B , and the present application does not limit this.

[0116] Exemplarily, the AI ​​encoding network and the AI ​​decoding network can be set in the NPU or GPU. Exemplarily, the entropy encoding network and the entropy decoding network can be set in the CPU. Exemplarily, the file saving module and the file loading module can be set in the CPU.

[0117] Referring to Figure 2B, illustratively, after the camera captures the video data, it can be input into the AI ​​coding network. Then, the AI ​​coding network can perform the following processing for each frame of the video data: perform prediction (including intra-frame prediction and inter-frame prediction, this application takes inter-frame prediction as an example to illustrate), determine the prediction information; and determine the residual information based on the prediction information and the original image, and determine the probability distribution of the residual information; then, input the residual information and the probability distribution of the residual information into the entropy coding network. Afterwards, the entropy coding network can entropy encode the residual information according to the probability distribution of the residual information to obtain a code stream; and input the code stream into the file saving module for saving to obtain a file.

[0118] Continuing with Figure 2B, for example, when a video needs to be played or edited, the file loading module can load the file and then input the code stream in the file into the entropy decoding network. The entropy decoding network can then obtain the probability distribution of the residual information from the AI ​​decoding network, and based on the probability distribution of the residual information, entropy decode the residual information from the code stream and output the residual information to the AI ​​decoding network. The AI ​​decoding network can then perform inter-frame prediction to obtain prediction information; subsequently, reconstruction is performed based on the prediction information and the residual information to obtain a reconstructed image; and further, reconstructed video data can be obtained.

[0119] FIG2C is a schematic diagram illustrating an exemplary transmission framework.

[0120] 2C , illustratively, the encoding process at the encoding end may be as follows:

[0121] For example, after the camera captures the video data, it can be input into the AI ​​coding network. Then, the AI ​​coding network can perform the following processing for each frame of the video data: perform prediction (including intra-frame prediction and inter-frame prediction, this application takes inter-frame prediction as an example), determine the prediction information; and determine the residual information based on the prediction information and the original image, and determine the probability distribution of the residual information; then, input the residual information and the probability distribution of the residual information into the entropy coding network. Afterwards, the entropy coding network can entropy encode the residual information according to the probability distribution of the residual information to obtain a code stream; and send the code stream to the server in the cloud.

[0122] Afterwards, the server can send the code stream to the decoding end. Among them, the server can be a single server or a server cluster, which is not limited in this application.

[0123] Continuing with FIG2C , illustratively, after the decoding end receives the code stream sent by the server, the decoding process may be as follows:

[0124] For example, the file loading module can load a file and then send the bitstream in the file to the entropy decoding network. The AI ​​decoding network can then perform inter-frame prediction to obtain prediction information. The entropy decoding network can then obtain the probability distribution of residual information from the AI ​​decoding network. Based on the probability distribution of residual information, the entropy decoding network can entropy decode the residual information from the bitstream and output the residual information to the AI ​​decoding network. Subsequently, reconstruction is performed based on the prediction information and residual information to obtain a reconstructed image, and thus, reconstructed video data can be obtained.

[0125] It should be understood that the encoding end can also directly send the code stream to the decoding end without forwarding through the server, and this application does not impose any restrictions on this.

[0126] FIG3A is a schematic diagram illustrating an exemplary training process.

[0127] 3A , illustratively, video data may be input into a video encoding network, encoded by the video encoding network, and output a code stream to a video decoding network; the video decoding network may decode the code stream and output reconstructed video data; then, a first loss may be determined based on the video data and the reconstructed video data; and back propagation may be performed using the first loss to train the video encoding network and the video decoding network.

[0128] Fig. 3B is a schematic diagram of the structure of an exemplary video coding network. In the embodiment of Fig. 3B, an inter-frame coding process is shown.

[0129] 3B , exemplarily, the AI ​​encoding network may include a motion estimation network, an optical flow encoding network, a first entropy estimation network, an optical flow decoding network, a prediction network, a feature extraction network, a residual encoding network, a second entropy estimation network, a residual decoding network, and a reconstruction network; it should be understood that the AI ​​encoding network may also include more or fewer networks than shown in FIG3B , and this application does not limit this.

[0130] 3B , illustratively, the entropy coding network may include a first entropy coding network and a second entropy coding network.

[0131] 3B , illustratively, the video encoding network may further include an entropy decoding network, and the entropy decoding network may include a first entropy decoding network and a second entropy decoding network.

[0132] It should be noted that the feature extraction network in the dotted box in FIG3B is an optional network.

[0133] Continuing with FIG3B , the inter-frame coding of an image frame in video data is used as an example for description. The AI ​​video coding process may be as follows:

[0134] Exemplarily, the original image of the current frame and the reconstructed image of the reference frame corresponding to the current frame can be input into the motion estimation network. Then, the motion estimation network can perform motion estimation based on the original image of the current frame and the reconstructed image of the reference frame to determine the optical flow of the current frame (hereinafter referred to as the first optical flow); and output the first optical flow of the current frame to the optical flow encoding network. The optical flow encoding network (also referred to as the optical flow feature extraction network) can encode the first optical flow of the current frame (it should be noted that the encoding performed by the optical flow encoding network is essentially feature extraction), obtain the feature information of the first optical flow of the current frame (also referred to as the compressed feature of the first optical flow of the current frame) and output it to the first entropy estimation network.

[0135] Exemplarily, the first entropy estimation network can perform probability estimation on the feature information of the first optical flow of the current frame, determine the probability distribution of the feature information of the first optical flow of the current frame (hereinafter referred to as the optical flow probability distribution of the current frame); then, output the feature information of the first optical flow of the current frame and the optical flow probability distribution of the current frame to the first entropy coding network.

[0136] Exemplarily, the first entropy coding network may perform entropy coding on feature information of the first optical flow of the current frame according to the optical flow probability distribution of the current frame to obtain an optical flow code stream.

[0137] For example, on the one hand, the optical flow code stream can be sent / saved (for example, the optical flow code stream is sent to a server or a decoding end). On the other hand, the optical flow code stream can be output to the first entropy decoding network; in addition, the first entropy estimation network can also output the optical flow probability distribution of the current frame to the first entropy decoding network.

[0138] Exemplarily, the first entropy decoding network can entropy decode the optical flow code stream according to the optical flow probability distribution of the current frame to obtain the feature information of the first optical flow after decoding the current frame (which can also be called the feature information of the second optical flow of the current frame); and output the feature information of the second optical flow of the current frame to the optical flow decoding network.

[0139] Exemplarily, the optical flow decoding network can decode the feature information of the second optical flow of the current frame (it should be noted that the decoding performed by the optical flow decoding network is essentially feature recovery), obtain the second optical flow of the current frame and output the second optical flow of the current frame to the prediction network.

[0140] In one possible way, when the AI ​​coding network includes a feature extraction network, the reconstructed image of the reference frame can be input into the feature extraction network, and the feature extraction network extracts the features of the reconstructed image of the reference frame to obtain the feature information of the reconstructed image of the reference frame; and the feature information of the reconstructed image of the reference frame is output to the prediction network. The prediction network can make a prediction based on the second optical flow of the current frame and the feature information of the reconstructed image of the reference frame to obtain the prediction information of the current frame; and output the prediction information of the current frame to the residual coding network. Correspondingly, the original image of the current frame can be input into the feature extraction network, and the feature extraction network extracts the features of the original image of the current frame to obtain the feature information of the original image of the current frame, and output the feature information of the original image of the current frame to the residual coding network. The residual coding network outputs the residual information of the current frame (hereinafter referred to as the first residual information of the current frame) to the second entropy estimation network based on the feature information of the original image of the current frame and the prediction information of the current frame. Among them, the first residual information of the current frame can be the first residual of the current frame (wherein, the first residual can refer to the residual between the feature information of the original image of the current frame and the prediction information of the current frame, or it can be the feature information of the first residual of the current frame (wherein, the feature information of the first residual can also be called the compressed feature of the first residual, which can be obtained by encoding the first residual of the residual coding network (it should be noted that the encoding performed by the residual coding network is essentially feature extraction)), and this application does not impose any restrictions on this. This application takes the first residual information of the current frame as the feature information of the first residual of the current frame as an example for explanation.

[0141] In one possible approach, when the AI ​​encoding network does not include a feature extraction network, the reconstructed image of the reference frame can be directly input into the prediction network. The prediction network then predicts the current frame based on the second optical flow of the current frame and the reconstructed image of the reference frame, obtaining prediction information for the current frame. This prediction information is then output to the residual encoding network. Alternatively, the original image of the current frame can be directly input into the residual encoding network. The residual encoding network then outputs the first residual information of the current frame to the second entropy estimation network based on the original image of the current frame and the prediction information of the current frame.

[0142] Exemplarily, the second entropy estimation network can perform probability estimation on the first residual information of the current frame to determine the probability distribution of the first residual information of the current frame (hereinafter referred to as the residual probability distribution of the current frame); then, the first residual information of the current frame and the residual probability distribution of the current frame are output to the second entropy coding network.

[0143] Exemplarily, the second entropy coding network may perform entropy coding on the first residual information of the current frame according to the residual probability distribution of the current frame to obtain a residual code stream.

[0144] For example, on the one hand, the residual bitstream can be sent / saved (for example, the residual bitstream is sent to a server or a decoding end). On the other hand, the residual bitstream can be output to the second entropy decoding network; in addition, the second entropy estimation network can also output the residual probability distribution of the current frame to the second entropy decoding network.

[0145] Exemplarily, the second entropy decoding network may entropy decode the residual code stream according to the residual probability distribution of the current frame to obtain the first residual information after entropy decoding of the current frame. When the first residual information of the current frame is feature information of the first residual of the current frame, the first residual information after entropy decoding of the current frame may be output to the residual decoding network.

[0146] Exemplarily, the residual decoding network can decode the first residual information after entropy decoding of the current frame (it should be noted that the decoding performed by the residual decoding network is essentially feature recovery), obtain the second residual information of the current frame and output the second residual information of the current frame to the reconstruction network. It should be noted that when the first residual information of the current frame is the first residual of the current frame, the AI ​​video encoding framework may not include a residual decoding network; at this time, the second residual information of the current frame is the decoded first residual information obtained by entropy decoding of the current frame. In addition, it can also be said that the second residual information of the current frame is obtained by decoding the encoding result of the first residual information of the current frame (i.e., the residual code stream); when the first residual information of the current frame is the feature information of the first residual of the current frame, the decoding in this step includes entropy decoding and feature recovery; when the first residual information of the current frame is the first residual of the current frame, the decoding in this step is entropy decoding.

[0147] Exemplarily, the prediction network can also output the prediction information of the current frame to the reconstruction network. When the AI ​​encoding network includes a feature extraction network, the reconstruction network can first reconstruct based on the second residual information of the current frame and the prediction information of the current frame to obtain the feature information of the reconstructed image of the current frame; then, the reconstruction network can perform feature transformation on the feature information of the reconstructed image of the current frame to obtain the reconstructed image of the current frame. When the AI ​​encoding network does not include a feature extraction network, the reconstruction network can reconstruct based on the second residual information of the current frame and the prediction information of the current frame to obtain the reconstructed image of the current frame.

[0148] Optionally, the present application may also encode information for determining the optical flow probability distribution of the current frame and send it to the decoding end; so that the decoding end determines the optical flow probability distribution to obtain the second optical flow of the current frame; and encode information for determining the residual probability distribution of the current frame and send it to the decoding end; so that the decoding end determines the residual probability distribution to obtain the second residual information of the current frame.

[0149] FIG3C is a schematic diagram showing the structure of an exemplary video decoding network.

[0150] 3C , exemplarily, the AI ​​decoding network may include: a first entropy estimation network, a second entropy estimation network, an optical flow decoding network, a prediction network, a feature extraction network, a residual decoding network, and a reconstruction network; it should be understood that the AI ​​encoding network may also include more or fewer networks than shown in FIG3C , and this application does not impose any restrictions on this.

[0151] 3C , illustratively, the entropy decoding network may include a first entropy decoding network and a second entropy encoding network.

[0152] Continuing with FIG3C , the inter-frame decoding of an image frame in the video data is taken as an example for description; wherein the AI ​​video decoding process may be as follows:

[0153] Exemplarily, the decoding end may receive a code stream, which may include an optical flow code stream and a residual code stream. When the code stream received by the decoding end does not include a code stream containing information for determining the optical flow probability distribution of the current frame, the first entropy estimation network may perform probability estimation based on the reconstructed image of the reference frame, the prediction information of the reference frame, or the feature information of the reconstructed image of the reference frame, etc., determine the optical flow probability distribution of the current frame, and output it to the first entropy decoding network. When the received code stream includes a code stream containing information for determining the optical flow probability distribution of the current frame, the first entropy decoding network may entropy decode the information for determining the optical flow probability distribution of the current frame from the code stream and output it to the first entropy estimation network; then, the first entropy estimation network may perform probability estimation based on the information for determining the optical flow probability distribution of the current frame, determine the optical flow probability distribution of the current frame, and output it to the first entropy decoding network.

[0154] Exemplarily, the first entropy decoding network may perform entropy decoding on the optical flow code stream according to the optical flow probability distribution of the current frame, obtain feature information of the second optical flow of the current frame, and output the feature information to the optical flow decoding network.

[0155] Exemplarily, the optical flow decoding network can perform feature recovery on the feature information of the second optical flow of the current frame, obtain the second optical flow of the current frame, and output the second optical flow of the current frame to the prediction network.

[0156] Exemplarily, when the received bitstream does not include information for determining the residual probability distribution of the current frame, the second entropy estimation network may perform probability estimation based on the reconstructed image of the reference frame, prediction information of the reference frame, or feature information of the reconstructed image of the reference frame, etc., to determine the residual probability distribution of the current frame and output it to the second entropy decoding network. When the received bitstream includes information for determining the residual probability distribution of the current frame, the second entropy decoding network may entropy decode the information for determining the residual probability distribution of the current frame from the bitstream; then, the second entropy estimation network may perform probability estimation based on the information for determining the residual probability distribution of the current frame, determine the residual probability distribution of the current frame, and output it to the second entropy decoding network.

[0157] Exemplarily, the second entropy decoding network can perform entropy decoding on the code stream of information used to determine the residual probability distribution, obtain information used to determine the residual probability distribution of the current frame, and output the information used to determine the residual probability distribution of the current frame to the second entropy estimation network; then, the second entropy estimation network performs probability estimation based on the information for determining the residual probability distribution of the current frame, determines the residual probability distribution of the current frame and outputs it to the second entropy decoding network.

[0158] Exemplarily, the second entropy decoding network may perform entropy decoding on the residual code stream according to the residual probability distribution of the current frame, obtain the first residual information after entropy decoding of the current frame, and output it to the residual decoding network.

[0159] Exemplarily, the residual decoding network can perform feature recovery on the first residual information after entropy decoding of the current frame, obtain the second residual information of the current frame, and output the second residual information of the current frame to the reconstruction network.

[0160] In one possible approach, when the AI ​​decoding network includes a feature extraction network, the reconstructed image of the reference frame can be input into the feature extraction network, and the feature extraction network extracts the features of the reconstructed image of the reference frame to obtain feature information of the reconstructed image of the reference frame; and the feature information of the reconstructed image of the reference frame is output to the prediction network. The prediction network can make a prediction based on the second optical flow of the current frame and the feature information of the reconstructed image of the reference frame to obtain prediction information of the current frame; and output the prediction information of the current frame to the reconstruction network. Then, the reconstruction network can first reconstruct based on the second residual information of the current frame and the prediction information of the current frame to obtain feature information of the reconstructed image of the current frame; thereafter, the reconstruction network can perform feature transformation (such as feature recovery) on the feature information of the reconstructed image of the current frame to obtain the reconstructed image of the current frame.

[0161] In one possible approach, when the AI ​​decoding network does not include a feature extraction network, the reconstructed image of the reference frame can be directly input into the prediction network. The prediction network then predicts the current frame based on the second optical flow of the current frame and the reconstructed image of the reference frame, obtaining prediction information for the current frame. This prediction information is then output to the reconstruction network. The reconstruction network then reconstructs the current frame based on the second residual information and the prediction information, obtaining a reconstructed image of the current frame.

[0162] It should be understood that the present invention does not limit the names of the various networks included in the AI ​​video encoding framework shown in Figure 3B, nor the names of the various networks included in the AI ​​video decoding framework shown in Figure 3C.

[0163] The following describes the training process of the first video processing network based on FIG. 3A to FIG. 3C .

[0164] FIG4 is a schematic diagram illustrating an exemplary training process.

[0165] S401, acquiring video data, where the video data includes multiple frames of original images.

[0166] Exemplarily, multiple pieces of video data may be obtained as training data for training the first video processing network.

[0167] Exemplarily, a piece of video data may include at least one group of pictures (a group of pictures, i.e., GOP, refers to a group of pictures between I frames, where I frame is Intra frame, i.e., intra-frame coded frame), and each group of pictures may include N (N is an integer greater than 1) frame images (hereinafter referred to as original images).

[0168] Next, the N frames of original images included in one image group can be used as a batch to train the first video processing network. The following S402 to S405 are described by taking the example of using one image group to train the first video processing network.

[0169] S402 , encoding the video data based on the first video processing network and then decoding the encoded video data based on the first video processing network to obtain reconstructed video data; wherein the reconstructed video data includes multiple frames of reconstructed images.

[0170] It should be noted that the first video processing network involved in this application is a network for inter-frame encoding and inter-frame decoding. Therefore, for the first original image of the image group, it can be encoded based on the AI ​​image encoding framework (including the AI ​​image encoding network), and the encoding result of the first original image can be decoded based on the AI ​​image decoding framework (including the AI ​​image decoding network) to obtain the first reconstructed image. For the second to Nth original images in the image group, they can be input into the first video processing network to obtain the second to Nth reconstructed images.

[0171] For example, for one of the last N-1 frames of original images in the image group, the original image can be input into the first video processing network. After the video encoding network included in the first video processing network encodes the original image, the video decoding network included in the first video processing network decodes the original image based on the encoding result of the original image and outputs a reconstructed image corresponding to the original image. The specific encoding and decoding process can be referred to the description in Figures 3A and 3B above and will not be repeated here. In this way, N reconstructed images output by the first video processing network can be obtained, and these N reconstructed images belong to one image group.

[0172] S403 : Determine a first loss of each reconstructed image frame in the multiple reconstructed image frames.

[0173] Exemplarily, the first loss may refer to a loss used to measure the image quality of the reconstructed image, such as distortion loss (distortion loss may refer to the difference (such as error) between the original image and the reconstructed image), etc. This application does not impose any restrictions on this.

[0174] For example, a first loss of the multiple reconstructed image frames can be determined based on the multiple original image frames and the multiple reconstructed image frames. Specifically, for each reconstructed image frame, the reconstructed image frame can be compared with a corresponding original image frame to determine the first loss of the reconstructed image frame.

[0175] S404, determining a first loss weight of each reconstructed image in the multiple reconstructed image frames; wherein, if the first loss weight of the i-th reconstructed image frame is greater than the first loss weight of the k-th reconstructed image frame, then the cumulative error of the i-th reconstructed image frame is greater than the cumulative error of the k-th reconstructed image frame; the i-th reconstructed image frame and the k-th reconstructed image frame belong to the same image group, and i and k are positive integers.

[0176] Exemplarily, the first loss weight of each reconstructed image frame can be determined based on the cumulative error of each reconstructed image frame. If the first loss weight of the reconstructed image of the i-th frame is greater than the first loss weight of the reconstructed image of the k-th frame, then the cumulative error of the reconstructed image of the i-th frame is greater than the cumulative error of the reconstructed image of the k-th frame. There are two ways of understanding here: only when the cumulative error of the i-th reconstructed image is greater than the cumulative error of the reconstructed image of the k-th frame, the first loss weight of the reconstructed image of the i-th frame is greater than the first loss weight of the reconstructed image of the k-th frame; or, when the cumulative error of the i-th reconstructed image is much larger than the cumulative error of the reconstructed image of the k-th frame (for example, the difference between the cumulative error of the i-th reconstructed image and the cumulative error of the reconstructed image of the k-th frame is greater than 2δ, where δ is a preset value), the first loss weight of the reconstructed image of the i-th frame is greater than the first loss weight of the reconstructed image of the k-th frame.

[0177] Here, i and k are integers between 1 and N (the values ​​of i and k can be 1 or N). That is, a reconstructed image with a large cumulative error has a large first loss weight.

[0178] S405 , training the first video processing network according to the target loss of the multi-frame reconstructed image.

[0179] The target loss of the j-th reconstructed image in the multi-frame reconstructed image is determined according to the product of the first loss of the j-th reconstructed image and the first loss weight of the j-th reconstructed image, where j is a positive integer.

[0180] For example, the target loss of the image group can be determined by referring to the following formula (1); and then the target loss of the image group is used to train the first video processing network:

[0181] Among them, in formula (1), Loss is the target loss of the image group, Loss1 j is the first loss of the reconstructed image of the jth (an integer between 1 and N, where j can be N or 1) frame in the image group, β j The first loss weight for reconstructing the image for the j-th frame.

[0182] Referring to formula (1), that is, the first loss of each frame of the reconstructed image is multiplied by the first loss weight to obtain the target loss of each frame of the reconstructed image; then the target losses of the N frames of the reconstructed image in the image group are added together to obtain the target loss of the image group; then, the target loss of the image group is used to train the first video processing network.

[0183] It should be noted that when j is equal to 1, β1*Loss11 can be used to train the AI ​​image encoding network and the AI ​​image decoding network. When j is 2 to N, β2*Loss12+β3*Loss13+......+β N *Loss1 N The first video processing network is trained.

[0184] It should be noted that, using β2*Loss12+β3*Loss13+......+β N *Loss1 N During the training of the first video processing network, the network parameters of the first video processing network are adjusted according to the target loss of each frame of the reconstructed image.

[0185] In this way, for reconstructed images with large cumulative errors (such as the reconstructed image of the i-th frame), the first loss weight of the reconstructed image of the i-th frame is determined to be large, and correspondingly, the target loss of the reconstructed image of the i-th frame will also be increased; and in the training process, the first video processing network is usually trained with the goal of minimizing the loss (that is, minimizing the target loss of each frame of the reconstructed image); therefore, after training the first video processing network with the goal of minimizing the target loss of the reconstructed image of the i-th frame, the cumulative error of the reconstructed image of the i-th frame obtained by encoding and decoding using the trained first video processing network will be reduced, and then the compression performance of the trained first video processing network frame for the original image of the i-th frame will also be improved. In this way, the compression performance of each frame can be balanced. Subsequently, using the trained first video processing network for video encoding and decoding can reduce the cumulative error of the reconstructed video data and improve the compression performance of the video data.

[0186] FIG5A is a schematic diagram illustrating an exemplary training process.

[0187] 5A , the present application can also obtain the bit rate loss (bit rate loss can refer to the bit rate estimated during the encoding process, and the bit rate loss is hereinafter referred to as the second loss) and the corresponding loss weight (hereinafter referred to as the second loss weight) of each reconstructed image in the multiple reconstructed image frames. Then, the first loss and the first loss weight of each reconstructed image frame, as well as the second loss and the second loss weight of each reconstructed image frame, can be combined to train the first video processing network. The first loss can be a distortion loss.

[0188] FIG5B is a schematic diagram of an exemplary training process. FIG5B is a training process shown based on FIG5A.

[0189] S501, obtaining video data, where the video data includes multiple frames of original images.

[0190] S502 , encoding the video data based on the first video processing network and then decoding the encoded video data based on the first video processing network to obtain reconstructed video data; wherein the reconstructed video data includes multiple frames of reconstructed images.

[0191] S503 : Determine a first loss of each reconstructed image frame in the multiple reconstructed image frames.

[0192] S504, determining a first loss weight of each reconstructed image in the multiple reconstructed image frames; wherein, the first loss weight of the i-th reconstructed image is greater than the first loss weight of the k-th reconstructed image, the cumulative error of the i-th reconstructed image is greater than the cumulative error of the k-th reconstructed image, and the i-th reconstructed image and the k-th reconstructed image belong to the same image group.

[0193] For example, S501 to S504 may refer to the description of S401 to S404 and will not be repeated here.

[0194] S505 , obtaining a second loss of each reconstructed image frame and a second loss weight of each reconstructed image frame in the multiple reconstructed image frames.

[0195] Exemplarily, the second loss may include the bit rate loss corresponding to the optical stream and the bit rate loss corresponding to the residual stream; refer to the following formula (2): Loss2 j =Loss j v +Loss j r (2)

[0196] In formula (2), Loss2 j The second loss for reconstructing the image of the jth frame, Loss j v is the bit rate loss corresponding to the optical flow code stream of the j-th frame reconstructed image, Loss j r The bit rate loss corresponding to the residual code stream of the reconstructed image of the jth frame.

[0197] Among them, the bit rate loss corresponding to the optical flow code stream can be determined and output by the first entropy estimation network during the entropy estimation process; and the bit rate loss corresponding to the residual code stream can be determined and output by the second entropy estimation network during the entropy estimation process.

[0198] For example, the second loss weight may be a fixed value, such as 1, which is not limited in the present application. In this case, the second loss weight of each reconstructed image frame is the same.

[0199] S506 , performing weighted calculation on the first loss of the j-th reconstructed image and the second loss of the j-th reconstructed image based on the first loss weight of the j-th reconstructed image and the second loss weight of the j-th reconstructed image.

[0200] S507: Training the first video processing network based on the weighted calculation result of the multiple frames of reconstructed images.

[0201] Among them, the result of the weighted calculation of the j-th frame reconstructed image is the target loss of the j-th frame reconstructed image.

[0202] For example, the first loss of the j-th frame reconstructed image and the second loss of the j-th frame reconstructed image can be weightedly calculated by referring to the content in the brackets of the following formula (3):

[0203] In formula (3), Loss is the target loss of the image group, Loss1 j is the first loss for reconstructing the image of the jth frame in the image group, β j The first loss weight for reconstructing the image for the jth frame; Loss2 j is the second loss for reconstructing the image of the jth frame in the image group, α j The second loss weight for reconstructing the image for the j-th frame.

[0204] Referring to formula (3), for the j-th frame reconstructed image, the first loss of the j-th frame reconstructed image and the second loss of the j-th frame reconstructed image are weighted and calculated based on the first loss weight of the j-th frame reconstructed image and the second loss weight of the j-th frame reconstructed image to obtain the target loss of the j-th frame reconstructed image; then, the target losses of the N-frame reconstructed images in the image group are added together to obtain the target loss of the image group; then, the target loss of the image group is used to train the first video processing network. In this way, the distortion of the reconstructed image output by the first video processing network and the bit rate of the bitstream obtained by encoding the original image can be balanced.

[0205] The following describes the process of determining the first loss weight of each reconstructed image frame based on the embodiment of FIG. 5A and FIG. 5B .

[0206] In one possible approach, the timing information of each reconstructed image frame can be used to characterize the cumulative error of each reconstructed image frame; thus, the first loss weight of each reconstructed image frame can be determined based on the timing information of each reconstructed image frame.

[0207] FIG6A is a schematic diagram illustrating an exemplary training process.

[0208] S601, acquiring video data, where the video data includes multiple frames of original images.

[0209] S602 , encoding the video data based on the first video processing network and then decoding the encoded video data based on the first video processing network to obtain reconstructed video data; wherein the reconstructed video data includes multiple frames of reconstructed images.

[0210] For example, S601 to S602 may refer to the description of S401 to S402 above, which will not be repeated here.

[0211] S603 : Determine a first loss of each reconstructed image frame in the multiple reconstructed image frames.

[0212] This application is described by taking the first loss as distortion loss as an example.

[0213] In one possible approach, a point-to-point loss (i.e., point-based loss) can be calculated between each original image frame and the corresponding reconstructed image as the distortion loss. The point-to-point loss can include Ln distances (such as L1 distance (Manhattan distance), L2 distance (Euclidean distance), L-Inf distance (Chebyshev distance), etc.), and this application does not limit this.

[0214] In one possible way, the feature-based loss (i.e., feature-based loss) between each frame of the original image and the corresponding reconstructed image can be calculated as a rate loss weight. Among them, the feature-based loss may include structural similarity (SSIM), multi-scale structural similarity (MSSSIM), learned perceptual image patch similarity (LPIPS), etc., and this application does not limit this. In addition, each frame of the original image and the corresponding reconstructed image can be input into a neural network (such as a convolutional network, VGG (Visual Geometry Group Network, Visual Geometry Group Network) network, etc.), and the neural network outputs the first feature of each frame of the original image and the second feature of the corresponding reconstructed image; then, the distance between the first feature and the second feature is calculated to obtain the feature-based loss.

[0215] It should be understood that the present application does not limit the manner of determining the distortion loss.

[0216] For example, the process of determining the first loss weight of each reconstructed image frame in the multi-frame reconstructed image frame may refer to the following S604 to S605:

[0217] S604 : Determine the time sequence information corresponding to each reconstructed image frame in the multiple reconstructed image frames, where the time sequence information of each reconstructed image frame is used to represent the accumulated error of each reconstructed image frame.

[0218] S605 : Determine a first loss weight of each reconstructed image frame in the multiple reconstructed image frames based on the timing information corresponding to each reconstructed image frame in the multiple reconstructed image frames.

[0219] For example, since during the inter-frame coding process, the original image of the current frame is encoded based on the reconstructed images of other frames before it, the cumulative error of the reconstructed image that is later in time in an image group is greater; furthermore, the present application can determine the timing information corresponding to each frame of the reconstructed image, and determine the first loss weight of each frame of the reconstructed image based on the timing information of each frame of the reconstructed image.

[0220] Exemplarily, the relationship between the timing information and the first loss weight can be set in advance; further, during the training process, the first loss weight of each frame of reconstructed image can be determined based on the timing information of each frame of reconstructed image, and the relationship between the timing information and the first loss weight.

[0221] Exemplarily, the timing information may be a frame number or time information, which is not limited in this application.

[0222] For example, the relationship between the timing information and the first loss weight can be expressed as follows:

[0223] In formula (4), β j is the first loss weight of the j-th (j is an integer between 1 and N, and the value of j can be 1 or N) frame reconstructed image, β is the preset weight, η is a hyperparameter set according to experience, and j is the timing information (i.e., frame number) of the j-th frame reconstructed image.

[0224] For example, the relationship between the timing information and the first loss weight can be expressed as follows: β j =β*(1+log A j) (5)

[0225] In formula (5), β j is the first loss weight of the j-th frame reconstructed image, β is the preset weight, A is a hyperparameter set according to experience (A>1), and j is the timing information (i.e., frame number) of the j-th frame reconstructed image.

[0226] It should be noted that β in formula (4) and formula (5) can also be the first loss weight of the reconstructed image of the jth frame in the previous iteration cycle. In this case, β jThe first loss weight of the reconstructed image of the j-th frame in the current frame iteration cycle.

[0227] It should be understood that the present application does not limit the specific expression of the relationship between the timing information and the first loss weight, and it is sufficient to indicate that the larger the timing is, the larger the first loss weight is.

[0228] Exemplarily, when the reconstructed image includes multiple components, the first loss of each frame of the reconstructed image includes the first losses of the multiple components of each frame of the reconstructed image, and the first loss weight of each frame of the reconstructed image includes the first loss weights of the multiple components of each frame of the reconstructed image.

[0229] Thus, assuming that the reconstructed image is a YUV image, for formula (4), the first loss weights of the three YUV components of each frame of the reconstructed image are respectively as shown in the following formulas (6), (7) and (8):

[0230] Among them, in formula (6) to formula (8), β j Y , β j U and β j V are the first loss weight of the Y component of the j-th frame reconstructed image, the first loss weight of the U component of the j-th frame reconstructed image, and the first loss weight of the V component of the j-th frame reconstructed image; β Y , β Y and β Y are the preset weights of the Y component, the U component, and the V component, respectively, and η Y ,η U and η U are the hyperparameters corresponding to the Y component, the U component, and the V component set based on experience.

[0231] Similarly, assuming that the reconstructed image is a YUV image, for formula (5), the first loss weights of the three YUV components of each frame of the reconstructed image are respectively as shown in the following formulas (9), (10) and (11):

[0232] Among them, in formula (9) to formula (11), β j Y , β j U and β j Vare the first loss weight of the Y component of the j-th frame reconstructed image, the first loss weight of the U component of the j-th frame reconstructed image, and the first loss weight of the V component of the j-th frame reconstructed image; β Y , β U and β V are the preset weights for the Y component, the U component, and the V component, respectively. a, b, and c are the hyperparameters for the Y component, the U component, and the V component, respectively, which are set based on experience. a, b, and c are all greater than 1.

[0233] In this embodiment, i is greater than k, that is, the i-th reconstructed image is located after the k-th reconstructed image.

[0234] Fig. 6B is a schematic diagram of a first loss weight of a multi-frame reconstructed image, wherein the reconstructed image is a YUV image having a Y component, a U component, and a V component.

[0235] Referring to FIG6B , the first loss weights of the first 10 reconstructed images in an image group are shown; the upper curve represents the first loss weight of the UV component of each of the 10 reconstructed image frames, and the lower curve represents the first loss weight of the Y component of each of the 10 reconstructed image frames. It should be understood that FIG6B is merely an example of the present application, and the first loss weight of the U component and the first loss weight of the V component of each reconstructed image frame may be different, and the present application does not impose any limitation on this.

[0236] It should be noted that the reconstructed image in Figure 6B is a YUV420 image with a Y component:U component:V component ratio of 8:1:1. Therefore, the cumulative error between the U and V components of the reconstructed image and those of the corresponding original image is greater. Correspondingly, the ratio of the first loss weight of the Y component, the first loss weight of the U component, and the first loss weight of the V component of the reconstructed image is 8:1:1. The first loss weights of the U component and the first loss weight of the V component of the reconstructed image are greater than the first loss weight of the V component.

[0237] S606 , obtaining a second loss of each reconstructed image frame and a second loss weight of each reconstructed image frame in the multiple reconstructed image frames.

[0238] S607 , performing weighted calculation on the first loss of the j-th reconstructed image and the second loss of the j-th reconstructed image based on the first loss weight of the j-th reconstructed image and the second loss weight of the j-th reconstructed image.

[0239] S608 : Training the first video processing network based on the weighted calculation result of the multi-frame reconstructed image.

[0240] For example, S606 to S608 may refer to the description of S505 to S507 above, which will not be repeated here.

[0241] The following test set is used to test the first video processing network, the traditional encoding and decoding method of the prior art (such as H266) and the AI ​​encoding and decoding method of the prior art (that is, the video processing network of the prior art) to compare the compression performance of the first video processing network of the present application and the video processing network of the prior art.

[0242] For example, the test set includes 5: two 4k test sets (wherein, one 4K (A1) test set has a complex motion type and one 4K test set (A2) has a simple motion type), a 1080p test set, an 832*480 test set (target object motion), and a 720p test set (such as conference video); the peak signal-to-noise ratio (PSNR) of the first video processing network of the present application relative to H266, and the PSNR of the video processing network of the prior art relative to H266 can be shown in Table 1 below:

[0243] Table 1

[0244] Referring to Table 1, Y-PSNR represents the PSNR of the Y component, U-PSNR represents the PSNR of the U component, and V-PSNR represents the PSNR of the V component.

[0245] Compared with the video processing network using the prior art, the first video processing network of the present application can be used for encoding and decoding to improve the compression effect; in particular, the compression effect is more significant on the UV component.

[0246] In one possible approach, the error of each frame of reconstructed image can be characterized by the difference between the first loss of each frame of reconstructed image and the first loss of the reference reconstructed image corresponding to each frame of reconstructed image; in this way, the first loss weight of the reconstructed image of each frame can be determined based on the difference between the first loss of each frame of reconstructed image and the first loss of the reference reconstructed image corresponding to each frame of reconstructed image.

[0247] FIG7A is a schematic diagram illustrating an exemplary training process.

[0248] S701, acquiring video data, where the video data includes multiple frames of original images.

[0249] S702 , encoding the video data based on the first video processing network and then decoding the encoded video data based on the first video processing network to obtain reconstructed video data; wherein the reconstructed video data includes multiple frames of reconstructed images.

[0250] S703 : Determine a first loss of each reconstructed image frame in the multiple reconstructed image frames.

[0251] For example, S701 to S703 may refer to the description of S501 to S503 above, which will not be repeated here.

[0252] For example, the process of determining the first loss weight of each reconstructed image frame may refer to S704 to S706 below. In S704 to S706, the first loss weight of the j-th reconstructed image is determined as an example for explanation:

[0253] S704, determining the difference between the first loss of the j-th frame reconstructed image and the first loss of the reference reconstructed image corresponding to the j-th frame reconstructed image, and obtaining the loss difference of the j-th frame reconstructed image. The loss difference of the j-th frame reconstructed image is used to characterize the cumulative error of the reconstructed image.

[0254] In one possible embodiment, the reference reconstructed image corresponding to the j-th frame reconstructed image is a reference reconstructed image obtained by encoding the j-th frame original image based on the second video processing network and then decoding the encoded result of the j-th frame original image based on the second video processing network; wherein the cumulative error of the reference reconstructed image corresponding to the j-th frame reconstructed image is less than the cumulative error of the j-th frame reconstructed image.

[0255] Exemplarily, a video processing network whose output reconstructed image has a cumulative error that is smaller than the cumulative error of the reconstructed image output by the first video processing network can be pre-selected as the second video processing network; for example, a video processing network composed of an AI condition I video encoding network and an AI condition I video decoding network (the AI ​​condition I video encoding network encodes the feature information of the original image, and the calculated first optical flow is not used to calculate the residual information, but only to calculate the probability distribution of the feature information of the original image, so there is no cumulative error in the reconstructed image output by the AI ​​condition I video encoding network and the AI ​​condition I video decoding network) is selected as the second video processing network.

[0256] FIG7B is a schematic diagram illustrating an exemplary training process.

[0257] Referring to Figure 7B , during training, while video data is being fed into the first video processing network, it can also be fed into the second video processing network. The second video processing network encodes each frame of the original image and then decodes it based on the encoding results, outputting multiple frames of reference reconstructed images. In this way, each frame of the original image has a corresponding reconstructed image and a reference reconstructed image.

[0258] In Figure 7B, X_2, X_3, X_4, ..., X_N are the last N frames of original images in an image group, Y1_2, Y1_3, Y1_4, ..., Y1_N are the last N frames of original images X_2, X_3, X_4, ..., X_N in an image group, which are respectively input into the first video processing network and the reconstructed images output by the first video processing network; Y2_2, Y2_3, Y2_4, ..., Y2_N are the last N frames of original images X_2, X_3, X_4, ..., X_N in an image group, which are respectively input into the second video processing network and the reconstructed images output by the second video processing network. Among them, the reference reconstructed image corresponding to the second frame reconstructed image Y1_2 is Y2_2, the reference reconstructed image corresponding to the third frame reconstructed image Y1_3 is Y2_3, the reference reconstructed image corresponding to the fourth frame reconstructed image Y1_4 is Y2_4,..., and the reference reconstructed image corresponding to the Nth frame reconstructed image Y1_N is Y2_N.

[0259] Exemplarily, the first loss of the reference reconstructed image corresponding to the j-th frame reconstructed image can be determined based on the j-th frame original image and the j-th frame reference reconstructed image output by the second video processing network (that is, the reference reconstructed image corresponding to the j-th frame reconstructed image); the specific method is similar to that of determining the first loss of the j-th frame reconstructed image, and will not be repeated here.

[0260] In one possible approach, the reference reconstructed image corresponding to the j-th reconstructed image is the m-frame reconstructed image preceding the j-th reconstructed image in the image group. Here, m is a positive integer. In this case, the first loss of the reference reconstructed image corresponding to the j-th reconstructed image can be the average of the first losses of the m-frame reconstructed images preceding the j-th reconstructed image, as shown in the following formula (12):

[0261] In formula (12), Loss1 j The first loss of the reconstructed image for the jth frame, Loss1 j ref is the first loss of the reference reconstructed image corresponding to the j-th frame reconstructed image.

[0262] Subsequently, the difference between the first loss of the j-th frame reconstructed image and the first loss of the reference reconstructed image corresponding to the j-th frame reconstructed image is determined to obtain the loss difference of the j-th frame reconstructed image; the loss difference of the j-th frame reconstructed image is used to characterize the cumulative error of the reconstructed image. When the cumulative error of the reconstructed image is larger, if the loss difference of the reconstructed image is the first loss of the reconstructed image minus the first loss of the reference reconstructed image corresponding to the reconstructed image, then the loss difference of the reconstructed image is larger; if the loss difference of the reconstructed image is the first loss of the reconstructed image corresponding to the reference reconstructed image minus the first loss of the reconstructed image, then the loss difference of the reconstructed image is smaller.

[0263] Afterwards, the weight coefficient of the j-th frame reconstructed image can be determined based on the loss difference of the j-th frame reconstructed image and the preset range. Specifically, the loss difference of the j-th frame reconstructed image can be compared with the preset range to determine the weight coefficient of the j-th frame reconstructed image; refer to the following S705 to S706:

[0264] S705 , determining a weight coefficient of the j-th frame reconstructed image based on the loss difference of the j-th frame reconstructed image and a preset range.

[0265] In one possible approach, the loss difference of the j-th frame reconstructed image is the first loss of the j-th frame reconstructed image minus the first loss of the reference reconstructed image corresponding to the j-th frame reconstructed image; this can be shown in the following formula (13): ΔLoss j =Loss1 j -Loss1 j ref (13)

[0266] In formula (13), ΔLoss j Loss1 is the loss difference of the reconstructed image of the jth frame j The first loss of the reconstructed image for the jth frame, Loss1 j ref is the first loss of the reference reconstructed image corresponding to the j-th frame reconstructed image.

[0267] In this case, when the loss difference of the reconstructed image of the j-th frame belongs to the preset range, the first preset coefficient is used as the weight coefficient of the reconstructed image of the j-th frame; when the loss difference of the reconstructed image of the j-th frame is greater than the second boundary value, the second preset coefficient is used as the weight coefficient of the reconstructed image of the j-th frame; when the loss difference of the reconstructed image of the j-th frame is less than the first boundary value, the third preset coefficient is used as the weight coefficient of the reconstructed image of the j-th frame.

[0268] Assume that the preset range is set to [-δ, +δ], where δ is a positive integer; -δ is the first boundary value, and +δ is the second boundary value. That is, when ΔLoss1 j∈[-δ,+δ], the weight coefficient of the j-th frame reconstructed image is the first preset coefficient; when ΔLoss1 j >+δ, the weight coefficient of the j-th frame reconstructed image is the second preset coefficient; when ΔLoss1 j When ≤-δ, the weight coefficient of the j-th frame reconstructed image is the third preset coefficient.

[0269] The second preset coefficient is greater than 1, for example, the second preset coefficient is 1.1; the third preset coefficient is less than 1, for example, the third preset coefficient is 0.9; the first preset coefficient is greater than the third preset coefficient and less than the second preset coefficient, for example, the first preset coefficient may be 1. In this way, the weight coefficient of the reconstructed image with a greater difference between the first loss and the first loss of the corresponding reference reconstructed image is greater, and the weight coefficient of the reconstructed image with a smaller difference between the first loss and the first loss of the corresponding reference reconstructed image is smaller.

[0270] In one possible approach, the loss difference of the j-th frame reconstructed image is the first loss of the reference reconstructed image corresponding to the j-th frame reconstructed image minus the first loss of the j-th frame reconstructed image; this can be shown in the following formula (14): ΔLoss j =Loss1 j ref -Loss1 j (14)

[0271] In formula (14), ΔLoss j Loss1 is the loss difference of the reconstructed image of the jth frame j The first loss of the reconstructed image for the jth frame, Loss1 j ref is the first loss of the reference reconstructed image corresponding to the j-th frame reconstructed image.

[0272] In this case, when the loss difference of the reconstructed image of the j-th frame belongs to the preset range, the first preset coefficient is used as the weight coefficient of the reconstructed image of the j-th frame; when the loss difference of the reconstructed image of the j-th frame is greater than the second boundary value, the third preset coefficient is used as the weight coefficient of the reconstructed image of the j-th frame; when the loss difference of the reconstructed image of the j-th frame is less than the first boundary value, the second preset coefficient is used as the weight coefficient of the reconstructed image of the j-th frame.

[0273] Assume that the preset range is set to [-δ, +δ], where δ is a positive integer; -δ is the first boundary value, and +δ is the second boundary value. That is, when ΔLoss1 j ∈[-δ,+δ], the weight coefficient of the j-th frame reconstructed image is the first preset coefficient; when ΔLoss1 j >+δ, the weight coefficient of the j-th frame reconstructed image is the third preset coefficient; when ΔLoss1 jWhen ≤-δ, the weight coefficient of the j-th frame reconstructed image is the second preset coefficient.

[0274] S706 , taking the product of the weight coefficient of the j-th frame reconstructed image and the first loss weight of the j-th frame reconstructed image in the previous iteration cycle as the first loss weight of the j-th frame reconstructed image in the current iteration cycle.

[0275] Afterwards, the product of the weight coefficient of the j-th frame reconstructed image and the first loss weight of the j-th frame reconstructed image in the previous iteration cycle is used as the first loss weight of the j-th frame reconstructed image in the current iteration cycle. It should be understood that using one image group to train the first video processing network corresponds to one iteration cycle.

[0276] For example, assuming that the loss difference of the reconstructed image of the jth frame is calculated according to formula (13), when ΔLoss1 j ∈[-δ,+δ], the first loss weight β of the reconstructed image of the jth frame in the current iteration cycle j H =β j H-1 *x1, where β j H-1 is the first loss weight of the reconstructed image of the jth frame in the previous iteration cycle, and x1 is the first preset coefficient. j > +δ, the first loss weight β of the reconstructed image of the jth frame in the current iteration cycle j H =β j H-1 *x2, where β j H-1 is the first loss weight of the reconstructed image of the jth frame in the previous iteration cycle, and x2 is the second preset coefficient. j <-δ, the first loss weight β of the reconstructed image of the jth frame in the current iteration cycle j H =β j H-1 *x1, where β j H-1 is the first loss weight of the j-th frame reconstructed image in the previous iteration cycle, and x3 is the third preset coefficient. Similarly, when the loss difference of the j-th frame reconstructed image is calculated according to formula (13), the method for determining the first loss weight of the j-th frame reconstructed image is not repeated.

[0277] Where H is a positive integer. When H=1, β j 0 The weights can be preset.

[0278] In this way, the greater the difference between the first loss and the first loss of the corresponding reference reconstructed image, the greater the first loss weight of the reconstructed image; the smaller the difference between the first loss and the first loss of the corresponding reference reconstructed image, the smaller the first loss weight of the reconstructed image.

[0279] For example, when the reconstructed image includes multiple components, a corresponding preset range can be set for each component in advance. Assuming that the reconstructed image is a YUV image, the preset range corresponding to the Y component can be expressed as: [-δ Y ,+δ Y ], the preset range corresponding to the U component can be expressed as: [-δ U ,+δ U ], the preset range corresponding to the V component can be expressed as: [-δ V ,+δ V ].

[0280] Assuming that the loss difference of the reconstructed image of the jth frame is calculated according to formula (13), then

[0281] When the loss difference of the Y component of the reconstructed image of the jth frame ΔLoss1 j Y ∈[-δ Y ,+δ Y ], the first loss weight β of the Y component of the reconstructed image of the jth frame in the current iteration cycle j HY =β j (H-1)Y *x1 Y , where β j (H-1)Y The first loss weight for the Y component of the reconstructed image of the jth frame in the previous iteration cycle, x1 Y is the first preset coefficient of the Y component. j Y >+δ Y When the first loss weight β of the Y component of the reconstructed image of the jth frame in the current iteration cycle is j HY =β j (H-1)Y *x2Y, where β j (H-1)Y The first loss weight for the Y component of the reconstructed image of the jth frame in the previous iteration cycle, x2 Y is the second preset coefficient of the Y component. j Y <-δ Y When the first loss weight β of the reconstructed image of the jth frame in the current iteration cycle is j HY =β j (H-1)Y*x3 Y , where β j (H-1)Y The first loss weight for the Y component of the reconstructed image of the jth frame in the previous iteration cycle, x3 Y is the third preset coefficient of the Y component.

[0282] When the loss difference of the U component of the reconstructed image of the jth frame is ΔLoss1 j U ∈[-δ U ,+δ U ], the first loss weight β of the U component of the reconstructed image of the jth frame in the current iteration cycle j HU =β j (H-1)U *x1 U , where β j (H-1)U The first loss weight of the U component of the reconstructed image of the jth frame in the previous iteration cycle, x1 U is the first preset coefficient of the U component. j U >+δ U When the first loss weight β of the U component of the reconstructed image of the jth frame in the current iteration cycle is j HU =β j (H-1)U *x2 U , where β j (H-1)U The first loss weight of the U component of the reconstructed image of the jth frame in the previous iteration cycle, x2 U is the second preset coefficient of the U component. j U <-δ U When the first loss weight β of the reconstructed image of the jth frame in the current iteration cycle is j HU =β j (H-1)U *x3 U , where β j (H-1)U The first loss weight of the U component of the reconstructed image of the jth frame in the previous iteration cycle, x3 U is the third preset coefficient of the U component.

[0283] When the loss difference of the V component of the reconstructed image of the jth frame ΔLoss1 j V ∈[-δ V ,+δ V ], the first loss weight β of the V component of the reconstructed image of the jth frame in the current iteration cycle jHV =β j (H-1)V *x1 V , where β j (H-1)V is the first loss weight of the V component of the j-th frame reconstructed image in the previous iteration cycle, and x1V is the first preset coefficient of the V component. j V > +δV, the first loss weight β of the V component of the reconstructed image of the jth frame in the current iteration cycle j HV =β j (H-1)V *x2 V , where β j (H-1)V The first loss weight of the V component of the reconstructed image of the jth frame in the previous iteration cycle, x2 V is the second preset coefficient of the V component. j V <-δ V When the first loss weight β of the reconstructed image of the jth frame in the current iteration cycle is j HV =β j (H-1)V *x3 V , where β j (H-1)V The first loss weight of the V component of the reconstructed image of the jth frame in the previous iteration cycle, x3 V is the third preset coefficient of the V component.

[0284] By analogy, when the loss difference of the j-th frame reconstructed image is calculated according to formula (14), the method for determining the first loss weight of each component of the j-th frame reconstructed image is no longer repeated.

[0285] S707 , obtaining a second loss of each reconstructed image frame and a second loss weight of each reconstructed image frame in the multiple reconstructed image frames.

[0286] S708 , performing weighted calculation on the first loss of the j-th reconstructed image and the second loss of the j-th reconstructed image based on the first loss weight of the j-th reconstructed image and the second loss weight of the j-th reconstructed image.

[0287] S709: Training the first video processing network based on the weighted calculation result of the multi-frame reconstructed image.

[0288] For example, S707 to S709 may refer to the description of S505 to S507 above, which will not be repeated here.

[0289] Figure 7C is a schematic diagram showing an exemplary compression performance comparison. The unit of the horizontal axis in Figure 7C is frame, and Figure 7C shows two image groups with a total of 100 frames of reconstructed images.

[0290] Referring to Figure 7C(1), in Figure 7C(1), the solid curve is the loss curve of the video processing network of the prior art, and the dotted curve is the loss curve of the first video processing network involved in the present application; by comparing the two curves, it can be seen that after the 20th frame in an image group, the target loss of the reconstructed image output by the first video processing network involved in the present application is less than the target loss of the reconstructed image output by the video processing network of the prior art. It can be seen that the training method of the embodiment of Figure 7A of the present application can improve the compression performance of the first video processing network for image frames using inter-frame coding in the image group.

[0291] Referring to FIG7C(2), in FIG7C(2), the solid curve is the PSNR curve of the video processing network of the prior art, and the dashed curve is the PSNR curve of the first video processing network involved in the present application. Comparing the two curves, it can be seen that, under the same bit rate, the PSNR of the reconstructed image output by the first video processing network involved in the present application is better than the PSNR of the reconstructed image output by the video processing network of the prior art. It can be seen that the training method of the embodiment of FIG7A of the present application can improve the image quality of the first video processing network for the image frames using inter-frame coding in the image group.

[0292] Exemplarily, after the first video processing network is trained, encoding can be performed using the video encoding network in the trained first video processing network. Specifically, video data to be encoded can be obtained, and then, based on the trained video encoding network in the first video processing network, the video data is encoded to obtain a bitstream. Decoding can also be performed using the video decoding network in the trained first video processing network. Specifically, the bitstream can be received and, based on the video decoding network, the bitstream is decoded to obtain reconstructed video data.

[0293] An embodiment of the present application also provides a network training device, which can be used to execute the training method of the aforementioned embodiment. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding method provided above, and will not be repeated here.

[0294] FIG8 is a schematic diagram of an exemplary network training device. The network training device may include:

[0295] An acquisition module 801 is configured to acquire video data, where the video data includes multiple frames of original images.

[0296] A compression module 802 is configured to encode the video data based on the first video processing network and then decode the encoded video data based on the first video processing network to obtain reconstructed video data; wherein the reconstructed video data includes multiple frames of reconstructed images;

[0297] a loss and weight determination module 803 for determining a first loss for each reconstructed image frame in the plurality of reconstructed image frames, and determining a first loss weight for each reconstructed image frame; wherein if the first loss weight of the i-th reconstructed image frame is greater than the first loss weight of the k-th reconstructed image frame, then the cumulative error of the i-th reconstructed image frame is greater than the cumulative error of the k-th reconstructed image frame; the i-th reconstructed image frame and the k-th reconstructed image frame belong to the same image group, where i and k are positive integers;

[0298] The training module 804 is used to train the first video processing network based on the target loss of the multi-frame reconstructed image; the target loss of the j-th frame reconstructed image in the multi-frame reconstructed image is determined according to the product of the first loss of the j-th frame reconstructed image and the first loss weight of the j-th frame reconstructed image, where j is a positive integer.

[0299] Exemplarily, the loss and weight determination module 803 is specifically configured to determine a first loss weight of each frame of reconstructed image based on the accumulated error of each frame of reconstructed image.

[0300] Exemplarily, the loss and weight determination module 803 is specifically configured to reconstruct an image for the j-th frame among the multiple reconstructed images:

[0301] Determine the difference between the first loss of the j-th frame reconstructed image and the first loss of the reference reconstructed image corresponding to the j-th frame reconstructed image to obtain the loss difference of the j-th frame reconstructed image; wherein the loss difference of the j-th frame reconstructed image is used to characterize the cumulative error of the j-th frame reconstructed image; based on the loss difference of the j-th frame reconstructed image and a preset range, determine the first loss weight of the j-th frame reconstructed image.

[0302] Exemplarily, the loss and weight determination module 803 is specifically configured to determine a weight coefficient of the j-th frame reconstructed image based on a loss difference of the j-th frame reconstructed image and a preset range;

[0303] The product value of the weight coefficient of the j-th frame reconstructed image and the first loss weight of the j-th frame reconstructed image in the previous iteration cycle is used as the first loss weight of the j-th frame reconstructed image in the current iteration cycle.

[0304] Exemplarily, the preset range includes a first boundary value and a second boundary value, and the first boundary value is less than the second boundary value; the loss and weight determination module 803 is specifically configured to use the first preset coefficient as the weight coefficient of the j-th frame reconstructed image when the loss difference of the j-th frame reconstructed image falls within the preset range;

[0305] When the loss difference of the j-th frame reconstructed image is greater than the second boundary value, the second preset coefficient is used as the weight coefficient of the j-th frame reconstructed image, and the second preset coefficient is greater than 1;

[0306] When the loss difference of the j-th frame reconstructed image is less than the first boundary value, the third preset coefficient is used as the weight coefficient of the j-th frame reconstructed image, and the third preset coefficient is less than 1;

[0307] The first preset coefficient is greater than the third preset coefficient and less than the second preset coefficient, and j is a positive integer.

[0308] Exemplarily, the reference reconstructed image corresponding to the j-th frame reconstructed image includes at least one of the following: a reference reconstructed image obtained by encoding the j-th frame original image based on the second video processing network and then decoding the encoded result of the j-th frame original image based on the second video processing network; wherein the cumulative error of the reference reconstructed image corresponding to the j-th frame reconstructed image is less than the cumulative error of the j-th frame reconstructed image; or, the m-frame reconstructed images preceding the j-th frame reconstructed image in the image group; wherein the first loss of the reference reconstructed image corresponding to the j-th frame reconstructed image is the average of the first losses of the m-frame reconstructed images preceding the j-th frame reconstructed image, and m is a positive integer.

[0309] Exemplarily, the loss and weight determination module 803 is specifically configured to determine the timing information corresponding to each frame of the reconstructed image, where the timing information of each frame of the reconstructed image is used to characterize the cumulative error of each frame of the reconstructed image;

[0310] Based on the timing information corresponding to each frame of the reconstructed image, a first loss weight of each frame of the reconstructed image is determined; wherein i is greater than k.

[0311] Exemplarily, the loss and weight determination module 803 is further used to obtain the second loss of each reconstructed image frame and the second loss weight of each reconstructed image frame in the multiple reconstructed images; wherein the first loss is the distortion loss and the second loss is the bit rate loss; wherein the target loss of the j-th reconstructed image frame is determined by weighted calculation of the first loss of the j-th reconstructed image frame and the second loss weight of the j-th reconstructed image frame.

[0312] Exemplarily, the reconstructed image includes multiple components; the first loss of each frame of the reconstructed image includes the first losses of the multiple components of each frame of the reconstructed image, and the first loss weight of each frame of the reconstructed image includes the first loss weights of the multiple components of each frame of the reconstructed image.

[0313] Exemplarily, the first video processing network includes a video encoding network and a video decoding network. The video encoding network is used to encode video data, and the video decoding network is used to decode the video data based on the encoding result.

[0314] In an example, FIG9 shows a schematic block diagram of a device 900 according to an embodiment of the present application. The device 900 may include: a processor 901 and a transceiver / transceiver pin 902 , and optionally, a memory 903 .

[0315] The various components of the device 900 are coupled together via a bus 904, wherein the bus 904 includes, in addition to a data bus, a power bus, a control bus, and a status signal bus. However, for the sake of clarity, all buses are referred to as bus 904 in the figure.

[0316] Optionally, the memory 903 may be used to store instructions in the aforementioned method embodiment. The processor 901 may be used to execute the instructions in the memory 903 and control the receiving pin to receive a signal and control the transmitting pin to send a signal.

[0317] The apparatus 900 may be the electronic device or a chip of the electronic device in the above method embodiment.

[0318] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0319] The present application also provides a chip including one or more interface circuits and one or more processors; the one or more processors receive or send data via the one or more interface circuits, and when the one or more processors execute computer instructions, the steps of the above-mentioned related methods are implemented. The interface circuit is a transceiver / transceiver pin 902.

[0320] This embodiment further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the method in the above-mentioned embodiment.

[0321] This embodiment further provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer executes the above-mentioned related steps to implement the method in the above-mentioned embodiment.

[0322] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the methods in the above-mentioned method embodiments.

[0323] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0324] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0325] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0326] Units described as separate components may or may not be physically separate, and components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0327] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0328] Any content of each embodiment of this application, as well as any content of the same embodiment, can be freely combined. Any combination of the above content is within the scope of this application.

[0329] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0330] The steps of the method or algorithm described in conjunction with the disclosure of the embodiments of the present application can be implemented in a hardware manner, or can be implemented by a processor executing a software instruction. The software instruction can be composed of corresponding software modules, and the software module can be stored in a random access memory (Random Access Memory, RAM), a flash memory, a read-only memory (Read Only Memory, ROM), an erasable programmable read-only memory (Erasable Programmable ROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), a register, a hard disk, a mobile hard disk, a read-only compact disc (CD-ROM) or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0331] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer-readable storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0332] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A network training method, characterized in that: The method comprises: Acquire video data, wherein the video data includes multiple frames of original images; After encoding the video data based on a first video processing network, decoding the encoding result of the video data based on the first video processing network to obtain reconstructed video data; wherein the reconstructed video data includes multiple frames of reconstructed images; Determine a first loss of each reconstructed image in the multiple reconstructed image frames, and determine a first loss weight of each reconstructed image frame; wherein, if the first loss weight of the i-th reconstructed image frame is greater than the first loss weight of the k-th reconstructed image frame, then the cumulative error of the i-th reconstructed image frame is greater than the cumulative error of the k-th reconstructed image frame; the i-th reconstructed image frame and the k-th reconstructed image frame belong to the same image group, and i and k are positive integers; The first video processing network is trained based on the target loss of the multiple-frame reconstructed images; wherein the target loss of the j-th frame reconstructed image in the multiple-frame reconstructed images is determined according to the product of the first loss of the j-th frame reconstructed image and the first loss weight of the j-th frame reconstructed image, and j is a positive integer.

2. The method according to claim 1, characterized in that The determining of the first loss weight of the reconstructed image per frame comprises: Based on the accumulated error of the reconstructed image of each frame, a first loss weight of the reconstructed image of each frame is determined.

3. The method according to claim 2, characterized in that The determining, based on the accumulated error of the reconstructed image of each frame, a first loss weight of the reconstructed image of each frame comprises: Reconstruct an image for the jth frame in the multiple reconstructed images: Determine the difference between the first loss of the j-th frame reconstructed image and the first loss of the reference reconstructed image corresponding to the j-th frame reconstructed image, and obtain the loss difference of the j-th frame reconstructed image; wherein the loss difference of the j-th frame reconstructed image is used to characterize the cumulative error of the j-th frame reconstructed image; Based on the loss difference of the j-th frame reconstructed image and a preset range, a first loss weight of the j-th frame reconstructed image is determined.

4. The method according to claim 3, characterized in that: The determining, based on the loss difference of the j-th frame reconstructed image and a preset range, a first loss weight of the j-th frame reconstructed image comprises: Determining a weight coefficient of the j-th frame reconstructed image based on a loss difference of the j-th frame reconstructed image and a preset range; The product value of the weight coefficient of the j-th frame reconstructed image and the first loss weight of the j-th frame reconstructed image in the previous iteration cycle is used as the first loss weight of the j-th frame reconstructed image in the current iteration cycle.

5. The method according to claim 4, characterized in that The preset range includes a first boundary value and a second boundary value, and the first boundary value is smaller than the second boundary value; The step of determining a weight coefficient of the j-th frame reconstructed image based on the loss difference of the j-th frame reconstructed image and a preset range includes: When the loss difference of the j-th reconstructed image falls within a preset range, using the first preset coefficient as the weight coefficient of the j-th reconstructed image; When the loss difference of the j-th frame reconstructed image is greater than a second boundary value, using a second preset coefficient as a weight coefficient of the j-th frame reconstructed image, and the second preset coefficient is greater than 1; When the loss difference of the j-th frame reconstructed image is less than the first boundary value, using a third preset coefficient as a weight coefficient of the j-th frame reconstructed image, wherein the third preset coefficient is less than 1; The first preset coefficient is greater than the third preset coefficient and less than the second preset coefficient, and j is a positive integer.

6. The method according to any one of claims 3 to 5, characterized in that: The reference reconstructed image corresponding to the j-th reconstructed image includes at least one of the following: A reference reconstructed image obtained by encoding the j-th frame original image based on a second video processing network and then decoding the encoding result of the j-th frame original image based on the second video processing network; wherein the cumulative error of the reference reconstructed image corresponding to the j-th frame reconstructed image is less than the cumulative error of the j-th frame reconstructed image; or, The m reconstructed images before the j-th reconstructed image in the image group; wherein the reference reconstructed image corresponding to the j-th reconstructed image The first loss of the image is the average of the first losses of the m frames of reconstructed images before the j-th frame of reconstructed image, where m is a positive integer.

7. The method according to claim 2, characterized in that The determining, based on the accumulated error of the reconstructed image of each frame, a first loss weight of the reconstructed image of each frame comprises: Determine the timing information corresponding to each frame of the reconstructed image, wherein the timing information of each frame of the reconstructed image is used to characterize the accumulated error of each frame of the reconstructed image; Based on the timing information corresponding to each frame of the reconstructed image, a first loss weight of each frame of the reconstructed image is determined; wherein i is greater than k.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: Obtaining a second loss of each reconstructed image in the multiple reconstructed image frames and a second loss weight of each reconstructed image frame; wherein the first loss is a distortion loss, and the second loss is a bit rate loss; Among them, the target loss of the j-th frame reconstructed image is determined by weighted calculation of the first loss of the j-th frame reconstructed image and the second loss of the j-th frame reconstructed image according to the first loss weight of the j-th frame reconstructed image and the second loss weight of the j-th frame reconstructed image.

9. The method according to any one of claims 1 to 8, characterized in that: The reconstructed image includes multiple components; the first loss of each frame of the reconstructed image includes the first losses of the multiple components of the reconstructed image of each frame, and the first loss weight of each frame of the reconstructed image includes the first loss weights of the multiple components of the reconstructed image of each frame.

10. A coding method, characterized in that: The method comprises: Get video data; The video data is encoded based on a video coding network to obtain a code stream; wherein the video coding network is trained based on a target loss of multiple reconstructed images, and the target loss of the j-th reconstructed image in the multiple reconstructed images is determined according to the product of the first loss of the j-th reconstructed image and the first loss weight of the j-th reconstructed image, where j is a positive integer; if the first loss weight of the i-th reconstructed image is greater than the first loss weight of the k-th reconstructed image, then the cumulative error of the i-th reconstructed image is greater than the cumulative error of the k-th reconstructed image; the i-th reconstructed image and the k-th reconstructed image belong to the same image group, and i and k are positive integers.

11. A decoding method, characterized in that: The method comprises: Receive code stream; The code stream is decoded based on a video decoding network to obtain reconstructed video data; wherein the video decoding network is trained based on a target loss of multiple frames of reconstructed images, and the target loss of the j-th frame of the multiple frames of reconstructed images is determined according to the product of the first loss of the j-th frame of reconstructed image and the first loss weight of the j-th frame of reconstructed image, where j is a positive integer; if the first loss weight of the i-th frame of reconstructed image is greater than the first loss weight of the k-th frame of reconstructed image, then the cumulative error of the i-th frame of reconstructed image is greater than the cumulative error of the k-th frame of reconstructed image; the i-th frame of reconstructed image and the k-th frame of reconstructed image belong to the same image group, and i and k are positive integers.

12. A network training device, characterized in that: The device comprises: An acquisition module, used for acquiring video data, wherein the video data includes multiple frames of original images; A compression module, configured to encode the video data based on a first video processing network and then decode the encoding result of the video data based on the first video processing network to obtain reconstructed video data; wherein the reconstructed video data includes a plurality of frames of reconstructed images; a loss and weight determination module, used to determine a first loss of each reconstructed image frame in the plurality of reconstructed image frames, and to determine a first loss weight of each reconstructed image frame; wherein, if the first loss weight of the i-th reconstructed image frame is greater than the first loss weight of the k-th reconstructed image frame, then the cumulative error of the i-th reconstructed image frame is greater than the cumulative error of the k-th reconstructed image frame; the i-th reconstructed image frame and the k-th reconstructed image frame belong to the same image group, and i and k are positive integers; A training module is used to train the first video processing network based on the target loss of the multiple frames of reconstructed images; wherein the target loss of the j-th frame reconstructed image in the multiple frames of reconstructed images is determined according to the product of the first loss of the j-th frame reconstructed image and the first loss weight of the j-th frame reconstructed image, and j is a positive integer.

13. An electronic device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 9.

14. A coding device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the encoding device executes the method according to claim 10 .

15. A decoding device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the decoding device executes the method according to claim 11.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program runs on a computer or a processor, the computer or the processor executes the method according to any one of claims 1 to 11.

17. A computer program product, characterized in that The computer program product comprises computer instructions, and when the computer instructions are executed by a computer or a processor, the steps of the method according to any one of claims 1 to 11 are performed.

Images